This is an automated email from the ASF dual-hosted git repository.
lostluck pushed a change to branch prismGrace1
in repository https://gitbox.apache.org/repos/asf/beam.git
from 54a235ff1c8 [prism] worker shutdown, cleanup, log fail
add eb732077f59 Moving experimental Swift SDK code into Beam repo
structure. Cleaning up and adding tests and Apache License labels where needed.
add 029ba71be57 Adding more of the Swift SDK implementation
add f77d6ae9ae6 Refactor Unary and Tuple SerializableFn into a single
ClosureFn since one is just a special case of the other. Add the wordcount
integration test to verify the refactoring does indeed work and start moving
the pipeline code into the branch.
add d48fc9f51c9 Moving some stuff around to make the proto construction
code easier to read
add 292632e6341 Add missing file
add d1605ed80fa Add the rest of the code. Basic word count integration
test now completes successfully with the Python Portable Runner.
add fc4aedecd67 Add licenses everywhere
add edff60efc7d Add the generated proto files so other people don't have
to do it along with the typealias data
add 0221d9a6fb2 Use a single advance in varint implementation
add 0dd6adb7c60 Small change to try to stop crashing on Linux
add 840f2c618bd Trying to stop advanced(by:) crashes on non-macOS platforms
add a20a0728e46 safeAdvance to try to address issues with zero-length data
elements on non-mac platforms.
add 1144716a510 Move typealiases for model protos to make it easier to
find and less likely to be ignored by .gitignore
add e368cb229ad Turns out String interpolation breaks the #file and #line
macros. Try a new approach with optional "real" names to get more stable
transform names. Start adding "ObjectFn" for more complicated DoFn
implementations.
add a5c5adaece9 Second attempt and fixing the line values.
add 82af74a6c57 Remove random output file
add be9b2aefef6 Start stubbing out more of the composite transform
implementation. Start stubbing out External Transforms and Schema support.
Improve automatic naming of pardos and make explicit name implementation
cleaner. Add S3 and GCS dependencies to implement basic internal IOs.
add 337e6d02235 Added backwards links in PCollections to support Composite
transforms. Added a very basic composite version of WordCount to the test
suite. Still need to hook up the graph generation, which requires searching for
the roots from the leaves to give it more of that "SwiftUI" style.
add c3ce7d6bab8 Added the primitives for a FileIO implementation with
Google Storage support to start. Doesn't do anything fancy, but does implement
a true version of wordcount with the classic Shakespeare input.
add 12b1a281600 Rename previous stream-based pardo implementation to
pstream and start to introduce an element-wise pardo. This better matches other
SDK conventions (i.e. processElement) and allows us to implicitly transfer the
timestamp and window from the input to an output. Repurposes the PInput and
POutput protocols to handle this functionality. Also update the map and flatmap
implementations to use pardo rather than pstream to get a sense of how well it
works (well).
add ec0de9a2f9b Changes to allow Swift SDK to operate successfully with
the Flink Portable Runner as well as the Python Portable Runner. Modified the
PInput/POutput functionality to just be structs, this allows us to use them for
both closures as well as the eventual DoFn interface and cleans up the function
signatures.
add 92b22b4b4a8 add swift to git ignore
add db58501585e Increase prism server receive size to max.
No new revisions were added by this update.
Summary of changes:
.gitignore | 4 +
.../beam/runners/prism/internal/worker/worker.go | 5 +-
sdks/swift/Package.resolved | 194 +
sdks/swift/Package.swift | 70 +
sdks/swift/README.md | 25 +
sdks/swift/Scripts/generate_protobuf.sh | 97 +
sdks/swift/Sources/ApacheBeam/ApacheBeam.swift | 20 +
.../ApacheBeam/Client/ApiServiceDescriptor.swift | 91 +
.../Sources/ApacheBeam/Coders/BeamValue.swift | 72 +
.../swift/Sources/ApacheBeam/Coders/Beamable.swift | 41 +
.../Sources/ApacheBeam/Coders/Coder+Decoding.swift | 82 +
.../Sources/ApacheBeam/Coders/Coder+Encoding.swift | 109 +
sdks/swift/Sources/ApacheBeam/Coders/Coder.swift | 213 +
.../Sources/ApacheBeam/Core/ArtifactInfo.swift | 29 +
.../ApacheBeam/Core/DynamicProperties.swift | 38 +
.../Sources/ApacheBeam/Core/Environment.swift | 94 +
.../Sources/ApacheBeam/Core/Fn/ClosureFn.swift | 71 +
sdks/swift/Sources/ApacheBeam/Core/Fn/DoFn.swift | 59 +
.../Sources/ApacheBeam/Core/Fn/ObjectFn.swift | 36 +
.../Core/Fn/ParameterizedClosureFn.swift | 74 +
.../ApacheBeam/Core/Fn/SerializableFn.swift | 40 +
sdks/swift/Sources/ApacheBeam/Core/KeyValue.swift | 66 +
.../Core/PCollection/AnyPCollection.swift | 92 +
.../Core/PCollection/AnyPCollectionStream.swift | 97 +
.../ApacheBeam/Core/PCollection/PCollection.swift | 83 +
.../Core/PCollection/PCollectionStream.swift | 120 +
.../ApacheBeam/Core/PTransform/AnyPTransform.swift | 43 +
.../ApacheBeam/Core/PTransform/Output.swift | 18 +
.../ApacheBeam/Core/PTransform/PTransform.swift | 52 +
.../Core/PTransform/PTransformBuilder.swift | 51 +
.../ApacheBeam/Core/PTransform/Transform.swift | 7 +
.../Core/PTransform/TuplePTransform.swift | 20 +
.../ApacheBeam/Core/Pipeline/Pipeline.swift | 282 +
.../ApacheBeam/Core/Pipeline/PipelineContext.swift | 38 +
.../ApacheBeam/Core/Pipeline/PipelineRunner.swift | 21 +
.../Core/Pipeline/PipelineTransform.swift | 56 +
sdks/swift/Sources/ApacheBeam/Core/Windowing.swift | 51 +
.../model/fn_execution/v1/beam_fn_api.grpc.swift | 2061 +++++
.../fn_execution/v1/beam_provision_api.grpc.swift | 332 +
.../job_management/v1/beam_artifact_api.grpc.swift | 1579 ++++
.../v1/beam_expansion_api.grpc.swift | 425 +
.../job_management/v1/beam_job_api.grpc.swift | 1237 +++
.../model/pipeline/v1/beam_runner_api.grpc.swift | 322 +
.../model/fn_execution/v1/beam_fn_api.pb.swift | 5317 ++++++++++++
.../fn_execution/v1/beam_provision_api.pb.swift | 383 +
.../job_management/v1/beam_artifact_api.pb.swift | 1372 +++
.../job_management/v1/beam_expansion_api.pb.swift | 413 +
.../model/job_management/v1/beam_job_api.pb.swift | 1708 ++++
.../model/pipeline/v1/beam_runner_api.pb.swift | 9135 ++++++++++++++++++++
.../beam/model/pipeline/v1/endpoints.pb.swift | 178 +
.../model/pipeline/v1/external_transforms.pb.swift | 482 ++
.../apache/beam/model/pipeline/v1/metrics.pb.swift | 1029 +++
.../apache/beam/model/pipeline/v1/schema.pb.swift | 2273 +++++
.../model/pipeline/v1/standard_window_fns.pb.swift | 515 ++
.../ApacheBeam/Internal/Array+Helpers.swift | 23 +
.../ApacheBeam/Internal/Data+Decoding.swift | 94 +
.../ApacheBeam/Internal/Data+Encoding.swift | 41 +
.../ApacheBeam/Internal/Date+Timestamp.swift | 27 +
.../ApacheBeam/Internal/Dictionary+Helpers.swift | 24 +
.../ApacheBeam/Internal/Model+Typealias.swift | 29 +
.../ApacheBeam/Internal/PipelineProto+Coder.swift | 46 +
.../Internal/PipelineProto+Environment.swift | 28 +
.../Internal/PipelineProto+Initializers.swift | 93 +
.../ApacheBeam/Internal/ProtoConversion.swift | 25 +
.../ApacheBeam/Internal/Server+Endpoint.swift | 35 +
.../Sources/ApacheBeam/Internal/String+Urns.swift | 38 +
.../ApacheBeam/Runners/PortableRunner.swift | 77 +
.../Runtime/Bundle/BundleProcessor.swift | 142 +
.../Sources/ApacheBeam/Runtime/Bundle/Sink.swift | 45 +
.../Sources/ApacheBeam/Runtime/Bundle/Source.swift | 71 +
.../ApacheBeam/Runtime/DataplaneClient.swift | 239 +
.../Runtime/Metrics/MetricReporter.swift | 4 +
.../Sources/ApacheBeam/Runtime/Worker/Worker.swift | 100 +
.../ApacheBeam/Runtime/Worker/WorkerProvider.swift | 83 +
sdks/swift/Sources/ApacheBeam/Schema/Row.swift | 28 +
.../Sources/ApacheBeam/Schema/RowProtocol.swift | 3 +
sdks/swift/Sources/ApacheBeam/Schema/Schema.swift | 22 +
.../ApacheBeam/Testing/PCollection+Testing.swift | 19 +
.../ApacheBeam/Testing/PCollectionTest.swift | 43 +
.../Sources/ApacheBeam/Transforms/Basic.swift | 100 +
.../ApacheBeam/Transforms/BuiltIn+Elements.swift | 56 +
.../Sources/ApacheBeam/Transforms/BuiltIn.swift | 179 +
.../Sources/ApacheBeam/Transforms/Combining.swift | 41 +
.../Sources/ApacheBeam/Transforms/Composite.swift | 20 +
.../Sources/ApacheBeam/Transforms/Grouping.swift | 26 +
.../Sources/ApacheBeam/Transforms/IO/DuckIO.swift | 13 +
.../Sources/ApacheBeam/Transforms/IO/FileIO.swift | 17 +
.../Transforms/IO/GoogleCloud/GoogleStorage.swift | 113 +
.../Documentation.docc/ApacheBeam.md | 37 +
.../Sources/Examples/Wordcount/Wordcount.swift | 2 +
.../Tests/ApacheBeamTests/ApacheBeamTests.swift | 30 +
.../Tests/ApacheBeamTests/Coders/CoderTests.swift | 71 +
.../Pipeline/CompositeIntegrationTests.swift | 77 +
.../ApacheBeamTests/Pipeline/FileIOTests.swift | 84 +
.../ApacheBeamTests/Pipeline/Fixtures/file1.txt | 4 +
.../ApacheBeamTests/Pipeline/Fixtures/file2.txt | 3 +
.../Pipeline/IntegrationTests.swift | 92 +
97 files changed, 33665 insertions(+), 1 deletion(-)
create mode 100644 sdks/swift/Package.resolved
create mode 100644 sdks/swift/Package.swift
create mode 100644 sdks/swift/README.md
create mode 100755 sdks/swift/Scripts/generate_protobuf.sh
create mode 100644 sdks/swift/Sources/ApacheBeam/ApacheBeam.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Client/ApiServiceDescriptor.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Coders/BeamValue.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Coders/Beamable.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Coders/Coder+Decoding.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Coders/Coder+Encoding.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Coders/Coder.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/ArtifactInfo.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/DynamicProperties.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Environment.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Fn/ClosureFn.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Fn/DoFn.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Fn/ObjectFn.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/Fn/ParameterizedClosureFn.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Fn/SerializableFn.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/KeyValue.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PCollection/AnyPCollection.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PCollection/AnyPCollectionStream.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PCollection/PCollection.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PCollection/PCollectionStream.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PTransform/AnyPTransform.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/PTransform/Output.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PTransform/PTransform.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PTransform/PTransformBuilder.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PTransform/Transform.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/PTransform/TuplePTransform.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Pipeline/Pipeline.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/Pipeline/PipelineContext.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/Pipeline/PipelineRunner.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Core/Pipeline/PipelineTransform.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Core/Windowing.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/fn_execution/v1/beam_fn_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/fn_execution/v1/beam_provision_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/job_management/v1/beam_artifact_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/job_management/v1/beam_expansion_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/job_management/v1/beam_job_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/GRPC/org/apache/beam/model/pipeline/v1/beam_runner_api.grpc.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/fn_execution/v1/beam_fn_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/fn_execution/v1/beam_provision_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/job_management/v1/beam_artifact_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/job_management/v1/beam_expansion_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/job_management/v1/beam_job_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/beam_runner_api.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/endpoints.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/external_transforms.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/metrics.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/schema.pb.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Generated/Model/org/apache/beam/model/pipeline/v1/standard_window_fns.pb.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Array+Helpers.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Data+Decoding.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Data+Encoding.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Date+Timestamp.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Internal/Dictionary+Helpers.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Model+Typealias.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Internal/PipelineProto+Coder.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Internal/PipelineProto+Environment.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Internal/PipelineProto+Initializers.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/ProtoConversion.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/Server+Endpoint.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Internal/String+Urns.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Runners/PortableRunner.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Runtime/Bundle/BundleProcessor.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Runtime/Bundle/Sink.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Runtime/Bundle/Source.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Runtime/DataplaneClient.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Runtime/Metrics/MetricReporter.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Runtime/Worker/Worker.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Runtime/Worker/WorkerProvider.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Schema/Row.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Schema/RowProtocol.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Schema/Schema.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Testing/PCollection+Testing.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Testing/PCollectionTest.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/Basic.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Transforms/BuiltIn+Elements.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/BuiltIn.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/Combining.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/Composite.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/Grouping.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/IO/DuckIO.swift
create mode 100644 sdks/swift/Sources/ApacheBeam/Transforms/IO/FileIO.swift
create mode 100644
sdks/swift/Sources/ApacheBeam/Transforms/IO/GoogleCloud/GoogleStorage.swift
create mode 100644
sdks/swift/Sources/ApacheBeamDocumentation/Documentation.docc/ApacheBeam.md
create mode 100644 sdks/swift/Sources/Examples/Wordcount/Wordcount.swift
create mode 100644 sdks/swift/Tests/ApacheBeamTests/ApacheBeamTests.swift
create mode 100644 sdks/swift/Tests/ApacheBeamTests/Coders/CoderTests.swift
create mode 100644
sdks/swift/Tests/ApacheBeamTests/Pipeline/CompositeIntegrationTests.swift
create mode 100644 sdks/swift/Tests/ApacheBeamTests/Pipeline/FileIOTests.swift
create mode 100644 sdks/swift/Tests/ApacheBeamTests/Pipeline/Fixtures/file1.txt
create mode 100644 sdks/swift/Tests/ApacheBeamTests/Pipeline/Fixtures/file2.txt
create mode 100644
sdks/swift/Tests/ApacheBeamTests/Pipeline/IntegrationTests.swift
