This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch spark-runner_structured-streaming in repository https://gitbox.apache.org/repos/asf/beam.git.
discard d5f235d Enable gradle build scan discard 2acdf67 Enable test mode discard a7d2328 Put all transform translators Serializable discard 9fad3d4 Simplify beam reader creation as it created once the source as already been partitioned discard 3be7f2d Fix SourceTest discard 002f0b4 Move SourceTest to same package as tested class discard 47c20c2 Add serialization test discard 43c737b Fix SerializationDebugger discard 5c9fcd3 Add SerializationDebugger discard bab9027 Fix serialization issues new c6618c5 First attempt for ParDo primitive implementation This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (d5f235d) \ N -- N -- N refs/heads/spark-runner_structured-streaming (c6618c5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/spark-structured-streaming/build.gradle | 4 - .../spark/structuredstreaming/SparkRunner.java | 2 +- .../translation/TransformTranslator.java | 3 +- .../translation/TranslationContext.java | 23 ++- .../translation/batch/DatasetSourceBatch.java | 80 +++++----- .../translation/batch/DoFnFunction.java | 137 ++++++++++++++++ .../translation/batch/ParDoTranslatorBatch.java | 174 ++++++++++++++++++++- .../translation/batch}/SparkProcessContext.java | 32 +--- .../batch/functions/SparkNoOpStepContext.java} | 6 +- .../batch/functions/SparkSideInputReader.java} | 45 +++--- .../spark/structuredstreaming/SourceTest.java | 29 ++++ .../translation/batch/SourceTest.java | 79 ---------- .../utils/SerializationDebugger.java | 131 ---------------- .../structuredstreaming/utils/package-info.java | 20 --- 14 files changed, 427 insertions(+), 338 deletions(-) create mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnFunction.java copy runners/{spark/src/main/java/org/apache/beam/runners/spark/translation => spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch}/SparkProcessContext.java (88%) copy runners/{flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkNoOpStepContext.java => spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkNoOpStepContext.java} (85%) copy runners/{google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/LazilyInitializedSideInputReader.java => spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java} (55%) create mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SourceTest.java delete mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SourceTest.java delete mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/SerializationDebugger.java delete mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/package-info.java