This is an automated email from the ASF dual-hosted git repository. echauchot pushed a change to branch spark-runner_structured-streaming in repository https://gitbox.apache.org/repos/asf/beam.git.
discard c6618c5 First attempt for ParDo primitive implementation new bab9027 Fix serialization issues new 5c9fcd3 Add SerializationDebugger new 43c737b Fix SerializationDebugger new 47c20c2 Add serialization test new 002f0b4 Move SourceTest to same package as tested class new 3be7f2d Fix SourceTest new 9fad3d4 Simplify beam reader creation as it created once the source as already been partitioned new a7d2328 Put all transform translators Serializable new 2acdf67 Enable test mode new d5f235d Enable gradle build scan new af80e19 Add flatten test This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (c6618c5) \ N -- N -- N refs/heads/spark-runner_structured-streaming (af80e19) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. The 11 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/spark-structured-streaming/build.gradle | 4 + .../spark/structuredstreaming/SparkRunner.java | 2 +- .../translation/TransformTranslator.java | 3 +- .../translation/TranslationContext.java | 23 +-- .../translation/batch/DatasetSourceBatch.java | 80 +++++----- .../translation/batch/DoFnFunction.java | 137 ---------------- .../translation/batch/ParDoTranslatorBatch.java | 174 +-------------------- .../translation/batch/SparkProcessContext.java | 149 ------------------ .../batch/functions/SparkNoOpStepContext.java | 36 ----- .../batch/functions/SparkSideInputReader.java | 62 -------- .../spark/structuredstreaming/SourceTest.java | 29 ---- .../translation/batch/FlattenTest.java | 42 +++++ .../translation/batch/SourceTest.java | 79 ++++++++++ .../utils/SerializationDebugger.java | 131 ++++++++++++++++ .../structuredstreaming/utils}/package-info.java | 4 +- 15 files changed, 311 insertions(+), 644 deletions(-) delete mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/DoFnFunction.java delete mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SparkProcessContext.java delete mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkNoOpStepContext.java delete mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/SparkSideInputReader.java delete mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/SourceTest.java create mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/FlattenTest.java create mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/SourceTest.java create mode 100644 runners/spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/SerializationDebugger.java copy runners/{spark/src/main/java/org/apache/beam/runners/spark/translation => spark-structured-streaming/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils}/package-info.java (86%)