This is an automated email from the ASF dual-hosted git repository. echauchot pushed a change to branch spark-runner_structured-streaming in repository https://gitbox.apache.org/repos/asf/beam.git.
from 9fcc955 Fix getSideInputs new 46b9f1f Extract binary schema creation in a helper class new de750d9 First version of combinePerKey new d876f41 Improve type checking of Tuple2 encoder new 612fcfe Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey new 014e695 Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless The 19423 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ...TransformTranslator.java => SchemaHelpers.java} | 22 ++-- .../batch/CombinePerKeyTranslatorBatch.java | 43 +++++++- .../translation/batch/DatasetSourceBatch.java | 35 ++----- .../translation/batch/DoFnFunction.java | 2 +- .../batch/GroupByKeyTranslatorBatch.java | 14 +-- .../translation/batch/ParDoTranslatorBatch.java | 8 +- .../batch/ReadSourceTranslatorBatch.java | 25 ++--- .../batch/functions/AggregatorCombiner.java | 70 +++++++++++++ .../translation/{ => helpers}/EncoderHelpers.java | 16 ++- .../KVHelpers.java} | 37 ++++--- .../translation/helpers/WindowingHelpers.java | 111 +++++++++++++++++++++ .../streaming/ReadSourceTranslatorStreaming.java | 8 +- .../translation/batch/CombineTest.java | 70 ++++++++----- 13 files changed, 346 insertions(+), 115 deletions(-) copy runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{TransformTranslator.java => SchemaHelpers.java} (56%) create mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/AggregatorCombiner.java rename runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{ => helpers}/EncoderHelpers.java (87%) copy runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{batch/CombinePerKeyTranslatorBatch.java => helpers/KVHelpers.java} (50%) create mode 100644 runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/WindowingHelpers.java