This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 9fcc955 Fix getSideInputs
new 46b9f1f Extract binary schema creation in a helper class
new de750d9 First version of combinePerKey
new d876f41 Improve type checking of Tuple2 encoder
new 612fcfe Introduce WindowingHelpers (and helpers package) and use it
in Pardo, GBK and CombinePerKey
new 014e695 Fix combiner using KV as input, use binary encoders in place
of accumulatorEncoder and outputEncoder, use helpers, spotless
The 19423 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...TransformTranslator.java => SchemaHelpers.java} | 22 ++--
.../batch/CombinePerKeyTranslatorBatch.java | 43 +++++++-
.../translation/batch/DatasetSourceBatch.java | 35 ++-----
.../translation/batch/DoFnFunction.java | 2 +-
.../batch/GroupByKeyTranslatorBatch.java | 14 +--
.../translation/batch/ParDoTranslatorBatch.java | 8 +-
.../batch/ReadSourceTranslatorBatch.java | 25 ++---
.../batch/functions/AggregatorCombiner.java | 70 +++++++++++++
.../translation/{ => helpers}/EncoderHelpers.java | 16 ++-
.../KVHelpers.java} | 37 ++++---
.../translation/helpers/WindowingHelpers.java | 111 +++++++++++++++++++++
.../streaming/ReadSourceTranslatorStreaming.java | 8 +-
.../translation/batch/CombineTest.java | 70 ++++++++-----
13 files changed, 346 insertions(+), 115 deletions(-)
copy
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{TransformTranslator.java
=> SchemaHelpers.java} (56%)
create mode 100644
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/functions/AggregatorCombiner.java
rename
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{
=> helpers}/EncoderHelpers.java (87%)
copy
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/{batch/CombinePerKeyTranslatorBatch.java
=> helpers/KVHelpers.java} (50%)
create mode 100644
runners/spark-structured-streaming/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/WindowingHelpers.java
