This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c350188 Use "sparkMaster" in local mode to obtain number of shuffle
partitions + spotless apply
new 031754c Wrap Beam Coders into Spark Encoders using ExpressionEncoder:
serialization part
new d613d6b Wrap Beam Coders into Spark Encoders using ExpressionEncoder:
deserialization part
new 84f2cbd9 Fix EncoderHelpers.fromBeamCoder() visibility
new 95fd272 type erasure: spark encoders require a Class<T>, pass Object
and cast to Class<T>
new a5d49f5 Fix scala Product in Encoders to avoid StackEverflow
new 23735e4 Conform to spark ExpressionEncoders: pass classTags,
implement scala Product, pass children from within the ExpressionEncoder
new e96af88 Fix visibility of serializer and deserializer
new 2bf4cd9 Add a simple spark native test to test Beam coders wrapping
into Spark Encoders
new b11e100 Fix code generation in Beam coder wrapper
new a758985 Lazy init coder because coder instance cannot be interpolated
by catalyst
new 2c94eef Fix call to scala Fucntion1 in coder lazy init
new cbd7c2b Fix warning in coder construction by reflexion
new 668227b Fix ExpressionEncoder generated code: typos, try catch, fqcn
new 959664f Fix getting the output value in code generation
new 8bbf991 Fix beam coder lazy init using reflexion
new 723c004 Add try catch around reflexion call in lazy init of beam coder
new 91e923c Cast coder instanciated by reflection
new c4a4464 Remove lazy init of beam coder because there is no generic
way on instanciating a beam coder
new 4351304 Remove example code
new ef69410 Fix equal and hashcode
new d8b8b42 Fix generated code: uniform exceptions catching, fix
parenthesis and variable declarations
new 0fe6f9b Add an assert of equality in the encoders test
new b879123 Apply spotless and checkstyle and add javadocs
new cfdf4a4 Improve exceptions catching
The 24 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
runners/spark/build.gradle | 1 +
.../translation/helpers/EncoderHelpers.java | 290 +++++++++++++++++++++
.../structuredstreaming/utils/EncodersTest.java} | 38 +--
3 files changed, 313 insertions(+), 16 deletions(-)
copy
runners/{google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/apiary/ApiaryTest.java
=>
spark/src/test/java/org/apache/beam/runners/spark/structuredstreaming/utils/EncodersTest.java}
(51%)
