[beam] branch master updated (0b415fd -> 18059ee)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0b415fd Merge pull request #10160 More compartmentalization of bundle-based-runner only utilities. add 6f72c93 [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0 add 4ca7e55 [BEAM-8470] Fix missing dep add 0fc0d0a4 [BEAM-8470] Add SparkPipelineOptions add e8ca23e [BEAM-8470] Start pipeline translation add 00964d2 [BEAM-8470] Add global pipeline translation structure add a3b278e [BEAM-8470] Add nodes translators structure add ef4941a [BEAM-8470] Wire node translators with pipeline translator add 8a8dc1e [BEAM-8470] Renames: better differenciate pipeline translator for transform translator add cdfd589 [BEAM-8470] Organise methods in PipelineTranslator add 38eca95 [BEAM-8470] Initialise BatchTranslationContext add 80f2d8c [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components add baf210f [BEAM-8470] Make transform translation clearer: renaming, comments add b65a9da [BEAM-8470] Improve javadocs add 0434749 [BEAM-8470] Move SparkTransformOverrides to correct package add 4372c7e [BEAM-8470] Move common translation context components to superclass add 49b666b [BEAM-8470] apply spotless add 0d6906a [BEAM-8470] Make codestyle and firebug happy add ef97440 [BEAM-8470] Add TODOs add 9abf8ac [BEAM-8470] Post-pone batch qualifier in all classes names for readability add 11a6e19 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN add 47ed3d1 [BEAM-8470] Added SparkRunnerRegistrar add 022a0d0 [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline() add 96b3f36 [BEAM-8470] Create PCollections manipulation methods add b0c42af [BEAM-8470] Create Datasets manipulation methods add 2c5cb23 [BEAM-8470] Add Flatten transformation translator add 9f1bf60 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation add 98ea9fb [BEAM-8470] Use Iterators.transform() to return Iterable add 0b55323 [BEAM-8470] Implement read transform add 4c91a57 [BEAM-8470] update TODO add 4adf3bb [BEAM-8470] Apply spotless add 6b4b916 [BEAM-8470] start source instanciation add ff60578 [BEAM-8470] Improve exception flow add 2ee98da [BEAM-8470] Improve type enforcement in ReadSourceTranslator add fc3abf5 [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table add 4746d9b [BEAM-8470] Add source mocks add e45e48d [BEAM-8470] fix mock, wire mock in translators and create a main test. add 7d7fe77 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary) add 9d84a0f [BEAM-8470] clean deps add b4032aa [BEAM-8470] Move DatasetSourceMock to proper batch mode add a7ad1ab [BEAM-8470] Run pipeline in batch mode or in streaming mode add 141e4bc [BEAM-8470] Split batch and streaming sources and translators add 8954c50 [BEAM-8470] Use raw Encoder also in regular ReadSourceTranslatorBatch add 00ef268 [BEAM-8470] Clean add c426c98 [BEAM-8470] Add ReadSourceTranslatorStreaming add 1740dc4 [BEAM-8470] Move Source and translator mocks to a mock package. add 7819918 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings add b10aa53 [BEAM-8470] Refactor DatasetSource fields add c4bb08c [BEAM-8470] Wire real SourceTransform and not mock and update the test add 5bbea63 [BEAM-8470] Add missing 0-arg public constructor add 0dbe26f [BEAM-8470] Use new PipelineOptionsSerializationUtils add 43052d3 [BEAM-8470] Apply spotless and fix checkstyle add 17ca18b [BEAM-8470] Add a dummy schema for reader add 2e8393b [BEAM-8470] Add empty 0-arg constructor for mock source add 43ff919 [BEAM-8470] Clean add c8ad727 [BEAM-8470] Checkstyle and Findbugs add 6e3575d [BEAM-8470] Refactor SourceTest to a UTest instaed of a main add c221aaa [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset add c26c421 [BEAM-8470] improve readability of options passing to the source add ff69ded [BEAM-8470] Clean unneeded fields in DatasetReader add 524667e [BEAM-8470] Fix serialization issues add 638bdae [BEAM-8470] Add SerializationDebugger add fd354fa [BEAM-8470] Add serialization test add 163102b [BEAM-8470] Move SourceTest to same
[beam] branch master updated (0b415fd -> 18059ee)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0b415fd Merge pull request #10160 More compartmentalization of bundle-based-runner only utilities. add 6f72c93 [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0 add 4ca7e55 [BEAM-8470] Fix missing dep add 0fc0d0a4 [BEAM-8470] Add SparkPipelineOptions add e8ca23e [BEAM-8470] Start pipeline translation add 00964d2 [BEAM-8470] Add global pipeline translation structure add a3b278e [BEAM-8470] Add nodes translators structure add ef4941a [BEAM-8470] Wire node translators with pipeline translator add 8a8dc1e [BEAM-8470] Renames: better differenciate pipeline translator for transform translator add cdfd589 [BEAM-8470] Organise methods in PipelineTranslator add 38eca95 [BEAM-8470] Initialise BatchTranslationContext add 80f2d8c [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components add baf210f [BEAM-8470] Make transform translation clearer: renaming, comments add b65a9da [BEAM-8470] Improve javadocs add 0434749 [BEAM-8470] Move SparkTransformOverrides to correct package add 4372c7e [BEAM-8470] Move common translation context components to superclass add 49b666b [BEAM-8470] apply spotless add 0d6906a [BEAM-8470] Make codestyle and firebug happy add ef97440 [BEAM-8470] Add TODOs add 9abf8ac [BEAM-8470] Post-pone batch qualifier in all classes names for readability add 11a6e19 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN add 47ed3d1 [BEAM-8470] Added SparkRunnerRegistrar add 022a0d0 [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline() add 96b3f36 [BEAM-8470] Create PCollections manipulation methods add b0c42af [BEAM-8470] Create Datasets manipulation methods add 2c5cb23 [BEAM-8470] Add Flatten transformation translator add 9f1bf60 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation add 98ea9fb [BEAM-8470] Use Iterators.transform() to return Iterable add 0b55323 [BEAM-8470] Implement read transform add 4c91a57 [BEAM-8470] update TODO add 4adf3bb [BEAM-8470] Apply spotless add 6b4b916 [BEAM-8470] start source instanciation add ff60578 [BEAM-8470] Improve exception flow add 2ee98da [BEAM-8470] Improve type enforcement in ReadSourceTranslator add fc3abf5 [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table add 4746d9b [BEAM-8470] Add source mocks add e45e48d [BEAM-8470] fix mock, wire mock in translators and create a main test. add 7d7fe77 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary) add 9d84a0f [BEAM-8470] clean deps add b4032aa [BEAM-8470] Move DatasetSourceMock to proper batch mode add a7ad1ab [BEAM-8470] Run pipeline in batch mode or in streaming mode add 141e4bc [BEAM-8470] Split batch and streaming sources and translators add 8954c50 [BEAM-8470] Use raw Encoder also in regular ReadSourceTranslatorBatch add 00ef268 [BEAM-8470] Clean add c426c98 [BEAM-8470] Add ReadSourceTranslatorStreaming add 1740dc4 [BEAM-8470] Move Source and translator mocks to a mock package. add 7819918 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings add b10aa53 [BEAM-8470] Refactor DatasetSource fields add c4bb08c [BEAM-8470] Wire real SourceTransform and not mock and update the test add 5bbea63 [BEAM-8470] Add missing 0-arg public constructor add 0dbe26f [BEAM-8470] Use new PipelineOptionsSerializationUtils add 43052d3 [BEAM-8470] Apply spotless and fix checkstyle add 17ca18b [BEAM-8470] Add a dummy schema for reader add 2e8393b [BEAM-8470] Add empty 0-arg constructor for mock source add 43ff919 [BEAM-8470] Clean add c8ad727 [BEAM-8470] Checkstyle and Findbugs add 6e3575d [BEAM-8470] Refactor SourceTest to a UTest instaed of a main add c221aaa [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset add c26c421 [BEAM-8470] improve readability of options passing to the source add ff69ded [BEAM-8470] Clean unneeded fields in DatasetReader add 524667e [BEAM-8470] Fix serialization issues add 638bdae [BEAM-8470] Add SerializationDebugger add fd354fa [BEAM-8470] Add serialization test add 163102b [BEAM-8470] Move SourceTest to same
[beam] branch master updated (0b415fd -> 18059ee)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0b415fd Merge pull request #10160 More compartmentalization of bundle-based-runner only utilities. add 6f72c93 [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0 add 4ca7e55 [BEAM-8470] Fix missing dep add 0fc0d0a4 [BEAM-8470] Add SparkPipelineOptions add e8ca23e [BEAM-8470] Start pipeline translation add 00964d2 [BEAM-8470] Add global pipeline translation structure add a3b278e [BEAM-8470] Add nodes translators structure add ef4941a [BEAM-8470] Wire node translators with pipeline translator add 8a8dc1e [BEAM-8470] Renames: better differenciate pipeline translator for transform translator add cdfd589 [BEAM-8470] Organise methods in PipelineTranslator add 38eca95 [BEAM-8470] Initialise BatchTranslationContext add 80f2d8c [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components add baf210f [BEAM-8470] Make transform translation clearer: renaming, comments add b65a9da [BEAM-8470] Improve javadocs add 0434749 [BEAM-8470] Move SparkTransformOverrides to correct package add 4372c7e [BEAM-8470] Move common translation context components to superclass add 49b666b [BEAM-8470] apply spotless add 0d6906a [BEAM-8470] Make codestyle and firebug happy add ef97440 [BEAM-8470] Add TODOs add 9abf8ac [BEAM-8470] Post-pone batch qualifier in all classes names for readability add 11a6e19 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN add 47ed3d1 [BEAM-8470] Added SparkRunnerRegistrar add 022a0d0 [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline() add 96b3f36 [BEAM-8470] Create PCollections manipulation methods add b0c42af [BEAM-8470] Create Datasets manipulation methods add 2c5cb23 [BEAM-8470] Add Flatten transformation translator add 9f1bf60 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation add 98ea9fb [BEAM-8470] Use Iterators.transform() to return Iterable add 0b55323 [BEAM-8470] Implement read transform add 4c91a57 [BEAM-8470] update TODO add 4adf3bb [BEAM-8470] Apply spotless add 6b4b916 [BEAM-8470] start source instanciation add ff60578 [BEAM-8470] Improve exception flow add 2ee98da [BEAM-8470] Improve type enforcement in ReadSourceTranslator add fc3abf5 [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table add 4746d9b [BEAM-8470] Add source mocks add e45e48d [BEAM-8470] fix mock, wire mock in translators and create a main test. add 7d7fe77 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary) add 9d84a0f [BEAM-8470] clean deps add b4032aa [BEAM-8470] Move DatasetSourceMock to proper batch mode add a7ad1ab [BEAM-8470] Run pipeline in batch mode or in streaming mode add 141e4bc [BEAM-8470] Split batch and streaming sources and translators add 8954c50 [BEAM-8470] Use raw Encoder also in regular ReadSourceTranslatorBatch add 00ef268 [BEAM-8470] Clean add c426c98 [BEAM-8470] Add ReadSourceTranslatorStreaming add 1740dc4 [BEAM-8470] Move Source and translator mocks to a mock package. add 7819918 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings add b10aa53 [BEAM-8470] Refactor DatasetSource fields add c4bb08c [BEAM-8470] Wire real SourceTransform and not mock and update the test add 5bbea63 [BEAM-8470] Add missing 0-arg public constructor add 0dbe26f [BEAM-8470] Use new PipelineOptionsSerializationUtils add 43052d3 [BEAM-8470] Apply spotless and fix checkstyle add 17ca18b [BEAM-8470] Add a dummy schema for reader add 2e8393b [BEAM-8470] Add empty 0-arg constructor for mock source add 43ff919 [BEAM-8470] Clean add c8ad727 [BEAM-8470] Checkstyle and Findbugs add 6e3575d [BEAM-8470] Refactor SourceTest to a UTest instaed of a main add c221aaa [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset add c26c421 [BEAM-8470] improve readability of options passing to the source add ff69ded [BEAM-8470] Clean unneeded fields in DatasetReader add 524667e [BEAM-8470] Fix serialization issues add 638bdae [BEAM-8470] Add SerializationDebugger add fd354fa [BEAM-8470] Add serialization test add 163102b [BEAM-8470] Move SourceTest to same
