This closes #1663: Merge master (b3de17b) into gearpump-runner Adjustments in gearpump-runner:
[BEAM-79] Upgrade to beam-0.5.0-incubating-SNAPSHOT [BEAM-79] Update to latest Gearpump API >From master: Disable automatic archiving of Maven builds [BEAM-59] initial interfaces and classes of Beam FileSystem. Change counter name in TestDataflowRunner More escaping in Jenkins timestamp spec Add RunnableOnService test for Metrics Fix seed job fetch spec Show timestamps on log lines in Jenkins [BEAM-1165] Fix unexpected file creation when checking dependencies [BEAM-1178] Make naming of logger objects consistent [BEAM-716] Fix javadoc on with* methods [BEAM-959] Improve check preconditions in JmsIO [BEAM-716] Use AutoValue in JmsIO Fix grammar error (repeated for) Empty TestPipeline need not be run [BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage Change counter name in TestDataflowRunner BigQueryIO: fix streaming write, typo in API [BEAM-853] Force streaming execution on batch pipelines for testing. Expose the adapted source. Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle Add timer support to DoFnRunner(s) Make TimerSpec and StateSpec fields accessible View.asMap: minor javadoc fixes Revert "Move InMemoryTimerInternals to runners-core" Revert "Moves DoFnAdapters to runners-core" Revert "Removes ArgumentProvider.windowingInternals" Revert "Removes code for wrapping DoFn as an OldDoFn" checkstyle: missed newline in DistributionCell Make {Metric,Counter,Distribution}Cell public Add PTransformOverrideFactory to the Core SDK Move ActiveWindowSet and implementations to runners-core Update Dataflow worker to beam-master-20161216 [BEAM-1108] Remove outdated language about experimental autoscaling [BEAM-450] Shade modules to separate paths [BEAM-362] Port runners to runners-core AggregatoryFactory Move InMemoryTimerInternals to runners-core Delete deprecated TimerCallback Remove deprecated methods of InMemoryTimerInternals Don't incorrectly log error in MetricsEnvironment Renames ParDo.getNewFn to getFn Moves DoFnAdapters to runners-core Removes unused code from NoOpOldDoFn Removes ArgumentProvider.windowingInternals Removes code for wrapping DoFn as an OldDoFn Removes OldDoFn from ParDo Pushes uses of OldDoFn deeper inside Flink runner Remove ParDo.of(OldDoFn) from Apex runner Converts all easy OldDoFns to DoFn [BEAM-1022] Add testing coverage for BigQuery streaming writes Fix mvn command args in Apex postcommit Jenkins job [BEAM-932] Enable findbugs validation (and fix existing issues) Fail to split in FileBasedSource if filePattern expands to empty. [BEAM-1154] Get side input from proper window in ReduceFn [BEAM-1153] GcsUtil: use non-batch API for single file size requests. Fix NPE in StatefulParDoEvaluatorFactoryTest mocking [BEAM-1033] Retry Bigquery Verifier when Query Fails Implement GetDefaultOutputCoder in DirectGroupByKey SimpleDoFnRunner observes window if SideInputReader is nonempty Better comments and cleanup Allow empty string value for ValueProvider types. starter: fix typo in pom.xml Revert "Allow stateful DoFn in DataflowRunner" Re-exclude UsesStatefulParDo tests for Dataflow Some minor changes and fixes for sorter module [BEAM-1149] Explode windows when fn uses side inputs Add Jenkins postcommit for RunnableOnService in Apex runner Update version from 0.5.0-SNAPSHOT to 0.5.0-incubating-SNAPSHOT Update Maven Archetype versions after cutting the release branch Move PerKeyCombineFnRunner to runners-core Update Dataflow worker to beam-master-20161212 [maven-release-plugin] prepare for next development iteration [maven-release-plugin] prepare branch release-0.4.0-incubating Fix version of Kryo in examples/java jenkins-precommit profile Revert 91cc606 "This closes #1586": Kryo + UBRFBS [BEAM-909] improve starter archetype Fix JDom malformed comment in Apex runner. [BEAM-927] Fix findbugs and re-enable Maven plugin in JmsIO [BEAM-807] Replace OldDoFn with DoFn. [BEAM-757] Use DoFnRunner in the implementation of DoFn via FlatMapFunction. FileBasedSinkTest: fix tests in Windows OS by using IOChannelUtils.resolve(). FileBasedSink: ignore exceptions when removing temp output files for issues in Windows OS. [BEAM-1142] Upgrade maven-invoker to address maven bug ARCHETYPE-488. Add Tests for Kryo Serialization of URFBS Add no-arg constructor for UnboundedReadFromBoundedSource Revise WindowedWordCount for runner and execution mode portability Factor out ShardedFile from FileChecksumMatcher Add IntervalWindow coder to the standard registry Stop expanding PValues in DirectRunner visitors Migrate AppliedPTransform to use AutoValue Enable and fix DirectRunnerTest case missing @Test [BEAM-1130] SparkRunner ResumeFromCheckpointStreamingTest Failing. [BEAM-1133] Add maxNumRecords per micro-batch for Spark runner options. BigQueryIO.Write: support runtime schema and table Fix handling of null ValueProviders in DisplayData [BEAM-551] Fix handling of default for VP [BEAM-1120] Move some DataflowRunner configurations from code to properties [BEAM-551] Fix toString for FileBasedSource [BEAM-921] spark-runner: register sources and coders to serialize with java serializer [BEAM-551] Fix handling of TextIO.Sink ... Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/4c445dd0 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/4c445dd0 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/4c445dd0 Branch: refs/heads/gearpump-runner Commit: 4c445dd0b6de0f5045c02579cb432da4fbc5d486 Parents: b6e7bb6 647034c Author: Kenneth Knowles <k...@google.com> Authored: Wed Dec 21 12:20:11 2016 -0800 Committer: Kenneth Knowles <k...@google.com> Committed: Wed Dec 21 12:20:11 2016 -0800 ---------------------------------------------------------------------- .jenkins/common_job_properties.groovy | 7 + ...ostCommit_Java_RunnableOnService_Apex.groovy | 41 ++ README.md | 2 +- examples/java/pom.xml | 20 +- .../apache/beam/examples/WindowedWordCount.java | 177 +++--- .../org/apache/beam/examples/WordCount.java | 2 +- .../examples/common/WriteWindowedFilesDoFn.java | 77 +++ .../beam/examples/complete/AutoComplete.java | 6 +- .../apache/beam/examples/complete/TfIdf.java | 6 +- .../examples/complete/TopWikipediaSessions.java | 15 +- .../examples/complete/TrafficMaxLaneFlow.java | 4 +- .../beam/examples/complete/TrafficRoutes.java | 4 +- .../examples/cookbook/BigQueryTornadoes.java | 2 +- .../cookbook/CombinePerKeyExamples.java | 2 +- .../beam/examples/cookbook/FilterExamples.java | 2 +- .../examples/cookbook/MaxPerKeyExamples.java | 2 +- .../beam/examples/cookbook/TriggerExample.java | 4 +- .../beam/examples/WindowedWordCountIT.java | 182 +++++- .../examples/complete/AutoCompleteTest.java | 2 +- examples/java8/pom.xml | 2 +- .../beam/examples/complete/game/GameStats.java | 2 +- .../examples/complete/game/LeaderBoard.java | 4 +- .../beam/examples/complete/game/UserScore.java | 2 +- .../complete/game/utils/WriteToBigQuery.java | 2 +- .../game/utils/WriteWindowedToBigQuery.java | 7 +- examples/pom.xml | 2 +- pom.xml | 5 +- runners/apex/pom.xml | 66 ++- .../apache/beam/runners/apex/ApexRunner.java | 111 ++-- .../beam/runners/apex/ApexRunnerResult.java | 50 +- .../beam/runners/apex/ApexYarnLauncher.java | 395 +++++++++++++ .../translation/ApexPipelineTranslator.java | 2 + .../translation/ParDoBoundMultiTranslator.java | 22 +- .../apex/translation/ParDoBoundTranslator.java | 22 +- .../apex/translation/TranslationContext.java | 4 +- .../apex/translation/WindowBoundTranslator.java | 78 +++ .../operators/ApexGroupByKeyOperator.java | 4 +- .../operators/ApexParDoOperator.java | 27 +- .../beam/runners/apex/ApexYarnLauncherTest.java | 138 +++++ .../FlattenPCollectionTranslatorTest.java | 15 +- .../translation/GroupByKeyTranslatorTest.java | 21 +- .../translation/ParDoBoundTranslatorTest.java | 38 +- .../translation/ReadUnboundTranslatorTest.java | 15 +- runners/core-java/pom.xml | 6 +- .../beam/runners/core/ActiveWindowSet.java | 173 ++++++ .../apache/beam/runners/core/AssignWindows.java | 46 -- .../apache/beam/runners/core/DoFnRunner.java | 24 +- .../apache/beam/runners/core/DoFnRunners.java | 2 - .../core/GroupAlsoByWindowViaWindowSetDoFn.java | 12 +- .../GroupAlsoByWindowsViaOutputBufferDoFn.java | 48 +- .../core/GroupByKeyViaGroupByKeyOnly.java | 8 +- .../apache/beam/runners/core/KeyedWorkItem.java | 44 ++ .../beam/runners/core/KeyedWorkItemCoder.java | 130 +++++ .../beam/runners/core/KeyedWorkItems.java | 122 ++++ .../core/LateDataDroppingDoFnRunner.java | 34 +- .../runners/core/MergingActiveWindowSet.java | 428 ++++++++++++++ .../runners/core/NonMergingActiveWindowSet.java | 91 +++ .../beam/runners/core/PaneInfoTracker.java | 4 + .../runners/core/PerKeyCombineFnRunner.java | 149 +++++ .../runners/core/PerKeyCombineFnRunners.java | 1 - .../core/PushbackSideInputDoFnRunner.java | 31 +- .../runners/core/ReduceFnContextFactory.java | 26 +- .../beam/runners/core/ReduceFnRunner.java | 496 ++++++++++------ .../beam/runners/core/SimpleDoFnRunner.java | 252 ++++++++- .../beam/runners/core/SimpleOldDoFnRunner.java | 9 +- .../beam/runners/core/SplittableParDo.java | 28 +- .../core/UnboundedReadFromBoundedSource.java | 2 +- .../apache/beam/runners/core/WatermarkHold.java | 5 + .../TriggerStateMachineContextFactory.java | 2 +- .../triggers/TriggerStateMachineRunner.java | 14 +- .../runners/core/KeyedWorkItemCoderTest.java | 64 +++ .../core/LateDataDroppingDoFnRunnerTest.java | 3 + .../core/MergingActiveWindowSetTest.java | 262 +++++++++ .../core/PushbackSideInputDoFnRunnerTest.java | 55 +- .../beam/runners/core/ReduceFnRunnerTest.java | 133 ++--- .../beam/runners/core/ReduceFnTester.java | 77 ++- .../beam/runners/core/SimpleDoFnRunnerTest.java | 301 ++++++++++ .../beam/runners/core/SplittableParDoTest.java | 2 - .../triggers/TriggerStateMachineTester.java | 25 +- runners/direct-java/pom.xml | 3 +- .../runners/direct/AggregatorContainer.java | 2 +- .../CopyOnAccessInMemoryStateInternals.java | 467 +++++++++++++++ .../runners/direct/DirectExecutionContext.java | 1 - ...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 9 +- .../beam/runners/direct/DirectGraphVisitor.java | 23 +- .../beam/runners/direct/DirectGroupByKey.java | 46 +- .../direct/DirectGroupByKeyOverrideFactory.java | 3 +- .../beam/runners/direct/DirectRunner.java | 7 +- .../beam/runners/direct/EvaluationContext.java | 1 - .../direct/ExecutorServiceParallelExecutor.java | 4 +- .../runners/direct/ForwardingPTransform.java | 6 +- .../GroupAlsoByWindowEvaluatorFactory.java | 8 +- .../direct/GroupByKeyOnlyEvaluatorFactory.java | 4 +- .../direct/KeyedPValueTrackingVisitor.java | 4 +- .../direct/PTransformOverrideFactory.java | 35 -- .../beam/runners/direct/ParDoEvaluator.java | 1 - .../runners/direct/ParDoEvaluatorFactory.java | 2 +- .../direct/ParDoMultiOverrideFactory.java | 29 +- .../ParDoSingleViaMultiOverrideFactory.java | 17 +- ...littableProcessElementsEvaluatorFactory.java | 2 +- .../direct/StatefulParDoEvaluatorFactory.java | 4 +- .../runners/direct/StepTransformResult.java | 1 - .../direct/TestStreamEvaluatorFactory.java | 7 +- .../beam/runners/direct/TransformResult.java | 1 - .../runners/direct/ViewEvaluatorFactory.java | 8 +- .../direct/WriteWithShardingFactory.java | 10 +- .../direct/BoundedReadEvaluatorFactoryTest.java | 18 +- .../runners/direct/CommittedResultTest.java | 2 +- .../CopyOnAccessInMemoryStateInternalsTest.java | 562 +++++++++++++++++++ .../runners/direct/DirectGraphVisitorTest.java | 71 ++- .../beam/runners/direct/DirectGraphs.java | 35 ++ .../beam/runners/direct/DirectRunnerTest.java | 36 -- .../runners/direct/EvaluationContextTest.java | 83 +-- .../direct/FlattenEvaluatorFactoryTest.java | 15 +- .../direct/ForwardingPTransformTest.java | 4 +- .../direct/GroupByKeyEvaluatorFactoryTest.java | 6 +- .../GroupByKeyOnlyEvaluatorFactoryTest.java | 7 +- .../ImmutabilityEnforcementFactoryTest.java | 2 +- .../direct/KeyedPValueTrackingVisitorTest.java | 4 +- .../beam/runners/direct/ParDoEvaluatorTest.java | 3 +- .../StatefulParDoEvaluatorFactoryTest.java | 11 +- .../runners/direct/StepTransformResultTest.java | 2 +- .../direct/TestStreamEvaluatorFactoryTest.java | 14 +- .../runners/direct/TransformExecutorTest.java | 9 +- .../UnboundedReadEvaluatorFactoryTest.java | 24 +- .../direct/ViewEvaluatorFactoryTest.java | 4 +- .../direct/WatermarkCallbackExecutorTest.java | 6 +- .../runners/direct/WatermarkManagerTest.java | 237 ++++---- .../direct/WriteWithShardingFactoryTest.java | 4 +- runners/flink/examples/pom.xml | 14 +- .../beam/runners/flink/examples/TFIDF.java | 6 +- .../beam/runners/flink/examples/WordCount.java | 2 +- .../flink/examples/streaming/AutoComplete.java | 6 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 10 +- .../FlinkPipelineExecutionEnvironment.java | 12 + .../runners/flink/FlinkPipelineOptions.java | 5 + .../apache/beam/runners/flink/FlinkRunner.java | 24 +- .../FlinkBatchPipelineTranslator.java | 4 +- .../FlinkBatchTransformTranslators.java | 59 +- .../FlinkStreamingPipelineTranslator.java | 7 +- .../FlinkStreamingTransformTranslators.java | 77 ++- .../functions/FlinkDoFnFunction.java | 10 +- .../FlinkMergingNonShuffleReduceFunction.java | 2 +- .../FlinkMergingPartialReduceFunction.java | 2 +- .../functions/FlinkMergingReduceFunction.java | 2 +- .../functions/FlinkMultiOutputDoFnFunction.java | 10 +- .../functions/FlinkPartialReduceFunction.java | 2 +- .../functions/FlinkProcessContextBase.java | 21 +- .../functions/FlinkReduceFunction.java | 2 +- .../wrappers/streaming/DoFnOperator.java | 46 +- .../streaming/SingletonKeyedWorkItem.java | 2 +- .../streaming/SingletonKeyedWorkItemCoder.java | 4 +- .../wrappers/streaming/WindowDoFnOperator.java | 14 +- .../wrappers/streaming/WorkItemKeySelector.java | 2 +- .../beam/runners/flink/PipelineOptionsTest.java | 6 +- .../flink/streaming/DoFnOperatorTest.java | 13 +- .../flink/streaming/GroupByNullKeyTest.java | 18 +- .../streaming/TopWikipediaSessionsITCase.java | 10 +- runners/gearpump/pom.xml | 2 +- .../beam/runners/gearpump/GearpumpRunner.java | 2 +- .../gearpump/examples/StreamingWordCount.java | 14 +- .../translators/ParDoBoundMultiTranslator.java | 2 +- .../translators/ParDoBoundTranslator.java | 2 +- .../translators/TranslationContext.java | 3 +- .../translators/utils/DoFnRunnerFactory.java | 2 +- .../utils/NoOpAggregatorFactory.java | 2 +- runners/google-cloud-dataflow-java/pom.xml | 12 +- .../beam/runners/dataflow/DataflowClient.java | 140 +++++ .../runners/dataflow/DataflowPipelineJob.java | 31 +- .../dataflow/DataflowPipelineTranslator.java | 15 +- .../beam/runners/dataflow/DataflowRunner.java | 136 +++-- .../runners/dataflow/DataflowRunnerInfo.java | 92 +++ .../dataflow/internal/AssignWindows.java | 10 +- .../DataflowUnboundedReadFromBoundedSource.java | 2 +- .../DataflowPipelineWorkerPoolOptions.java | 57 +- .../options/DataflowProfilingOptions.java | 8 +- .../dataflow/testing/TestDataflowRunner.java | 35 +- .../dataflow/util/DataflowTemplateJob.java | 2 +- .../runners/dataflow/util/MonitoringUtil.java | 22 +- .../beam/runners/dataflow/dataflow.properties | 23 + .../dataflow/DataflowPipelineJobTest.java | 49 +- .../DataflowPipelineTranslatorTest.java | 61 +- .../dataflow/DataflowRunnerInfoTest.java | 51 ++ .../runners/dataflow/DataflowRunnerTest.java | 285 ++++++---- .../options/DataflowProfilingOptionsTest.java | 6 +- .../testing/TestDataflowRunnerTest.java | 52 +- .../transforms/DataflowGroupByKeyTest.java | 18 +- .../dataflow/transforms/DataflowViewTest.java | 20 +- .../dataflow/util/MonitoringUtilTest.java | 21 +- runners/pom.xml | 2 +- runners/spark/README.md | 59 +- runners/spark/pom.xml | 55 +- .../beam/runners/spark/EvaluationResult.java | 67 --- .../runners/spark/SparkPipelineOptions.java | 10 + .../beam/runners/spark/SparkPipelineResult.java | 193 +++++++ .../apache/beam/runners/spark/SparkRunner.java | 130 +++-- .../beam/runners/spark/TestSparkRunner.java | 91 ++- .../spark/aggregators/AccumulatorSingleton.java | 6 +- .../spark/aggregators/SparkAggregators.java | 126 +++++ .../metrics/WithNamedAggregatorsSupport.java | 4 +- .../coders/BeamSparkRunnerRegistrator.java | 60 +- .../beam/runners/spark/examples/WordCount.java | 15 +- .../apache/beam/runners/spark/io/ConsoleIO.java | 2 +- .../beam/runners/spark/io/CreateStream.java | 2 +- .../beam/runners/spark/io/SourceDStream.java | 21 +- .../runners/spark/io/SparkUnboundedSource.java | 17 +- .../beam/runners/spark/io/hadoop/HadoopIO.java | 4 +- .../runners/spark/translation/DoFnFunction.java | 110 ++-- .../spark/translation/EvaluationContext.java | 131 +---- .../translation/GroupCombineFunctions.java | 23 +- .../spark/translation/MultiDoFnFunction.java | 135 ++--- .../spark/translation/SparkAssignWindowFn.java | 69 +++ .../spark/translation/SparkContextFactory.java | 2 +- .../translation/SparkGroupAlsoByWindowFn.java | 214 +++++++ .../spark/translation/SparkProcessContext.java | 385 +++---------- .../spark/translation/SparkRuntimeContext.java | 62 +- .../translation/StorageLevelPTransform.java | 2 +- .../spark/translation/TransformTranslator.java | 67 +-- .../spark/translation/TranslationUtils.java | 33 ++ .../spark/translation/WindowingHelpers.java | 3 +- .../streaming/StreamingTransformTranslator.java | 45 +- .../util/SinglePrimitiveOutputPTransform.java | 2 +- .../beam/runners/spark/ForceStreamingTest.java | 123 ++++ .../runners/spark/ProvidedSparkContextTest.java | 6 +- .../runners/spark/SparkPipelineStateTest.java | 217 +++++++ .../spark/aggregators/ClearAggregatorsRule.java | 37 ++ .../metrics/sink/ClearAggregatorsRule.java | 33 -- .../metrics/sink/NamedAggregatorsTest.java | 3 +- .../coders/BeamSparkRunnerRegistratorTest.java | 57 ++ .../beam/runners/spark/io/AvroPipelineTest.java | 2 +- .../beam/runners/spark/io/NumShardsTest.java | 2 +- .../io/hadoop/HadoopFileFormatPipelineTest.java | 2 +- .../spark/translation/SideEffectsTest.java | 59 -- .../streaming/EmptyStreamAssertionTest.java | 4 + .../streaming/KafkaStreamingTest.java | 34 +- .../ResumeFromCheckpointStreamingTest.java | 32 +- .../streaming/utils/PAssertStreaming.java | 31 +- sdks/java/build-tools/pom.xml | 2 +- .../src/main/resources/beam/findbugs-filter.xml | 68 ++- sdks/java/core/pom.xml | 2 +- .../beam/sdk/AggregatorPipelineExtractor.java | 3 +- .../beam/sdk/annotations/Experimental.java | 5 +- .../apache/beam/sdk/coders/CoderRegistry.java | 2 + .../java/org/apache/beam/sdk/io/AvroIO.java | 4 +- .../sdk/io/BoundedReadFromUnboundedSource.java | 16 +- .../org/apache/beam/sdk/io/CountingInput.java | 52 +- .../org/apache/beam/sdk/io/CountingSource.java | 17 +- .../org/apache/beam/sdk/io/FileBasedSink.java | 77 ++- .../org/apache/beam/sdk/io/FileBasedSource.java | 12 +- .../java/org/apache/beam/sdk/io/FileSystem.java | 29 + .../apache/beam/sdk/io/FileSystemRegistrar.java | 49 ++ .../org/apache/beam/sdk/io/FileSystems.java | 155 +++++ .../org/apache/beam/sdk/io/LocalFileSystem.java | 27 + .../beam/sdk/io/LocalFileSystemRegistrar.java | 41 ++ .../java/org/apache/beam/sdk/io/PubsubIO.java | 4 +- .../apache/beam/sdk/io/PubsubUnboundedSink.java | 6 +- .../beam/sdk/io/PubsubUnboundedSource.java | 12 +- .../main/java/org/apache/beam/sdk/io/Read.java | 4 +- .../java/org/apache/beam/sdk/io/TextIO.java | 4 +- .../main/java/org/apache/beam/sdk/io/Write.java | 2 +- .../apache/beam/sdk/io/range/ByteKeyRange.java | 4 +- .../beam/sdk/io/range/ByteKeyRangeTracker.java | 8 +- .../apache/beam/sdk/metrics/CounterCell.java | 10 +- .../beam/sdk/metrics/DistributionCell.java | 14 +- .../org/apache/beam/sdk/metrics/MetricCell.java | 2 +- .../org/apache/beam/sdk/metrics/MetricName.java | 3 +- .../org/apache/beam/sdk/metrics/Metrics.java | 5 +- .../beam/sdk/metrics/MetricsEnvironment.java | 8 +- .../sdk/options/PipelineOptionsFactory.java | 110 +++- .../beam/sdk/options/StreamingOptions.java | 2 +- .../apache/beam/sdk/options/ValueProvider.java | 27 +- .../sdk/runners/PTransformOverrideFactory.java | 41 ++ .../apache/beam/sdk/runners/PipelineRunner.java | 2 +- .../beam/sdk/runners/TransformHierarchy.java | 26 +- .../beam/sdk/testing/BigqueryMatcher.java | 48 +- .../beam/sdk/testing/FileChecksumMatcher.java | 114 +--- .../apache/beam/sdk/testing/GatherAllPanes.java | 2 +- .../org/apache/beam/sdk/testing/PAssert.java | 12 +- .../apache/beam/sdk/testing/TestPipeline.java | 225 +++++++- .../org/apache/beam/sdk/testing/TestStream.java | 2 +- .../apache/beam/sdk/testing/UsesMetrics.java | 24 + .../beam/sdk/testing/UsesTimersInParDo.java | 25 + .../sdk/transforms/AggregatorRetriever.java | 2 +- .../beam/sdk/transforms/AppliedPTransform.java | 75 +-- .../beam/sdk/transforms/ApproximateUnique.java | 4 +- .../org/apache/beam/sdk/transforms/Combine.java | 14 +- .../beam/sdk/transforms/CombineWithContext.java | 2 +- .../org/apache/beam/sdk/transforms/Count.java | 2 +- .../org/apache/beam/sdk/transforms/Create.java | 4 +- .../apache/beam/sdk/transforms/Distinct.java | 4 +- .../org/apache/beam/sdk/transforms/DoFn.java | 22 + .../beam/sdk/transforms/DoFnAdapters.java | 12 + .../apache/beam/sdk/transforms/DoFnTester.java | 29 +- .../org/apache/beam/sdk/transforms/Filter.java | 2 +- .../beam/sdk/transforms/FlatMapElements.java | 2 +- .../org/apache/beam/sdk/transforms/Flatten.java | 4 +- .../apache/beam/sdk/transforms/GroupByKey.java | 2 +- .../org/apache/beam/sdk/transforms/Keys.java | 2 +- .../org/apache/beam/sdk/transforms/KvSwap.java | 2 +- .../org/apache/beam/sdk/transforms/Latest.java | 4 +- .../apache/beam/sdk/transforms/MapElements.java | 2 +- .../apache/beam/sdk/transforms/PTransform.java | 4 +- .../org/apache/beam/sdk/transforms/ParDo.java | 191 +------ .../apache/beam/sdk/transforms/Partition.java | 2 +- .../org/apache/beam/sdk/transforms/Regex.java | 14 +- .../org/apache/beam/sdk/transforms/Sample.java | 2 +- .../org/apache/beam/sdk/transforms/Values.java | 2 +- .../org/apache/beam/sdk/transforms/View.java | 16 +- .../apache/beam/sdk/transforms/WithKeys.java | 2 +- .../beam/sdk/transforms/WithTimestamps.java | 2 +- .../sdk/transforms/display/DisplayData.java | 22 + .../beam/sdk/transforms/join/CoGroupByKey.java | 2 +- .../reflect/ByteBuddyDoFnInvokerFactory.java | 11 + .../sdk/transforms/reflect/DoFnInvoker.java | 8 + .../sdk/transforms/reflect/DoFnSignature.java | 26 +- .../sdk/transforms/reflect/DoFnSignatures.java | 96 +++- .../beam/sdk/transforms/windowing/Window.java | 4 +- .../apache/beam/sdk/util/ActiveWindowSet.java | 173 ------ .../org/apache/beam/sdk/util/ApiSurface.java | 52 +- .../apache/beam/sdk/util/ExecutionContext.java | 8 +- .../beam/sdk/util/ExplicitShardedFile.java | 120 ++++ .../java/org/apache/beam/sdk/util/GcsUtil.java | 107 +++- .../org/apache/beam/sdk/util/KeyedWorkItem.java | 43 -- .../beam/sdk/util/KeyedWorkItemCoder.java | 128 ----- .../apache/beam/sdk/util/KeyedWorkItems.java | 121 ---- .../beam/sdk/util/MergingActiveWindowSet.java | 428 -------------- .../sdk/util/NonMergingActiveWindowSet.java | 91 --- .../beam/sdk/util/NumberedShardedFile.java | 220 ++++++++ .../beam/sdk/util/PerKeyCombineFnRunner.java | 148 ----- .../org/apache/beam/sdk/util/PropertyNames.java | 2 + .../org/apache/beam/sdk/util/ReleaseInfo.java | 4 - .../org/apache/beam/sdk/util/Reshuffle.java | 2 +- .../org/apache/beam/sdk/util/ShardedFile.java | 42 ++ .../CopyOnAccessInMemoryStateInternals.java | 453 --------------- .../sdk/util/state/InMemoryStateInternals.java | 33 +- .../sdk/util/state/InMemoryTimerInternals.java | 126 +++-- .../beam/sdk/util/state/StateInternals.java | 2 +- .../beam/sdk/util/state/TimerCallback.java | 35 -- .../org/apache/beam/sdk/values/PCollection.java | 4 +- .../beam/sdk/values/PCollectionTuple.java | 2 +- .../org/apache/beam/sdk/values/TypedPValue.java | 2 +- .../sdk/AggregatorPipelineExtractorTest.java | 8 +- .../java/org/apache/beam/sdk/PipelineTest.java | 6 +- .../beam/sdk/coders/CoderRegistryTest.java | 4 +- .../apache/beam/sdk/io/CountingInputTest.java | 63 ++- .../apache/beam/sdk/io/CountingSourceTest.java | 10 + .../apache/beam/sdk/io/FileBasedSinkTest.java | 11 +- .../apache/beam/sdk/io/FileBasedSourceTest.java | 26 + .../org/apache/beam/sdk/io/FileSystemsTest.java | 104 ++++ .../sdk/io/LocalFileSystemRegistrarTest.java | 44 ++ .../java/org/apache/beam/sdk/io/TextIOTest.java | 19 + .../java/org/apache/beam/sdk/io/WriteTest.java | 2 +- .../org/apache/beam/sdk/io/XmlSinkTest.java | 4 +- .../apache/beam/sdk/metrics/MetricMatchers.java | 47 +- .../apache/beam/sdk/metrics/MetricsTest.java | 63 ++- .../sdk/options/PipelineOptionsFactoryTest.java | 222 +++++++- .../beam/sdk/options/ValueProviderTest.java | 21 + .../sdk/runners/TransformHierarchyTest.java | 17 +- .../beam/sdk/runners/TransformTreeTest.java | 4 +- .../beam/sdk/testing/BigqueryMatcherTest.java | 82 +-- .../sdk/testing/FileChecksumMatcherTest.java | 77 --- .../beam/sdk/testing/TestPipelineTest.java | 186 +++++- .../beam/sdk/transforms/DoFnTesterTest.java | 2 +- .../sdk/transforms/FlatMapElementsTest.java | 2 +- .../beam/sdk/transforms/GroupByKeyTest.java | 4 +- .../beam/sdk/transforms/MapElementsTest.java | 2 +- .../apache/beam/sdk/transforms/NoOpOldDoFn.java | 72 --- .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++--- .../beam/sdk/transforms/PTransformTest.java | 2 +- .../beam/sdk/transforms/ParDoLifecycleTest.java | 28 +- .../apache/beam/sdk/transforms/ParDoTest.java | 342 ++++++----- .../beam/sdk/transforms/SplittableDoFnTest.java | 58 +- .../apache/beam/sdk/transforms/ViewTest.java | 4 +- .../display/DisplayDataEvaluatorTest.java | 2 +- .../display/DisplayDataMatchersTest.java | 6 +- .../sdk/transforms/display/DisplayDataTest.java | 90 ++- .../DoFnSignaturesSplittableDoFnTest.java | 3 +- .../transforms/reflect/DoFnSignaturesTest.java | 47 ++ .../sdk/transforms/windowing/WindowingTest.java | 2 +- .../org/apache/beam/sdk/util/GcsUtilTest.java | 85 ++- .../beam/sdk/util/KeyedWorkItemCoderTest.java | 62 -- .../sdk/util/MergingActiveWindowSetTest.java | 262 --------- .../beam/sdk/util/NumberedShardedFileTest.java | 181 ++++++ .../apache/beam/sdk/util/StringUtilsTest.java | 6 +- .../CopyOnAccessInMemoryStateInternalsTest.java | 552 ------------------ .../util/state/InMemoryTimerInternalsTest.java | 125 +++-- .../org/apache/beam/sdk/values/PDoneTest.java | 4 +- sdks/java/extensions/join-library/pom.xml | 2 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/README.md | 2 +- sdks/java/extensions/sorter/pom.xml | 10 +- .../sorter/BufferedExternalSorter.java | 6 +- .../sdk/extensions/sorter/ExternalSorter.java | 15 +- .../sdk/extensions/sorter/InMemorySorter.java | 26 +- .../beam/sdk/extensions/sorter/SortValues.java | 2 +- .../sorter/BufferedExternalSorterTest.java | 16 + .../extensions/sorter/ExternalSorterTest.java | 16 + .../extensions/sorter/InMemorySorterTest.java | 8 + sdks/java/io/google-cloud-platform/pom.xml | 8 +- .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 453 +++++++++++---- .../sdk/io/gcp/bigquery/BigQueryServices.java | 7 +- .../io/gcp/bigquery/BigQueryServicesImpl.java | 121 +++- .../io/gcp/bigquery/BigQueryTableInserter.java | 217 ------- .../beam/sdk/io/gcp/bigtable/BigtableIO.java | 28 +- .../io/gcp/bigtable/BigtableServiceImpl.java | 6 +- .../beam/sdk/io/gcp/datastore/DatastoreV1.java | 4 +- .../beam/sdk/io/gcp/storage/GcsFileSystem.java | 34 ++ .../io/gcp/storage/GcsFileSystemRegistrar.java | 42 ++ .../beam/sdk/io/gcp/storage/package-info.java | 21 + .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 2 + .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 543 ++++++++++++++---- .../gcp/bigquery/BigQueryServicesImplTest.java | 139 ++++- .../gcp/bigquery/BigQueryTableInserterTest.java | 245 -------- .../sdk/io/gcp/bigquery/BigQueryUtilTest.java | 50 +- .../gcp/storage/GcsFileSystemRegistrarTest.java | 51 ++ sdks/java/io/hdfs/pom.xml | 8 +- .../beam/sdk/io/hdfs/HadoopFileSystem.java | 29 + .../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 42 ++ .../io/hdfs/HadoopFileSystemRegistrarTest.java | 52 ++ sdks/java/io/jdbc/pom.xml | 2 +- .../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 4 +- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 4 +- sdks/java/io/jms/pom.xml | 22 +- .../java/org/apache/beam/sdk/io/jms/JmsIO.java | 342 +++++++---- .../org/apache/beam/sdk/io/jms/JmsRecord.java | 8 +- sdks/java/io/kafka/pom.xml | 2 +- .../org/apache/beam/sdk/io/kafka/KafkaIO.java | 20 +- sdks/java/io/kinesis/pom.xml | 2 +- sdks/java/io/mongodb/pom.xml | 2 +- .../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +- .../apache/beam/sdk/io/mongodb/MongoDbIO.java | 16 +- .../sdk/io/mongodb/MongoDBGridFSIOTest.java | 8 +- .../beam/sdk/io/mongodb/MongoDbIOTest.java | 8 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- .../maven-archetypes/examples-java8/pom.xml | 13 +- .../main/resources/archetype-resources/pom.xml | 122 +++- sdks/java/maven-archetypes/examples/pom.xml | 13 +- .../main/resources/archetype-resources/pom.xml | 47 +- sdks/java/maven-archetypes/pom.xml | 2 +- sdks/java/maven-archetypes/starter/pom.xml | 9 +- .../main/resources/archetype-resources/pom.xml | 16 +- .../resources/projects/basic/reference/pom.xml | 16 +- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 446 files changed, 13550 insertions(+), 7606 deletions(-) ----------------------------------------------------------------------