This closes #1676: Merge master (57d9bbd) into python-sdk No fixups. Commit from master:
Hold output watermark according to pending timers Allow setting timer by ID in DirectTimerInternals Add UsesTestStream for use with JUnit @Category Add static Window.withOutputTimeFn to match build method Use informative Instant formatter in WatermarkHold Add informative Instant formatter to BoundedWindow Actually propagate and commit state in direct runner Fix windowing in direct runner Stateful ParDo Support set and delete of timer by ID in InMemoryTimerInternals Move ExecutionContext and related classes to runners-core Update Dataflow worker to beam-master-20161221 Require TimeDomain to delete a timer Provide local tags in PInput, POutput expansions Remove deprecated InMemoryTimerInternals from SDK Remove deprecated AggregatorFactory from SDK Update Dataflow worker to beam-master-20161220 Fixup usage of canonical name with name since canonical name != name for inner classes. Provide a better error message for non-existing gcpTempLocation Restore SDK's InMemoryTimerInternals, deprecated Move InMemoryTimerInternals to runners-core Port direct runner StatefulParDo to KeyedWorkItem Propagate key through ParDo if DoFn is key-preserving Move responsibility for knowing about keyedness into EvaluationContext Add some key-preserving to KeyedPValueTrackingVisitor Migrated the beam-sdks-java-io-java8tests module to TestPipeline as a JUnit rule + fixed WithTimestampsJava8Test.withTimestampsLambdaShouldApplyTimestamps. Migrated the beam-sdks-java-io-mongodb module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-io-kinesis module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-io-kafka module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-io-jms module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-io-jdbc module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-io-google-cloud-platform module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-extensions-sorter module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-extensions-join-library module to TestPipeline as a JUnit rule. Migrated the beam-sdks-java-core module to TestPipeline as a JUnit rule. Plus, fixed some checkstyle errors from previous modules' migration. Migrated the beam-runners-direct-java module to TestPipeline as a JUnit rule. Migrated the beam-runners-core module to TestPipeline as a JUnit rule. Migrated the beam-examples-java8 module to TestPipeline as a JUnit rule. Migrated the beam-examples-java module to TestPipeline as a JUnit rule. Disable automatic archiving of Maven builds [BEAM-59] initial interfaces and classes of Beam FileSystem. Change counter name in TestDataflowRunner More escaping in Jenkins timestamp spec Add RunnableOnService test for Metrics Fix seed job fetch spec Show timestamps on log lines in Jenkins [BEAM-1165] Fix unexpected file creation when checking dependencies [BEAM-1178] Make naming of logger objects consistent [BEAM-716] Fix javadoc on with* methods [BEAM-959] Improve check preconditions in JmsIO [BEAM-716] Use AutoValue in JmsIO Fix grammar error (repeated for) Empty TestPipeline need not be run [BEAM-85, BEAM-298] Make TestPipeline a JUnit Rule checking proper usage Change counter name in TestDataflowRunner BigQueryIO: fix streaming write, typo in API [BEAM-853] Force streaming execution on batch pipelines for testing. Expose the adapted source. Use empty SideInputReader, fixes NPE in SimpleDoFnRunnerTest Test that SimpleDoFnRunner wraps exceptions in startBundle and finishBundle Add timer support to DoFnRunner(s) Make TimerSpec and StateSpec fields accessible View.asMap: minor javadoc fixes Revert "Move InMemoryTimerInternals to runners-core" Revert "Moves DoFnAdapters to runners-core" Revert "Removes ArgumentProvider.windowingInternals" Revert "Removes code for wrapping DoFn as an OldDoFn" checkstyle: missed newline in DistributionCell Make {Metric,Counter,Distribution}Cell public Add PTransformOverrideFactory to the Core SDK Move ActiveWindowSet and implementations to runners-core Update Dataflow worker to beam-master-20161216 [BEAM-1108] Remove outdated language about experimental autoscaling [BEAM-450] Shade modules to separate paths [BEAM-362] Port runners to runners-core AggregatoryFactory Move InMemoryTimerInternals to runners-core Delete deprecated TimerCallback Remove deprecated methods of InMemoryTimerInternals Don't incorrectly log error in MetricsEnvironment Renames ParDo.getNewFn to getFn Moves DoFnAdapters to runners-core Removes unused code from NoOpOldDoFn Removes ArgumentProvider.windowingInternals Removes code for wrapping DoFn as an OldDoFn Removes OldDoFn from ParDo Pushes uses of OldDoFn deeper inside Flink runner Remove ParDo.of(OldDoFn) from Apex runner Converts all easy OldDoFns to DoFn [BEAM-1022] Add testing coverage for BigQuery streaming writes Fix mvn command args in Apex postcommit Jenkins job [BEAM-932] Enable findbugs validation (and fix existing issues) Fail to split in FileBasedSource if filePattern expands to empty. [BEAM-1154] Get side input from proper window in ReduceFn [BEAM-1153] GcsUtil: use non-batch API for single file size requests. Fix NPE in StatefulParDoEvaluatorFactoryTest mocking [BEAM-1033] Retry Bigquery Verifier when Query Fails Implement GetDefaultOutputCoder in DirectGroupByKey SimpleDoFnRunner observes window if SideInputReader is nonempty Better comments and cleanup Allow empty string value for ValueProvider types. starter: fix typo in pom.xml Revert "Allow stateful DoFn in DataflowRunner" Re-exclude UsesStatefulParDo tests for Dataflow ... Project: http://git-wip-us.apache.org/repos/asf/incubator-beam/repo Commit: http://git-wip-us.apache.org/repos/asf/incubator-beam/commit/3454d691 Tree: http://git-wip-us.apache.org/repos/asf/incubator-beam/tree/3454d691 Diff: http://git-wip-us.apache.org/repos/asf/incubator-beam/diff/3454d691 Branch: refs/heads/python-sdk Commit: 3454d691f033bd0403058ff7cf0c140a7cf93f71 Parents: 3b4fd5c 7774297 Author: Kenneth Knowles <k...@google.com> Authored: Wed Dec 21 14:47:00 2016 -0800 Committer: Kenneth Knowles <k...@google.com> Committed: Wed Dec 21 14:48:32 2016 -0800 ---------------------------------------------------------------------- .gitignore | 4 + .jenkins/common_job_properties.groovy | 173 ++++++ ...job_beam_PostCommit_Java_MavenInstall.groovy | 42 ++ ...ostCommit_Java_RunnableOnService_Apex.groovy | 41 ++ ...ommit_Java_RunnableOnService_Dataflow.groovy | 39 ++ ...stCommit_Java_RunnableOnService_Flink.groovy | 38 ++ ...ommit_Java_RunnableOnService_Gearpump.groovy | 41 ++ ...stCommit_Java_RunnableOnService_Spark.groovy | 38 ++ .../job_beam_PostCommit_Python_Verify.groovy | 37 ++ .../job_beam_PreCommit_Java_MavenInstall.groovy | 42 ++ .../job_beam_Release_NightlySnapshot.groovy | 46 ++ .jenkins/job_seed.groovy | 47 ++ .travis.yml | 2 +- .travis/README.md | 2 +- README.md | 2 +- examples/java/pom.xml | 20 +- .../apache/beam/examples/WindowedWordCount.java | 177 +++--- .../org/apache/beam/examples/WordCount.java | 2 +- .../examples/common/WriteWindowedFilesDoFn.java | 77 +++ .../beam/examples/complete/AutoComplete.java | 6 +- .../org/apache/beam/examples/complete/README.md | 14 +- .../apache/beam/examples/complete/TfIdf.java | 6 +- .../examples/complete/TopWikipediaSessions.java | 15 +- .../examples/complete/TrafficMaxLaneFlow.java | 4 +- .../beam/examples/complete/TrafficRoutes.java | 4 +- .../examples/cookbook/BigQueryTornadoes.java | 2 +- .../cookbook/CombinePerKeyExamples.java | 2 +- .../beam/examples/cookbook/DeDupExample.java | 96 --- .../beam/examples/cookbook/DistinctExample.java | 96 +++ .../beam/examples/cookbook/FilterExamples.java | 2 +- .../examples/cookbook/MaxPerKeyExamples.java | 2 +- .../org/apache/beam/examples/cookbook/README.md | 16 +- .../beam/examples/cookbook/TriggerExample.java | 4 +- .../beam/examples/WindowedWordCountIT.java | 182 +++++- .../org/apache/beam/examples/WordCountIT.java | 2 +- .../org/apache/beam/examples/WordCountTest.java | 7 +- .../examples/complete/AutoCompleteTest.java | 13 +- .../beam/examples/complete/TfIdfTest.java | 6 +- .../complete/TopWikipediaSessionsTest.java | 7 +- .../examples/cookbook/DistinctExampleTest.java | 9 +- .../examples/cookbook/JoinExamplesTest.java | 6 +- .../examples/cookbook/TriggerExampleTest.java | 6 +- examples/java8/pom.xml | 2 +- .../beam/examples/complete/game/GameStats.java | 2 +- .../examples/complete/game/LeaderBoard.java | 4 +- .../beam/examples/complete/game/UserScore.java | 2 +- .../complete/game/utils/WriteToBigQuery.java | 2 +- .../game/utils/WriteWindowedToBigQuery.java | 7 +- .../examples/MinimalWordCountJava8Test.java | 6 +- .../examples/complete/game/GameStatsTest.java | 7 +- .../complete/game/HourlyTeamScoreTest.java | 5 +- .../examples/complete/game/LeaderBoardTest.java | 11 +- .../examples/complete/game/UserScoreTest.java | 10 +- examples/pom.xml | 2 +- pom.xml | 15 +- runners/apex/pom.xml | 69 ++- .../apache/beam/runners/apex/ApexRunner.java | 111 ++-- .../beam/runners/apex/ApexRunnerResult.java | 50 +- .../beam/runners/apex/ApexYarnLauncher.java | 395 ++++++++++++ .../translation/ApexPipelineTranslator.java | 14 +- .../translation/ParDoBoundMultiTranslator.java | 22 +- .../apex/translation/ParDoBoundTranslator.java | 22 +- .../apex/translation/TranslationContext.java | 10 +- .../apex/translation/WindowBoundTranslator.java | 78 +++ .../operators/ApexGroupByKeyOperator.java | 26 +- .../operators/ApexParDoOperator.java | 29 +- .../apex/translation/utils/NoOpStepContext.java | 3 +- .../beam/runners/apex/ApexYarnLauncherTest.java | 138 +++++ .../FlattenPCollectionTranslatorTest.java | 15 +- .../translation/GroupByKeyTranslatorTest.java | 21 +- .../translation/ParDoBoundTranslatorTest.java | 38 +- .../translation/ReadUnboundTranslatorTest.java | 15 +- runners/core-java/pom.xml | 6 +- .../beam/runners/core/ActiveWindowSet.java | 173 ++++++ .../beam/runners/core/AggregatorFactory.java | 1 - .../apache/beam/runners/core/AssignWindows.java | 46 -- .../beam/runners/core/BaseExecutionContext.java | 176 ++++++ .../apache/beam/runners/core/DoFnRunner.java | 24 +- .../apache/beam/runners/core/DoFnRunners.java | 4 +- .../core/ElementAndRestrictionCoder.java | 8 + .../beam/runners/core/ExecutionContext.java | 102 +++ .../runners/core/GBKIntoKeyedWorkItems.java | 55 -- .../core/GroupAlsoByWindowViaWindowSetDoFn.java | 12 +- .../GroupAlsoByWindowsViaOutputBufferDoFn.java | 55 +- .../core/GroupByKeyViaGroupByKeyOnly.java | 30 +- .../runners/core/InMemoryTimerInternals.java | 302 +++++++++ .../apache/beam/runners/core/KeyedWorkItem.java | 44 ++ .../beam/runners/core/KeyedWorkItemCoder.java | 130 ++++ .../beam/runners/core/KeyedWorkItems.java | 122 ++++ .../core/LateDataDroppingDoFnRunner.java | 34 +- .../runners/core/MergingActiveWindowSet.java | 428 +++++++++++++ .../runners/core/NonMergingActiveWindowSet.java | 91 +++ .../beam/runners/core/PaneInfoTracker.java | 4 + .../runners/core/PerKeyCombineFnRunner.java | 149 +++++ .../runners/core/PerKeyCombineFnRunners.java | 262 ++++++++ .../core/PushbackSideInputDoFnRunner.java | 11 + .../runners/core/ReduceFnContextFactory.java | 26 +- .../beam/runners/core/ReduceFnRunner.java | 496 +++++++++------ .../beam/runners/core/SimpleDoFnRunner.java | 328 +++++++++- .../beam/runners/core/SimpleOldDoFnRunner.java | 27 +- .../beam/runners/core/SplittableParDo.java | 392 +++++++++--- .../core/UnboundedReadFromBoundedSource.java | 2 +- .../apache/beam/runners/core/WatermarkHold.java | 9 +- .../TriggerStateMachineContextFactory.java | 2 +- .../triggers/TriggerStateMachineRunner.java | 14 +- .../core/GroupAlsoByWindowsProperties.java | 590 ++++++++++-------- .../core/InMemoryTimerInternalsTest.java | 197 ++++++ .../runners/core/KeyedWorkItemCoderTest.java | 64 ++ .../core/LateDataDroppingDoFnRunnerTest.java | 3 + .../core/MergingActiveWindowSetTest.java | 262 ++++++++ .../core/PushbackSideInputDoFnRunnerTest.java | 60 +- .../beam/runners/core/ReduceFnRunnerTest.java | 138 +++-- .../beam/runners/core/ReduceFnTester.java | 78 ++- .../beam/runners/core/SimpleDoFnRunnerTest.java | 301 +++++++++ .../runners/core/SimpleOldDoFnRunnerTest.java | 3 +- .../beam/runners/core/SplittableParDoTest.java | 158 ++++- .../UnboundedReadFromBoundedSourceTest.java | 6 +- .../beam/runners/core/WindowMatchers.java | 204 ++++++ .../beam/runners/core/WindowMatchersTest.java | 82 +++ .../triggers/TriggerStateMachineTester.java | 27 +- runners/direct-java/pom.xml | 3 +- .../direct/AbstractModelEnforcement.java | 2 +- .../runners/direct/AggregatorContainer.java | 4 +- .../direct/BoundedReadEvaluatorFactory.java | 8 +- .../beam/runners/direct/CommittedResult.java | 2 +- .../beam/runners/direct/CompletionCallback.java | 2 +- .../direct/ConsumerTrackingPipelineVisitor.java | 173 ------ .../CopyOnAccessInMemoryStateInternals.java | 467 ++++++++++++++ .../runners/direct/DirectExecutionContext.java | 7 +- ...ectGBKIntoKeyedWorkItemsOverrideFactory.java | 42 +- .../apache/beam/runners/direct/DirectGraph.java | 89 +++ .../beam/runners/direct/DirectGraphVisitor.java | 142 +++++ .../beam/runners/direct/DirectGroupByKey.java | 46 +- .../direct/DirectGroupByKeyOverrideFactory.java | 3 +- .../beam/runners/direct/DirectRunner.java | 62 +- .../runners/direct/DirectTimerInternals.java | 14 +- .../runners/direct/DisplayDataValidator.java | 6 +- .../runners/direct/DoFnLifecycleManager.java | 4 +- ...ecycleManagerRemovingTransformEvaluator.java | 2 +- .../runners/direct/EmptyTransformEvaluator.java | 4 +- .../beam/runners/direct/EvaluationContext.java | 105 ++-- .../direct/ExecutorServiceParallelExecutor.java | 29 +- .../runners/direct/FlattenEvaluatorFactory.java | 10 +- .../runners/direct/ForwardingPTransform.java | 6 +- .../GroupAlsoByWindowEvaluatorFactory.java | 13 +- .../direct/GroupByKeyOnlyEvaluatorFactory.java | 6 +- .../ImmutabilityCheckingBundleFactory.java | 21 +- .../direct/ImmutabilityEnforcementFactory.java | 2 +- .../direct/KeyedPValueTrackingVisitor.java | 64 +- .../beam/runners/direct/ModelEnforcement.java | 2 +- .../direct/PTransformOverrideFactory.java | 35 -- .../beam/runners/direct/ParDoEvaluator.java | 45 +- .../runners/direct/ParDoEvaluatorFactory.java | 97 ++- .../direct/ParDoMultiOverrideFactory.java | 193 +++++- .../ParDoSingleViaMultiOverrideFactory.java | 21 +- .../direct/PassthroughTransformEvaluator.java | 4 +- ...littableProcessElementsEvaluatorFactory.java | 145 +++++ .../direct/StatefulParDoEvaluatorFactory.java | 257 ++++++++ .../runners/direct/StepTransformResult.java | 39 +- .../direct/TestStreamEvaluatorFactory.java | 9 +- .../beam/runners/direct/TransformEvaluator.java | 2 +- .../direct/TransformEvaluatorRegistry.java | 7 + .../beam/runners/direct/TransformExecutor.java | 4 +- .../beam/runners/direct/TransformResult.java | 17 +- .../direct/UnboundedReadEvaluatorFactory.java | 3 +- .../runners/direct/ViewEvaluatorFactory.java | 10 +- .../direct/WatermarkCallbackExecutor.java | 34 + .../beam/runners/direct/WatermarkManager.java | 139 +++-- .../runners/direct/WindowEvaluatorFactory.java | 6 +- .../direct/WriteWithShardingFactory.java | 10 +- .../runners/direct/AggregatorContainerTest.java | 2 +- .../direct/BoundedReadEvaluatorFactoryTest.java | 47 +- .../direct/CloningBundleFactoryTest.java | 8 +- .../runners/direct/CommittedResultTest.java | 8 +- .../ConsumerTrackingPipelineVisitorTest.java | 287 --------- .../CopyOnAccessInMemoryStateInternalsTest.java | 565 +++++++++++++++++ .../runners/direct/DirectGraphVisitorTest.java | 265 ++++++++ .../beam/runners/direct/DirectGraphs.java | 35 ++ .../beam/runners/direct/DirectRunnerTest.java | 36 -- ...leManagerRemovingTransformEvaluatorTest.java | 4 +- .../runners/direct/EvaluationContextTest.java | 132 ++-- .../direct/FlattenEvaluatorFactoryTest.java | 27 +- .../direct/ForwardingPTransformTest.java | 4 +- .../direct/GroupByKeyEvaluatorFactoryTest.java | 11 +- .../GroupByKeyOnlyEvaluatorFactoryTest.java | 12 +- .../ImmutabilityCheckingBundleFactoryTest.java | 10 +- .../ImmutabilityEnforcementFactoryTest.java | 11 +- .../direct/ImmutableListBundleFactoryTest.java | 14 +- .../direct/KeyedPValueTrackingVisitorTest.java | 137 ++-- .../beam/runners/direct/ParDoEvaluatorTest.java | 11 +- .../runners/direct/SideInputContainerTest.java | 5 +- .../beam/runners/direct/SplittableDoFnTest.java | 231 ------- .../StatefulParDoEvaluatorFactoryTest.java | 323 ++++++++++ .../runners/direct/StepTransformResultTest.java | 32 +- .../direct/TestStreamEvaluatorFactoryTest.java | 29 +- .../runners/direct/TransformExecutorTest.java | 228 +++---- .../UnboundedReadEvaluatorFactoryTest.java | 53 +- .../direct/ViewEvaluatorFactoryTest.java | 9 +- .../direct/WatermarkCallbackExecutorTest.java | 11 +- .../runners/direct/WatermarkManagerTest.java | 260 ++++---- .../direct/WindowEvaluatorFactoryTest.java | 21 +- .../direct/WriteWithShardingFactoryTest.java | 18 +- runners/flink/examples/pom.xml | 14 +- .../beam/runners/flink/examples/TFIDF.java | 6 +- .../beam/runners/flink/examples/WordCount.java | 2 +- .../flink/examples/streaming/AutoComplete.java | 6 +- runners/flink/pom.xml | 2 +- runners/flink/runner/pom.xml | 15 +- .../FlinkPipelineExecutionEnvironment.java | 12 + .../runners/flink/FlinkPipelineOptions.java | 5 + .../apache/beam/runners/flink/FlinkRunner.java | 36 +- .../FlinkBatchPipelineTranslator.java | 18 +- .../FlinkBatchTransformTranslators.java | 55 +- .../FlinkStreamingPipelineTranslator.java | 23 +- .../FlinkStreamingTransformTranslators.java | 140 ++--- .../PipelineTranslationOptimizer.java | 10 +- .../functions/FlinkDoFnFunction.java | 10 +- .../FlinkMergingNonShuffleReduceFunction.java | 4 +- .../FlinkMergingPartialReduceFunction.java | 4 +- .../functions/FlinkMergingReduceFunction.java | 4 +- .../functions/FlinkMultiOutputDoFnFunction.java | 10 +- .../functions/FlinkPartialReduceFunction.java | 4 +- .../functions/FlinkProcessContextBase.java | 29 +- .../functions/FlinkReduceFunction.java | 4 +- .../wrappers/streaming/DoFnOperator.java | 48 +- .../streaming/SingletonKeyedWorkItem.java | 2 +- .../streaming/SingletonKeyedWorkItemCoder.java | 4 +- .../wrappers/streaming/WindowDoFnOperator.java | 25 +- .../wrappers/streaming/WorkItemKeySelector.java | 2 +- .../beam/runners/flink/PipelineOptionsTest.java | 6 +- .../flink/streaming/DoFnOperatorTest.java | 13 +- .../flink/streaming/GroupByNullKeyTest.java | 18 +- .../streaming/TopWikipediaSessionsITCase.java | 10 +- runners/google-cloud-dataflow-java/pom.xml | 14 +- .../beam/runners/dataflow/DataflowClient.java | 140 +++++ .../runners/dataflow/DataflowPipelineJob.java | 31 +- .../dataflow/DataflowPipelineTranslator.java | 36 +- .../beam/runners/dataflow/DataflowRunner.java | 210 ++++--- .../runners/dataflow/DataflowRunnerInfo.java | 92 +++ .../dataflow/internal/AssignWindows.java | 10 +- .../DataflowUnboundedReadFromBoundedSource.java | 2 +- .../options/DataflowPipelineOptions.java | 19 +- .../DataflowPipelineWorkerPoolOptions.java | 57 +- .../options/DataflowProfilingOptions.java | 8 +- .../dataflow/testing/TestDataflowRunner.java | 35 +- .../dataflow/util/DataflowTemplateJob.java | 2 +- .../runners/dataflow/util/MonitoringUtil.java | 22 +- .../beam/runners/dataflow/dataflow.properties | 23 + .../dataflow/DataflowPipelineJobTest.java | 49 +- .../DataflowPipelineTranslatorTest.java | 63 +- .../dataflow/DataflowRunnerInfoTest.java | 51 ++ .../runners/dataflow/DataflowRunnerTest.java | 333 ++++++---- .../dataflow/RecordingPipelineVisitor.java | 6 +- .../options/DataflowPipelineOptionsTest.java | 20 +- .../options/DataflowProfilingOptionsTest.java | 6 +- .../testing/TestDataflowRunnerTest.java | 52 +- .../transforms/DataflowGroupByKeyTest.java | 18 +- .../dataflow/transforms/DataflowViewTest.java | 20 +- .../dataflow/util/MonitoringUtilTest.java | 21 +- runners/pom.xml | 2 +- runners/spark/README.md | 59 +- runners/spark/pom.xml | 60 +- .../beam/runners/spark/EvaluationResult.java | 67 -- .../runners/spark/SparkPipelineOptions.java | 15 + .../beam/runners/spark/SparkPipelineResult.java | 193 ++++++ .../apache/beam/runners/spark/SparkRunner.java | 213 +++++-- .../beam/runners/spark/TestSparkRunner.java | 91 ++- .../spark/aggregators/AccumulatorSingleton.java | 8 +- .../spark/aggregators/SparkAggregators.java | 126 ++++ .../metrics/WithNamedAggregatorsSupport.java | 4 +- .../coders/BeamSparkRunnerRegistrator.java | 60 +- .../beam/runners/spark/examples/WordCount.java | 15 +- .../apache/beam/runners/spark/io/ConsoleIO.java | 2 +- .../beam/runners/spark/io/CreateStream.java | 2 +- .../beam/runners/spark/io/MicrobatchSource.java | 20 +- .../beam/runners/spark/io/SourceDStream.java | 22 +- .../runners/spark/io/SparkUnboundedSource.java | 17 +- .../beam/runners/spark/io/hadoop/HadoopIO.java | 4 +- .../spark/stateful/StateSpecFunctions.java | 2 +- .../spark/translation/BoundedDataset.java | 5 +- .../beam/runners/spark/translation/Dataset.java | 2 +- .../runners/spark/translation/DoFnFunction.java | 110 ++-- .../spark/translation/EvaluationContext.java | 137 +--- .../translation/GroupCombineFunctions.java | 23 +- .../spark/translation/MultiDoFnFunction.java | 135 ++-- .../spark/translation/SparkAssignWindowFn.java | 69 +++ .../spark/translation/SparkContextFactory.java | 2 +- .../translation/SparkGroupAlsoByWindowFn.java | 214 +++++++ .../spark/translation/SparkProcessContext.java | 390 +++--------- .../spark/translation/SparkRuntimeContext.java | 62 +- .../translation/StorageLevelPTransform.java | 43 ++ .../spark/translation/TransformTranslator.java | 91 ++- .../spark/translation/TranslationUtils.java | 33 + .../spark/translation/WindowingHelpers.java | 3 +- .../SparkRunnerStreamingContextFactory.java | 2 +- .../streaming/StreamingTransformTranslator.java | 47 +- .../translation/streaming/UnboundedDataset.java | 13 +- .../util/SinglePrimitiveOutputPTransform.java | 2 +- .../beam/runners/spark/ForceStreamingTest.java | 123 ++++ .../runners/spark/ProvidedSparkContextTest.java | 6 +- .../runners/spark/SparkPipelineStateTest.java | 217 +++++++ .../spark/aggregators/ClearAggregatorsRule.java | 37 ++ .../metrics/sink/ClearAggregatorsRule.java | 33 - .../metrics/sink/NamedAggregatorsTest.java | 3 +- .../coders/BeamSparkRunnerRegistratorTest.java | 57 ++ .../beam/runners/spark/io/AvroPipelineTest.java | 2 +- .../beam/runners/spark/io/NumShardsTest.java | 2 +- .../io/hadoop/HadoopFileFormatPipelineTest.java | 2 +- .../spark/translation/SideEffectsTest.java | 59 -- .../spark/translation/StorageLevelTest.java | 56 ++ .../streaming/EmptyStreamAssertionTest.java | 6 + .../streaming/FlattenStreamingTest.java | 2 + .../streaming/KafkaStreamingTest.java | 34 +- .../ResumeFromCheckpointStreamingTest.java | 32 +- .../streaming/SimpleStreamingWordCountTest.java | 1 + .../streaming/utils/PAssertStreaming.java | 31 +- .../SparkTestPipelineOptionsForStreaming.java | 6 - sdks/java/build-tools/pom.xml | 2 +- .../src/main/resources/beam/findbugs-filter.xml | 68 +- sdks/java/core/pom.xml | 31 +- .../beam/sdk/AggregatorPipelineExtractor.java | 9 +- .../main/java/org/apache/beam/sdk/Pipeline.java | 151 +---- .../beam/sdk/annotations/Experimental.java | 5 +- .../apache/beam/sdk/coders/CoderRegistry.java | 2 + .../java/org/apache/beam/sdk/io/AvroIO.java | 4 +- .../sdk/io/BoundedReadFromUnboundedSource.java | 16 +- .../org/apache/beam/sdk/io/CountingInput.java | 52 +- .../org/apache/beam/sdk/io/CountingSource.java | 17 +- .../org/apache/beam/sdk/io/FileBasedSink.java | 95 ++- .../org/apache/beam/sdk/io/FileBasedSource.java | 12 +- .../java/org/apache/beam/sdk/io/FileSystem.java | 29 + .../apache/beam/sdk/io/FileSystemRegistrar.java | 49 ++ .../org/apache/beam/sdk/io/FileSystems.java | 155 +++++ .../org/apache/beam/sdk/io/LocalFileSystem.java | 27 + .../beam/sdk/io/LocalFileSystemRegistrar.java | 41 ++ .../java/org/apache/beam/sdk/io/PubsubIO.java | 180 +++++- .../apache/beam/sdk/io/PubsubUnboundedSink.java | 27 +- .../beam/sdk/io/PubsubUnboundedSource.java | 48 +- .../main/java/org/apache/beam/sdk/io/Read.java | 4 +- .../java/org/apache/beam/sdk/io/TextIO.java | 32 +- .../main/java/org/apache/beam/sdk/io/Write.java | 2 +- .../java/org/apache/beam/sdk/io/XmlSink.java | 4 +- .../apache/beam/sdk/io/range/ByteKeyRange.java | 4 +- .../beam/sdk/io/range/ByteKeyRangeTracker.java | 8 +- .../apache/beam/sdk/metrics/CounterCell.java | 10 +- .../beam/sdk/metrics/DistributionCell.java | 14 +- .../org/apache/beam/sdk/metrics/MetricCell.java | 2 +- .../org/apache/beam/sdk/metrics/MetricName.java | 3 +- .../org/apache/beam/sdk/metrics/Metrics.java | 5 +- .../beam/sdk/metrics/MetricsEnvironment.java | 8 +- .../org/apache/beam/sdk/options/GcpOptions.java | 19 +- .../beam/sdk/options/PipelineOptions.java | 7 + .../sdk/options/PipelineOptionsFactory.java | 111 +++- .../sdk/options/ProxyInvocationHandler.java | 26 + .../beam/sdk/options/StreamingOptions.java | 2 +- .../apache/beam/sdk/options/ValueProvider.java | 29 +- .../sdk/runners/PTransformOverrideFactory.java | 41 ++ .../apache/beam/sdk/runners/PipelineRunner.java | 2 +- .../beam/sdk/runners/TransformHierarchy.java | 345 +++++++++-- .../beam/sdk/runners/TransformTreeNode.java | 257 -------- .../beam/sdk/testing/BigqueryMatcher.java | 48 +- .../beam/sdk/testing/FileChecksumMatcher.java | 124 ++-- .../apache/beam/sdk/testing/GatherAllPanes.java | 88 +++ .../org/apache/beam/sdk/testing/PAssert.java | 87 +-- .../apache/beam/sdk/testing/PaneExtractors.java | 55 +- .../apache/beam/sdk/testing/TestPipeline.java | 225 ++++++- .../org/apache/beam/sdk/testing/TestStream.java | 2 +- .../apache/beam/sdk/testing/UsesMetrics.java | 24 + .../beam/sdk/testing/UsesSplittableParDo.java | 25 + .../apache/beam/sdk/testing/UsesTestStream.java | 24 + .../beam/sdk/testing/UsesTimersInParDo.java | 25 + .../beam/sdk/testing/ValueInSingleWindow.java | 134 ++++ .../apache/beam/sdk/transforms/Aggregator.java | 19 - .../sdk/transforms/AggregatorRetriever.java | 2 +- .../beam/sdk/transforms/AppliedPTransform.java | 75 +-- .../beam/sdk/transforms/ApproximateUnique.java | 4 +- .../org/apache/beam/sdk/transforms/Combine.java | 51 +- .../beam/sdk/transforms/CombineWithContext.java | 2 +- .../org/apache/beam/sdk/transforms/Count.java | 2 +- .../org/apache/beam/sdk/transforms/Create.java | 4 +- .../apache/beam/sdk/transforms/Distinct.java | 4 +- .../org/apache/beam/sdk/transforms/DoFn.java | 38 +- .../beam/sdk/transforms/DoFnAdapters.java | 14 + .../apache/beam/sdk/transforms/DoFnTester.java | 387 ++++++------ .../org/apache/beam/sdk/transforms/Filter.java | 2 +- .../beam/sdk/transforms/FlatMapElements.java | 2 +- .../org/apache/beam/sdk/transforms/Flatten.java | 4 +- .../apache/beam/sdk/transforms/GroupByKey.java | 2 +- .../org/apache/beam/sdk/transforms/Keys.java | 2 +- .../org/apache/beam/sdk/transforms/KvSwap.java | 2 +- .../org/apache/beam/sdk/transforms/Latest.java | 4 +- .../apache/beam/sdk/transforms/MapElements.java | 2 +- .../org/apache/beam/sdk/transforms/OldDoFn.java | 8 +- .../apache/beam/sdk/transforms/PTransform.java | 4 +- .../org/apache/beam/sdk/transforms/ParDo.java | 201 +----- .../apache/beam/sdk/transforms/Partition.java | 2 +- .../org/apache/beam/sdk/transforms/Regex.java | 14 +- .../org/apache/beam/sdk/transforms/Sample.java | 2 +- .../org/apache/beam/sdk/transforms/Values.java | 2 +- .../org/apache/beam/sdk/transforms/View.java | 16 +- .../apache/beam/sdk/transforms/WithKeys.java | 2 +- .../beam/sdk/transforms/WithTimestamps.java | 2 +- .../sdk/transforms/display/DisplayData.java | 22 + .../beam/sdk/transforms/join/CoGroupByKey.java | 2 +- .../transforms/join/KeyedPCollectionTuple.java | 9 +- .../reflect/ByteBuddyDoFnInvokerFactory.java | 39 +- .../reflect/ByteBuddyOnTimerInvokerFactory.java | 10 +- .../sdk/transforms/reflect/DoFnInvoker.java | 11 + .../sdk/transforms/reflect/DoFnInvokers.java | 16 +- .../sdk/transforms/reflect/DoFnSignature.java | 34 +- .../sdk/transforms/reflect/DoFnSignatures.java | 96 ++- .../sdk/transforms/windowing/BoundedWindow.java | 31 + .../beam/sdk/transforms/windowing/Window.java | 13 +- .../apache/beam/sdk/util/ActiveWindowSet.java | 173 ------ .../org/apache/beam/sdk/util/ApiSurface.java | 52 +- .../beam/sdk/util/BaseExecutionContext.java | 174 ------ .../apache/beam/sdk/util/ExecutionContext.java | 100 --- .../beam/sdk/util/ExplicitShardedFile.java | 120 ++++ .../apache/beam/sdk/util/GatherAllPanes.java | 86 --- .../apache/beam/sdk/util/GcsPathValidator.java | 3 +- .../java/org/apache/beam/sdk/util/GcsUtil.java | 107 +++- .../apache/beam/sdk/util/IdentityWindowFn.java | 2 +- .../org/apache/beam/sdk/util/KeyedWorkItem.java | 43 -- .../beam/sdk/util/KeyedWorkItemCoder.java | 128 ---- .../apache/beam/sdk/util/KeyedWorkItems.java | 121 ---- .../beam/sdk/util/MergingActiveWindowSet.java | 428 ------------- .../sdk/util/NonMergingActiveWindowSet.java | 91 --- .../beam/sdk/util/NumberedShardedFile.java | 220 +++++++ .../beam/sdk/util/PerKeyCombineFnRunner.java | 148 ----- .../beam/sdk/util/PerKeyCombineFnRunners.java | 258 -------- .../org/apache/beam/sdk/util/PropertyNames.java | 2 + .../org/apache/beam/sdk/util/ReleaseInfo.java | 4 - .../org/apache/beam/sdk/util/Reshuffle.java | 2 +- .../org/apache/beam/sdk/util/ShardedFile.java | 42 ++ .../apache/beam/sdk/util/TimerInternals.java | 120 ++-- .../beam/sdk/util/WindowingInternals.java | 10 - .../CopyOnAccessInMemoryStateInternals.java | 453 -------------- .../sdk/util/state/InMemoryStateInternals.java | 33 +- .../sdk/util/state/InMemoryTimerInternals.java | 247 -------- .../beam/sdk/util/state/StateInternals.java | 2 +- .../beam/sdk/util/state/TimerCallback.java | 35 -- .../sdk/util/state/TimerInternalsFactory.java | 36 ++ .../java/org/apache/beam/sdk/values/PBegin.java | 4 +- .../org/apache/beam/sdk/values/PCollection.java | 4 +- .../apache/beam/sdk/values/PCollectionList.java | 65 +- .../beam/sdk/values/PCollectionTuple.java | 30 +- .../java/org/apache/beam/sdk/values/PDone.java | 4 +- .../java/org/apache/beam/sdk/values/PInput.java | 4 +- .../org/apache/beam/sdk/values/POutput.java | 4 +- .../java/org/apache/beam/sdk/values/PValue.java | 10 + .../org/apache/beam/sdk/values/PValueBase.java | 11 +- .../apache/beam/sdk/values/TaggedPValue.java | 42 ++ .../org/apache/beam/sdk/values/TypedPValue.java | 2 +- .../sdk/AggregatorPipelineExtractorTest.java | 28 +- .../java/org/apache/beam/sdk/PipelineTest.java | 43 +- .../org/apache/beam/sdk/WindowMatchers.java | 204 ------ .../org/apache/beam/sdk/WindowMatchersTest.java | 82 --- .../apache/beam/sdk/coders/AvroCoderTest.java | 11 +- .../beam/sdk/coders/CoderRegistryTest.java | 10 +- .../beam/sdk/coders/SerializableCoderTest.java | 7 +- .../java/org/apache/beam/sdk/io/AvroIOTest.java | 16 +- .../io/BoundedReadFromUnboundedSourceTest.java | 6 +- .../beam/sdk/io/CompressedSourceTest.java | 12 +- .../apache/beam/sdk/io/CountingInputTest.java | 69 ++- .../apache/beam/sdk/io/CountingSourceTest.java | 21 +- .../apache/beam/sdk/io/FileBasedSinkTest.java | 11 +- .../apache/beam/sdk/io/FileBasedSourceTest.java | 30 +- .../org/apache/beam/sdk/io/FileSystemsTest.java | 104 ++++ .../sdk/io/LocalFileSystemRegistrarTest.java | 44 ++ .../org/apache/beam/sdk/io/PubsubIOTest.java | 43 +- .../beam/sdk/io/PubsubUnboundedSinkTest.java | 30 +- .../beam/sdk/io/PubsubUnboundedSourceTest.java | 26 +- .../java/org/apache/beam/sdk/io/TextIOTest.java | 44 +- .../java/org/apache/beam/sdk/io/WriteTest.java | 4 +- .../org/apache/beam/sdk/io/XmlSinkTest.java | 10 +- .../org/apache/beam/sdk/io/XmlSourceTest.java | 10 +- .../apache/beam/sdk/metrics/MetricMatchers.java | 47 +- .../apache/beam/sdk/metrics/MetricsTest.java | 63 +- .../apache/beam/sdk/options/GcpOptionsTest.java | 32 +- .../sdk/options/PipelineOptionsFactoryTest.java | 222 ++++++- .../beam/sdk/options/PipelineOptionsTest.java | 24 + .../sdk/options/ProxyInvocationHandlerTest.java | 5 +- .../beam/sdk/options/ValueProviderTest.java | 39 ++ .../sdk/runners/TransformHierarchyTest.java | 208 +++++-- .../beam/sdk/runners/TransformTreeTest.java | 27 +- .../beam/sdk/testing/BigqueryMatcherTest.java | 82 +-- .../sdk/testing/FileChecksumMatcherTest.java | 66 +- .../beam/sdk/testing/GatherAllPanesTest.java | 141 +++++ .../apache/beam/sdk/testing/PAssertTest.java | 32 +- .../beam/sdk/testing/PaneExtractorsTest.java | 133 ++-- .../beam/sdk/testing/TestPipelineTest.java | 186 +++++- .../apache/beam/sdk/testing/TestStreamTest.java | 19 +- .../testing/ValueInSingleWindowCoderTest.java | 51 ++ .../transforms/ApproximateQuantilesTest.java | 12 +- .../sdk/transforms/ApproximateUniqueTest.java | 6 +- .../beam/sdk/transforms/CombineFnsTest.java | 5 +- .../apache/beam/sdk/transforms/CombineTest.java | 25 +- .../apache/beam/sdk/transforms/CountTest.java | 13 +- .../apache/beam/sdk/transforms/CreateTest.java | 27 +- .../beam/sdk/transforms/DistinctTest.java | 12 +- .../apache/beam/sdk/transforms/DoFnTest.java | 4 +- .../beam/sdk/transforms/DoFnTesterTest.java | 46 +- .../apache/beam/sdk/transforms/FilterTest.java | 18 +- .../sdk/transforms/FlatMapElementsTest.java | 12 +- .../apache/beam/sdk/transforms/FlattenTest.java | 35 +- .../beam/sdk/transforms/GroupByKeyTest.java | 34 +- .../apache/beam/sdk/transforms/KeysTest.java | 9 +- .../apache/beam/sdk/transforms/KvSwapTest.java | 9 +- .../apache/beam/sdk/transforms/LatestTest.java | 12 +- .../beam/sdk/transforms/MapElementsTest.java | 16 +- .../apache/beam/sdk/transforms/NoOpOldDoFn.java | 72 --- .../apache/beam/sdk/transforms/OldDoFnTest.java | 125 ++-- .../beam/sdk/transforms/PTransformTest.java | 2 +- .../beam/sdk/transforms/ParDoLifecycleTest.java | 45 +- .../apache/beam/sdk/transforms/ParDoTest.java | 617 ++++++++++++------- .../beam/sdk/transforms/PartitionTest.java | 8 +- .../apache/beam/sdk/transforms/RegexTest.java | 25 +- .../apache/beam/sdk/transforms/SampleTest.java | 34 +- .../beam/sdk/transforms/SplittableDoFnTest.java | 453 ++++++++++++++ .../org/apache/beam/sdk/transforms/TopTest.java | 15 +- .../apache/beam/sdk/transforms/ValuesTest.java | 7 +- .../apache/beam/sdk/transforms/ViewTest.java | 88 +-- .../beam/sdk/transforms/WithKeysTest.java | 8 +- .../beam/sdk/transforms/WithTimestampsTest.java | 9 +- .../display/DisplayDataEvaluator.java | 8 +- .../display/DisplayDataEvaluatorTest.java | 2 +- .../display/DisplayDataMatchersTest.java | 6 +- .../sdk/transforms/display/DisplayDataTest.java | 90 ++- .../sdk/transforms/join/CoGroupByKeyTest.java | 11 +- .../DoFnSignaturesSplittableDoFnTest.java | 3 +- .../transforms/reflect/DoFnSignaturesTest.java | 47 ++ .../sdk/transforms/windowing/WindowTest.java | 22 +- .../sdk/transforms/windowing/WindowingTest.java | 13 +- .../beam/sdk/util/GatherAllPanesTest.java | 143 ----- .../beam/sdk/util/GcsPathValidatorTest.java | 15 +- .../org/apache/beam/sdk/util/GcsUtilTest.java | 85 ++- .../beam/sdk/util/KeyedWorkItemCoderTest.java | 62 -- .../sdk/util/MergingActiveWindowSetTest.java | 262 -------- .../beam/sdk/util/NumberedShardedFileTest.java | 181 ++++++ .../apache/beam/sdk/util/ReleaseInfoTest.java | 45 ++ .../org/apache/beam/sdk/util/ReshuffleTest.java | 11 +- .../apache/beam/sdk/util/StringUtilsTest.java | 6 +- .../beam/sdk/util/TimerInternalsTest.java | 4 +- .../CopyOnAccessInMemoryStateInternalsTest.java | 552 ----------------- .../util/state/InMemoryTimerInternalsTest.java | 116 ---- .../beam/sdk/values/PCollectionListTest.java | 117 ++++ .../beam/sdk/values/PCollectionTupleTest.java | 80 ++- .../org/apache/beam/sdk/values/PDoneTest.java | 13 +- .../apache/beam/sdk/values/TypedPValueTest.java | 10 +- sdks/java/extensions/join-library/pom.xml | 2 +- .../extensions/joinlibrary/InnerJoinTest.java | 9 +- .../joinlibrary/OuterLeftJoinTest.java | 10 +- .../joinlibrary/OuterRightJoinTest.java | 10 +- sdks/java/extensions/pom.xml | 2 +- sdks/java/extensions/sorter/README.md | 2 +- sdks/java/extensions/sorter/pom.xml | 10 +- .../sorter/BufferedExternalSorter.java | 12 +- .../sdk/extensions/sorter/ExternalSorter.java | 21 +- .../sdk/extensions/sorter/InMemorySorter.java | 26 +- .../beam/sdk/extensions/sorter/SortValues.java | 2 +- .../sorter/BufferedExternalSorterTest.java | 74 ++- .../extensions/sorter/ExternalSorterTest.java | 69 ++- .../extensions/sorter/InMemorySorterTest.java | 8 + .../sdk/extensions/sorter/SortValuesTest.java | 7 +- sdks/java/io/google-cloud-platform/pom.xml | 8 +- .../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 453 ++++++++++---- .../sdk/io/gcp/bigquery/BigQueryServices.java | 7 +- .../io/gcp/bigquery/BigQueryServicesImpl.java | 121 +++- .../io/gcp/bigquery/BigQueryTableInserter.java | 217 ------- .../beam/sdk/io/gcp/bigtable/BigtableIO.java | 28 +- .../io/gcp/bigtable/BigtableServiceImpl.java | 6 +- .../beam/sdk/io/gcp/datastore/DatastoreV1.java | 4 +- .../beam/sdk/io/gcp/storage/GcsFileSystem.java | 34 + .../io/gcp/storage/GcsFileSystemRegistrar.java | 42 ++ .../beam/sdk/io/gcp/storage/package-info.java | 21 + .../apache/beam/sdk/io/gcp/ApiSurfaceTest.java | 2 + .../sdk/io/gcp/bigquery/BigQueryIOTest.java | 568 +++++++++++++---- .../gcp/bigquery/BigQueryServicesImplTest.java | 139 ++++- .../gcp/bigquery/BigQueryTableInserterTest.java | 245 -------- .../sdk/io/gcp/bigquery/BigQueryUtilTest.java | 50 +- .../sdk/io/gcp/bigtable/BigtableIOTest.java | 13 +- .../gcp/storage/GcsFileSystemRegistrarTest.java | 51 ++ sdks/java/io/hdfs/pom.xml | 8 +- .../beam/sdk/io/hdfs/HadoopFileSystem.java | 29 + .../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 42 ++ .../io/hdfs/HadoopFileSystemRegistrarTest.java | 52 ++ sdks/java/io/jdbc/pom.xml | 2 +- .../org/apache/beam/sdk/io/jdbc/JdbcIO.java | 56 +- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 38 +- sdks/java/io/jms/pom.xml | 22 +- .../java/org/apache/beam/sdk/io/jms/JmsIO.java | 342 ++++++---- .../org/apache/beam/sdk/io/jms/JmsRecord.java | 8 +- .../org/apache/beam/sdk/io/jms/JmsIOTest.java | 9 +- sdks/java/io/kafka/pom.xml | 2 +- .../org/apache/beam/sdk/io/kafka/KafkaIO.java | 20 +- .../apache/beam/sdk/io/kafka/KafkaIOTest.java | 25 +- sdks/java/io/kinesis/pom.xml | 2 +- .../sdk/io/kinesis/KinesisMockReadTest.java | 7 +- .../beam/sdk/io/kinesis/KinesisReaderIT.java | 6 +- sdks/java/io/mongodb/pom.xml | 2 +- .../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +- .../apache/beam/sdk/io/mongodb/MongoDbIO.java | 22 +- .../sdk/io/mongodb/MongoDBGridFSIOTest.java | 17 +- .../beam/sdk/io/mongodb/MongoDbIOTest.java | 15 +- sdks/java/io/pom.xml | 2 +- sdks/java/java8tests/pom.xml | 2 +- .../beam/sdk/transforms/CombineJava8Test.java | 8 +- .../beam/sdk/transforms/DistinctJava8Test.java | 5 +- .../beam/sdk/transforms/FilterJava8Test.java | 9 +- .../transforms/FlatMapElementsJava8Test.java | 7 +- .../sdk/transforms/MapElementsJava8Test.java | 9 +- .../beam/sdk/transforms/PartitionJava8Test.java | 7 +- .../beam/sdk/transforms/WithKeysJava8Test.java | 6 +- .../sdk/transforms/WithTimestampsJava8Test.java | 14 +- .../examples-java8/generate-sources.sh | 82 +++ .../maven-archetypes/examples-java8/pom.xml | 182 ++++++ .../META-INF/maven/archetype-metadata.xml | 39 ++ .../main/resources/archetype-resources/pom.xml | 295 +++++++++ .../projects/basic/archetype.properties | 21 + .../src/test/resources/projects/basic/goal.txt | 1 + .../examples/generate-sources.sh | 68 ++ sdks/java/maven-archetypes/examples/pom.xml | 32 +- .../main/resources/archetype-resources/pom.xml | 168 ++++- .../src/main/java/DebuggingWordCount.java | 164 ----- .../src/main/java/MinimalWordCount.java | 118 ---- .../src/main/java/WindowedWordCount.java | 229 ------- .../src/main/java/WordCount.java | 186 ------ .../common/ExampleBigQueryTableOptions.java | 55 -- .../src/main/java/common/ExampleOptions.java | 37 -- ...xamplePubsubTopicAndSubscriptionOptions.java | 45 -- .../java/common/ExamplePubsubTopicOptions.java | 45 -- .../src/main/java/common/ExampleUtils.java | 352 ----------- .../src/test/java/DebuggingWordCountTest.java | 52 -- .../src/test/java/WordCountTest.java | 85 --- sdks/java/maven-archetypes/pom.xml | 40 +- sdks/java/maven-archetypes/starter/pom.xml | 9 +- .../main/resources/archetype-resources/pom.xml | 20 +- .../resources/projects/basic/reference/pom.xml | 20 +- .../update-examples-archetype.sh | 59 -- sdks/java/pom.xml | 2 +- sdks/pom.xml | 2 +- 641 files changed, 23199 insertions(+), 14088 deletions(-) ----------------------------------------------------------------------