This is an automated email from the ASF dual-hosted git repository.

ccy pushed a change to branch charlesccychen-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git.


 discard e8f9064  Update interactive_utils.py
 discard 04e3eb5  Update interactive_utils.py
 discard a4705e0  [BEAM-9277] Fix exception when running in IPython notebook
     add 616fb53  [BEAM-7092] Fix invalid import of Guava coming from 
transitive Spark dep
     add 2ea6e00  Merge pull request #10812: [BEAM-7092] Fix invalid import of 
Guava coming from transitive Spark dep
     add cb1d4e7  [BEAM-9268] SpannerIO: Add more documentation and warnings 
for unknown tables. (#10690)
     add d39a37a  Reject unsupported WindowFns and Window types.
     add 995d0ae  Merge pull request #10805 Reject unsupported WindowFns and 
Window types.
     add 065ada9  [BEAM-9266] Remove unused fields from provisioning API.
     add 3bd2911  Remove one more reference to provision resources.
     add aa1142a  Merge pull request #10798 from robertwb/provision
     add aab5ae5  [BEAM-9241] Fix inconsistent proto nullability
     add 7fc5fb5  Merge pull request #10746: [BEAM-9241] Fix inconsistent proto 
nullability
     add e96dbf3  [BEAM-9231] Add Experimental(Kind.PORTABILITY) and tag 
related classes
     add 650e6cd  [BEAM-9231] Tag SplittableDoFn related classes/methods as 
Experimental
     add 9108832  [BEAM-9231] Make Experimental annotations homogeneous in 
beam-sdks-java-core
     add cb7cb53  [BEAM-9231] Untag Experimental/Internal classes not needed to 
write pipelines
     add 1574a81  [BEAM-9231] Tag beam-sdks-java-core internal classes as 
Internal
     add 0b2d2f5  [BEAM-9231] Tag DoFn.OnTimerContext as 
Experimental(Kind.TIMERS)
     add 486d230  [BEAM-9231] Tag Experimental/Internal packages in 
beam-sdks-java-core
     add a90ef1a  [BEAM-9231] Tag Experimental/Internal packages in IOs and 
extensions
     add ce35107  [BEAM-9231] Tag public but internal IOs and extensions 
classes as Internal
     add da9e172  Merge pull request #10739: [BEAM-9231] Annotate as 
Experimental/Internal missing classes in beam-sdks-java-core
     add e527bec  [BEAM-8630] Validate prepared expression on expand
     add e24d1e5  Merge pull request #10820: [BEAM-8630] Validate prepared 
expression on expand
     add 1fee64c  [BEAM-9160] Update AWS SDK to support Pod Level Identity
     add 930b5ef  Merge pull request #10825: [BEAM-9160] Update AWS SDK to 
support Pod Level Identity
     add 0a15bca  Move external PubsubIO hooks outside of PubsubIO.
     add 6d721c1  Merge pull request #10476: [BEAM-8932][Cleanup] Move external 
PubsubIO hooks outside of PubsubIO.
     add cd6e54b  Move ThreadsafeRestrictionTracker and RestrictionTrackerView 
out from iobase.py
     add bcc3e13  Merge pull request #10802 from boyuanzz/refactor
     add 9165828  Remove tables and refer to dependency locations in code 
(#10745)
     add 6df1c74  Cleanup MappingProxy reducer since dill supports it natively 
now.
     add 07aab1f  Merge pull request #10837 [BEAM-7284] Cleanup MappingProxy 
reducer since dill supports it now.
     add f72841d  [BEAM-8095] Remove no_xdist for test
     add c7daa7c  fix lint
     add f6633d8  Merge pull request #10759 from udim/xdist-fixup
     add 94fc1ad  [website] Update Java 11 and Spark roadmap
     add cc151c3  Merge pull request #10819 from iemejia/website-roadmap
     add ef337d8  Enable probot autolabeler action to label github pull 
requests automatically
     add 355094b  Remove prefixes in autolabeler configuration to improve 
readability
     add d300d6f  Merge pull request #10824: Enable probot autolabeler
     add 9c59d65  [BEAM-9160] Removed WebIdentityTokenCredentialsProvider 
explicit json (de)serialization in AWS module
     add ba5fc1f  Merge pull request #10836: [BEAM-9160] Removed 
WebIdentityTokenCredentialsProvider explicit json (de)serialization in AWS 
module
     add 30063fb  Fix method name in Combine and coGBK tests
     add 6b395f8  Move imports and variables out of global namespace
     add ffeea60  Merge pull request #10800 from kamilwu/load-tests-fix
     add d15e50f  [BEAM-3221] Clarify documentation for 
StandardTransforms.Primitives, Pipeline, and PTransform. (#10779)
     add 283ecff  [BEAM-8201] Add other endpoint fields to provision API. 
(#10839)
     add dd9f5d2  [BEAM-9269] Add commit deadline for Spanner writes. (#10752)
     add 7eb0da5  [AVRO-2737] Exclude a buggy avro version from requirements 
spec.
     add 097a66b  Merge pull request #10844 [AVRO-2737] Exclude a buggy avro 
version from requirements spec.
     add b0c09e8  Update roadmap page for python 3 support
     add 96f2039  Merge pull request #10848 Update roadmap page for python 3 
support
     add 4c3b8c4  [BEAM-7198] rename ToStringCoder to ToBytesCoder for proper 
representation of its role
     add ea7d45f  [BEAM-7198] add comment
     add 47c5759  [BEAM-7198] fixup: reformatted with yapf
     add b3e0612  Merge pull request #10828 [BEAM-7198] rename ToStringCoder to 
ToBytesCoder for proper representation of its role
     add 2a8db6c  Refine labels/categories for PR autolabeling
     add 346f716  Merge pull request #10845 from Refine labels/categories for 
PR autolabeling
     add 5a1b318  [BEAM-9160] Removed WebIdentityTokenCredentialsProvider 
explicit json (de)serialization in AWS2 module
     add 1a964ff  Merge pull request #10846: [BEAM-9160] Removed 
WebIdentityTokenCredentialsProvider explicit json (de)serialization in AWS2 
module
     add 38de950  [BEAM-9292] Provide an ability to specify additional maven 
repositories for published POMs
     add 4a550df  [BEAM-9292] KafkaIO: add io.confluent repository to published 
POM
     add 3be2060  Merge pull request #10832: [BEAM-9292] Provide an ability to 
specify additional maven repositories for published POMs
     add 6054ae1  [BEAM-9281] Update commons-csv to version 1.8
     add df33058  Merge pull request #10818: [BEAM-9281] Update commons-csv to 
version 1.8
     add 5140e7d  Remove unused ReduceFnRunnerHelper class
     add ecec607  Do not set options.filesToStage in case of spark local 
execution in StructuredStreaming runner
     add bcb7bbd  Do not set options.filesToStage in case of spark local 
execution in current spark runner
     add ee1f1c7  Merge pull request #10831: Avoid staging jar files in current 
both spark runners when running on spark local + minor cleaning on new spark 
runner
     add ac3d237  beam-linkage-check.sh
     add 06731ef  copyright
     add 7b748f2  Merge pull request #10841: [BEAM-9301] Check in 
beam-linkage-check.sh
     add 49863a4  [BEAM-6522] [BEAM-7455] Unskip Avro IO tests that are now 
passing. (#10838)
     add c1d054f  [BEAM-5605] Convert all BoundedSources to SplittableDoFns 
when using beam_fn_api experiment. (#10576)
     add e5710ba  [BEAM-8758] Google-cloud-spanner upgrade to 1.49.1 (#10765)
     add bb99ce5  Ensuring appropriate write_disposition and create_disposition 
for jobs (#10668)
     add b7a75f1  [BEAM-3545] Return metrics as MonitoringInfos (#10777)
     add 9fd3a62  Modify the TestStreamFileRecord to use TestStreamPayload 
events. (#10826)
     add 87f8ff1  [BEAM-9290] Support runner_harness_container_image in 
released python sdks.
     add cec1094  Merge pull request #10827 from 
angoenka/runner_harness_image_dev
     add 9705a56  [BEAM-9280] Update commons-compress to version 1.20
     add 58b8705  Merge pull request #10817: [BEAM-9280] Update 
commons-compress to version 1.20
     add 6818560  Bump dataflow container version (#10861)
     add aa63a87  [BEAM-9311] ZetaSQL Named Parameters are case-insensitive
     add 972329b  Merge pull request #10855: [BEAM-9311] ZetaSQL Named 
Parameters are case-insensitive
     add f922fc1  [BEAM-8335] Update StreamingCache with new Protos (#10856)
     add 6278e14  [BEAM-9317] Fix portable test executions to specify the 
beam_fn_api argument (#10867)
     add 3d0bab8  Add attributes defined in operations.pxd but missing in 
operations.py
     add acaaa4d  Merge pull request #10783 [BEAM-7746] Add attributes missing 
in operations.py
     add b1a02e9  [BEAM-9289] Improve performance for metrics update of samza 
runner (#10833)
     add cc4054c  [BEAM-9212] fix zetasql struct exception
     add c1817f4  Merge pull request #10707 from ibzib/zetasql-error-str
     add 8654f20  [BEAM-9265] @RequiresTimeSortedInput respects allowedLateness
     add 9710fce  Merge pull request #10795: [BEAM-9265] 
@RequiresTimeSortedInput respects allowedLateness
     add b6b6312  [BEAM-9317] Fix Dataflow tests to not perform SplittableDoFn 
expansion at the moment. (#10868)
     add 85eeb10  [BEAM-9273] Explicitly disable @RequiresTimeSortedInput on 
unsupported runners
     add 9e45f9d  [BEAM-9273] code review - to be squashed
     add a149b6b  Merge pull request #10816: [BEAM-9273] Explicitly disable 
@RequiresTimeSortedInput on unsupported runners
     add e68492d  [BEAM-9315] Allow multiple paths via HADOOP_CONF_DIR in 
HadoopFileSystemOptions
     add 7a62fb7  Merge pull request #10866: [BEAM-9315] Allow multiple paths 
via HADOOP_CONF_DIR in HadoopFileSystemOptions
     add ef65363  [BEAM-6857] Classify unbounded dynamic timers tests in the 
UsesUnboundedPCollections category
     add 8c96f71  Merge pull request #10871: [BEAM-6857] Classify unbounded 
dynamic timers tests in the UsesUnboundedPCollections category
     add 7fcc92e  Pin Avro dependency in Python SDK to be consistent with Avro 
versioning
     add 92d73aa  Merge pull request #10851: Pin Avro dependency in Python SDK 
to be consistent with Avro versioning
     add cbde736  Update container images used by Dataflow runner with 
unreleased SDKs. These are based on python:slim-stretch images. (#10858)
     add 290bd9f  [BEAM-9314] Make dot output deterministic (#10864)
     add 79a2327  [BEAM-9277] Fix exception when running in IPython notebook.

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (e8f9064)
            \
             N -- N -- N   refs/heads/charlesccychen-patch-1 (79a2327)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

No new revisions were added by this update.

Summary of changes:
 .github/autolabeler.yml                            |  89 ++++
 CHANGES.md                                         |   2 +
 .../org/apache/beam/gradle/BeamModulePlugin.groovy |  29 +-
 .../src/main/proto/beam_provision_api.proto        |  51 +--
 .../src/main/proto/beam_interactive_api.proto      |  12 +-
 .../pipeline/src/main/proto/beam_runner_api.proto  | 130 ++++--
 .../runners/apex/translation/ParDoTranslator.java  |  13 +-
 .../runners/core/construction/CoderTranslator.java |   3 -
 .../core/construction/CoderTranslators.java        |   3 -
 .../beam/runners/core/construction/External.java   |   3 +
 .../core/construction/JavaReadViaImpulse.java      | 176 --------
 .../core/construction/PTransformMatchers.java      |   3 -
 .../runners/core/construction/SdkComponents.java   |   2 -
 .../runners/core/construction/SplittableParDo.java |   2 -
 .../core/construction/TriggerTranslation.java      |   2 -
 .../construction/expansion/ExpansionService.java   |   6 +-
 .../construction/graph/GreedyPipelineFuser.java    |   5 +-
 .../core/construction/graph/ProtoOverrides.java    |   2 -
 .../core/construction/JavaReadViaImpulseTest.java  | 180 --------
 .../beam/runners/core/InMemoryStateInternals.java  |   3 -
 .../beam/runners/core/MergingStateAccessor.java    |   3 -
 .../apache/beam/runners/core/StateAccessor.java    |   3 -
 .../apache/beam/runners/core/StateInternals.java   |   3 -
 .../beam/runners/core/StateInternalsFactory.java   |   3 -
 .../org/apache/beam/runners/core/StateTag.java     |   3 -
 .../org/apache/beam/runners/core/StateTags.java    |   3 -
 .../beam/runners/core/StatefulDoFnRunner.java      |  29 +-
 .../beam/runners/core/TimerInternalsFactory.java   |   3 -
 .../beam/runners/core/metrics/CounterCell.java     |   5 -
 .../beam/runners/core/metrics/DirtyState.java      |   5 -
 .../runners/core/metrics/DistributionCell.java     |   5 -
 .../beam/runners/core/metrics/GaugeCell.java       |   4 -
 .../beam/runners/core/metrics/MetricCell.java      |   3 -
 .../beam/runners/core/metrics/MetricUpdates.java   |   3 -
 .../runners/core/metrics/MetricsContainerImpl.java |   3 -
 .../beam/runners/core/metrics/MetricsMap.java      |   3 -
 .../beam/runners/core/metrics/MetricsPusher.java   |   2 -
 .../runners/core/metrics/MetricsTranslation.java   |   3 -
 .../core/triggers/AfterAllStateMachine.java        |   2 -
 .../AfterDelayFromFirstElementStateMachine.java    |   2 -
 .../core/triggers/AfterFirstStateMachine.java      |   2 -
 .../core/triggers/AfterPaneStateMachine.java       |   2 -
 .../triggers/AfterProcessingTimeStateMachine.java  |   2 -
 .../core/triggers/AfterWatermarkStateMachine.java  |   2 -
 .../core/triggers/DefaultTriggerStateMachine.java  |   2 -
 .../beam/runners/core/StatefulDoFnRunnerTest.java  |  23 +-
 .../java/org/apache/beam/runners/direct/Clock.java |   2 -
 .../beam/runners/direct/DirectTestOptions.java     |   2 -
 .../beam/runners/direct/WatermarkManager.java      |   2 -
 .../extensions/metrics/MetricsGraphiteSink.java    |   2 -
 .../extensions/metrics/MetricsHttpSink.java        |   2 -
 runners/flink/flink_runner.gradle                  |   2 +-
 .../beam/runners/flink/PortableExecutionTest.java  |   7 +-
 .../runners/flink/PortableStateExecutionTest.java  |   7 +-
 .../runners/flink/PortableTimersExecutionTest.java |   6 +-
 .../beam/runners/flink/ReadSourcePortableTest.java |   7 +-
 .../translators/ParDoMultiOutputTranslator.java    |   9 +
 runners/google-cloud-dataflow-java/build.gradle    |   4 +-
 .../examples/build.gradle                          |   2 +-
 .../dataflow/BatchStatefulParDoOverrides.java      |   4 +-
 .../dataflow/DataflowPipelineTranslator.java       |   9 +-
 .../beam/runners/dataflow/DataflowRunner.java      |  71 ++--
 .../beam/runners/dataflow/TransformTranslator.java |   2 -
 .../options/DataflowPipelineWorkerPoolOptions.java |   5 +-
 .../beam/runners/dataflow/util/PackageUtil.java    |   2 -
 .../dataflow/DataflowPipelineTranslatorTest.java   |  17 +-
 runners/java-fn-execution/build.gradle             |   1 +
 .../jobsubmission/PortablePipelineResult.java      |   2 -
 .../runners/fnexecution/provisioning/JobInfo.java  |   2 -
 ...gleEnvironmentInstanceJobBundleFactoryTest.java |   7 +-
 .../StaticGrpcProvisionServiceTest.java            |  19 +-
 .../java/org/apache/beam/runners/jet/Utils.java    |   8 +-
 .../beam/runners/portability/PortableRunner.java   |   3 -
 runners/samza/build.gradle                         |   1 -
 .../runners/samza/adapter/BoundedSourceSystem.java |   2 +-
 .../samza/adapter/UnboundedSourceSystem.java       |   2 +-
 .../samza/metrics/DoFnRunnerWithMetrics.java       |  16 +-
 .../samza/metrics/FnWithMetricsWrapper.java        |   7 +-
 .../samza/metrics/SamzaMetricsContainer.java       |   6 +-
 .../runners/samza/runtime/SamzaDoFnRunners.java    |   2 +-
 .../beam/runners/samza/state/SamzaMapState.java    |   2 -
 .../beam/runners/samza/state/SamzaSetState.java    |   2 -
 .../translation/ParDoBoundMultiTranslator.java     |  11 +-
 .../beam/runners/spark/SparkPipelineOptions.java   |  38 +-
 .../org/apache/beam/runners/spark/SparkRunner.java |   3 +-
 .../SparkStructuredStreamingRunner.java            |   5 +-
 .../translation/PipelineTranslator.java            |   7 +-
 .../translation/batch/ParDoTranslatorBatch.java    |  15 +-
 .../translation/helpers/ReduceFnRunnerHelpers.java |  88 ----
 .../runners/spark/translation/SparkCombineFn.java  |   2 -
 .../beam/runners/spark/util/SparkCompat.java       |   2 -
 .../runners/spark/SparkPortableExecutionTest.java  |   8 +-
 .../spark/translation/TransformTranslatorTest.java |   2 +-
 sdks/go/pkg/beam/core/runtime/exec/datasource.go   |  21 +-
 sdks/go/pkg/beam/core/runtime/exec/plan.go         |  85 +---
 sdks/go/pkg/beam/core/runtime/exec/translate.go    |   3 +
 sdks/go/pkg/beam/core/runtime/harness/harness.go   |  12 +-
 .../go/pkg/beam/core/runtime/harness/monitoring.go | 196 +++++++++
 sdks/go/pkg/beam/core/util/dot/dot.go              |   7 +-
 sdks/java/build-tools/beam-linkage-check.sh        | 115 +++++
 .../src/main/resources/beam/checkstyle.xml         |   8 +
 sdks/java/container/boot.go                        |   3 -
 .../apache/beam/sdk/annotations/Experimental.java  |   4 +
 .../org/apache/beam/sdk/annotations/Internal.java  |   1 +
 .../apache/beam/sdk/annotations/package-info.java  |   1 +
 .../sdk/expansion/ExternalTransformRegistrar.java  |   3 +-
 .../apache/beam/sdk/expansion/package-info.java    |   3 +
 .../org/apache/beam/sdk/harness/package-info.java  |   2 +
 .../main/java/org/apache/beam/sdk/io/AvroIO.java   |   4 +-
 .../java/org/apache/beam/sdk/io/AvroSource.java    |   7 +-
 .../org/apache/beam/sdk/io/BlockBasedSource.java   |   7 +-
 .../java/org/apache/beam/sdk/io/BoundedSource.java |   3 +-
 .../org/apache/beam/sdk/io/CompressedSource.java   |   3 +-
 .../main/java/org/apache/beam/sdk/io/FileIO.java   |   7 +-
 .../src/main/java/org/apache/beam/sdk/io/Read.java | 154 +++++++
 .../beam/sdk/io/ReadAllViaFileBasedSource.java     |   3 +-
 .../main/java/org/apache/beam/sdk/io/Source.java   |   3 +-
 .../org/apache/beam/sdk/io/UnboundedSource.java    |   3 +-
 .../java/org/apache/beam/sdk/io/WriteFiles.java    |   3 +-
 .../org/apache/beam/sdk/io/fs/MatchResult.java     |   3 +-
 .../org/apache/beam/sdk/io/fs/MetadataCoderV2.java |   3 +-
 .../apache/beam/sdk/metrics/DelegatingCounter.java |   5 +-
 .../java/org/apache/beam/sdk/metrics/Gauge.java    |   3 +-
 .../apache/beam/sdk/metrics/MetricsOptions.java    |   3 +-
 .../org/apache/beam/sdk/metrics/MetricsSink.java   |   3 +-
 .../org/apache/beam/sdk/metrics/package-info.java  |   3 +
 .../options/ManualDockerEnvironmentOptions.java    |   3 +-
 .../beam/sdk/options/PipelineOptionsFactory.java   |   2 +-
 .../beam/sdk/options/PortablePipelineOptions.java  |   3 +
 .../beam/sdk/options/RemoteEnvironmentOptions.java |   3 +-
 .../apache/beam/sdk/options/SdkHarnessOptions.java |   3 +-
 .../org/apache/beam/sdk/runners/package-info.java  |   2 +
 .../org/apache/beam/sdk/schemas/package-info.java  |   3 +
 .../java/org/apache/beam/sdk/state/TimeDomain.java |   3 +-
 .../main/java/org/apache/beam/sdk/state/Timer.java |   3 +-
 .../java/org/apache/beam/sdk/state/TimerMap.java   |   3 +-
 .../java/org/apache/beam/sdk/state/Timers.java     |   3 +-
 .../org/apache/beam/sdk/state/package-info.java    |   3 +
 .../testing/DataflowPortabilityApiUnsupported.java |   3 +
 .../testing/FlattenWithHeterogeneousCoders.java    |   2 +
 .../sdk/testing/UsesRequiresTimeSortedInput.java   |   2 +-
 .../java/org/apache/beam/sdk/transforms/DoFn.java  |   1 +
 .../beam/sdk/transforms/DoFnOutputReceivers.java   |   2 +
 .../beam/sdk/transforms/DoFnSchemaInformation.java |   2 +
 .../sdk/transforms/ExternalTransformBuilder.java   |   3 +-
 .../beam/sdk/transforms/FlatMapElements.java       |   9 +-
 .../apache/beam/sdk/transforms/MapElements.java    |   4 +-
 .../java/org/apache/beam/sdk/transforms/Watch.java |   3 +-
 .../apache/beam/sdk/transforms/WithFailures.java   |   3 +-
 .../reflect/ByteBuddyDoFnInvokerFactory.java       |   2 +-
 .../beam/sdk/transforms/reflect/DoFnInvokers.java  |   2 +
 .../beam/sdk/transforms/reflect/DoFnSignature.java |   2 +
 .../sdk/transforms/reflect/DoFnSignatures.java     |  52 +++
 .../sdk/transforms/reflect/OnTimerInvoker.java     |   2 +
 .../beam/sdk/transforms/reflect/package-info.java  |   2 +
 .../splittabledofn/ByteKeyRangeTracker.java        |   3 +
 .../splittabledofn/HasDefaultTracker.java          |   4 +
 .../splittabledofn/OffsetRangeTracker.java         |   3 +
 .../splittabledofn/RestrictionTracker.java         |   3 +
 .../beam/sdk/transforms/splittabledofn/Sizes.java  |   4 +
 .../sdk/transforms/splittabledofn/SplitResult.java |   3 +
 .../transforms/splittabledofn/package-info.java    |   3 +
 .../beam/sdk/transforms/windowing/AfterAll.java    |   3 +-
 .../beam/sdk/transforms/windowing/AfterEach.java   |   3 +-
 .../beam/sdk/transforms/windowing/AfterFirst.java  |   3 +-
 .../beam/sdk/transforms/windowing/AfterPane.java   |   3 +-
 .../transforms/windowing/AfterProcessingTime.java  |   3 +-
 .../sdk/transforms/windowing/AfterWatermark.java   |   3 +-
 .../sdk/transforms/windowing/DefaultTrigger.java   |   3 +-
 .../transforms/windowing/TimestampCombiner.java    |   3 +-
 .../beam/sdk/transforms/windowing/Trigger.java     |   3 +-
 .../java/org/apache/beam/sdk/util/ApiSurface.java  |   2 +
 .../org/apache/beam/sdk/util/AppliedCombineFn.java |   2 +
 .../util/BufferedElementCountingOutputStream.java  |   2 +
 .../beam/sdk/util/CombineContextFactory.java       |   2 +
 .../org/apache/beam/sdk/util/CombineFnUtil.java    |   2 +
 .../java/org/apache/beam/sdk/util/DoFnInfo.java    |   2 +
 .../sdk/util/DoFnWithExecutionInformation.java     |   2 +
 .../apache/beam/sdk/util/ExplicitShardedFile.java  |   2 +
 .../beam/sdk/util/ExposedByteArrayInputStream.java |   2 +
 .../sdk/util/ExposedByteArrayOutputStream.java     |   2 +
 .../sdk/util/FilePatternMatchingShardedFile.java   |   2 +
 .../org/apache/beam/sdk/util/IdentityWindowFn.java |   2 +
 .../org/apache/beam/sdk/util/MutationDetector.java |   3 +
 .../apache/beam/sdk/util/MutationDetectors.java    |   2 +
 .../java/org/apache/beam/sdk/util/NameUtils.java   |   2 +
 .../apache/beam/sdk/util/NumberedShardedFile.java  |   2 +
 .../java/org/apache/beam/sdk/util/ReleaseInfo.java |   2 +
 .../java/org/apache/beam/sdk/util/ShardedFile.java |   2 +
 .../beam/sdk/util/ShardingWritableByteChannel.java |   2 +
 .../java/org/apache/beam/sdk/util/StreamUtils.java |   2 +
 .../java/org/apache/beam/sdk/util/StringUtils.java |   2 +
 .../apache/beam/sdk/util/SystemDoFnInternal.java   |   2 +
 .../apache/beam/sdk/util/UnownedInputStream.java   |   2 +
 .../apache/beam/sdk/util/UnownedOutputStream.java  |   2 +
 .../apache/beam/sdk/util/UserCodeException.java    |   2 +
 .../java/org/apache/beam/sdk/util/Weighted.java    |   3 +
 .../org/apache/beam/sdk/util/WeightedValue.java    |   3 +
 .../org/apache/beam/sdk/util/WindowTracing.java    |   2 +
 .../org/apache/beam/sdk/util/WindowedValue.java    |   2 +
 .../java/org/apache/beam/sdk/util/ZipFiles.java    |   2 +
 .../common/ElementByteSizeObservableIterable.java  |   2 +
 .../common/ElementByteSizeObservableIterator.java  |   2 +
 .../sdk/util/common/ElementByteSizeObserver.java   |   2 +
 .../apache/beam/sdk/util/common/Reiterable.java    |   3 +
 .../apache/beam/sdk/util/common/Reiterator.java    |   2 +
 .../org/apache/beam/sdk/util/package-info.java     |   3 +
 .../beam/sdk/values/ValueInSingleWindow.java       |   2 +
 .../apache/beam/sdk/values/WindowingStrategy.java  |   5 +-
 .../java/org/apache/beam/sdk/io/AvroIOTest.java    |   6 +
 .../test/java/org/apache/beam/sdk/io/ReadTest.java |   8 +
 .../org/apache/beam/sdk/io/TextIOReadTest.java     |   6 +
 .../org/apache/beam/sdk/testing/PAssertTest.java   |   4 +-
 .../org/apache/beam/sdk/transforms/ParDoTest.java  |  56 ++-
 .../sdk/transforms/reflect/DoFnSignaturesTest.java | 240 +++++++++++
 .../core/client/operator/OutputValues.java         |   2 -
 .../sdk/extensions/gcp/storage/package-info.java   |   4 +
 .../beam/sdk/extensions/jackson/AsJsons.java       |   7 +-
 .../beam/sdk/extensions/jackson/ParseJsons.java    |   7 +-
 .../beam/sdk/extensions/kryo/package-info.java     |   3 +
 .../extensions/protobuf/ProtoSchemaTranslator.java |  42 +-
 .../protobuf/ProtoMessageSchemaTest.java           |  50 ++-
 .../protobuf/ProtoSchemaTranslatorTest.java        |  28 ++
 .../sdk/extensions/protobuf/TestProtoSchemas.java  | 136 ++++--
 ...messages.proto => proto2_schema_messages.proto} |  38 +-
 .../sdk/extensions/sketching/package-info.java     |   3 +
 .../sdk/extensions/sql/example/package-info.java   |   3 +
 .../meta/provider/datacatalog/package-info.java    |   3 +
 .../sql/meta/provider/hcatalog/package-info.java   |   3 +
 .../beam/sdk/extensions/sql/jdbc/package-info.java |   3 +
 .../beam/sdk/extensions/sql/impl/QueryPlanner.java |  13 +-
 .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java |   5 +-
 .../sdk/extensions/sql/zetasql/package-info.java   |   3 +
 .../zetasql/translation/ExpressionConverter.java   |  11 +-
 .../sql/zetasql/translation/package-info.java      |   3 +
 .../sql/zetasql/ZetaSQLDialectSpecTest.java        |  13 +
 .../sdk/extensions/zetasketch/package-info.java    |   3 +
 .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java       |   3 +-
 .../beam/sdk/io/aws/dynamodb/package-info.java     |   4 +
 .../beam/sdk/io/aws/options/package-info.java      |   4 +
 .../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java  |   3 +-
 .../apache/beam/sdk/io/aws/s3/package-info.java    |   4 +
 .../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java |   3 +-
 .../apache/beam/sdk/io/aws/sns/package-info.java   |   4 +
 .../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java |   3 +-
 .../apache/beam/sdk/io/aws/sqs/package-info.java   |   4 +
 .../apache/beam/sdk/io/aws/sns/AmazonSNSMock.java  | 385 -----------------
 .../beam/sdk/io/aws/sns/AmazonSNSMockErrors.java   |  30 --
 .../beam/sdk/io/aws/sns/AmazonSNSMockSuccess.java  |  39 --
 .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java  |  43 +-
 .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java      |   3 +-
 .../beam/sdk/io/aws2/dynamodb/package-info.java    |   4 +
 .../apache/beam/sdk/io/aws2/options/AwsModule.java |   3 +-
 .../beam/sdk/io/aws2/options/AwsOptions.java       |   3 +-
 .../beam/sdk/io/aws2/options/package-info.java     |   4 +
 .../org/apache/beam/sdk/io/aws2/sns/SnsIO.java     |   3 +-
 .../apache/beam/sdk/io/aws2/sns/package-info.java  |   4 +
 .../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java     |   3 +-
 .../apache/beam/sdk/io/aws2/sqs/package-info.java  |   5 +
 .../java/org/apache/beam/sdk/io/amqp/AmqpIO.java   |   3 +-
 .../org/apache/beam/sdk/io/amqp/package-info.java  |   4 +
 .../apache/beam/sdk/io/cassandra/CassandraIO.java  |   3 +-
 .../org/apache/beam/sdk/io/cassandra/Mapper.java   |   3 +-
 .../apache/beam/sdk/io/cassandra/package-info.java |   4 +
 .../beam/sdk/io/clickhouse/ClickHouseIO.java       |   3 +-
 .../beam/sdk/io/clickhouse/ClickHouseWriter.java   |   3 +-
 .../apache/beam/sdk/io/clickhouse/TableSchema.java |   3 +-
 .../beam/sdk/io/clickhouse/package-info.java       |   4 +
 .../beam/sdk/io/elasticsearch/ElasticsearchIO.java |   3 +-
 .../beam/sdk/io/elasticsearch/package-info.java    |   4 +
 .../beam/sdk/io/gcp/bigquery/BigQueryIO.java       |  48 +--
 .../beam/sdk/io/gcp/bigquery/BigQueryServices.java |   5 +-
 .../sdk/io/gcp/bigquery/BigQueryServicesImpl.java  |   3 +-
 .../sdk/io/gcp/bigquery/BigQuerySourceDef.java     |   3 +-
 .../gcp/bigquery/BigQueryStorageQuerySource.java   |   3 +-
 .../io/gcp/bigquery/BigQueryStorageSourceBase.java |   3 +-
 .../gcp/bigquery/BigQueryStorageStreamSource.java  |   5 +-
 .../gcp/bigquery/BigQueryStorageTableSource.java   |   3 +-
 .../beam/sdk/io/gcp/bigtable/BigtableIO.java       |   6 +-
 .../beam/sdk/io/gcp/bigtable/package-info.java     |   4 +
 .../gcp/common/GcpIoPipelineOptionsRegistrar.java  |   2 +
 .../beam/sdk/io/gcp/datastore/DatastoreIO.java     |   3 +-
 .../beam/sdk/io/gcp/datastore/package-info.java    |   4 +
 .../beam/sdk/io/gcp/pubsub/ExternalRead.java       | 138 ++++++
 .../beam/sdk/io/gcp/pubsub/ExternalWrite.java      | 102 +++++
 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java    | 172 +-------
 .../beam/sdk/io/gcp/spanner/BatchSpannerRead.java  |   4 +-
 .../sdk/io/gcp/spanner/CreateTransactionFn.java    |   2 +-
 .../sdk/io/gcp/spanner/MutationKeyEncoder.java     |  46 +-
 .../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java  |   2 +-
 .../beam/sdk/io/gcp/spanner/ReadSpannerSchema.java |   2 +-
 .../beam/sdk/io/gcp/spanner/SpannerAccessor.java   |  97 ++++-
 .../beam/sdk/io/gcp/spanner/SpannerConfig.java     |  71 ++--
 .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java  | 230 ++++++++--
 .../beam/sdk/io/gcp/spanner/package-info.java      |   4 +
 .../beam/sdk/io/gcp/testing/BigqueryClient.java    |   2 +
 .../sdk/io/gcp/testing/FakeBigQueryServices.java   |   4 +-
 .../sdk/io/gcp/testing/FakeDatasetService.java     |   2 +
 .../beam/sdk/io/gcp/testing/FakeJobService.java    |   4 +-
 .../beam/sdk/io/gcp/testing/package-info.java      |   3 +
 .../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java  |  13 +-
 .../sdk/io/gcp/spanner/MutationKeyEncoderTest.java |  61 ++-
 .../sdk/io/gcp/spanner/SpannerIOWriteTest.java     | 104 +++++
 .../apache/beam/sdk/io/hadoop/package-info.java    |   4 +
 .../beam/sdk/io/hdfs/HadoopFileSystemOptions.java  |  23 +
 .../org/apache/beam/sdk/io/hdfs/package-info.java  |   4 +
 .../sdk/io/hdfs/HadoopFileSystemOptionsTest.java   |  80 ++++
 .../beam/sdk/io/hadoop/format/HadoopFormatIO.java  |   3 +-
 .../beam/sdk/io/hadoop/format/package-info.java    |   4 +
 .../java/org/apache/beam/sdk/io/hbase/HBaseIO.java |   4 +-
 .../org/apache/beam/sdk/io/hbase/package-info.java |   4 +
 .../apache/beam/sdk/io/hcatalog/HCatalogIO.java    |   3 +-
 .../apache/beam/sdk/io/hcatalog/package-info.java  |   4 +
 .../io/hcatalog/test/EmbeddedMetastoreService.java |   2 +
 .../sdk/io/hcatalog/test/HCatalogIOTestUtils.java  |   2 +
 .../beam/sdk/io/hcatalog/test/package-info.java    |   3 +
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   |   7 +-
 .../org/apache/beam/sdk/io/jdbc/package-info.java  |   4 +
 .../java/org/apache/beam/sdk/io/jms/JmsIO.java     |   3 +-
 .../org/apache/beam/sdk/io/jms/package-info.java   |   4 +
 sdks/java/io/kafka/build.gradle                    |   7 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |  11 +-
 .../org/apache/beam/sdk/io/kafka/package-info.java |   4 +
 .../kafka/serialization/InstantDeserializer.java   |   3 +
 .../io/kafka/serialization/InstantSerializer.java  |   3 +
 .../sdk/io/kafka/serialization/package-info.java   |   4 +
 .../org/apache/beam/sdk/io/kinesis/KinesisIO.java  |   3 +-
 .../apache/beam/sdk/io/kinesis/package-info.java   |   4 +
 .../java/org/apache/beam/sdk/io/kudu/KuduIO.java   |   3 +-
 .../org/apache/beam/sdk/io/kudu/package-info.java  |   4 +
 .../beam/sdk/io/mongodb/AggregationQuery.java      |   3 +-
 .../org/apache/beam/sdk/io/mongodb/FindQuery.java  |   3 +-
 .../beam/sdk/io/mongodb/MongoDbGridFSIO.java       |   3 +-
 .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java  |   3 +-
 .../apache/beam/sdk/io/mongodb/package-info.java   |   4 +
 .../java/org/apache/beam/sdk/io/mqtt/MqttIO.java   |   3 +-
 .../org/apache/beam/sdk/io/mqtt/package-info.java  |   4 +
 .../org/apache/beam/sdk/io/parquet/ParquetIO.java  |   3 +-
 .../apache/beam/sdk/io/parquet/package-info.java   |   4 +
 .../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java    |   3 +-
 .../apache/beam/sdk/io/rabbitmq/package-info.java  |   4 +
 .../java/org/apache/beam/sdk/io/redis/RedisIO.java |   3 +-
 .../org/apache/beam/sdk/io/redis/package-info.java |   4 +
 .../java/org/apache/beam/sdk/io/solr/SolrIO.java   |   3 +-
 .../org/apache/beam/sdk/io/solr/package-info.java  |   4 +
 .../sdk/io/synthetic/SyntheticBoundedSource.java   |   3 +-
 .../sdk/io/synthetic/SyntheticUnboundedSource.java |   3 +-
 .../beam/sdk/io/synthetic/delay/package-info.java  |   4 +
 .../apache/beam/sdk/io/synthetic/package-info.java |   4 +
 .../java/org/apache/beam/sdk/io/tika/TikaIO.java   |   3 +-
 .../org/apache/beam/sdk/io/tika/package-info.java  |   4 +
 .../org/apache/beam/sdk/io/xml/package-info.java   |   4 +
 sdks/python/apache_beam/coders/coders.py           |   8 +-
 .../apache_beam/coders/coders_test_common.py       |   2 +-
 .../apache_beam/examples/fastavro_it_test.py       |   8 +-
 .../apache_beam/examples/snippets/snippets_test.py |   4 +-
 sdks/python/apache_beam/internal/pickler.py        |   8 -
 sdks/python/apache_beam/io/avroio_test.py          |   4 -
 sdks/python/apache_beam/io/filebasedsink_test.py   |  26 +-
 .../apache_beam/io/gcp/bigquery_file_loads.py      |  29 +-
 .../apache_beam/io/gcp/bigquery_write_perf_test.py |   2 +-
 sdks/python/apache_beam/io/iobase.py               | 128 ------
 sdks/python/apache_beam/io/iobase_test.py          |  86 +---
 sdks/python/apache_beam/io/textio.py               |   6 +-
 sdks/python/apache_beam/runners/common.py          |  69 +--
 .../runners/dataflow/dataflow_runner.py            |  21 +
 .../runners/dataflow/dataflow_runner_test.py       |  26 +-
 .../runners/dataflow/internal/apiclient.py         |  11 +-
 .../runners/dataflow/internal/apiclient_test.py    |  26 +-
 .../apache_beam/runners/dataflow/internal/names.py |   4 +-
 .../runners/interactive/caching/streaming_cache.py |  46 +-
 .../interactive/caching/streaming_cache_test.py    |  52 ++-
 .../runners/portability/fn_api_runner.py           |  11 +-
 .../runners/portability/fn_api_runner_test.py      |   8 +-
 .../runners/portability/local_job_service.py       |   9 +-
 sdks/python/apache_beam/runners/sdf_utils.py       | 176 ++++++++
 sdks/python/apache_beam/runners/sdf_utils_test.py  | 114 +++++
 .../apache_beam/runners/worker/bundle_processor.py |  16 +-
 .../apache_beam/runners/worker/operations.py       |   4 +
 .../testing/load_tests/co_group_by_key_test.py     |  30 +-
 .../apache_beam/testing/load_tests/combine_test.py |   2 +-
 sdks/python/setup.py                               |   8 +-
 .../src/documentation/sdks/java-dependencies.md    | 355 ++--------------
 .../src/documentation/sdks/python-dependencies.md  | 473 ++-------------------
 website/src/roadmap/index.md                       |   5 +-
 website/src/roadmap/java-sdk.md                    |   5 +-
 website/src/roadmap/spark-runner.md                |   9 +-
 387 files changed, 3652 insertions(+), 3089 deletions(-)
 create mode 100644 .github/autolabeler.yml
 delete mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/JavaReadViaImpulse.java
 delete mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/JavaReadViaImpulseTest.java
 delete mode 100644 
runners/spark/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/helpers/ReduceFnRunnerHelpers.java
 create mode 100644 sdks/go/pkg/beam/core/runtime/harness/monitoring.go
 create mode 100755 sdks/java/build-tools/beam-linkage-check.sh
 copy 
sdks/java/extensions/protobuf/src/test/proto/{proto2_coder_test_messages.proto 
=> proto2_schema_messages.proto} (59%)
 delete mode 100644 
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMock.java
 delete mode 100644 
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMockErrors.java
 delete mode 100644 
sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMockSuccess.java
 create mode 100644 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalRead.java
 create mode 100644 
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalWrite.java
 create mode 100644 sdks/python/apache_beam/runners/sdf_utils.py
 create mode 100644 sdks/python/apache_beam/runners/sdf_utils_test.py

Reply via email to