This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1f790fc [BEAM-11182][Website revamp] Implemented Stay up to date with
Beam and Changed Works with components (#13406)
add b0dd257 Squash, merge PR #13277 - [BEAM-10921] Add the
UserPipelineTracker to track user pipelines for derived pipelines
add 9107a12 Increase the deadline for CreateReadSession to 2 hours
add 4e0925b Merge pull request #13295: [BEAM-11226] Increase the deadline
for CreateReadSession to 2 hours
add 746cdfa Revert "Merge pull request #13321 from [BEAM-11255] Adding
upper bound on urllib3 dependency"
add d09768e [BEAM-11255] Revert "Merge pull request #13321 from
[BEAM-11255] Adding upper bound on urllib3 dependency" (#13328)
add fc4f1c7 add xvr dataflow badge
add beee718 Merge pull request #13336 from ihji/add_xvr_dataflow_badge
add 11c58f2 [BEAM-11211] Allow pyarrow up to 2.x, raise helpful error
when trying to write wit… (#13302)
add 8e70cf6 More helpful error when we can't find a way to construct a
POJO
add d9fe3f8 More helpful error when we can't construct a Java Bean
add 17c9d49 Add @SchemaCaseFormat, test with AutoValue beans and POJOs
add a6cd84c lint
add 47a6022 [BEAM-10989] Add @SchemaCaseFormat annotation (#13003)
add a8337b8 Update SDF related documentation.
add d2ff41e Merge pull request #13313 from [BEAM-10480] Update SDF
related documentation.
add 31fd516 [BEAM-9980] do not hardcode Python version for cross language
tests and make it switchable. (#13316)
add f46f050 use guava version specified in beam module
add 31bffe8 add guava dependency management to archetype
add a66c54d Merge pull request #13315: [BEAM-11228] Add guava to
dependency management in generated pom
add 656c21b [BEAM-11192] Add use_runner_v2 to experiments.
add 43e8b99 Merge pull request #13339 from [BEAM-11192] Add use_runner_v2
to experiments.
add 7a2e401 Allow null next index for last report.
add 53a0954 Merge pull request #13343 Allow null next index for last
report.
add 7aec62b Reshuffle in pd.read_* (#13346)
add 3d36122 [BEAM-11196] Ensure parent of fused stages is not one of its
transforms (#13303)
add 4fd7dfd [BEAM-11188] Refactor Go xlang expansion from runners to
core. (#13325)
add 01ccb08 Enable OrderedListState Dataflow tests.
add 4925b83 Merge pull request #13301: [BEAM-10650] Enable
OrderedListState Dataflow tests.
add 945fa4b Change hbase-shaded-client as provided dependency
add 5ea2244 Update change.md file
add 483ab67 Merge pull request #13351: [BEAM-9278] hbase-shaded-client as
provided dependency
add fb23d3f [BEAM-8876] Run hadoop tests with different versions and
enable HadoopFormatIT test in Java Postcommit
add 0ab83e9 Merge pull request #13287: [BEAM-8876] Run hadoop tests with
different versions and enable HadoopFormatIT test on Postcommit
add b0ac11d [BEAM-8615 BEAM-8569 BEAM-7937] Add hadoop 3 compatibility
tests
add 171c351 Merge pull request #13352: [BEAM-8615 BEAM-8569 BEAM-7937]
Add hadoop 3 compatibility tests
add 143f8f9 Add trigger commands for Direct, Spark XVR Postcommits.
(#13337)
add 2a22b77 [BEAM-9980] use constants of python versions in
PythonTestProperties.groovy (#13349)
add 2f2ffda [BEAM-10139][BEAM-10140] Add cross-language support for Java
SpannerIO with python wrapper (#12611)
add 6edc50c Use non-underscore methods for series and dataframe
comparators.
add 83070b4 Only force indexing for multi-input elementwise operations.
add eba648c Merge pull request #13341 Avoid unnecessary shuffling for
single-input elementwise operations.
add 252d2bc Fix Java ValidatesRunner V2 task dependency and update test
exclusion.
add d66c99c Merge pull request #13259: Fix Java ValidatesRunner V2 task
dependency.
add 1ae67c4 [BEAM-11265] Update quickstart-java.md (#13128)
add 78b9d05 [BEAM-11263] Java cleanUpDockerImages now force removes
container images.
add 7ac82a8 Merge pull request #13344 from [BEAM-11263] Java
cleanUpDockerImages now force removes container images.
add a7ea758 Further dataframe batch consolidation.
add 9ef95b5 Add another test.
add 6300c33 Update sdks/python/apache_beam/dataframe/transforms.py
add b18784f Merge pull request #13333 Further dataframe batch
consolidation.
add 09a96db Upgrade gcsio to 2.1.6
add 41bdd91 Merge pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6
add 028becb Go redundant type cleanup. (#13348)
add baae15a [BEAM-11207] Metric Extraction via proto RPC API (#13272)
add 6c5011a [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest
(#13340)
add 9a3d3ea [BEAM-11270] Dataflow Java on runner v2 tests are failing
because sdk docker container is cleaned up incorrectly
add 39b95c4 Merge pull request #13356 from [BEAM-11270] Fix container
clean-up logic
add 5c8a748 Update Beam Dataflow container versions for Python
add 15cdc43 Merge pull request #13358: Update Beam Dataflow container
versions for Python
add 689d89f Fix failing seed job
add 85b3023 Merge pull request #13363 from kamilwu/fix-seed-job
add 890332e [BEAM-4136] Keep strong reference to loggers to avoid
potential NPE (#13345)
add a872aca Fix NPE in CountingSource
add fc41a9d Merge pull request #13361 from Fix NPE in CountingSource
add a95dbc8 Upgrade Conscrypt to 2.5.1 in GrpcVendoring_1_26_0
add ed09e96 Merge pull request #13150 from [BEAM-5748] Upgrade Conscrypt
to 2.5.1 in GrpcVendoring_1_26_0
add fd87be2f add validate runner dataflow v2 java badge
add cf6f1d1 Merge pull request #13360 from ihji/add_dataflow_v2_java_badge
add 30c1eef [BEAM-8106] Separate Java8/11 container image build tasks
(#13211)
add 1653a97 [BEAM-9782] Update README for instructions on updating katas
(#13329)
add 62e8aeb [BEAM-8569] Add changes note about Hadoop 3 support
add 010af43 Merge pull request #13354: [BEAM-8569] Add changes note about
Hadoop 3 support
add ed33e8c [BEAM-11207] Adding PipelineResult to session runner. (#13371)
add 5f133ca [BEAM-11211] parquetio_test using multiple pyarrow versions
add 1a6b9d8 Merge pull request #13369: [BEAM-11211] parquetio_test using
multiple pyarrow versions
add d4c82ec Make UsesTestStream extend UsesUnboundedPCollections for
exclusion in batch only runners
add 434bce5 Merge pull request #13376 from Make UsesTestStream extend
UsesUnboundedPCollections for exclusion in batch-only runners
add 53a26ae [BEAM-11092] Add bigquery io request count metric,
implementing HarnessMonitoringInfos and process_wide metrics
add 7c28db2 Merge pull request #13217 from [BEAM-11092] Add bigquery io
request count metric, implementing HarnessMonitoringInfos and process_wide
metrics
add cdb882d [BEAM-11303] Use sum as the post-agg for size (#13379)
add fc52b63 Update SDF programming guide.
add 774185b Merge pull request #13326 from [BEAM-10480] Update SDF
programming guide
add a267d96 [BEAM-9615] Switch options for logical types. (#13366)
add 532b024 [BEAM-10475]Add WithShardedKey variation of GroupIntoBatches
transform in Python SDK. (#13292)
add 70ddf84 [BEAM-11188] Adding unit test for mergeExpandedWithPipeline
(#13370)
add 37deb55 [BEAM-11075] Pardo tests for Go SDK (#13362)
add d5eaa3f [BEAM-11304] groupby(Series).size() on a filtered dataframe
is incorrect (#13380)
add c92a743 [BEAM-11308] Update deprecated Google GitHub action
add 18b82c6 Merge pull request #13384 from
TobKed/beam-11308-update-deprecated-google-github-action
add b9f0538 [BEAM-10925] Move general-usage helper methods from
DoFnSignatures to ReflectHelpers.
add 019ebb5 Merge pull request #13304 from ibzib/reflect-helpers
add 909a519 [BEAM-10379] Add bit_and back into Calcite SQL. (#13062)
add c4ac644 Fix Windmill timer tag output timestamp decoding
add 92ea33e Merge pull request #13373 from nehsyc/fix_timestamp_decoding
add 54225bd [BEAM-9547] Add support for Series.rename (#13385)
add fa5c512 [BEAM-113100] Exclude
org.apache.beam.sdk.transforms.ViewTest.testWindowedSideInputNotPresent from
flink validates runner test
add 2fed655 Merge pull request #13392 from [BEAM-11310] Exclude
org.apache.beam.sdk.transforms.ViewTest.testWin
add 626551e [BEAM-11254] New documentation on multi-language pipelines
(#13317)
add be9ef8c Updated shared.py comments (#13187)
add a0f1a54 [BEAM-10974] Skip GroupByKeyTest.testLargeKeys10MB.
add 1a34854 Merge pull request #13218 from ibzib/BEAM-10974
add 60e0221 Restore original str() behaviour to MetricName, for user
counter(namespace+name) style MetricNames
add 6851109 Merge pull request #13400 from Restore original str()
behaviour to MetricName, for user counter(namespace+name) style MetricNames
add 9054c7a set_index does not preserve partitioning (#13398)
add 07cbed5 Enable more runner v2 tests.
add 208690c Merge pull request #13372 from boyuanzz/enable_test
add b29102e [BEAM-11070] Use self-checkpoint to enfore finalization
happens.
add 9b51d4b Merge pull request #13338 from [BEAM-11070] Use
self-checkpoint to enforce finalization happens.
add 353ca39 fix java 11 container task
add 213385d add java 11 examples test
add 19eaf71 add java 11 image to released images
add fa7c5fc Merge pull request #13374: [BEAM-8106] Add Java 11 container
to release and Java 11+V2+Dataflow testing
add 67339a9 [BEAM-11266] Python IO MongoDB: add bucket_auto aggregation
option for bundling in Atlas. (#13350)
add ab2e94d Add use_unified_worker to pipeline options.
add ae1df91 Merge pull request #13396 from boyuanzz/taxi_cmd
add f5ba29f [BEAM-10593] Add Jenkins job for creating regular snapshots
of the Beam SDK Harness Container Images (#13393)
add 6cc3a92 Support ZetaSQL IN operator
add 8b4b78e Merge pull request #13381 from robinyqiu/in
add 6122341 Nexmark: fix typos and clean unused options
add 4507139 Merge pull request #13403 from
rHermes/nexmark-misc-improvements
add b51d64e Add an option to GroupIntoBatches to output ShardedKeys.
Update Dataflow pipeline translation accordingly.
add f4d889f Merge pull request #13208 from nehsyc/fix_override
add ab1d27c [BEAM-10124] Announce ContextualTextIO in Beam 2.26.0.
(#13402)
add 6a0fbe2 [BEAM-10896] Support ZetaSQL UNNEST syntax on array of struct
type (#13377)
add 99b4669 add to dashboard
add 4f34b9d fix Java 11 HOME
add 245374a Merge pull request #13416: Fix Java 11 V2 CI test and add V2
tests to dashboard
add bd825f5 [BEAM-11324] Add additional verification in
PartitioningSession (#13401)
add d7655c1 [BEAM-10921]: Fix BEAM-10921 and underlying issues (#13335)
new 0850259 Merge remote-tracking branch 'origin/master' into
website-revamp
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 4 +-
.github/workflows/build_wheels.yml | 8 +-
.github/workflows/java_tests.yml | 2 +-
.github/workflows/python_tests.yml | 2 +-
.test-infra/jenkins/CommonTestProperties.groovy | 12 +-
...Properties.groovy => JavaTestProperties.groovy} | 4 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 15 +-
.test-infra/jenkins/PythonTestProperties.groovy | 10 +
.test-infra/jenkins/README.md | 6 +-
.../jenkins/job_LoadTests_ParDo_Flink_Go.groovy | 150 +++++
...it_CrossLanguageValidatesRunner_Dataflow.groovy | 38 +-
...mmit_CrossLanguageValidatesRunner_Direct.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 23 +-
...tCommit_Java_Examples_Dataflow_V2_Java11.groovy | 50 ++
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 1 -
.../jenkins/job_Publish_Docker_Snapshots.groovy | 8 +-
.../jenkins/job_Publish_SDK_Image_Snapshots.groovy | 55 ++
CHANGES.md | 9 +-
README.md | 4 +-
build.gradle | 28 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 20 +-
.../apache/beam/gradle/GrpcVendoring_1_26_0.groovy | 2 +-
learning/katas/go/README.md | 29 +
.../src/main/scripts/build_release_candidate.sh | 6 +-
release/src/main/scripts/publish_docker_images.sh | 2 +-
.../core/construction/PTransformMatchers.java | 2 +-
runners/flink/job-server/flink_job_server.gradle | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 78 ++-
.../dataflow/DataflowPipelineTranslator.java | 14 +-
.../beam/runners/dataflow/DataflowRunner.java | 29 +-
.../runners/dataflow/GroupIntoBatchesOverride.java | 132 ++++-
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../dataflow/DataflowPipelineTranslatorTest.java | 67 ++-
.../beam/runners/dataflow/DataflowRunnerTest.java | 64 ++-
.../dataflow/worker/WindmillTimerInternals.java | 13 +-
.../dataflow/worker/WorkItemStatusClient.java | 2 +-
.../worker/WindmillTimerInternalsTest.java | 4 +-
runners/portability/java/build.gradle | 9 +-
runners/spark/build.gradle | 66 ++-
sdks/go/pkg/beam/core/metrics/metrics.go | 118 ++++
.../core/runtime/graphx/schema/logicaltypes.go | 121 ++++
.../pkg/beam/core/runtime/graphx/schema/schema.go | 115 ++--
.../beam/core/runtime/graphx/schema/schema_test.go | 55 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 44 +-
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 210 +++++++
sdks/go/pkg/beam/core/runtime/graphx/xlang_test.go | 218 +++++++
.../go/pkg/beam/core/runtime/harness/monitoring.go | 187 +-----
.../beam/core/runtime/harness/monitoring_test.go | 33 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 172 ++++++
.../beam/core/runtime/metricsx/metricsx_test.go | 166 ++++++
sdks/go/pkg/beam/core/runtime/metricsx/urns.go | 170 ++++++
sdks/go/pkg/beam/core/runtime/options.go | 1 +
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 69 ++-
sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 2 +-
.../pkg/beam/core/runtime/xlangx/namespace_test.go | 348 +++++------
sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 228 --------
sdks/go/pkg/beam/doc_test.go | 2 +-
sdks/go/pkg/beam/io/textio/sdf_test.go | 2 +-
sdks/go/pkg/beam/options/jobopts/options.go | 3 +
sdks/go/pkg/beam/pipeline.go | 6 +
sdks/go/pkg/beam/runner.go | 6 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 25 +-
sdks/go/pkg/beam/runners/direct/direct.go | 17 +-
sdks/go/pkg/beam/runners/dot/dot.go | 10 +-
sdks/go/pkg/beam/runners/flink/flink.go | 3 +-
sdks/go/pkg/beam/runners/session/session.go | 6 +-
sdks/go/pkg/beam/runners/spark/spark.go | 3 +-
.../beam/runners/universal/runnerlib/execute.go | 48 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 3 +
sdks/go/pkg/beam/runners/universal/universal.go | 28 +-
sdks/go/pkg/beam/runners/vet/vet.go | 8 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 3 +-
sdks/go/pkg/beam/x/beamx/run.go | 8 +
sdks/go/pkg/beam/xlang.go | 59 +-
sdks/go/test/load/build.gradle | 66 +++
sdks/go/test/load/pardo/pardo.go | 112 ++++
sdks/go/test/load/util.go | 227 ++++++++
sdks/java/container/build.gradle | 63 +-
sdks/java/container/common.gradle | 104 ++++
.../java/container/java11/build.gradle | 11 +-
.../java/container/java8/build.gradle | 11 +-
.../org/apache/beam/sdk/io/CountingSource.java | 3 +-
.../sdk/schemas/FieldValueTypeInformation.java | 32 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 27 +-
.../sdk/schemas/annotations/SchemaCaseFormat.java | 73 +++
.../sdk/schemas/annotations/SchemaFieldName.java | 11 +-
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 39 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 5 +-
.../apache/beam/sdk/testing/UsesTestStream.java | 2 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 4 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 70 ++-
.../sdk/transforms/reflect/DoFnSignatures.java | 77 +--
.../splittabledofn/RestrictionTracker.java | 23 +-
.../beam/sdk/util/common/ReflectHelpers.java | 60 ++
.../beam/sdk/schemas/AutoValueSchemaTest.java | 57 ++
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 73 +++
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 55 ++
.../beam/sdk/schemas/utils/SchemaTestUtils.java | 22 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 129 +++++
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 56 ++
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 82 +++
sdks/java/extensions/sorter/build.gradle | 46 +-
sdks/java/extensions/sql/build.gradle | 44 +-
.../impl/transform/BeamBuiltinAggregations.java | 143 ++---
.../provider/bigquery/BeamBigQuerySqlDialect.java | 11 +
.../extensions/sql/BeamSqlDslAggregationTest.java | 263 ++++-----
.../zetasql/SupportedZetaSqlBuiltinFunctions.java | 4 +-
.../sql/zetasql/ZetaSQLQueryPlanner.java | 8 +
.../translation/ArrayScanColumnRefToUncollect.java | 4 +-
.../ArrayScanLiteralToUncollectConverter.java | 4 +-
.../translation/ArrayScanToJoinConverter.java | 5 +-
.../translation/SqlOperatorMappingTable.java | 1 +
.../zetasql/unnest/BeamZetaSqlUncollectRel.java | 120 ++++
.../zetasql/unnest/BeamZetaSqlUncollectRule.java | 54 ++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRel.java | 164 ++++++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRule.java | 120 ++++
.../sql/zetasql/unnest/ZetaSqlUnnest.java | 140 +++++
.../sql/zetasql/unnest/package-info.java | 8 +-
.../beam/sdk/extensions/sql/zetasql/TestInput.java | 66 ++-
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 137 ++++-
.../extensions/sql/zetasql/ZetaSqlTestBase.java | 8 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 50 +-
.../harness/logging/BeamFnLoggingClientTest.java | 50 +-
.../expansion-service/build.gradle | 37 ++
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 22 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 320 +++++++++++
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 6 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 8 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 112 +++-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 374 ++++++++++++
.../beam/sdk/io/gcp/spanner/StructUtils.java | 387 +++++++++++++
.../beam/sdk/io/gcp/spanner/MutationUtilsTest.java | 285 +++++++++
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 5 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 258 +++++++++
sdks/java/io/hadoop-common/build.gradle | 50 +-
sdks/java/io/hadoop-file-system/build.gradle | 57 +-
sdks/java/io/hadoop-format/build.gradle | 104 +++-
.../io/hadoop/format/HadoopFormatIOElasticIT.java | 20 +-
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 58 +-
.../hadoop/format/HadoopFormatIOTestOptions.java | 6 +
sdks/java/io/hbase/build.gradle | 2 +-
sdks/java/io/hcatalog/build.gradle | 42 +-
sdks/java/io/parquet/build.gradle | 43 +-
sdks/java/maven-archetypes/examples/build.gradle | 1 +
sdks/java/maven-archetypes/examples/sample.txt | 102 ++++
.../src/main/resources/archetype-resources/pom.xml | 8 +-
.../beam/sdk/nexmark/NexmarkConfiguration.java | 2 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 4 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 30 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 2 +-
sdks/python/apache_beam/dataframe/expressions.py | 92 ++-
sdks/python/apache_beam/dataframe/frame_base.py | 8 +-
sdks/python/apache_beam/dataframe/frames.py | 65 ++-
sdks/python/apache_beam/dataframe/frames_test.py | 91 +--
sdks/python/apache_beam/dataframe/io.py | 1 +
.../apache_beam/dataframe/pandas_doctests_test.py | 3 +-
sdks/python/apache_beam/dataframe/partitionings.py | 26 +
sdks/python/apache_beam/dataframe/transforms.py | 62 +-
.../apache_beam/dataframe/transforms_test.py | 17 +-
.../apache_beam/examples/kafkataxi/README.md | 6 +-
sdks/python/apache_beam/internal/metrics/metric.py | 90 +++
.../apache_beam/internal/metrics/metric_test.py | 41 ++
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 36 +-
.../gcp/resource_identifiers.py} | 34 +-
sdks/python/apache_beam/io/gcp/spanner.py | 635 +++++++++++++++++++++
.../io/gcp/tests/xlang_spannerio_it_test.py | 339 +++++++++++
sdks/python/apache_beam/io/iobase.py | 12 +-
sdks/python/apache_beam/io/mongodbio.py | 166 ++++--
sdks/python/apache_beam/io/mongodbio_it_test.py | 94 ++-
sdks/python/apache_beam/io/mongodbio_test.py | 235 +++++++-
sdks/python/apache_beam/io/parquetio.py | 8 +
sdks/python/apache_beam/io/parquetio_test.py | 4 +
sdks/python/apache_beam/metrics/cells.py | 22 +-
sdks/python/apache_beam/metrics/execution.pxd | 4 +-
sdks/python/apache_beam/metrics/execution.py | 71 ++-
sdks/python/apache_beam/metrics/metric.py | 18 +-
sdks/python/apache_beam/metrics/metric_test.py | 16 +
sdks/python/apache_beam/metrics/metricbase.py | 46 +-
.../python/apache_beam/metrics/monitoring_infos.py | 21 +-
.../apache_beam/metrics/monitoring_infos_test.py | 35 ++
sdks/python/apache_beam/pipeline.py | 12 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/interactive/background_caching_job.py | 11 +-
.../interactive/background_caching_job_test.py | 42 +-
.../runners/interactive/interactive_environment.py | 49 +-
.../runners/interactive/interactive_runner.py | 4 +-
.../runners/interactive/interactive_runner_test.py | 3 -
.../runners/interactive/pipeline_fragment.py | 9 +-
.../runners/interactive/pipeline_instrument.py | 29 +-
.../runners/interactive/recording_manager.py | 20 +-
.../runners/interactive/recording_manager_test.py | 61 +-
.../runners/interactive/user_pipeline_tracker.py | 150 +++++
.../interactive/user_pipeline_tracker_test.py | 209 +++++++
.../runners/portability/flink_runner_test.py | 3 +-
.../portability/fn_api_runner/fn_runner_test.py | 81 ++-
.../portability/fn_api_runner/translations.py | 29 +-
.../portability/fn_api_runner/translations_test.py | 39 ++
.../apache_beam/runners/worker/sdk_worker.py | 38 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 60 ++
sdks/python/apache_beam/transforms/core.py | 28 +-
sdks/python/apache_beam/transforms/util.py | 49 +-
sdks/python/apache_beam/transforms/util_test.py | 17 +
sdks/python/apache_beam/utils/sharded_key.py | 2 +-
sdks/python/apache_beam/utils/shared.py | 24 +-
sdks/python/setup.py | 4 +-
sdks/python/test-suites/portable/common.gradle | 6 +-
sdks/python/test-suites/tox/common.gradle | 3 +-
sdks/python/test-suites/tox/py38/build.gradle | 7 +
sdks/python/tox.ini | 21 +
settings.gradle | 4 +
.../content/en/documentation/programming-guide.md | 365 +++++++++++-
.../en/documentation/runtime/environments.md | 18 +-
.../site/content/en/get-started/quickstart-java.md | 20 +-
.../partials/section-menu/en/documentation.html | 10 +
.../images/multi-language-pipelines-diagram.svg | 20 +
216 files changed, 10389 insertions(+), 2166 deletions(-)
copy .test-infra/jenkins/{PythonTestProperties.groovy =>
JavaTestProperties.groovy} (88%)
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Flink_Go.groovy
create mode 100644
.test-infra/jenkins/job_PostCommit_Java_Examples_Dataflow_V2_Java11.groovy
create mode 100644 .test-infra/jenkins/job_Publish_SDK_Image_Snapshots.groovy
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/schema/logicaltypes.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/metricsx_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/urns.go
delete mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/translate.go
create mode 100644 sdks/go/test/load/build.gradle
create mode 100644 sdks/go/test/load/pardo/pardo.go
create mode 100644 sdks/go/test/load/util.go
create mode 100644 sdks/java/container/common.gradle
copy .test-infra/jenkins/PythonTestProperties.groovy =>
sdks/java/container/java11/build.gradle (72%)
copy .test-infra/jenkins/PythonTestProperties.groovy =>
sdks/java/container/java8/build.gradle (72%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaCaseFormat.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRel.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRule.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUnnestRel.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUnnestRule.java
create mode 100644
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/ZetaSqlUnnest.java
copy .test-infra/jenkins/PythonTestProperties.groovy =>
sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/package-info.java
(78%)
create mode 100644
sdks/java/io/google-cloud-platform/expansion-service/build.gradle
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java
create mode 100644
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/StructUtils.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtilsTest.java
create mode 100644
sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/StructUtilsTest.java
create mode 100644 sdks/java/maven-archetypes/examples/sample.txt
copy sdks/python/apache_beam/{metrics/execution.pxd =>
io/gcp/resource_identifiers.py} (56%)
create mode 100644 sdks/python/apache_beam/io/gcp/spanner.py
create mode 100644
sdks/python/apache_beam/io/gcp/tests/xlang_spannerio_it_test.py
create mode 100644
sdks/python/apache_beam/runners/interactive/user_pipeline_tracker.py
create mode 100644
sdks/python/apache_beam/runners/interactive/user_pipeline_tracker_test.py
create mode 100644
website/www/site/static/images/multi-language-pipelines-diagram.svg
