This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 56c2082 [BEAM-13459] Update CHANGES.md, add note about artifact
caching python jobs #16416
add e9e49ff [BEAM-13015] Use 20% of memory when the maximum has been
configured. (#16420)
add 38b833a [BEAM-13567] Consolidate runner flag definition. (#16426)
add 7d1d169 Upgrade python library versions in base_image_requirements.txt
add 91ce87d Merge pull request #16424 from y1chi/bump_pubsub
add 2a4a4cd [BEAM-13601] Don't cache Row types for a schema. (#16427)
add 3df986f [BEAM-13430] Re-enable checkerframework (#16429)
add 27908b4 [BEAM-13430] Ensure that testRuntimeMigration depends on
"default" configuration allowing us to get the main "jar" as a dependency.
(#16430)
add e2fadfb Merge pull request #16277 from [BEAM-13124][Playground]
Create readiness endpoint
add 84a2902 Merge pull request #16314 from
[BEAM-13260][Playground]Implement setup of CI test environment
add 5962f59 Merge pull request #16383 from [BEAM-13566][Playground] Add
logic of sending preparation's output on the backend side
add 6612983 [BEAM-13565][Playground] Add GetPreparationOutput method into
the .proto files; Regenerate files;
add 56a3c57 Merge pull request #16382 from [BEAM-13565][Playground] Add
GetPreparationOutput API method to .proto file
add 2c4f1d9 Merge pull request #16365 from [BEAM-13559][Playground]
Remove tag in examples CD
add d733e85 Merge pull request #16360 from [BEAM-13546][Playground]
Update nginx configuration to enable embedding iframes
add f51e1a6 Merge pull request #16192 from [BEAM-13395] [Playground] Tag
katas
add 5ea425b Merge pull request #16254 from [BEAM-13249][Playground]
Security – Mock Network
add b778f7d update bom
add 46b3ef5 Merge pull request #16377: [BEAM-13576] update bom
add 21d9f6d Merge pull request #16347: fix: move connector to use v1
BigQuery Storage Write API
add 9642de1 [BEAM-12879] Prevented missing permission from failing GCS I/O
add 70d9e2a [BEAM-12879] Prevented missing permission from failing GCS I/O
add c67bed4 [BEAM-13603] Fix bug in apache_beam.utils.Shared (#16437)
add 90f1675 [BEAM-10345] Add an import guard to support recent
google-cloud-spanner versions. (#16434)
No new revisions were added by this update.
Summary of changes:
.github/workflows/playground_deploy_examples.yml | 129 +++
.test-infra/validate-runner/build.gradle | 1 -
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 43 +-
examples/java/build.gradle | 1 -
examples/kotlin/build.gradle | 1 -
.../commontransforms/aggregation/count/Task.java | 8 +
.../commontransforms/aggregation/max/Task.java | 8 +
.../commontransforms/aggregation/mean/Task.java | 8 +
.../commontransforms/aggregation/min/Task.java | 8 +
.../commontransforms/aggregation/sum/Task.java | 8 +
.../katas/commontransforms/filter/filter/Task.java | 8 +
.../katas/commontransforms/filter/pardo/Task.java | 8 +
.../katas/commontransforms/withkeys/Task.java | 8 +
.../katas/coretransforms/branching/Task.java | 10 +
.../combine/binarycombinefnlambda/Task.java | 9 +
.../combine/binarycombinefn/Task.java | 9 +
.../coretransforms/combine/combineperkey/Task.java | 9 +
.../coretransforms/combine/combinefn/Task.java | 9 +
.../katas/coretransforms/combine/simple/Task.java | 9 +
.../katas/coretransforms/composite/Task.java | 11 +
.../katas/coretransforms/flatten/Task.java | 10 +
.../katas/coretransforms/groupbykey/Task.java | 9 +
.../coretransforms/map/flatmapelements/Task.java | 9 +
.../katas/coretransforms/map/mapelements/Task.java | 8 +
.../coretransforms/map/pardoonetomany/Task.java | 9 +
.../katas/coretransforms/map/pardo/Task.java | 8 +
.../katas/coretransforms/partition/Task.java | 10 +
.../katas/coretransforms/sideoutput/Task.java | 10 +
.../learning/katas/examples/wordcount/Task.java | 8 +
.../beam/learning/katas/intro/hello/Task.java | 8 +
.../learning/katas/windowing/fixedwindow/Task.java | 9 +
.../Common Transforms/Aggregation/Count/task.py | 8 +
.../Common Transforms/Aggregation/Largest/task.py | 8 +
.../Common Transforms/Aggregation/Mean/task.py | 8 +
.../Common Transforms/Aggregation/Smallest/task.py | 8 +
.../Common Transforms/Aggregation/Sum/task.py | 8 +
.../python/Common Transforms/Filter/Filter/task.py | 8 +
.../python/Common Transforms/Filter/ParDo/task.py | 8 +
.../Common Transforms/WithKeys/WithKeys/task.py | 8 +
.../Core Transforms/Branching/Branching/task.py | 10 +
.../CoGroupByKey/CoGroupByKey/task.py | 9 +
.../Core Transforms/Combine/Combine PerKey/task.py | 8 +
.../Core Transforms/Combine/CombineFn/task.py | 8 +
.../Combine/Simple Function/task.py | 8 +
.../Composite Transform/task.py | 9 +
.../python/Core Transforms/Flatten/Flatten/task.py | 8 +
.../Core Transforms/GroupByKey/GroupByKey/task.py | 8 +
.../python/Core Transforms/Map/FlatMap/task.py | 9 +
.../katas/python/Core Transforms/Map/Map/task.py | 8 +
.../Core Transforms/Map/ParDo OneToMany/task.py | 9 +
.../katas/python/Core Transforms/Map/ParDo/task.py | 8 +
.../Core Transforms/Partition/Partition/task.py | 9 +
.../Core Transforms/Side Input/Side Input/task.py | 8 +
.../Side Output/Side Output/task.py | 9 +
.../python/Examples/Word Count/Word Count/task.py | 8 +
.../Introduction/Hello Beam/Hello Beam/task.py | 8 +
.../Windowing/Adding Timestamp/ParDo/task.py | 8 +
.../Fixed Time Window/Fixed Time Window/task.py | 9 +
playground/api/v1/api.proto | 14 +-
playground/backend/cmd/server/controller.go | 17 +
playground/backend/cmd/server/http.go | 13 +-
playground/backend/cmd/server/server.go | 2 +-
playground/backend/containers/go/Dockerfile | 14 +-
playground/backend/containers/go/build.gradle | 12 +
playground/backend/containers/go/entrypoint.sh | 25 +
playground/backend/containers/java/Dockerfile | 15 +-
playground/backend/containers/java/build.gradle | 12 +
playground/backend/containers/java/entrypoint.sh | 25 +
playground/backend/containers/python/Dockerfile | 15 +-
playground/backend/containers/python/build.gradle | 12 +
playground/backend/containers/python/entrypoint.sh | 26 +
playground/backend/internal/api/v1/api.pb.go | 729 +++++++-----
playground/backend/internal/api/v1/api_grpc.pb.go | 38 +
playground/backend/internal/cache/cache.go | 3 +
.../backend/internal/cache/redis/redis_cache.go | 2 +-
.../internal/code_processing/code_processing.go | 22 +-
playground/backend/internal/environment/beam.go | 16 +-
.../internal/environment/environment_service.go | 22 +-
.../environment/environment_service_test.go | 11 +-
.../backend/internal/executors/executor_test.go | 2 +-
.../internal/preparators/java_preparators.go | 8 +-
.../setup_tools/builder/setup_builder_test.go | 4 +-
playground/backend/internal/utils/system_utils.go | 45 +
.../backend/internal/utils/system_utils_test.go | 86 ++
playground/categories.yaml | 6 +
playground/frontend/Dockerfile | 1 +
playground/frontend/lib/api/v1/api.pb.dart | 1211 ++++++++++++++------
playground/frontend/lib/api/v1/api.pbgrpc.dart | 363 +++---
playground/frontend/lib/api/v1/api.pbjson.dart | 166 ++-
playground/frontend/nginx_default.conf | 12 +-
playground/infrastructure/config.py | 4 +-
playground/infrastructure/helm/Chart.yaml | 20 +
.../infrastructure/helm/templates/_helpers.tpl | 76 ++
.../helm/templates/deployment-backend-go.yaml | 71 ++
.../helm/templates/deployment-backend-java.yaml | 71 ++
.../helm/templates/deployment-backend-python.yaml | 71 ++
.../infrastructure/helm/templates/service-go.yaml | 32 +
.../helm/templates/service-java.yaml | 32 +
.../helm/templates/service-python.yaml | 32 +
playground/infrastructure/helm/values.yaml | 65 ++
playground/infrastructure/helper.py | 36 +-
playground/infrastructure/proxy/allow_list.py | 29 +
.../infrastructure/proxy/allow_list_proxy.py | 30 +
playground/infrastructure/test_cd_helper.py | 1 +
playground/infrastructure/test_helper.py | 34 +-
.../modules/gke/main.tf} | 45 +-
.../modules/gke/output.tf} | 23 +-
.../modules/gke/variables.tf} | 37 +-
runners/flink/job-server/flink_job_server.gradle | 2 -
runners/google-cloud-dataflow-java/build.gradle | 3 -
.../examples-streaming/build.gradle | 1 -
.../examples/build.gradle | 1 -
runners/portability/java/build.gradle | 1 -
runners/samza/job-server/build.gradle | 2 -
runners/spark/job-server/spark_job_server.gradle | 2 -
runners/spark/spark_runner.gradle | 1 -
.../go/pkg/beam/runners/flag.go | 19 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 21 +-
sdks/go/pkg/beam/x/beamx/run.go | 21 +-
.../container/license_scripts/dep_urls_java.yaml | 2 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 54 +-
.../beam/sdk/options/SdkHarnessOptionsTest.java | 48 +-
sdks/java/extensions/ml/build.gradle | 1 -
sdks/java/extensions/sql/build.gradle | 2 -
sdks/java/extensions/sql/jdbc/build.gradle | 1 -
sdks/java/extensions/sql/perf-tests/build.gradle | 2 -
sdks/java/io/amazon-web-services/build.gradle | 1 -
.../apache/beam/sdk/io/aws/options/AwsOptions.java | 4 +-
sdks/java/io/amazon-web-services2/build.gradle | 1 -
.../beam/sdk/io/aws2/options/AwsOptions.java | 4 +-
sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 -
.../elasticsearch-tests-6/build.gradle | 1 -
sdks/java/io/file-based-io-tests/build.gradle | 3 -
sdks/java/io/google-cloud-platform/build.gradle | 1 -
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 12 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 35 +-
.../sdk/io/gcp/bigquery/SplittingIterable.java | 2 +-
.../gcp/bigquery/StorageApiFinalizeWritesDoFn.java | 8 +-
.../bigquery/StorageApiFlushAndFinalizeDoFn.java | 4 +-
.../bigquery/StorageApiWriteUnshardedRecords.java | 6 +-
.../bigquery/StorageApiWritesShardedRecords.java | 6 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 14 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 -
sdks/java/io/hadoop-format/build.gradle | 3 -
sdks/java/io/jdbc/build.gradle | 1 -
sdks/java/io/kafka/build.gradle | 1 -
sdks/java/io/mongodb/build.gradle | 1 -
.../apache_beam/io/gcp/experimental/spannerio.py | 12 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 26 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 47 +
sdks/python/apache_beam/typehints/schemas.py | 24 +-
sdks/python/apache_beam/utils/shared.py | 1 +
sdks/python/apache_beam/utils/shared_test.py | 11 +
.../container/py36/base_image_requirements.txt | 18 +-
.../container/py37/base_image_requirements.txt | 28 +-
.../container/py38/base_image_requirements.txt | 28 +-
156 files changed, 3562 insertions(+), 1214 deletions(-)
create mode 100644 .github/workflows/playground_deploy_examples.yml
create mode 100755 playground/backend/containers/go/entrypoint.sh
create mode 100755 playground/backend/containers/java/entrypoint.sh
create mode 100755 playground/backend/containers/python/entrypoint.sh
create mode 100644 playground/infrastructure/helm/Chart.yaml
create mode 100644 playground/infrastructure/helm/templates/_helpers.tpl
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-go.yaml
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-java.yaml
create mode 100644
playground/infrastructure/helm/templates/deployment-backend-python.yaml
create mode 100644 playground/infrastructure/helm/templates/service-go.yaml
create mode 100644 playground/infrastructure/helm/templates/service-java.yaml
create mode 100644 playground/infrastructure/helm/templates/service-python.yaml
create mode 100644 playground/infrastructure/helm/values.yaml
create mode 100644 playground/infrastructure/proxy/allow_list.py
create mode 100644 playground/infrastructure/proxy/allow_list_proxy.py
copy playground/{categories.yaml => terraform/modules/gke/main.tf} (52%)
copy playground/{categories.yaml => terraform/modules/gke/output.tf} (74%)
copy playground/{categories.yaml => terraform/modules/gke/variables.tf} (64%)
copy playground/backend/internal/utils/system_utils.go =>
sdks/go/pkg/beam/runners/flag.go (72%)
