[beam] branch nightly-refs/heads/master updated (7d78572baa2 -> 3cdb9fe2b2d)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch nightly-refs/heads/master in repository https://gitbox.apache.org/repos/asf/beam.git from 7d78572baa2 Add GoogleAdsIO to the list of built-in connectors (#28667) add 5c4bea2fdbb add YAML SDK component to issue template (#28678) add 725a2d62b27 Feature/add retry to gcp auth (#28475) add f35a4130ef3 Make SerializableConfiguration cacheable (#28590) add ced3de3feae Require --sdk_location for Dataflow pipelines running with dev SDKs. (#28670) add 1c50fd2e6a8 Upgrade transforms without upgrading the pipelines (#28210) add 3cdb9fe2b2d Bump Python FnAPI beam-master container. (#28704) No new revisions were added by this update. Summary of changes: .github/ISSUE_TEMPLATE/bug.yml | 1 + .github/ISSUE_TEMPLATE/failing_test.yml| 1 + .github/ISSUE_TEMPLATE/feature.yml | 1 + .github/ISSUE_TEMPLATE/task.yml| 1 + .github/issue-rules.yml| 2 + runners/core-construction-java/build.gradle| 1 + .../core/construction/CombineTranslation.java | 6 +- .../CreatePCollectionViewTranslation.java | 2 +- .../beam/runners/core/construction/External.java | 10 +- .../construction/ExternalTranslationOptions.java | 32 +- .../ExternalTranslationOptionsRegistrar.java | 9 +- .../core/construction/FlattenTranslator.java | 2 +- .../core/construction/GroupByKeyTranslation.java | 2 +- .../construction/GroupIntoBatchesTranslation.java | 4 +- .../core/construction/ImpulseTranslation.java | 2 +- .../core/construction/PTransformTranslation.java | 102 +- .../core/construction/PipelineTranslation.java | 15 + .../runners/core/construction/ReadTranslation.java | 4 +- .../core/construction/ReshuffleTranslation.java| 2 +- .../runners/core/construction/SplittableParDo.java | 2 +- .../core/construction/TestStreamTranslation.java | 2 +- .../core/construction/TransformUpgrader.java | 330 ++ .../core/construction/WindowIntoTranslation.java | 5 +- .../core/construction/WriteFilesTranslation.java | 2 +- .../core/construction/TransformUpgraderTest.java | 369 + .../flink/FlinkStreamingTransformTranslators.java | 2 +- .../beam/runners/dataflow/DataflowRunner.java | 6 + .../dataflow/PrimitiveParDoSingleFactory.java | 2 +- .../beam/runners/dataflow/DataflowRunnerTest.java | 2 +- .../samza/translation/SamzaPublishView.java| 2 +- .../streaming/StreamingTransformTranslator.java| 4 +- .../sdk/expansion/service/ExpansionService.java| 65 sdks/java/harness/build.gradle | 2 + .../io/gcp/pubsub/PubSubPayloadTranslation.java| 9 +- .../sdk/io/hadoop/SerializableConfiguration.java | 19 +- sdks/python/apache_beam/internal/gcp/auth.py | 14 +- sdks/python/apache_beam/internal/gcp/auth_test.py | 135 .../runners/dataflow/dataflow_runner.py| 10 + .../runners/dataflow/dataflow_runner_test.py | 17 +- .../apache_beam/runners/dataflow/internal/names.py | 2 +- .../apache_beam/runners/portability/stager.py | 8 +- 41 files changed, 1131 insertions(+), 77 deletions(-) copy sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java => runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslationOptions.java (52%) copy sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransformOptionsRegistrar.java => runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslationOptionsRegistrar.java (84%) create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformUpgrader.java create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TransformUpgraderTest.java create mode 100644 sdks/python/apache_beam/internal/gcp/auth_test.py
[beam] branch dependabot/github_actions/actions/checkout-4 updated (72fec284295 -> 22daca4eedb)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch
dependabot/github_actions/actions/checkout-4
in repository https://gitbox.apache.org/repos/asf/beam.git
omit 72fec284295 Bump actions/checkout from 3 to 4
add b74a0dc8e65 Add Load Tests Combine Dataflow Batch Java workflow
(#28676)
add 22daca4eedb Bump actions/checkout from 3 to 4
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (72fec284295)
\
N -- N -- N
refs/heads/dependabot/github_actions/actions/checkout-4 (22daca4eedb)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
...beam_LoadTests_Java_Combine_Dataflow_Batch.yml} | 47 +-
.../config_Combine_Java_Dataflow_Batch_10b.txt}| 19 ++---
...onfig_Combine_Java_Dataflow_Batch_Fanout_4.txt} | 21 ++
...onfig_Combine_Java_Dataflow_Batch_Fanout_8.txt} | 21 ++
4 files changed, 68 insertions(+), 40 deletions(-)
copy .github/workflows/{beam_LoadTests_Go_SideInput_Dataflow_Batch.yml =>
beam_LoadTests_Java_Combine_Dataflow_Batch.yml} (62%)
copy .github/workflows/{performance-tests-job-configs/TFRecordIOIT_HDFS.txt =>
load-tests-job-configs/config_Combine_Java_Dataflow_Batch_10b.txt} (66%)
copy .github/workflows/{performance-tests-job-configs/TFRecordIOIT_HDFS.txt =>
load-tests-job-configs/config_Combine_Java_Dataflow_Batch_Fanout_4.txt} (65%)
copy .github/workflows/{performance-tests-job-configs/TFRecordIOIT_HDFS.txt =>
load-tests-job-configs/config_Combine_Java_Dataflow_Batch_Fanout_8.txt} (65%)
[beam] branch dependabot/github_actions/actions/checkout-4 updated (231ffe3a00a -> 72fec284295)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/github_actions/actions/checkout-4 in repository https://gitbox.apache.org/repos/asf/beam.git omit 231ffe3a00a Bump actions/checkout from 3 to 4 add 0f2bb5aaef7 Bump google.golang.org/api from 0.142.0 to 0.143.0 in /sdks (#28659) add d52b077ad50 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#28641) add c86a9bb5dbf Move Python dep upgrade instructions into release guide (#28648) add a7e12db9b59 Filter out some warnings that aren't useful in pytest (#28622) add 70979bfa361 Move committer onboarding and committer guide to contributor-docs (#28643) add 57954bd9726 Run arm suite against all validates_container tests (#28627) add 7d78572baa2 Add GoogleAdsIO to the list of built-in connectors (#28667) add 5c4bea2fdbb add YAML SDK component to issue template (#28678) add 725a2d62b27 Feature/add retry to gcp auth (#28475) add f35a4130ef3 Make SerializableConfiguration cacheable (#28590) add ced3de3feae Require --sdk_location for Dataflow pipelines running with dev SDKs. (#28670) add 1c50fd2e6a8 Upgrade transforms without upgrading the pipelines (#28210) add 3cdb9fe2b2d Bump Python FnAPI beam-master container. (#28704) add 218bda98624 added jobs to GitHub Actions (#28679) add 72fec284295 Bump actions/checkout from 3 to 4 This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (231ffe3a00a) \ N -- N -- N refs/heads/dependabot/github_actions/actions/checkout-4 (72fec284295) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/ISSUE_TEMPLATE/bug.yml | 1 + .github/ISSUE_TEMPLATE/failing_test.yml| 1 + .github/ISSUE_TEMPLATE/feature.yml | 1 + .github/ISSUE_TEMPLATE/task.yml| 1 + .github/issue-rules.yml| 2 + .github/workflows/beam_PerformanceTests_Cdap.yml | 108 ++ .../beam_PerformanceTests_HadoopFormat.yml | 108 ++ .../beam_PerformanceTests_MongoDBIO_IT.yml | 108 ++ ...erformanceTests_PubsubIOIT_Python_Streaming.yml | 94 ++ .../beam_PerformanceTests_SparkReceiver_IO.yml | 108 ++ .../beam_PerformanceTests_TFRecordIOIT_HDFS.yml| 112 +++ ...PerformanceTests_WordCountIT_PythonVersions.yml | 105 ++ .../TFRecordIOIT_HDFS.txt | 23 ++ .../performance-tests-job-configs/cdap.txt | 29 ++ .../performance-tests-job-configs/hadoopFormat.txt | 29 ++ .../performance-tests-job-configs/mongoDBIO_IT.txt | 28 ++ .../pubsubIOIT_Python_Streaming.txt| 28 ++ .../sparkReceiver_IO.txt | 26 ++ .../wordCountIT_Python.txt | 27 ++ .../committer-guide.md | 90 ++--- contributor-docs/committer-onboarding.md | 37 +++ contributor-docs/release-guide.md | 11 +- runners/core-construction-java/build.gradle| 1 + .../core/construction/CombineTranslation.java | 6 +- .../CreatePCollectionViewTranslation.java | 2 +- .../beam/runners/core/construction/External.java | 10 +- .../construction/ExternalTranslationOptions.java | 43 +++ .../ExternalTranslationOptionsRegistrar.java | 36 ++ .../core/construction/FlattenTranslator.java | 2 +- .../core/construction/GroupByKeyTranslation.java | 2 +- .../construction/GroupIntoBatchesTranslation.java | 4 +- .../core/construction/ImpulseTranslation.java | 2 +- .../core/construction/PTransformTranslation.java | 102 +- .../core/construction/PipelineTranslation.java | 15 + .../runners/core/construction/ReadTranslation.java | 4 +- .../core/construction/ReshuffleTranslation.java| 2 +- .../runners/core/construction/SplittableParDo.java | 2 +- .../core/construction/TestStreamTranslation.java | 2 +- .../core/construction/TransformUpgrader.java | 330 ++ .../core/construction/WindowIntoTranslation.java | 5 +- .../core/construction/WriteFilesTranslation.java | 2 +- .../core/construction/TransformUpgraderTest.java | 369 + .../flink/FlinkStreamingTransformTranslators.java | 2 +- .../beam/runners/dataflow/DataflowRunner.java
[beam] branch master updated: Add Load Tests Combine Dataflow Batch Java workflow (#28676)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new b74a0dc8e65 Add Load Tests Combine Dataflow Batch Java workflow
(#28676)
b74a0dc8e65 is described below
commit b74a0dc8e6592503909530850f1fb6ec826f4a65
Author: Vitaly Terentyev
AuthorDate: Thu Sep 28 07:50:22 2023 +0400
Add Load Tests Combine Dataflow Batch Java workflow (#28676)
---
.../beam_LoadTests_Java_Combine_Dataflow_Batch.yml | 104 +
.../config_Combine_Java_Dataflow_Batch_10b.txt | 30 ++
...config_Combine_Java_Dataflow_Batch_Fanout_4.txt | 30 ++
...config_Combine_Java_Dataflow_Batch_Fanout_8.txt | 30 ++
4 files changed, 194 insertions(+)
diff --git a/.github/workflows/beam_LoadTests_Java_Combine_Dataflow_Batch.yml
b/.github/workflows/beam_LoadTests_Java_Combine_Dataflow_Batch.yml
new file mode 100644
index 000..758d196f273
--- /dev/null
+++ b/.github/workflows/beam_LoadTests_Java_Combine_Dataflow_Batch.yml
@@ -0,0 +1,104 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Load Tests Combine Dataflow Batch Java
+
+on:
+ issue_comment:
+types: [created]
+ schedule:
+- cron: '35 7 * * *'
+ workflow_dispatch:
+
+#Setting explicit permissions for the action to avoid the default permissions
which are `write-all` in case of pull_request_target event
+permissions:
+ actions: write
+ pull-requests: read
+ checks: read
+ contents: read
+ deployments: read
+ id-token: none
+ issues: read
+ discussions: read
+ packages: read
+ pages: read
+ repository-projects: read
+ security-events: read
+ statuses: read
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+ group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.sha
|| github.head_ref || github.ref }}-${{ github.event.schedule ||
github.event.comment.body || github.event.sender.login }}'
+ cancel-in-progress: true
+
+env:
+ GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
+ GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
+ GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
+
+jobs:
+ beam_LoadTests_Java_Combine_Dataflow_Batch:
+if: |
+ github.event_name == 'workflow_dispatch' ||
+ github.event_name == 'schedule' ||
+ github.event.comment.body == 'Run Load Tests Java Combine Dataflow Batch'
+runs-on: [self-hosted, ubuntu-20.04, main]
+timeout-minutes: 720
+name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+strategy:
+ matrix:
+job_name: ["beam_LoadTests_Java_Combine_Dataflow_Batch"]
+job_phrase: ["Run Load Tests Java Combine Dataflow Batch"]
+steps:
+ - uses: actions/checkout@v3
+ - name: Setup repository
+uses: ./.github/actions/setup-action
+with:
+ comment_phrase: ${{ matrix.job_phrase }}
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+ - name: Prepare test arguments
+uses: ./.github/actions/test-arguments-action
+with:
+ test-type: load
+ test-language: java
+ argument-file-paths: |
+${{ github.workspace
}}/.github/workflows/load-tests-job-configs/config_Combine_Java_Dataflow_Batch_10b.txt
+${{ github.workspace
}}/.github/workflows/load-tests-job-configs/config_Combine_Java_Dataflow_Batch_Fanout_4.txt
+${{ github.workspace
}}/.github/workflows/load-tests-job-configs/config_Combine_Java_Dataflow_Batch_Fanout_8.txt
+ - name: run Combine Dataflow Batch Java Load Test 1 (10 b records)
+uses: ./.github/actions/gradle-command-self-hosted-action
+with:
+ gradle-command: :sdks:java:testing:load-tests:run
+ arguments: |
+-PloadTest.mainClass=org.apache.beam.sdk.loadtests.CombineLoadTest
\
+-Prunner=:runners:google-cloud-dataflow-java \
+'-PloadTest.args=${{
env.beam_LoadTests_Java_Combine_Dataflow_Batch_test_arguments_1 }}' \
+ - name: run
[beam] branch master updated: added jobs to GitHub Actions (#28679)
This is an automated email from the ASF dual-hosted git repository.
yhu pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 218bda98624 added jobs to GitHub Actions (#28679)
218bda98624 is described below
commit 218bda9862407c20adfeffb6723d0de07f2e4db3
Author: Andrey Devyatkin
AuthorDate: Thu Sep 28 05:49:24 2023 +0200
added jobs to GitHub Actions (#28679)
---
.github/workflows/beam_PerformanceTests_Cdap.yml | 108
.../beam_PerformanceTests_HadoopFormat.yml | 108
.../beam_PerformanceTests_MongoDBIO_IT.yml | 108
...erformanceTests_PubsubIOIT_Python_Streaming.yml | 94 +
.../beam_PerformanceTests_SparkReceiver_IO.yml | 108
.../beam_PerformanceTests_TFRecordIOIT_HDFS.yml| 112 +
...PerformanceTests_WordCountIT_PythonVersions.yml | 105 +++
.../TFRecordIOIT_HDFS.txt | 23 +
.../performance-tests-job-configs/cdap.txt | 29 ++
.../performance-tests-job-configs/hadoopFormat.txt | 29 ++
.../performance-tests-job-configs/mongoDBIO_IT.txt | 28 ++
.../pubsubIOIT_Python_Streaming.txt| 28 ++
.../sparkReceiver_IO.txt | 26 +
.../wordCountIT_Python.txt | 27 +
14 files changed, 933 insertions(+)
diff --git a/.github/workflows/beam_PerformanceTests_Cdap.yml
b/.github/workflows/beam_PerformanceTests_Cdap.yml
new file mode 100644
index 000..0dfd095ecf5
--- /dev/null
+++ b/.github/workflows/beam_PerformanceTests_Cdap.yml
@@ -0,0 +1,108 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: PerformanceTests Cdap
+
+on:
+ issue_comment:
+types: [created]
+ schedule:
+- cron: '13 4/16 * * *'
+ workflow_dispatch:
+
+#Setting explicit permissions for the action to avoid the default permissions
which are `write-all` in case of pull_request_target event
+permissions:
+ actions: write
+ pull-requests: write
+ checks: write
+ contents: read
+ deployments: read
+ id-token: none
+ issues: write
+ discussions: read
+ packages: read
+ pages: read
+ repository-projects: read
+ security-events: read
+ statuses: read
+
+# This allows a subsequently queued workflow run to interrupt previous runs
+concurrency:
+ group: '${{ github.workflow }} @ ${{ github.event.issue.number || github.sha
|| github.head_ref || github.ref }}-${{ github.event.schedule ||
github.event.comment.body || github.event.sender.login }}'
+ cancel-in-progress: true
+
+env:
+ GRADLE_ENTERPRISE_ACCESS_KEY: ${{ secrets.GE_ACCESS_TOKEN }}
+ GRADLE_ENTERPRISE_CACHE_USERNAME: ${{ secrets.GE_CACHE_USERNAME }}
+ GRADLE_ENTERPRISE_CACHE_PASSWORD: ${{ secrets.GE_CACHE_PASSWORD }}
+ beam_PerformanceTests_Cdap_test_arguments_1: ''
+
+jobs:
+ beam_PerformanceTests_Cdap:
+if: |
+ github.event_name == 'workflow_dispatch' ||
+ github.event_name == 'schedule' ||
+ github.event.comment.body == 'Run Java CdapIO Performance Test'
+runs-on: [self-hosted, ubuntu-20.04, main]
+timeout-minutes: 100
+name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+strategy:
+ matrix:
+job_name: ["beam_PerformanceTests_Cdap"]
+job_phrase: ["Run Java CdapIO Performance Test"]
+steps:
+ - uses: actions/checkout@v4
+ - name: Setup repository
+uses: ./.github/actions/setup-action
+with:
+ comment_phrase: ${{ matrix.job_phrase }}
+ github_token: ${{ secrets.GITHUB_TOKEN }}
+ github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+ - name: Authenticate on GCP
+id: auth
+uses: google-github-actions/auth@v1
+with:
+ credentials_json: ${{ secrets.GCP_SA_KEY }}
+ project_id: ${{ secrets.GCP_PROJECT_ID }}
+ - name: Set k8s access
+uses: ./.github/actions/setup-k8s-access
+with:
+ cluster_name: io-datastores
+ k8s_namespace: ${{ matrix.job_name }}-${{ github.run_id }}
+ - name: Install postgres
+
[beam] branch asf-site updated: Publishing website 2023/09/28 00:47:49 at commit 3cdb9fe
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 14ca69ea694 Publishing website 2023/09/28 00:47:49 at commit 3cdb9fe 14ca69ea694 is described below commit 14ca69ea694c0b78edaa81cb126a7bae392eb044 Author: runner AuthorDate: Thu Sep 28 00:47:49 2023 + Publishing website 2023/09/28 00:47:49 at commit 3cdb9fe --- website/generated-content/sitemap.xml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/generated-content/sitemap.xml b/website/generated-content/sitemap.xml index cbb9ed431ec..0991f099d3f 100644 --- a/website/generated-content/sitemap.xml +++ b/website/generated-content/sitemap.xml @@ -1 +1 @@ -http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.50.0/2023-09-27T17:14:59+02:00/categories/blog/2023-09-27T17:14:59+02:00/blog/2023-09-27T17:14:59+02:00/categories/2023-09-27T17:14:59+02:00/catego [...] \ No newline at end of file +http://www.sitemaps.org/schemas/sitemap/0.9; xmlns:xhtml="http://www.w3.org/1999/xhtml;>/blog/beam-2.50.0/2023-09-27T19:58:09-04:00/categories/blog/2023-09-27T19:58:09-04:00/blog/2023-09-27T19:58:09-04:00/categories/2023-09-27T19:58:09-04:00/catego [...] \ No newline at end of file
[beam] branch master updated: Bump Python FnAPI beam-master container. (#28704)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 3cdb9fe2b2d Bump Python FnAPI beam-master container. (#28704) 3cdb9fe2b2d is described below commit 3cdb9fe2b2d8c17d02920ccb3bf81b56695179cf Author: Pranav Bhandari AuthorDate: Wed Sep 27 19:58:09 2023 -0400 Bump Python FnAPI beam-master container. (#28704) --- sdks/python/apache_beam/runners/dataflow/internal/names.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/runners/dataflow/internal/names.py b/sdks/python/apache_beam/runners/dataflow/internal/names.py index 57c0bcdff20..8347a597e96 100644 --- a/sdks/python/apache_beam/runners/dataflow/internal/names.py +++ b/sdks/python/apache_beam/runners/dataflow/internal/names.py @@ -34,6 +34,6 @@ SERIALIZED_SOURCE_KEY = 'serialized_source' # Unreleased sdks use container image tag specified below. # Update this tag whenever there is a change that # requires changes to SDK harness container or SDK harness launcher. -BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20230915' +BEAM_DEV_SDK_CONTAINER_TAG = 'beam-master-20230927' DATAFLOW_CONTAINER_IMAGE_REPOSITORY = 'gcr.io/cloud-dataflow/v1beta3'
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new c1815d2b05d Updating config from bot c1815d2b05d is described below commit c1815d2b05d4a9f1de4f00f1a9490f81805c96cb Author: github-actions AuthorDate: Wed Sep 27 23:34:38 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index 21b84f237bd..ce74c3aec5b 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -6,7 +6,7 @@ "chamikaramj": 1695735303862, "damccorm": 1695754582189, "johnjcasey": 1695803889063, -"jrmccluskey": 1695321488837, +"jrmccluskey": 1695857675658, "kennknowles": 1695807301067, "lostluck": 1695654376375, "riteshghorse": 1695652495992,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 459bf5107ae Updating config from bot
459bf5107ae is described below
commit 459bf5107aee1d7c344961f0f15322c8b90312b3
Author: github-actions
AuthorDate: Wed Sep 27 23:34:44 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 7098514517b..e7c996f9707 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1695828897035,
+"damccorm": 1695857680041,
"Abacn": 1695841218903
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 1283bf5f492 Updating config from bot
1283bf5f492 is described below
commit 1283bf5f492d010952bf6de728a6e5204528b6d7
Author: github-actions
AuthorDate: Wed Sep 27 23:34:41 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28706.json | 12
1 file changed, 12 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28706.json
b/scripts/ci/pr-bot/state/pr-state/pr-28706.json
new file mode 100644
index 000..e91f611ad5f
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28706.json
@@ -0,0 +1,12 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"java": "robertwb",
+"build": "damccorm",
+"io": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new e236d5d8512 Updating config from bot
e236d5d8512 is described below
commit e236d5d851238e2eb9ea9342ce10a73a9cf3c870
Author: github-actions
AuthorDate: Wed Sep 27 23:34:45 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-io.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
index ab61ae23798..be06587b038 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-io.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-io.json
@@ -2,7 +2,7 @@
"label": "io",
"dateOfLastReviewAssignment": {
"chamikaramj": 1695391548648,
-"johnjcasey": 1695212037351,
+"johnjcasey": 1695857680042,
"pabloem": 1691787951165,
"Abacn": 1695375280928,
"ahmedabu98": 1695669912043,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c944f36e1a5 Updating config from bot
c944f36e1a5 is described below
commit c944f36e1a5e625de62166fe83f548784679
Author: github-actions
AuthorDate: Wed Sep 27 23:34:42 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-java.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
index 4d689b70e82..a23002abba2 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-java.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-java.json
@@ -3,7 +3,7 @@
"dateOfLastReviewAssignment": {
"lukecwik": 1680701852535,
"kennknowles": 1695212036819,
-"robertwb": 1695137718373,
+"robertwb": 1695857680041,
"kileys": 1674428678843,
"apilloud": 1678822446183,
"Abacn": 1695375280928,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2bebab66040 Updating config from bot
2bebab66040 is described below
commit 2bebab66040ad60095280533125dbb23f3e99a25
Author: github-actions
AuthorDate: Wed Sep 27 23:34:37 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28707.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28707.json
b/scripts/ci/pr-bot/state/pr-state/pr-28707.json
new file mode 100644
index 000..cd352f44205
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28707.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"no-matching-label": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch dependabot/npm_and_yarn/sdks/typescript/get-func-name-2.0.2 created (now d7793355f19)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/sdks/typescript/get-func-name-2.0.2 in repository https://gitbox.apache.org/repos/asf/beam.git at d7793355f19 Bump get-func-name from 2.0.0 to 2.0.2 in /sdks/typescript No new revisions were added by this update.
[beam-starter-typescript] branch dependabot/npm_and_yarn/get-func-name-2.0.2 created (now fba57cd)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/get-func-name-2.0.2 in repository https://gitbox.apache.org/repos/asf/beam-starter-typescript.git at fba57cd Bump get-func-name from 2.0.0 to 2.0.2 No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ecb5c24b5a9 Updating config from bot
ecb5c24b5a9 is described below
commit ecb5c24b5a9582276c3b96d14e5fbc5996f91f46
Author: github-actions
AuthorDate: Wed Sep 27 22:05:49 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28704.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28704.json
b/scripts/ci/pr-bot/state/pr-state/pr-28704.json
new file mode 100644
index 000..321b31c0f05
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28704.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "tvalentyn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 855d3ed265a Updating config from bot 855d3ed265a is described below commit 855d3ed265a5d3097bb61b16ad0112f4cd10db45 Author: github-actions AuthorDate: Wed Sep 27 22:05:50 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 2cb0d2a05ff..e4aa55bc99c 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -5,7 +5,7 @@ "yeandy": 1665802753763, "TheNeuralBit": 1667896849319, "ryanthompson591": 1670002443548, -"tvalentyn": 1695418481493, +"tvalentyn": 1695852347296, "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1695841213234,
[beam] branch master updated (ced3de3feae -> 1c50fd2e6a8)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ced3de3feae Require --sdk_location for Dataflow pipelines running with dev SDKs. (#28670) add 1c50fd2e6a8 Upgrade transforms without upgrading the pipelines (#28210) No new revisions were added by this update. Summary of changes: runners/core-construction-java/build.gradle| 1 + .../core/construction/CombineTranslation.java | 6 +- .../CreatePCollectionViewTranslation.java | 2 +- .../beam/runners/core/construction/External.java | 10 +- .../construction/ExternalTranslationOptions.java | 32 +- .../ExternalTranslationOptionsRegistrar.java | 9 +- .../core/construction/FlattenTranslator.java | 2 +- .../core/construction/GroupByKeyTranslation.java | 2 +- .../construction/GroupIntoBatchesTranslation.java | 4 +- .../core/construction/ImpulseTranslation.java | 2 +- .../core/construction/PTransformTranslation.java | 102 +- .../core/construction/PipelineTranslation.java | 15 + .../runners/core/construction/ReadTranslation.java | 4 +- .../core/construction/ReshuffleTranslation.java| 2 +- .../runners/core/construction/SplittableParDo.java | 2 +- .../core/construction/TestStreamTranslation.java | 2 +- .../core/construction/TransformUpgrader.java | 330 ++ .../core/construction/WindowIntoTranslation.java | 5 +- .../core/construction/WriteFilesTranslation.java | 2 +- .../core/construction/TransformUpgraderTest.java | 369 + .../flink/FlinkStreamingTransformTranslators.java | 2 +- .../beam/runners/dataflow/DataflowRunner.java | 6 + .../dataflow/PrimitiveParDoSingleFactory.java | 2 +- .../beam/runners/dataflow/DataflowRunnerTest.java | 2 +- .../samza/translation/SamzaPublishView.java| 2 +- .../streaming/StreamingTransformTranslator.java| 4 +- .../sdk/expansion/service/ExpansionService.java| 65 sdks/java/harness/build.gradle | 2 + .../io/gcp/pubsub/PubSubPayloadTranslation.java| 9 +- 29 files changed, 935 insertions(+), 62 deletions(-) copy sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogPipelineOptions.java => runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslationOptions.java (52%) copy sdks/java/extensions/python/src/main/java/org/apache/beam/sdk/extensions/python/PythonExternalTransformOptionsRegistrar.java => runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExternalTranslationOptionsRegistrar.java (84%) create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/TransformUpgrader.java create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/TransformUpgraderTest.java
[beam] branch release-2.51.0 updated: Cherry picking PR #28618 into 2.51.0 (setting numShards for Python BigQuery xlang) (#28631)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch release-2.51.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.51.0 by this push:
new 2420c904f66 Cherry picking PR #28618 into 2.51.0 (setting numShards
for Python BigQuery xlang) (#28631)
2420c904f66 is described below
commit 2420c904f66a8882eb454416f4029604c9160502
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Wed Sep 27 17:43:54 2023 -0400
Cherry picking PR #28618 into 2.51.0 (setting numShards for Python BigQuery
xlang) (#28631)
---
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 37 ++
.../io/external/xlang_bigqueryio_it_test.py| 44 ++
sdks/python/apache_beam/io/gcp/bigquery.py | 9 +
.../documentation/io/built-in/google-bigquery.md | 2 +-
6 files changed, 69 insertions(+), 27 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
index d1ee27088c7..1280fcb4e23 100644
---
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Dataflow',
'Run Python_Xlang_Gcp_Dataflow PostCommit', 'Python_Xlang_Gcp_Dataflow
(\"Run Python_Xlang_Gcp_Dataflow PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner. \"Run Python_Xlang_Gcp_Dataflow PostCommit\"')
// Set common parameters.
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
index 438b735fba7..e4bf771be1a 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Direct',
'Run Python_Xlang_Gcp_Direct PostCommit', 'Python_Xlang_Gcp_Direct (\"Run
Python_Xlang_Gcp_Direct PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Direct
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Direct
runner. \"Run Python_Xlang_Gcp_Direct PostCommit\"')
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
index e4461793011..1b9eb309ec4 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
@@ -176,6 +176,13 @@ public class BigQueryStorageWriteApiSchemaTransformProvider
!Strings.isNullOrEmpty(this.getErrorHandling().getOutput()),
invalidConfigMessage + "Output must not be empty if error handling
specified.");
}
+
+ if (this.getAutoSharding() != null && this.getAutoSharding()) {
+checkArgument(
+this.getNumStreams() == 0,
+invalidConfigMessage
++ "Cannot set a fixed number of streams when auto-sharding is
enabled. Please pick only one of the two options.");
+ }
}
/**
@@ -218,11 +225,17 @@ public class
BigQueryStorageWriteApiSchemaTransformProvider
public abstract Boolean getUseAtLeastOnceSemantics();
@SchemaFieldDescription(
-"This option enables using a dynamically determined number of shards
to write to "
+"This option enables using a dynamically determined number of Storage
Write API streams to write to "
+ "BigQuery. Only applicable to unbounded data.")
@Nullable
public abstract Boolean getAutoSharding();
+
[beam] branch release-2.51.0 updated: CP for #28624 into release 2.51.0 (Bigtable Python timestamp bug fix) (#28634)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch release-2.51.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.51.0 by this push:
new 70f4a1ae26d CP for #28624 into release 2.51.0 (Bigtable Python
timestamp bug fix) (#28634)
70f4a1ae26d is described below
commit 70f4a1ae26d71fc89c3f5a99cae4e417cd1ee446
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Wed Sep 27 17:42:18 2023 -0400
CP for #28624 into release 2.51.0 (Bigtable Python timestamp bug fix)
(#28634)
---
.../BigtableWriteSchemaTransformProvider.java | 13 +++--
.../BigtableWriteSchemaTransformProviderIT.java | 19 +++
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +++
sdks/python/apache_beam/io/gcp/bigtableio_it_test.py | 18 ++
4 files changed, 39 insertions(+), 18 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
index d38bdae2f09..b99b69621a8 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
@@ -179,12 +179,13 @@ public class BigtableWriteSchemaTransformProvider
.setColumnQualifier(
ByteString.copyFrom(ofNullable(mutation.get("column_qualifier")).get()))
.setFamilyNameBytes(
-
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()));
-if (mutation.containsKey("timestamp_micros")) {
- setMutation =
- setMutation.setTimestampMicros(
-
Longs.fromByteArray(ofNullable(mutation.get("timestamp_micros")).get()));
-}
+
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()))
+// Use timestamp if provided, else default to -1 (current
Bigtable server time)
+.setTimestampMicros(
+mutation.containsKey("timestamp_micros")
+? Longs.fromByteArray(
+
ofNullable(mutation.get("timestamp_micros")).get())
+: -1);
bigtableMutation =
Mutation.newBuilder().setSetCell(setMutation.build()).build();
break;
case "DeleteFromColumn":
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
index 14bb04b0315..1a60fe661b5 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
@@ -154,8 +154,8 @@ public class BigtableWriteSchemaTransformProviderIT {
public void testSetMutationsExistingColumn() {
RowMutation rowMutation =
RowMutation.create(tableId, "key-1")
-.setCell(COLUMN_FAMILY_NAME_1, "col_a", "val-1-a")
-.setCell(COLUMN_FAMILY_NAME_2, "col_c", "val-1-c");
+.setCell(COLUMN_FAMILY_NAME_1, "col_a", 1000, "val-1-a")
+.setCell(COLUMN_FAMILY_NAME_2, "col_c", 1000, "val-1-c");
dataClient.mutateRow(rowMutation);
List> mutations = new ArrayList<>();
@@ -165,13 +165,15 @@ public class BigtableWriteSchemaTransformProviderIT {
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-a".getBytes(StandardCharsets.UTF_8),
"column_qualifier", "col_a".getBytes(StandardCharsets.UTF_8),
-"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8)));
+"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8),
+"timestamp_micros", Longs.toByteArray(2000)));
mutations.add(
ImmutableMap.of(
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-c".getBytes(StandardCharsets.UTF_8),
"column_qualifier", "col_c".getBytes(StandardCharsets.UTF_8),
-"family_name",
COLUMN_FAMILY_NAME_2.getBytes(StandardCharsets.UTF_8)));
+"family_name",
COLUMN_FAMILY_NAME_2.getBytes(StandardCharsets.UTF_8),
+"timestamp_micros", Longs.toByteArray(2000)));
Row mutationRow =
[beam] branch master updated (f35a4130ef3 -> ced3de3feae)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from f35a4130ef3 Make SerializableConfiguration cacheable (#28590) add ced3de3feae Require --sdk_location for Dataflow pipelines running with dev SDKs. (#28670) No new revisions were added by this update. Summary of changes: .../apache_beam/runners/dataflow/dataflow_runner.py | 10 ++ .../runners/dataflow/dataflow_runner_test.py| 17 - sdks/python/apache_beam/runners/portability/stager.py | 8 +++- 3 files changed, 25 insertions(+), 10 deletions(-)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new f98e560d44c Updating config from bot f98e560d44c is described below commit f98e560d44c1b20d9158f24cab319120f8b54ef7 Author: github-actions AuthorDate: Wed Sep 27 19:12:27 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28666.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28666.json b/scripts/ci/pr-bot/state/pr-state/pr-28666.json index 9c51466b9ee..b232230e5a0 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28666.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28666.json @@ -4,7 +4,7 @@ "build": "Abacn" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": false } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 34b3180ee60 Updating config from bot 34b3180ee60 is described below commit 34b3180ee606414eb4bf5c7ad4e663bbd781c8c8 Author: github-actions AuthorDate: Wed Sep 27 19:04:24 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28670.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28670.json b/scripts/ci/pr-bot/state/pr-state/pr-28670.json index 88d1920a8bb..24d0e10a64c 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28670.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28670.json @@ -6,5 +6,5 @@ "nextAction": "Reviewers", "stopReviewerNotifications": false, "remindAfterTestsPass": [], - "committerAssigned": false + "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 060f5a28838 Updating config from bot 060f5a28838 is described below commit 060f5a28838c3d36e7d4b5ca5c9482b2ad694940 Author: github-actions AuthorDate: Wed Sep 27 19:01:06 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index dd393f42f95..2cb0d2a05ff 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -9,7 +9,7 @@ "pabloem": 1681281324703, "y1chi": 1667002607045, "damccorm": 1695841213234, -"jrmccluskey": 1695396898795, +"jrmccluskey": 1695841262547, "riteshghorse": 1695758756941, "liferoad": 1695422080469 }
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ed6b3e7f13d Updating config from bot
ed6b3e7f13d is described below
commit ed6b3e7f13d7caeabe76bf15c362a9d4b5a07aae
Author: github-actions
AuthorDate: Wed Sep 27 19:01:04 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28680.json | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28680.json
b/scripts/ci/pr-bot/state/pr-state/pr-28680.json
index 242a48d7d3b..8411c9d3126 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28680.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28680.json
@@ -1,7 +1,9 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"python": "jrmccluskey"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new b6aef9ff4c6 Updating config from bot
b6aef9ff4c6 is described below
commit b6aef9ff4c6dc2ffd63731dc12f0207fdbe943bc
Author: github-actions
AuthorDate: Wed Sep 27 19:00:22 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index a6d9e9834ff..7098514517b 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1695828897035,
-"Abacn": 1695823562628
+"Abacn": 1695841218903
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 09b21d326a5 Updating config from bot 09b21d326a5 is described below commit 09b21d326a5f6d41de1dac2d89f02dea04e581db Author: github-actions AuthorDate: Wed Sep 27 19:00:17 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-python.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-python.json b/scripts/ci/pr-bot/state/reviewers-for-label-python.json index 2a37a20d80f..dd393f42f95 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-python.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-python.json @@ -8,7 +8,7 @@ "tvalentyn": 1695418481493, "pabloem": 1681281324703, "y1chi": 1667002607045, -"damccorm": 1695335680296, +"damccorm": 1695841213234, "jrmccluskey": 1695396898795, "riteshghorse": 1695758756941, "liferoad": 1695422080469
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 0f3f7d120ef Updating config from bot
0f3f7d120ef is described below
commit 0f3f7d120ef789627ac7b69f2dd930fb9d407b6e
Author: github-actions
AuthorDate: Wed Sep 27 19:00:20 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28666.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28666.json
b/scripts/ci/pr-bot/state/pr-state/pr-28666.json
new file mode 100644
index 000..9c51466b9ee
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28666.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new ae55446bfe8 Updating config from bot
ae55446bfe8 is described below
commit ae55446bfe86480db0fe735c3a540fa6fab2c2c6
Author: github-actions
AuthorDate: Wed Sep 27 19:00:15 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28670.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28670.json
b/scripts/ci/pr-bot/state/pr-state/pr-28670.json
new file mode 100644
index 000..88d1920a8bb
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28670.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"python": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 3c2b8d3fe64 Updating config from bot 3c2b8d3fe64 is described below commit 3c2b8d3fe6420ff214a5847080c86840e746e0a5 Author: github-actions AuthorDate: Wed Sep 27 17:22:32 2023 + Updating config from bot --- scripts/ci/pr-bot/state/pr-state/pr-28564.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28564.json b/scripts/ci/pr-bot/state/pr-state/pr-28564.json index 8a078ee492b..08c2e443178 100644 --- a/scripts/ci/pr-bot/state/pr-state/pr-28564.json +++ b/scripts/ci/pr-bot/state/pr-state/pr-28564.json @@ -4,7 +4,7 @@ "python": "jrmccluskey" }, "nextAction": "Reviewers", - "stopReviewerNotifications": false, + "stopReviewerNotifications": true, "remindAfterTestsPass": [], "committerAssigned": true } \ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 81a40ca8123 Updating config from bot
81a40ca8123 is described below
commit 81a40ca8123b296b2b826c4a2cabf7ffad46d964
Author: github-actions
AuthorDate: Wed Sep 27 16:36:29 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28680.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28680.json
b/scripts/ci/pr-bot/state/pr-state/pr-28680.json
new file mode 100644
index 000..242a48d7d3b
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28680.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": true,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new c18523d1065 Updating config from bot
c18523d1065 is described below
commit c18523d1065b18b2910dfb4b831efdde6344e81b
Author: github-actions
AuthorDate: Wed Sep 27 15:35:00 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index 0d647c6cc90..a6d9e9834ff 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1695823556954,
+"damccorm": 1695828897035,
"Abacn": 1695823562628
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new a4b666b1ca0 Updating config from bot
a4b666b1ca0 is described below
commit a4b666b1ca0485ba44d9bcca96e679b2353b38cb
Author: github-actions
AuthorDate: Wed Sep 27 15:34:58 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28679.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28679.json
b/scripts/ci/pr-bot/state/pr-state/pr-28679.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28679.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch master updated (725a2d62b27 -> f35a4130ef3)
This is an automated email from the ASF dual-hosted git repository. yhu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 725a2d62b27 Feature/add retry to gcp auth (#28475) add f35a4130ef3 Make SerializableConfiguration cacheable (#28590) No new revisions were added by this update. Summary of changes: .../beam/sdk/io/hadoop/SerializableConfiguration.java | 19 ++- 1 file changed, 18 insertions(+), 1 deletion(-)
[beam] branch master updated: Feature/add retry to gcp auth (#28475)
This is an automated email from the ASF dual-hosted git repository. johncasey pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 725a2d62b27 Feature/add retry to gcp auth (#28475) 725a2d62b27 is described below commit 725a2d62b27f657a0f5ab8d279508753de0908d1 Author: johnjcasey <95318300+johnjca...@users.noreply.github.com> AuthorDate: Wed Sep 27 11:14:06 2023 -0400 Feature/add retry to gcp auth (#28475) * Update 2.50 release notes to include new Kafka topicPattern feature * Create groovy class for io performance tests Create gradle task and github actions config for GCS using this. * delete unnecessary class * fix env call * fix call to gradle * run on hosted runner for testing * add additional checkout * add destination for triggered tests * move env variables to correct location * try uploading against separate dataset * try without a user * update branch checkout, try to view the failure log * run on failure * update to use correct BigQuery instance * convert to matrix * add result reporting * add failure clause * remove failure clause, update to run on self-hosted * address comments, clean up build * clarify branching * Update auth to retry getting credentials from GCE * Re-order imports * Add test case * Update exception log * Add failure test * Update removal of retrying method * rework via mock * Clear credentials cache for idempotent tests * Remove handler after test Change retry timeout to facilitate shorter retrys for anonymous access cases * Change retry timeout to facilitate shorter retrys for anonymous access cases * reset credentials before and after test --- sdks/python/apache_beam/internal/gcp/auth.py | 14 ++- sdks/python/apache_beam/internal/gcp/auth_test.py | 135 ++ 2 files changed, 146 insertions(+), 3 deletions(-) diff --git a/sdks/python/apache_beam/internal/gcp/auth.py b/sdks/python/apache_beam/internal/gcp/auth.py index 7e54ba0a4ba..b2fda2c6e89 100644 --- a/sdks/python/apache_beam/internal/gcp/auth.py +++ b/sdks/python/apache_beam/internal/gcp/auth.py @@ -26,6 +26,7 @@ from typing import Optional from apache_beam.options.pipeline_options import GoogleCloudOptions from apache_beam.options.pipeline_options import PipelineOptions +from apache_beam.utils import retry # google.auth is only available when Beam is installed with the gcp extra. try: @@ -152,8 +153,7 @@ class _Credentials(object): try: # pylint: disable=c-extension-no-member - credentials, _ = google.auth.default( - scopes=pipeline_options.view_as(GoogleCloudOptions).gcp_oauth_scopes) + credentials = _Credentials._get_credentials_with_retrys(pipeline_options) credentials = _Credentials._add_impersonation_credentials( credentials, pipeline_options) credentials = _ApitoolsCredentialsAdapter(credentials) @@ -164,10 +164,18 @@ class _Credentials(object): except Exception as e: _LOGGER.warning( 'Unable to find default credentials to use: %s\n' - 'Connecting anonymously.', + 'Connecting anonymously. This is expected if no ' + 'credentials are needed to access GCP resources.', e) return None + @staticmethod + @retry.with_exponential_backoff(num_retries=4, initial_delay_secs=2) + def _get_credentials_with_retrys(pipeline_options): +credentials, _ = google.auth.default( + scopes=pipeline_options.view_as(GoogleCloudOptions).gcp_oauth_scopes) +return credentials + @staticmethod def _add_impersonation_credentials(credentials, pipeline_options): gcs_options = pipeline_options.view_as(GoogleCloudOptions) diff --git a/sdks/python/apache_beam/internal/gcp/auth_test.py b/sdks/python/apache_beam/internal/gcp/auth_test.py new file mode 100644 index 000..98fb828875b --- /dev/null +++ b/sdks/python/apache_beam/internal/gcp/auth_test.py @@ -0,0 +1,135 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +#
[beam] branch master updated: add YAML SDK component to issue template (#28678)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 5c4bea2fdbb add YAML SDK component to issue template (#28678) 5c4bea2fdbb is described below commit 5c4bea2fdbb46e2055b26c5affe8362c0599423d Author: Jeff Kinard <35542536+pol...@users.noreply.github.com> AuthorDate: Wed Sep 27 11:11:48 2023 -0400 add YAML SDK component to issue template (#28678) Signed-off-by: Jeffrey Kinard --- .github/ISSUE_TEMPLATE/bug.yml | 1 + .github/ISSUE_TEMPLATE/failing_test.yml | 1 + .github/ISSUE_TEMPLATE/feature.yml | 1 + .github/ISSUE_TEMPLATE/task.yml | 1 + .github/issue-rules.yml | 2 ++ 5 files changed, 6 insertions(+) diff --git a/.github/ISSUE_TEMPLATE/bug.yml b/.github/ISSUE_TEMPLATE/bug.yml index 1ff96082c03..67f8b21445d 100644 --- a/.github/ISSUE_TEMPLATE/bug.yml +++ b/.github/ISSUE_TEMPLATE/bug.yml @@ -63,6 +63,7 @@ body: - label: "Component: Go SDK" - label: "Component: Typescript SDK" - label: "Component: IO connector" + - label: "Component: Beam YAML" - label: "Component: Beam examples" - label: "Component: Beam playground" - label: "Component: Beam katas" diff --git a/.github/ISSUE_TEMPLATE/failing_test.yml b/.github/ISSUE_TEMPLATE/failing_test.yml index 0904719e6d2..44e1cd72074 100644 --- a/.github/ISSUE_TEMPLATE/failing_test.yml +++ b/.github/ISSUE_TEMPLATE/failing_test.yml @@ -69,6 +69,7 @@ body: - label: "Component: Go SDK" - label: "Component: Typescript SDK" - label: "Component: IO connector" + - label: "Component: Beam YAML" - label: "Component: Beam examples" - label: "Component: Beam playground" - label: "Component: Beam katas" diff --git a/.github/ISSUE_TEMPLATE/feature.yml b/.github/ISSUE_TEMPLATE/feature.yml index deaa14287b8..11234a5e150 100644 --- a/.github/ISSUE_TEMPLATE/feature.yml +++ b/.github/ISSUE_TEMPLATE/feature.yml @@ -57,6 +57,7 @@ body: - label: "Component: Go SDK" - label: "Component: Typescript SDK" - label: "Component: IO connector" + - label: "Component: Beam YAML" - label: "Component: Beam examples" - label: "Component: Beam playground" - label: "Component: Beam katas" diff --git a/.github/ISSUE_TEMPLATE/task.yml b/.github/ISSUE_TEMPLATE/task.yml index 4fa1b241207..477b91b181b 100644 --- a/.github/ISSUE_TEMPLATE/task.yml +++ b/.github/ISSUE_TEMPLATE/task.yml @@ -58,6 +58,7 @@ body: - label: "Component: Go SDK" - label: "Component: Typescript SDK" - label: "Component: IO connector" + - label: "Component: Beam YAML" - label: "Component: Beam examples" - label: "Component: Beam playground" - label: "Component: Beam katas" diff --git a/.github/issue-rules.yml b/.github/issue-rules.yml index c26cb84bf6b..b01a22dafd7 100644 --- a/.github/issue-rules.yml +++ b/.github/issue-rules.yml @@ -36,6 +36,8 @@ rules: addLabels: ['typescript'] - contains: '[x] Component: IO' addLabels: ['io'] +- contains: '[x] Component: Beam YAML' + addLabels: ['yaml'] - contains: '[x] Component: Beam examples' addLabels: ['examples'] - contains: '[x] Component: Beam playground'
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new feb4b7148ee Updating config from bot
feb4b7148ee is described below
commit feb4b7148ee3045eee3f93b934ddf45bac4dee5e
Author: github-actions
AuthorDate: Wed Sep 27 14:50:51 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28678.json | 8
1 file changed, 8 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28678.json
b/scripts/ci/pr-bot/state/pr-state/pr-28678.json
new file mode 100644
index 000..9c2aa5aa212
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28678.json
@@ -0,0 +1,8 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {},
+ "nextAction": "Author",
+ "stopReviewerNotifications": true,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch users/damccorm/armIT updated (3b113cd5eb0 -> d9eb38b24b6)
This is an automated email from the ASF dual-hosted git repository. damccorm pushed a change to branch users/damccorm/armIT in repository https://gitbox.apache.org/repos/asf/beam.git from 3b113cd5eb0 Run other arm tests on Dataflow Java add d9eb38b24b6 Set max workers No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 1 + 1 file changed, 1 insertion(+)
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new f0010b6a915 Updating config from bot
f0010b6a915 is described below
commit f0010b6a915559c96c549672106fc9c7ea90b76d
Author: github-actions
AuthorDate: Wed Sep 27 14:06:06 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index be64bf298ca..0d647c6cc90 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -2,6 +2,6 @@
"label": "build",
"dateOfLastReviewAssignment": {
"damccorm": 1695823556954,
-"Abacn": 1695792881158
+"Abacn": 1695823562628
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 17072266445 Updating config from bot
17072266445 is described below
commit 170722664458cc026d96f3c189987227837d684b
Author: github-actions
AuthorDate: Wed Sep 27 14:06:00 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/reviewers-for-label-build.json | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
index f2be2ff71f9..be64bf298ca 100644
--- a/scripts/ci/pr-bot/state/reviewers-for-label-build.json
+++ b/scripts/ci/pr-bot/state/reviewers-for-label-build.json
@@ -1,7 +1,7 @@
{
"label": "build",
"dateOfLastReviewAssignment": {
-"damccorm": 1695730444830,
+"damccorm": 1695823556954,
"Abacn": 1695792881158
}
}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 65f0b7e0125 Updating config from bot
65f0b7e0125 is described below
commit 65f0b7e01250456d32a4bb0e05aad30da2ae99c8
Author: github-actions
AuthorDate: Wed Sep 27 14:06:04 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28676.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28676.json
b/scripts/ci/pr-bot/state/pr-state/pr-28676.json
new file mode 100644
index 000..9c51466b9ee
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28676.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "Abacn"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 2251dc4fec8 Updating config from bot
2251dc4fec8 is described below
commit 2251dc4fec80d29e23b5718ffca19ebd214e616a
Author: github-actions
AuthorDate: Wed Sep 27 14:05:59 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28677.json | 10 ++
1 file changed, 10 insertions(+)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28677.json
b/scripts/ci/pr-bot/state/pr-state/pr-28677.json
new file mode 100644
index 000..8a92900947e
--- /dev/null
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28677.json
@@ -0,0 +1,10 @@
+{
+ "commentedAboutFailingChecks": false,
+ "reviewersAssignedForLabels": {
+"build": "damccorm"
+ },
+ "nextAction": "Reviewers",
+ "stopReviewerNotifications": false,
+ "remindAfterTestsPass": [],
+ "committerAssigned": false
+}
\ No newline at end of file
[beam-starter-typescript] branch dependabot/npm_and_yarn/types/mocha-10.0.2 created (now a577fc8)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to branch dependabot/npm_and_yarn/types/mocha-10.0.2 in repository https://gitbox.apache.org/repos/asf/beam-starter-typescript.git at a577fc8 Bump @types/mocha from 10.0.1 to 10.0.2 No new revisions were added by this update.
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 7024951e1d4 Updating config from bot 7024951e1d4 is described below commit 7024951e1d4002a0dec56ffb8916b011a3585bfc Author: github-actions AuthorDate: Wed Sep 27 09:35:04 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index d5629739ac9..21b84f237bd 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -7,7 +7,7 @@ "damccorm": 1695754582189, "johnjcasey": 1695803889063, "jrmccluskey": 1695321488837, -"kennknowles": 1695314212617, +"kennknowles": 1695807301067, "lostluck": 1695654376375, "riteshghorse": 1695652495992, "robertwb": 1695656095176,
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 452aa805218 Updating config from bot
452aa805218 is described below
commit 452aa805218f1c5cadded4ad969f3f6bd51d4750
Author: github-actions
AuthorDate: Wed Sep 27 09:35:03 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28614.json | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28614.json
b/scripts/ci/pr-bot/state/pr-state/pr-28614.json
index 242a48d7d3b..e28594629cb 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28614.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28614.json
@@ -1,7 +1,9 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"no-matching-label": "kennknowles"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a commit to branch pr-bot-state
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/pr-bot-state by this push:
new 7adc8a3ec2f Updating config from bot
7adc8a3ec2f is described below
commit 7adc8a3ec2f750b38afb4370518515e22643a9b2
Author: github-actions
AuthorDate: Wed Sep 27 08:38:10 2023 +
Updating config from bot
---
scripts/ci/pr-bot/state/pr-state/pr-28609.json | 6 --
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/scripts/ci/pr-bot/state/pr-state/pr-28609.json
b/scripts/ci/pr-bot/state/pr-state/pr-28609.json
index 242a48d7d3b..bc5c5902960 100644
--- a/scripts/ci/pr-bot/state/pr-state/pr-28609.json
+++ b/scripts/ci/pr-bot/state/pr-state/pr-28609.json
@@ -1,7 +1,9 @@
{
"commentedAboutFailingChecks": true,
- "reviewersAssignedForLabels": {},
- "nextAction": "Author",
+ "reviewersAssignedForLabels": {
+"no-matching-label": "johnjcasey"
+ },
+ "nextAction": "Reviewers",
"stopReviewerNotifications": false,
"remindAfterTestsPass": [],
"committerAssigned": false
[beam] branch pr-bot-state updated: Updating config from bot
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a commit to branch pr-bot-state in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/pr-bot-state by this push: new 2992bf20fab Updating config from bot 2992bf20fab is described below commit 2992bf20fabc6813115da1ba704a1611d13f2c25 Author: github-actions AuthorDate: Wed Sep 27 08:38:13 2023 + Updating config from bot --- scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json index 18d78566634..d5629739ac9 100644 --- a/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json +++ b/scripts/ci/pr-bot/state/reviewers-for-label-no-matching-label.json @@ -5,7 +5,7 @@ "AnandInguva": 1695657969899, "chamikaramj": 1695735303862, "damccorm": 1695754582189, -"johnjcasey": 1695259567443, +"johnjcasey": 1695803889063, "jrmccluskey": 1695321488837, "kennknowles": 1695314212617, "lostluck": 1695654376375,
