(beam) branch master updated (6de9a60dc9e -> 2196758c20b)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6de9a60dc9e Skip tests using TFT in Beam ML unit test suite on Python 3.11+ (#31288) add 041a8dd1173 Touch Samza ValidatesRunner trigger files add f58a9d6680e Reapply "Add Redistribute translation to Samza runner" new 2196758c20b Merge pull request #31270: Reapply "Add Redistribute translation to Samza runner" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam_PostCommit_Java_PVR_Samza.json| 3 +- ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +- .../translation/RedistributeByKeyTranslator.java | 64 ++ .../samza/translation/ReshuffleTranslator.java | 14 - .../samza/translation/SamzaPipelineTranslator.java | 1 + 5 files changed, 81 insertions(+), 4 deletions(-) create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
(beam) 01/01: Merge pull request #31270: Reapply "Add Redistribute translation to Samza runner"
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2196758c20bd176cc1e202644a96ae5da708cc75 Merge: 6de9a60dc9e f58a9d6680e Author: Kenneth Knowles AuthorDate: Wed May 15 13:51:09 2024 -0400 Merge pull request #31270: Reapply "Add Redistribute translation to Samza runner" .../beam_PostCommit_Java_PVR_Samza.json| 3 +- ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +- .../translation/RedistributeByKeyTranslator.java | 64 ++ .../samza/translation/ReshuffleTranslator.java | 14 - .../samza/translation/SamzaPipelineTranslator.java | 1 + 5 files changed, 81 insertions(+), 4 deletions(-)
(beam) branch master updated (99714991b00 -> 7f9264ad91d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform" add 8869175a887 Fix nullness errors in TestPipeline add 7f9264ad91d Merge pull request #31230: Fix nullness errors in TestPipeline No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/testing/TestPipeline.java | 85 -- 1 file changed, 46 insertions(+), 39 deletions(-)
(beam) 01/01: Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform"
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 99714991b004ae391568635bba67d45d6a73b65a Merge: c1d1bd642d1 238792d7c65 Author: Kenneth Knowles AuthorDate: Tue May 14 11:18:33 2024 -0400 Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform" ...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 +- ...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +- ...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +- ...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +- .../dataflow/DataflowPipelineTranslator.java | 37 ++ .../beam/runners/dataflow/util/PropertyNames.java | 1 + 6 files changed, 46 insertions(+), 4 deletions(-)
(beam) branch master updated (c1d1bd642d1 -> 99714991b00)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c1d1bd642d1 Fix a typo in skip condition. (#31267) add 4a15af019cc Touch trigger files for Dataflow ValidatesRunner tests add 238792d7c65 Reapply "Add direct Dataflow translation of Redistribute transform" new 99714991b00 Merge pull request #31268: Reapply "Add direct Dataflow translation of Redistribute transform" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: ...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 +- ...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +- ...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +- ...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +- .../dataflow/DataflowPipelineTranslator.java | 37 ++ .../beam/runners/dataflow/util/PropertyNames.java | 1 + 6 files changed, 46 insertions(+), 4 deletions(-)
(beam) branch master updated (66da760357c -> 737be230347)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 66da760357c Update yaml to non-beta link (#31210)
add a0c08272d42 Touch trigger files for all relevant ValidatesRunner and
PVR suites
add bf347fe898b Sickbay extremely flaky Flink ParDoLifecycleTest
add ef431ec90f2 Reapply "Add Redistribute transform to Java SDK"
add 737be230347 Merge pull request #31156: Reapply "Add Redistribute
transform to Java SDK"
No new revisions were added by this update.
Summary of changes:
.../beam_PostCommit_Java_PVR_Flink_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Docker.json | 4 +
.../beam_PostCommit_Java_PVR_Flink_Streaming.json | 4 +
.../beam_PostCommit_Java_PVR_Samza.json| 4 +
.../beam_PostCommit_Java_PVR_Spark3_Batch.json | 4 +
.../beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +-
...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 +-
...Java_ValidatesRunner_Dataflow_JavaVersions.json | 4 +
...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 +-
...ostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +-
...Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +-
...eam_PostCommit_Java_ValidatesRunner_Direct.json | 4 +
...t_Java_ValidatesRunner_Direct_JavaVersions.json | 4 +
...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 +-
...stCommit_Java_ValidatesRunner_Flink_Java11.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 +-
...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 +-
...a_ValidatesRunner_SparkStructuredStreaming.json | 3 +-
...stCommit_Java_ValidatesRunner_Spark_Java11.json | 4 +
...m_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +-
.../beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +-
runners/flink/flink_runner.gradle | 3 +
runners/flink/job-server/flink_job_server.gradle | 2 +
runners/portability/java/build.gradle | 2 +
runners/samza/job-server/build.gradle | 2 +
runners/spark/job-server/spark_job_server.gradle | 8 +
runners/spark/spark_runner.gradle | 7 +
.../apache/beam/sdk/transforms/Redistribute.java | 300 +
.../util/construction/PTransformTranslation.java | 8 +
.../{ReshuffleTest.java => RedistributeTest.java} | 159 +--
30 files changed, 453 insertions(+), 110 deletions(-)
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Batch.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Docker.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Flink_Streaming.json
create mode 100644 .github/trigger_files/beam_PostCommit_Java_PVR_Samza.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Batch.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Dataflow_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Direct_JavaVersions.json
create mode 100644
.github/trigger_files/beam_PostCommit_Java_ValidatesRunner_Spark_Java11.json
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java
copy
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/{ReshuffleTest.java
=> RedistributeTest.java} (73%)
(beam) branch master updated (057eb06514a -> 13708eaedeb)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 057eb06514a Add a container for Python 3.12. (#31216) add 736f2b963dd Turn off abandoned node enforcement in some Reshuffle unit tests where it is not needed add 13708eaedeb Merge pull request #31229: Turn off abandoned node enforcement in some Reshuffle unit tests where it is not needed No new revisions were added by this update. Summary of changes: .../src/test/java/org/apache/beam/sdk/transforms/ReshuffleTest.java | 4 1 file changed, 4 insertions(+)
(beam) branch master updated (2ad45739acd -> 8328d073ab8)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 2ad45739acd Targeted dependency update for Pandas 2.1 (#31209) add 0b36f8e23fc Use already-existing and more appropriately alarmist experiment in Python new 8328d073ab8 Merge pull request #31212: Use already-existing and more appropriately alarmist experiment in Python The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Merge pull request #31212: Use already-existing and more appropriately alarmist experiment in Python
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8328d073ab8492927927ad0a49d3d2f2169023bf Merge: 2ad45739acd 0b36f8e23fc Author: Kenneth Knowles AuthorDate: Tue May 7 21:05:11 2024 -0400 Merge pull request #31212: Use already-existing and more appropriately alarmist experiment in Python sdks/python/apache_beam/runners/dataflow/dataflow_runner.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Merge pull request #31143: Limit elasticsearch testing to basic tests when core SDK is modified
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 294aa65af93865e0b30060ebaf69b62a5b1c7bf4 Merge: a2987006ded 2f570145fd9 Author: Kenneth Knowles AuthorDate: Thu May 2 09:48:32 2024 -0400 Merge pull request #31143: Limit elasticsearch testing to basic tests when core SDK is modified build.gradle.kts | 5 - 1 file changed, 5 deletions(-)
(beam) branch master updated (a2987006ded -> 294aa65af93)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a2987006ded update Changes.md for bigquery metrics (#31155) add 2f570145fd9 Limit elasticsearch testing from core SDK change to just one suite (not full elasticsearch compat suite which is very flaky) new 294aa65af93 Merge pull request #31143: Limit elasticsearch testing to basic tests when core SDK is modified The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle.kts | 5 - 1 file changed, 5 deletions(-)
(beam) branch master updated (ea4770bf59c -> 8ef90be1b60)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ea4770bf59c Add controlling multiple models across processes to CHANGES (#31147) add f498cdfc2a8 Revert "Add Redistribute transform to Java SDK" add 3b272d2c6c9 Revert "Add direct Dataflow translation of Redistribute transform" add ece456c1a91 Revert "Add Redistribute translation to Spark runner" add ab9af72aeed Revert "Add Redistribute translation to Samza runner" add 49056fd7ed1 Revert "Add Redistribute translation to FlinkRunner" new 8ef90be1b60 Merge pull request #31140: Revert addition of Redistribute and tests to Java SDK, and translations from runners The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/flink_runner.gradle | 1 - runners/flink/job-server/flink_job_server.gradle | 2 - .../FlinkBatchPortablePipelineTranslator.java | 17 +- .../flink/FlinkBatchTransformTranslators.java | 78 .../FlinkStreamingPortablePipelineTranslator.java | 22 -- .../flink/FlinkStreamingTransformTranslators.java | 37 -- .../dataflow/DataflowPipelineTranslator.java | 37 -- .../beam/runners/dataflow/util/PropertyNames.java | 1 - .../translation/RedistributeByKeyTranslator.java | 64 .../samza/translation/ReshuffleTranslator.java | 14 +- .../samza/translation/SamzaPipelineTranslator.java | 1 - .../spark/translation/TransformTranslator.java | 66 .../streaming/StreamingTransformTranslator.java| 70 .../apache/beam/sdk/transforms/Redistribute.java | 300 --- .../util/construction/PTransformTranslation.java | 8 - .../beam/sdk/transforms/RedistributeTest.java | 410 - 16 files changed, 3 insertions(+), 1125 deletions(-) delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java delete mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/RedistributeTest.java
(beam) 01/01: Merge pull request #31140: Revert addition of Redistribute and tests to Java SDK, and translations from runners
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8ef90be1b6042d5ea79a249d61179df9ca0f6ea2 Merge: ea4770bf59c 49056fd7ed1 Author: Kenneth Knowles AuthorDate: Wed May 1 13:10:34 2024 -0400 Merge pull request #31140: Revert addition of Redistribute and tests to Java SDK, and translations from runners runners/flink/flink_runner.gradle | 1 - runners/flink/job-server/flink_job_server.gradle | 2 - .../FlinkBatchPortablePipelineTranslator.java | 17 +- .../flink/FlinkBatchTransformTranslators.java | 78 .../FlinkStreamingPortablePipelineTranslator.java | 22 -- .../flink/FlinkStreamingTransformTranslators.java | 37 -- .../dataflow/DataflowPipelineTranslator.java | 37 -- .../beam/runners/dataflow/util/PropertyNames.java | 1 - .../translation/RedistributeByKeyTranslator.java | 64 .../samza/translation/ReshuffleTranslator.java | 14 +- .../samza/translation/SamzaPipelineTranslator.java | 1 - .../spark/translation/TransformTranslator.java | 66 .../streaming/StreamingTransformTranslator.java| 70 .../apache/beam/sdk/transforms/Redistribute.java | 300 --- .../util/construction/PTransformTranslation.java | 8 - .../beam/sdk/transforms/RedistributeTest.java | 410 - 16 files changed, 3 insertions(+), 1125 deletions(-)
(beam) branch master updated (413af128937 -> e3b4d41f6b8)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 413af128937 Allow users to configure wait options for new Neo4j databases (#31129) add 22535521e7b Bump com.gradle.enterprise from 3.15.1 to 3.17.2 new e3b4d41f6b8 Merge pull request #31032: Bump com.gradle.enterprise from 3.15.1 to 3.17.2 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: settings.gradle.kts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) 01/01: Merge pull request #31032: Bump com.gradle.enterprise from 3.15.1 to 3.17.2
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e3b4d41f6b8a7a36891a268c5ee513e70fe5530c Merge: 413af128937 22535521e7b Author: Kenneth Knowles AuthorDate: Tue Apr 30 10:40:11 2024 -0400 Merge pull request #31032: Bump com.gradle.enterprise from 3.15.1 to 3.17.2 settings.gradle.kts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated (673da546c14 -> 6fa6a28b397)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 673da546c14 [#30083][prism] Factor out hold tracking to dedicated
structures (#31105)
add 08fc33b26cc Touch trigger files (add some comments as well)
add 7ade06f6f77 Add Redistribute to Beam model protos
add f49ac7d10a0 Add Redistribute transform to Java SDK
add 18ebc454928 Add direct Dataflow translation of Redistribute transform
add 21e3fa1eedf Add Redistribute translation to Spark runner
add 8f1d3da4399 Add Redistribute translation to Samza runner
add bdfc7812fdf Add Redistribute translation to FlinkRunner
new 6fa6a28b397 Merge pull request #30545: Add Redistribute transform to
model, Java SDK, and most active runners
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
..._PostCommit_Java_ValidatesRunner_Dataflow.json} | 0
...t_Java_ValidatesRunner_Dataflow_Streaming.json} | 0
...eam_PostCommit_Java_ValidatesRunner_Flink.json} | 0
...eam_PostCommit_Java_ValidatesRunner_Samza.json} | 0
...eam_PostCommit_Java_ValidatesRunner_Spark.json} | 0
.../beam/model/pipeline/v1/beam_runner_api.proto | 8 +
runners/flink/flink_runner.gradle | 1 +
runners/flink/job-server/flink_job_server.gradle | 2 +
.../FlinkBatchPortablePipelineTranslator.java | 17 +-
.../flink/FlinkBatchTransformTranslators.java | 78 ++
.../FlinkStreamingPortablePipelineTranslator.java | 22 ++
.../flink/FlinkStreamingTransformTranslators.java | 37 +++
.../dataflow/DataflowPipelineTranslator.java | 37 +++
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../translation/RedistributeByKeyTranslator.java | 64 +
.../samza/translation/ReshuffleTranslator.java | 14 +-
.../samza/translation/SamzaPipelineTranslator.java | 1 +
.../spark/translation/TransformTranslator.java | 66 +
.../streaming/StreamingTransformTranslator.java| 70 +
.../apache/beam/sdk/transforms/Redistribute.java | 300 +
.../util/construction/PTransformTranslation.java | 8 +
.../{ReshuffleTest.java => RedistributeTest.java} | 155 +--
22 files changed, 784 insertions(+), 97 deletions(-)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark3_Streaming.json =>
beam_PostCommit_Java_ValidatesRunner_Dataflow.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark3_Streaming.json =>
beam_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark3_Streaming.json =>
beam_PostCommit_Java_ValidatesRunner_Flink.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark3_Streaming.json =>
beam_PostCommit_Java_ValidatesRunner_Samza.json} (100%)
copy .github/trigger_files/{beam_PostCommit_Java_PVR_Spark3_Streaming.json =>
beam_PostCommit_Java_ValidatesRunner_Spark.json} (100%)
create mode 100644
runners/samza/src/main/java/org/apache/beam/runners/samza/translation/RedistributeByKeyTranslator.java
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Redistribute.java
copy
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/{ReshuffleTest.java
=> RedistributeTest.java} (74%)
(beam) 01/01: Merge pull request #30545: Add Redistribute transform to model, Java SDK, and most active runners
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 6fa6a28b397b753ccfb81995509a44f1df32a155 Merge: 673da546c14 bdfc7812fdf Author: Kenneth Knowles AuthorDate: Mon Apr 29 08:52:49 2024 -0400 Merge pull request #30545: Add Redistribute transform to model, Java SDK, and most active runners ...m_PostCommit_Java_ValidatesRunner_Dataflow.json | 3 + ...it_Java_ValidatesRunner_Dataflow_Streaming.json | 3 + ...beam_PostCommit_Java_ValidatesRunner_Flink.json | 3 + ...beam_PostCommit_Java_ValidatesRunner_Samza.json | 3 + ...beam_PostCommit_Java_ValidatesRunner_Spark.json | 3 + .../beam/model/pipeline/v1/beam_runner_api.proto | 8 + runners/flink/flink_runner.gradle | 1 + runners/flink/job-server/flink_job_server.gradle | 2 + .../FlinkBatchPortablePipelineTranslator.java | 17 +- .../flink/FlinkBatchTransformTranslators.java | 78 .../FlinkStreamingPortablePipelineTranslator.java | 22 ++ .../flink/FlinkStreamingTransformTranslators.java | 37 ++ .../dataflow/DataflowPipelineTranslator.java | 37 ++ .../beam/runners/dataflow/util/PropertyNames.java | 1 + .../translation/RedistributeByKeyTranslator.java | 64 .../samza/translation/ReshuffleTranslator.java | 14 +- .../samza/translation/SamzaPipelineTranslator.java | 1 + .../spark/translation/TransformTranslator.java | 66 .../streaming/StreamingTransformTranslator.java| 70 .../apache/beam/sdk/transforms/Redistribute.java | 300 +++ .../util/construction/PTransformTranslation.java | 8 + .../beam/sdk/transforms/RedistributeTest.java | 410 + 22 files changed, 1148 insertions(+), 3 deletions(-)
(beam) branch master updated (ab27f6f0358 -> 485c5198384)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ab27f6f0358 Refactor Residuals to prepare for processing time scheduling. (#31057) add 1ebd5039a97 Simplify intermediate data in Iceberg sink; use manifest files add 485c5198384 Merge pull request #31086: Simplify intermediate data in Iceberg sink; use manifest files No new revisions were added by this update. Summary of changes: sdks/java/io/iceberg/build.gradle | 1 - .../beam/sdk/io/iceberg/AppendFilesToTables.java | 2 +- .../beam/sdk/io/iceberg/FileWriteResult.java | 210 - .../apache/beam/sdk/io/iceberg/RecordWriter.java | 20 +- .../sdk/io/iceberg/WriteGroupedRowsToFiles.java| 3 +- .../sdk/io/iceberg/WriteUngroupedRowsToFiles.java | 5 +- .../beam/sdk/io/iceberg/FileWriteResultTest.java | 166 7 files changed, 61 insertions(+), 346 deletions(-) delete mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/FileWriteResultTest.java
(beam) branch master updated (1a0591ea797 -> 96dc16aedb7)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 1a0591ea797 Bump transformers (#30922) add 631f6e7a832 Fix iceberg dependencies add 36f32280a1b Refactor iceberg/beam conversions for clarity add a169e6a2a1c Re-add Iceberg bounded source; test splitting new 96dc16aedb7 Merge pull request #30805: Re-add iceberg bounded source; test splitting The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/iceberg/build.gradle | 15 +- .../java/org/apache/beam/io/iceberg/IcebergIO.java | 43 .../apache/beam/io/iceberg/IcebergScanConfig.java | 166 + .../org/apache/beam/io/iceberg/RecordWriter.java | 3 +- .../java/org/apache/beam/io/iceberg/RowHelper.java | 100 .../org/apache/beam/io/iceberg/ScanSource.java | 127 ++ .../org/apache/beam/io/iceberg/ScanTaskReader.java | 186 ++ .../org/apache/beam/io/iceberg/ScanTaskSource.java | 87 +++ .../beam/io/iceberg/SchemaAndRowConversions.java | 270 + .../org/apache/beam/io/iceberg/SchemaHelper.java | 116 - .../beam/io/iceberg/FileWriteResultTest.java | 10 +- .../apache/beam/io/iceberg/IcebergIOReadTest.java | 120 + .../apache/beam/io/iceberg/IcebergIOWriteTest.java | 10 +- .../org/apache/beam/io/iceberg/RowHelperTest.java | 128 -- .../org/apache/beam/io/iceberg/ScanSourceTest.java | 180 ++ .../io/iceberg/SchemaAndRowConversionsTest.java| 226 + .../org/apache/beam/io/iceberg/TestFixtures.java | 2 +- 17 files changed, 1423 insertions(+), 366 deletions(-) create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergScanConfig.java delete mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/RowHelper.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanSource.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanTaskReader.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/ScanTaskSource.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/SchemaAndRowConversions.java delete mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/SchemaHelper.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/IcebergIOReadTest.java delete mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/RowHelperTest.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/ScanSourceTest.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/SchemaAndRowConversionsTest.java
(beam) 01/01: Merge pull request #30805: Re-add iceberg bounded source; test splitting
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 96dc16aedb7708cb949ba6308960fb832381165f Merge: 1a0591ea797 a169e6a2a1c Author: Kenneth Knowles AuthorDate: Fri Apr 12 14:41:05 2024 -0400 Merge pull request #30805: Re-add iceberg bounded source; test splitting sdks/java/io/iceberg/build.gradle | 15 +- .../java/org/apache/beam/io/iceberg/IcebergIO.java | 43 .../apache/beam/io/iceberg/IcebergScanConfig.java | 166 + .../org/apache/beam/io/iceberg/RecordWriter.java | 3 +- .../java/org/apache/beam/io/iceberg/RowHelper.java | 100 .../org/apache/beam/io/iceberg/ScanSource.java | 127 ++ .../org/apache/beam/io/iceberg/ScanTaskReader.java | 186 ++ .../org/apache/beam/io/iceberg/ScanTaskSource.java | 87 +++ .../beam/io/iceberg/SchemaAndRowConversions.java | 270 + .../org/apache/beam/io/iceberg/SchemaHelper.java | 116 - .../beam/io/iceberg/FileWriteResultTest.java | 10 +- .../apache/beam/io/iceberg/IcebergIOReadTest.java | 120 + .../apache/beam/io/iceberg/IcebergIOWriteTest.java | 10 +- .../org/apache/beam/io/iceberg/RowHelperTest.java | 128 -- .../org/apache/beam/io/iceberg/ScanSourceTest.java | 180 ++ .../io/iceberg/SchemaAndRowConversionsTest.java| 226 + .../org/apache/beam/io/iceberg/TestFixtures.java | 2 +- 17 files changed, 1423 insertions(+), 366 deletions(-)
(beam) branch iceberg-gha deleted (was 153c7db71d5)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch iceberg-gha in repository https://gitbox.apache.org/repos/asf/beam.git was 153c7db71d5 Fixup iceberg GitHub Action workflow The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) 01/01: Merge pull request #30935: Fixup iceberg GitHub Action workflow
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 983f4e8a4995e55e217870713974b0753078ef24 Merge: 0e7b047c636 153c7db71d5 Author: Kenneth Knowles AuthorDate: Thu Apr 11 14:04:10 2024 -0400 Merge pull request #30935: Fixup iceberg GitHub Action workflow .github/workflows/IO_Iceberg.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam) branch master updated (0e7b047c636 -> 983f4e8a499)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0e7b047c636 [yaml] Fix Beam YAML blog links (#30932) add 153c7db71d5 Fixup iceberg GitHub Action workflow new 983f4e8a499 Merge pull request #30935: Fixup iceberg GitHub Action workflow The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/IO_Iceberg.yml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
(beam) branch iceberg-gha updated (0365ee7b05b -> 153c7db71d5)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch iceberg-gha in repository https://gitbox.apache.org/repos/asf/beam.git discard 0365ee7b05b Fixup iceberg GitHub Action workflow add 153c7db71d5 Fixup iceberg GitHub Action workflow This update added new revisions after undoing existing revisions. That is to say, some revisions that were in the old version of the branch are not in the new version. This situation occurs when a user --force pushes a change and generates a repository containing something like this: * -- * -- B -- O -- O -- O (0365ee7b05b) \ N -- N -- N refs/heads/iceberg-gha (153c7db71d5) You should already have received notification emails for all of the O revisions, and so the following emails describe only the N revisions from the common base, B. Any revisions marked "omit" are not gone; other references still refer to them. Any revisions marked "discard" are gone forever. No new revisions were added by this update. Summary of changes: .github/workflows/IO_Iceberg.yml | 6 -- 1 file changed, 4 insertions(+), 2 deletions(-)
(beam) 01/01: Fixup iceberg GitHub Action workflow
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch iceberg-gha
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 0365ee7b05b771a9ba8f5bda814f23282da3676f
Author: Kenneth Knowles
AuthorDate: Tue Apr 9 21:07:04 2024 -0400
Fixup iceberg GitHub Action workflow
---
.github/workflows/IO_Iceberg.yml | 7 +++
1 file changed, 3 insertions(+), 4 deletions(-)
diff --git a/.github/workflows/IO_Iceberg.yml b/.github/workflows/IO_Iceberg.yml
index abc75836322..2490920808c 100644
--- a/.github/workflows/IO_Iceberg.yml
+++ b/.github/workflows/IO_Iceberg.yml
@@ -62,7 +62,7 @@ env:
jobs:
IO_Iceberg:
-name: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
+name: ${{ matrix.job_name }}
strategy:
matrix:
job_name: ["IO_Iceberg"]
@@ -71,8 +71,7 @@ jobs:
github.event_name == 'push' ||
github.event_name == 'pull_request_target' ||
(github.event_name == 'schedule' && github.repository == 'apache/beam')
||
- github.event_name == 'workflow_dispatch' ||
- github.event.comment.body == 'Run Java_Amqp_IO_Direct PreCommit'
+ github.event_name == 'workflow_dispatch'
runs-on: [self-hosted, ubuntu-20.04, main]
steps:
- uses: actions/checkout@v4
@@ -84,7 +83,7 @@ jobs:
github_job: ${{ matrix.job_name }} (${{ matrix.job_phrase }})
- name: Setup environment
uses: ./.github/actions/setup-environment-action
- - name: run Amqp IO build script
+ - name: run IcebergIO build script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:io:iceberg:build
(beam) branch iceberg-gha created (now 0365ee7b05b)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch iceberg-gha in repository https://gitbox.apache.org/repos/asf/beam.git at 0365ee7b05b Fixup iceberg GitHub Action workflow This branch includes the following new commits: new 0365ee7b05b Fixup iceberg GitHub Action workflow The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
(beam) branch master updated: [java-extension-sql] Beam Sql doesn't honor aliases because of `BeamAggregateProjectMergeRule` optimization (#30902)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c2188643d27 [java-extension-sql] Beam Sql doesn't honor aliases
because of `BeamAggregateProjectMergeRule` optimization (#30902)
c2188643d27 is described below
commit c2188643d27e1a95b3cb7e724953cc9aec4cdf5c
Author: Brachi Packter
AuthorDate: Wed Apr 10 22:32:06 2024 +0300
[java-extension-sql] Beam Sql doesn't honor aliases because of
`BeamAggregateProjectMergeRule` optimization (#30902)
---
.../extensions/sql/impl/planner/BeamRuleSets.java | 3 +-
.../beam/sdk/extensions/sql/BeamSqlAliasTest | 94 ++
.../rule/BeamAggregateProjectMergeRuleTest.java| 3 +
3 files changed, 98 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
index 9851b2fcbf2..8d5b4d4fa08 100644
---
a/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
+++
b/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/planner/BeamRuleSets.java
@@ -21,7 +21,6 @@ import java.util.Collection;
import java.util.List;
import org.apache.beam.sdk.annotations.Internal;
import org.apache.beam.sdk.extensions.sql.impl.rel.BeamRelNode;
-import
org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregateProjectMergeRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamAggregationRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamBasicAggregationRule;
import org.apache.beam.sdk.extensions.sql.impl.rule.BeamCalcMergeRule;
@@ -83,7 +82,7 @@ public class BeamRuleSets {
CoreRules.PROJECT_SET_OP_TRANSPOSE,
// aggregation and projection rules
- BeamAggregateProjectMergeRule.INSTANCE,
+ // BeamAggregateProjectMergeRule.INSTANCE,
// push a projection past a filter or vice versa
CoreRules.PROJECT_FILTER_TRANSPOSE,
CoreRules.FILTER_PROJECT_TRANSPOSE,
diff --git
a/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlAliasTest
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlAliasTest
new file mode 100644
index 000..790312b7e75
--- /dev/null
+++
b/sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/BeamSqlAliasTest
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.extensions.sql;
+
+import java.io.Serializable;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Objects;
+import org.apache.beam.sdk.coders.StringUtf8Coder;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.testing.PAssert;
+import org.apache.beam.sdk.testing.TestPipeline;
+import org.apache.beam.sdk.transforms.Create;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.junit.Rule;
+import org.junit.Test;
+import org.testcontainers.shaded.com.fasterxml.jackson.databind.MapperFeature;
+import org.testcontainers.shaded.com.fasterxml.jackson.databind.ObjectMapper;
+
+public class BeamSqlAliasTest implements Serializable {
+
+ @Rule public final transient TestPipeline pipeline = TestPipeline.create();
+
+ @Test
+ public void testSqlWithAliasIsNotIgnoredWithOptimizers() {
+String ID = "id";
+String EVENT = "event";
+
+Schema inputType =
Schema.builder().addStringField(ID).addStringField(EVENT).build();
+
+String sql =
+"select event as event_name, count(*) as c\n" + "from PCOLLECTION\n" +
"group by event";
+
+List inputRows =
+TestUtils.RowsBuilder.of(inputType).addRows("123",
"some_event").getRows();
+
+PCollection r
(beam) branch master updated (812e98fac24 -> 819e54c972c)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 812e98fac24 Bump GCP-BOM to 26.36.0 (#30868) add 0437a8dcfcc Initial Iceberg connector add fd08eb49f31 Fix up IcebergIO Write path add a7a6515cf3f Add IcebergIO GitHub Action workflow new 819e54c972c Merge pull request #30797: Initial Iceberg Sink The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/IO_Iceberg.yml | 118 +++ sdks/java/io/iceberg/build.gradle | 98 ++ .../beam/io/iceberg/AppendFilesToTables.java | 102 ++ .../apache/beam/io/iceberg/AssignDestinations.java | 65 .../beam/io/iceberg/DynamicDestinations.java | 36 +++ .../apache/beam/io/iceberg/FileWriteResult.java| 203 .../beam/io/iceberg/IcebergCatalogConfig.java | 233 ++ .../apache/beam/io/iceberg/IcebergDestination.java | 65 .../java/org/apache/beam/io/iceberg/IcebergIO.java | 50 +++ .../beam/io/iceberg/IcebergTableCreateConfig.java | 50 +++ .../apache/beam/io/iceberg/IcebergWriteResult.java | 65 .../io/iceberg/OneTableDynamicDestinations.java| 65 .../apache/beam/io/iceberg/PropertyBuilder.java| 38 +++ .../org/apache/beam/io/iceberg/RecordWriter.java | 99 ++ .../java/org/apache/beam/io/iceberg/RowHelper.java | 100 ++ .../org/apache/beam/io/iceberg/SchemaHelper.java | 116 +++ .../beam/io/iceberg/WriteGroupedRowsToFiles.java | 103 ++ .../beam/io/iceberg/WriteToDestinations.java | 123 .../beam/io/iceberg/WriteUngroupedRowsToFiles.java | 350 + .../org/apache/beam/io/iceberg/package-info.java | 20 ++ .../beam/io/iceberg/FileWriteResultTest.java | 147 + .../apache/beam/io/iceberg/IcebergIOWriteTest.java | 293 + .../org/apache/beam/io/iceberg/RowHelperTest.java | 128 .../apache/beam/io/iceberg/TestDataWarehouse.java | 146 + .../org/apache/beam/io/iceberg/TestFixtures.java | 101 ++ settings.gradle.kts| 2 + 26 files changed, 2916 insertions(+) create mode 100644 .github/workflows/IO_Iceberg.yml create mode 100644 sdks/java/io/iceberg/build.gradle create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/AppendFilesToTables.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/AssignDestinations.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/DynamicDestinations.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/FileWriteResult.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergCatalogConfig.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergDestination.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergTableCreateConfig.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteResult.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/OneTableDynamicDestinations.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/PropertyBuilder.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/RecordWriter.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/RowHelper.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/SchemaHelper.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteGroupedRowsToFiles.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteToDestinations.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/WriteUngroupedRowsToFiles.java create mode 100644 sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/package-info.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/FileWriteResultTest.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/IcebergIOWriteTest.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/RowHelperTest.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/TestDataWarehouse.java create mode 100644 sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/TestFixtures.java
(beam) 01/01: Merge pull request #30797: Initial Iceberg Sink
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 819e54c972c41a1ec4ae56b604fa052a5edc991f Merge: 812e98fac24 a7a6515cf3f Author: Kenneth Knowles AuthorDate: Tue Apr 9 11:36:52 2024 -0400 Merge pull request #30797: Initial Iceberg Sink .github/workflows/IO_Iceberg.yml | 118 +++ sdks/java/io/iceberg/build.gradle | 98 ++ .../beam/io/iceberg/AppendFilesToTables.java | 102 ++ .../apache/beam/io/iceberg/AssignDestinations.java | 65 .../beam/io/iceberg/DynamicDestinations.java | 36 +++ .../apache/beam/io/iceberg/FileWriteResult.java| 203 .../beam/io/iceberg/IcebergCatalogConfig.java | 233 ++ .../apache/beam/io/iceberg/IcebergDestination.java | 65 .../java/org/apache/beam/io/iceberg/IcebergIO.java | 50 +++ .../beam/io/iceberg/IcebergTableCreateConfig.java | 50 +++ .../apache/beam/io/iceberg/IcebergWriteResult.java | 65 .../io/iceberg/OneTableDynamicDestinations.java| 65 .../apache/beam/io/iceberg/PropertyBuilder.java| 38 +++ .../org/apache/beam/io/iceberg/RecordWriter.java | 99 ++ .../java/org/apache/beam/io/iceberg/RowHelper.java | 100 ++ .../org/apache/beam/io/iceberg/SchemaHelper.java | 116 +++ .../beam/io/iceberg/WriteGroupedRowsToFiles.java | 103 ++ .../beam/io/iceberg/WriteToDestinations.java | 123 .../beam/io/iceberg/WriteUngroupedRowsToFiles.java | 350 + .../org/apache/beam/io/iceberg/package-info.java | 20 ++ .../beam/io/iceberg/FileWriteResultTest.java | 147 + .../apache/beam/io/iceberg/IcebergIOWriteTest.java | 293 + .../org/apache/beam/io/iceberg/RowHelperTest.java | 128 .../apache/beam/io/iceberg/TestDataWarehouse.java | 146 + .../org/apache/beam/io/iceberg/TestFixtures.java | 101 ++ settings.gradle.kts| 2 + 26 files changed, 2916 insertions(+)
(beam) branch revert-30425-mt-shutdown-channels deleted (was 8681ab9901d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch revert-30425-mt-shutdown-channels in repository https://gitbox.apache.org/repos/asf/beam.git was 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)" The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) 01/01: Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually"
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit cdaae0a7fb63bd679823488f901603e135d6db67 Merge: f4d03d49713 8681ab9901d Author: Kenneth Knowles AuthorDate: Wed Mar 20 18:28:21 2024 -0400 Merge pull request #30691: Revert #30425 "add a way for channels to be closed manually" .../google-cloud-dataflow-java/worker/build.gradle | 12 -- .../dataflow/worker/StreamingDataflowWorker.java | 11 +- .../worker/windmill/WindmillConnection.java| 5 - .../windmill/client/grpc/GrpcDispatcherClient.java | 4 +- .../windmill/client/grpc/GrpcWindmillServer.java | 29 +--- .../client/grpc/StreamingEngineClient.java | 25 ++-- .../windmill/client/grpc/stubs/ChannelCache.java | 117 .../grpc/stubs/ChannelCachingStubFactory.java | 38 -- .../client/grpc/stubs/IsolationChannel.java| 2 +- ...Factory.java => RemoteWindmillStubFactory.java} | 38 +++--- .../client/grpc/stubs/WindmillChannelFactory.java | 2 +- .../client/grpc/GrpcWindmillServerTest.java| 3 +- .../client/grpc/StreamingEngineClientTest.java | 31 +++-- .../client/grpc/WindmillStreamSenderTest.java | 3 +- .../client/grpc/stubs/ChannelCacheTest.java| 150 - .../windmill/testing/FakeWindmillStubFactory.java | 27 ++-- .../budget/EvenGetWorkBudgetDistributorTest.java | 7 +- 17 files changed, 78 insertions(+), 426 deletions(-)
(beam) branch master updated (f4d03d49713 -> cdaae0a7fb6)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from f4d03d49713 Minimize scope of expensive lock (#30679)
add 8681ab9901d Revert "Cache and close windmill grpc channels (#30425)"
new cdaae0a7fb6 Merge pull request #30691: Revert #30425 "add a way for
channels to be closed manually"
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../google-cloud-dataflow-java/worker/build.gradle | 12 --
.../dataflow/worker/StreamingDataflowWorker.java | 11 +-
.../worker/windmill/WindmillConnection.java| 5 -
.../windmill/client/grpc/GrpcDispatcherClient.java | 4 +-
.../windmill/client/grpc/GrpcWindmillServer.java | 29 +---
.../client/grpc/StreamingEngineClient.java | 25 ++--
.../windmill/client/grpc/stubs/ChannelCache.java | 117
.../grpc/stubs/ChannelCachingStubFactory.java | 38 --
.../client/grpc/stubs/IsolationChannel.java| 2 +-
...Factory.java => RemoteWindmillStubFactory.java} | 38 +++---
.../client/grpc/stubs/WindmillChannelFactory.java | 2 +-
.../client/grpc/GrpcWindmillServerTest.java| 3 +-
.../client/grpc/StreamingEngineClientTest.java | 31 +++--
.../client/grpc/WindmillStreamSenderTest.java | 3 +-
.../client/grpc/stubs/ChannelCacheTest.java| 150 -
.../windmill/testing/FakeWindmillStubFactory.java | 27 ++--
.../budget/EvenGetWorkBudgetDistributorTest.java | 7 +-
17 files changed, 78 insertions(+), 426 deletions(-)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCache.java
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCachingStubFactory.java
rename
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/{ChannelCachingRemoteStubFactory.java
=> RemoteWindmillStubFactory.java} (67%)
delete mode 100644
runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/windmill/client/grpc/stubs/ChannelCacheTest.java
(beam) branch master updated (06c76b4e930 -> c79b6e157dd)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 06c76b4e930 Updates the test suite to use the transform service (#30605) add 17bbe8262b9 Remove MapState and SetState tests that mix v1 and v2 codepaths erroneously add 8945b8fc144 Remove GroupIntoBatches tests that mix v1 and v2 codepaths erroneously add bf5551bd118 Remove PubsubSink tests that conflate v1 and v2 codepaths erroneously add 0a0186de555 Skip v1 translation steps for pipelines that will run on v2 new c79b6e157dd Merge pull request #30604: Skip running Dataflow v1 logic when a job will run on v2 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../dataflow/DataflowPipelineTranslator.java | 6 +- .../beam/runners/dataflow/DataflowRunner.java | 11 ++- .../dataflow/DataflowPipelineTranslatorTest.java | 105 - .../beam/runners/dataflow/DataflowRunnerTest.java | 29 +- 4 files changed, 16 insertions(+), 135 deletions(-)
(beam) 01/01: Merge pull request #30604: Skip running Dataflow v1 logic when a job will run on v2
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c79b6e157dd2f31ff6131938bade3e60a51a826c Merge: 06c76b4e930 0a0186de555 Author: Kenneth Knowles AuthorDate: Tue Mar 12 21:04:40 2024 -0400 Merge pull request #30604: Skip running Dataflow v1 logic when a job will run on v2 .../dataflow/DataflowPipelineTranslator.java | 6 +- .../beam/runners/dataflow/DataflowRunner.java | 11 ++- .../dataflow/DataflowPipelineTranslatorTest.java | 105 - .../beam/runners/dataflow/DataflowRunnerTest.java | 29 +- 4 files changed, 16 insertions(+), 135 deletions(-)
(beam) branch master updated (0c8b78a9ff0 -> 9a6777ebe7e)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 0c8b78a9ff0 Remove Anand, add Shunping to reviewers (#30608) add 97f8dae92f2 Put an explanatory comment in trigger files that exist new 9a6777ebe7e Merge pull request #30586: Put an explanatory comment in trigger files that exist The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/trigger_files/beam_PostCommit_Java_Avro_Versions.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_DataflowV1.json| 3 +++ .github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_PVR_Spark_Batch.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Flink_Java11.json| 3 +++ ...beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.json | 3 +++ .../trigger_files/beam_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Examples_Direct.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Dataflow.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Direct.json| 4 +++- .github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json | 3 +++ .github/trigger_files/beam_PostCommit_XVR_Flink.json | 3 +++ .../beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json | 3 +++ 16 files changed, 48 insertions(+), 1 deletion(-) create mode 100644 .github/trigger_files/beam_PostCommit_XVR_Flink.json
(beam) 01/01: Merge pull request #30586: Put an explanatory comment in trigger files that exist
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 9a6777ebe7e0903f9ffb57f7ea9e84bb4e448170 Merge: 0c8b78a9ff0 97f8dae92f2 Author: Kenneth Knowles AuthorDate: Tue Mar 12 11:59:47 2024 -0400 Merge pull request #30586: Put an explanatory comment in trigger files that exist .github/trigger_files/beam_PostCommit_Java_Avro_Versions.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_DataflowV1.json| 3 +++ .github/trigger_files/beam_PostCommit_Java_PVR_Spark3_Streaming.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_PVR_Spark_Batch.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Dataflow_V2.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Dataflow_V2_Streaming.json | 3 +++ .../beam_PostCommit_Java_ValidatesRunner_Flink_Java11.json| 3 +++ ...beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming.json | 3 +++ .../trigger_files/beam_PostCommit_Java_ValidatesRunner_Twister2.json | 3 +++ .github/trigger_files/beam_PostCommit_Java_ValidatesRunner_ULR.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Examples_Direct.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Dataflow.json | 3 +++ .github/trigger_files/beam_PostCommit_Python_Xlang_Gcp_Direct.json| 4 +++- .github/trigger_files/beam_PostCommit_Python_Xlang_IO_Dataflow.json | 3 +++ .github/trigger_files/beam_PostCommit_XVR_Flink.json | 3 +++ .../beam_PostCommit_XVR_PythonUsingJavaSQL_Dataflow.json | 3 +++ 16 files changed, 48 insertions(+), 1 deletion(-)
(beam) branch master updated (9ec2d49af98 -> 89366bbb4aa)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 9ec2d49af98 Duet AI Prompt - Beam YAML (no links) (#30440) add 407d5077b9f Remove JsonIgore for usePublicIps add 89366bbb4aa Merge pull request #30484: Remove JsonIgore for usePublicIps No new revisions were added by this update. Summary of changes: .../runners/dataflow/options/DataflowPipelineWorkerPoolOptions.java | 2 -- 1 file changed, 2 deletions(-)
(beam) branch master updated (80d4c852f36 -> fbde0ce32c9)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 80d4c852f36 Merge pull request #30380 Fix updating of annotations for
explicitly named transforms.
add 2d85be72f31 Fix Avro Version PostCommit
add fbde0ce32c9 Merge pull request #30442: Fix Avro Version PostCommit
No new revisions were added by this update.
Summary of changes:
...m_PostCommit_Java.json => beam_PostCommit_Java_Avro_Versions.json} | 0
sdks/java/extensions/avro/build.gradle| 4 ++--
2 files changed, 2 insertions(+), 2 deletions(-)
rename .github/trigger_files/{beam_PostCommit_Java.json =>
beam_PostCommit_Java_Avro_Versions.json} (100%)
(beam) branch master updated (bcdac047062 -> 53c966dfd9e)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from bcdac047062 Update test-your-pipeline.md (#30328) add eef267f3cd5 Use canonical path when testing classpath scanner add f73ce82fb37 Merge runners-core-construction into sdks-java-core add 7c7820eb9cb CHANGES note: elimination of sdks/java/fn-execution and runners/core-construction-java add 53c966dfd9e Merge pull request #29924: Merge runners-core-construction into sdks-java-core No new revisions were added by this update. Summary of changes: CHANGES.md | 2 + build.gradle.kts | 1 - .../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 - .../backend/internal/preparers/java_preparers.go | 2 +- .../internal/preparers/java_preparers_test.go | 4 +- runners/core-construction-java/build.gradle| 77 - runners/core-java/build.gradle | 1 - .../core/GroupAlsoByWindowViaWindowSetNewDoFn.java | 2 +- .../core/SplittableParDoViaKeyedWorkItems.java | 16 +- .../core/metrics/MonitoringInfoConstants.java | 2 +- .../core/metrics/MonitoringInfoMetricName.java | 1 - .../apache/beam/runners/core/ReduceFnTester.java | 2 +- runners/direct-java/build.gradle | 10 +- .../direct/BoundedReadEvaluatorFactory.java| 4 +- ...DirectGBKIntoKeyedWorkItemsOverrideFactory.java | 4 +- .../apache/beam/runners/direct/DirectGraph.java| 2 +- .../beam/runners/direct/DirectGraphVisitor.java| 2 +- .../beam/runners/direct/DirectGroupByKey.java | 2 +- .../direct/DirectGroupByKeyOverrideFactory.java| 4 +- .../apache/beam/runners/direct/DirectRunner.java | 6 +- .../runners/direct/DirectTransformExecutor.java| 2 +- .../beam/runners/direct/ExecutableGraph.java | 2 +- .../direct/GroupAlsoByWindowEvaluatorFactory.java | 2 +- .../beam/runners/direct/MultiStepCombine.java | 6 +- .../beam/runners/direct/ParDoEvaluatorFactory.java | 2 +- .../runners/direct/ParDoMultiOverrideFactory.java | 8 +- .../beam/runners/direct/ReadEvaluatorFactory.java | 2 +- .../beam/runners/direct/RootProviderRegistry.java | 6 +- .../runners/direct/TestStreamEvaluatorFactory.java | 4 +- .../runners/direct/TransformEvaluatorRegistry.java | 20 +-- .../runners/direct/UnboundedReadDeduplicator.java | 2 +- .../direct/UnboundedReadEvaluatorFactory.java | 4 +- .../runners/direct/WindowEvaluatorFactory.java | 2 +- .../runners/direct/WriteWithShardingFactory.java | 6 +- .../direct/BoundedReadEvaluatorFactoryTest.java| 2 +- .../direct/StatefulParDoEvaluatorFactoryTest.java | 2 +- .../direct/UnboundedReadEvaluatorFactoryTest.java | 2 +- runners/flink/flink_runner.gradle | 1 - .../runners/flink/CreateStreamingFlinkView.java| 4 +- .../flink/FlinkBatchPipelineTranslator.java| 2 +- .../FlinkBatchPortablePipelineTranslator.java | 20 +-- .../flink/FlinkBatchTransformTranslators.java | 8 +- .../runners/flink/FlinkExecutionEnvironments.java | 2 +- .../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +- .../flink/FlinkPipelineExecutionEnvironment.java | 2 +- .../beam/runners/flink/FlinkPipelineRunner.java| 16 +- .../org/apache/beam/runners/flink/FlinkRunner.java | 4 +- .../flink/FlinkStreamingPipelineTranslator.java| 12 +- .../FlinkStreamingPortablePipelineTranslator.java | 26 +-- .../flink/FlinkStreamingTransformTranslators.java | 14 +- .../flink/FlinkStreamingTranslationContext.java| 2 +- .../runners/flink/FlinkTransformOverrides.java | 8 +- .../translation/functions/FlinkDoFnFunction.java | 2 +- .../functions/FlinkExecutableStageFunction.java| 10 +- .../FlinkExecutableStagePruningFunction.java | 2 +- .../FlinkMergingNonShuffleReduceFunction.java | 2 +- .../functions/FlinkMultiOutputPruningFunction.java | 2 +- .../functions/FlinkPartialReduceFunction.java | 2 +- .../translation/functions/FlinkReduceFunction.java | 2 +- .../functions/FlinkStatefulDoFnFunction.java | 2 +- .../translation/types/CoderTypeInformation.java| 2 +- .../translation/types/CoderTypeSerializer.java | 2 +- .../utils/FlinkPortableRunnerUtils.java| 2 +- .../translation/utils/LookupPipelineVisitor.java | 2 +- .../translation/wrappers/SourceInputFormat.java| 2 +- .../wrappers/streaming/DoFnOperator.java | 2 +- .../streaming/ExecutableStageDoFnOperator.java | 10 +- .../streaming/KvToByteBufferKeySelector.java | 2 +- .../streaming/SdfByteBufferKeySelector.java| 2 +- .../wrappers/streaming/WorkItemKeySelector.java| 2 +- .../wrappers/streaming/io/DedupingOperator.java| 2 +- .../streaming
(beam) branch master updated (3b09283e45d -> b5cfd9523cd)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 3b09283e45d Bump github.com/docker/go-connections from 0.4.0 to 0.5.0
in /sdks (#30247)
add ea68a4590d0 Register URN for SplittableParDo centrally, to avoid
conflicts
add 95d6ec049ae Remove avro dependency from runners/core-construction-java
new b5cfd9523cd Merge pull request #30235: Remove avro dependency from
runners-core-construction
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
runners/core-construction-java/build.gradle| 2 -
.../core/construction/CoderTranslation.java| 57 +
.../beam/runners/core/construction/External.java | 2 +-
.../core/construction/PTransformTranslation.java | 2 +
.../core/construction/CoderTranslationTest.java| 10 ++--
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 29 +++
runners/direct-java/build.gradle | 3 --
.../runners/direct/TransformEvaluatorRegistry.java | 4 --
.../flink/FlinkStreamingTransformTranslators.java | 4 --
sdks/java/extensions/avro/build.gradle | 5 +-
.../avro}/AvroGenericCoderRegistrar.java | 4 +-
.../avro}/AvroGenericCoderTranslator.java | 3 +-
.../avro/{schemas => }/package-info.java | 2 +-
.../extensions/avro/AvroCoderTranslationTest.java | 59 ++
15 files changed, 139 insertions(+), 49 deletions(-)
rename
{runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction
=>
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro}/AvroGenericCoderRegistrar.java
(90%)
rename
{runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction
=>
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro}/AvroGenericCoderTranslator.java
(94%)
copy
sdks/java/extensions/avro/src/main/java/org/apache/beam/sdk/extensions/avro/{schemas
=> }/package-info.java (95%)
create mode 100644
sdks/java/extensions/avro/src/test/java/org/apache/beam/sdk/extensions/avro/AvroCoderTranslationTest.java
(beam) 01/01: Merge pull request #30235: Remove avro dependency from runners-core-construction
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b5cfd9523cde3fee8610956682541d4e90ee967b Merge: 3b09283e45d 95d6ec049ae Author: Kenneth Knowles AuthorDate: Tue Feb 13 10:28:42 2024 -0500 Merge pull request #30235: Remove avro dependency from runners-core-construction runners/core-construction-java/build.gradle| 2 - .../core/construction/CoderTranslation.java| 57 + .../beam/runners/core/construction/External.java | 2 +- .../core/construction/PTransformTranslation.java | 2 + .../core/construction/CoderTranslationTest.java| 10 ++-- .../core/construction/PipelineTranslationTest.java | 2 +- .../core/SplittableParDoViaKeyedWorkItems.java | 29 +++ runners/direct-java/build.gradle | 3 -- .../runners/direct/TransformEvaluatorRegistry.java | 4 -- .../flink/FlinkStreamingTransformTranslators.java | 4 -- sdks/java/extensions/avro/build.gradle | 5 +- .../avro}/AvroGenericCoderRegistrar.java | 4 +- .../avro}/AvroGenericCoderTranslator.java | 3 +- .../beam/sdk/extensions/avro/package-info.java | 26 ++ .../extensions/avro/AvroCoderTranslationTest.java | 59 ++ 15 files changed, 164 insertions(+), 48 deletions(-)
(beam) branch master updated (a15dd7e7f84 -> 382c6dccfcc)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from a15dd7e7f84 [Go SDK] Use distroless:debian12 (no-ssl) as base image. (#30011) add 9b4b509e52c Fail later when duplicate transform translators are on the classpath new 382c6dccfcc Merge pull request #30252: Fail later when duplicate transform translators are on the classpath The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../core/construction/PTransformTranslation.java | 90 ++- .../sdk/expansion/service/ExpansionService.java| 650 ++--- .../ExpansionServiceSchemaTransformProvider.java | 1 - .../service/JavaClassLookupTransformProvider.java | 1 - .../sdk/expansion/service/TransformProvider.java | 167 ++ .../expansion/service/ExpansionServiceTest.java| 11 +- .../beam/sdk/expansion/service/ExternalTest.java | 2 +- .../testing/expansion/TestExpansionService.java| 10 +- 8 files changed, 541 insertions(+), 391 deletions(-) create mode 100644 sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/TransformProvider.java
(beam) 01/01: Merge pull request #30252: Fail later when duplicate transform translators are on the classpath
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 382c6dccfccabfe4a7b20db9a2c24369e8fce981 Merge: a15dd7e7f84 9b4b509e52c Author: Kenneth Knowles AuthorDate: Wed Feb 7 19:54:43 2024 -0500 Merge pull request #30252: Fail later when duplicate transform translators are on the classpath .../core/construction/PTransformTranslation.java | 90 ++- .../sdk/expansion/service/ExpansionService.java| 650 ++--- .../ExpansionServiceSchemaTransformProvider.java | 1 - .../service/JavaClassLookupTransformProvider.java | 1 - .../sdk/expansion/service/TransformProvider.java | 167 ++ .../expansion/service/ExpansionServiceTest.java| 11 +- .../beam/sdk/expansion/service/ExternalTest.java | 2 +- .../testing/expansion/TestExpansionService.java| 10 +- 8 files changed, 541 insertions(+), 391 deletions(-)
(beam) 01/01: Merge pull request #30237: Adjust transform-service module names and deps with sdks/java/core
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 13a70844abb349d0ae5c6d84097e5cf4d0e9cc21 Merge: 4645a76ee2a f250a843230 Author: Kenneth Knowles AuthorDate: Wed Feb 7 12:33:47 2024 -0500 Merge pull request #30237: Adjust transform-service module names and deps with sdks/java/core sdks/java/transform-service/app/build.gradle | 2 +- sdks/java/transform-service/build.gradle | 2 +- sdks/java/transform-service/launcher/build.gradle | 7 ++- 3 files changed, 4 insertions(+), 7 deletions(-)
(beam) branch master updated (4645a76ee2a -> 13a70844abb)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 4645a76ee2a gcsio: reduce number of get requests in function calls (#30205) add f250a843230 Adjust transform-service module names and deps with sdks/java/core new 13a70844abb Merge pull request #30237: Adjust transform-service module names and deps with sdks/java/core The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/transform-service/app/build.gradle | 2 +- sdks/java/transform-service/build.gradle | 2 +- sdks/java/transform-service/launcher/build.gradle | 7 ++- 3 files changed, 4 insertions(+), 7 deletions(-)
(beam) branch master updated (30a778b9cbe -> 8ec51f181a0)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 30a778b9cbe Bump github.com/containerd/containerd from 1.7.7 to 1.7.11 in /sdks (#29823) add f03b115830b Remove extraneous mocking that was causing NPEs in DataflowWorkUnitClientTest add 00f07864784 Remove sdks/java/fn-execution new 8ec51f181a0 Merge pull request #30193: Remove sdks/java/fn-execution The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: build.gradle.kts | 1 - runners/direct-java/build.gradle | 2 - .../worker/DataflowWorkUnitClientTest.java | 99 +- sdks/java/expansion-service/build.gradle | 2 - sdks/java/fn-execution/build.gradle| 34 sdks/java/harness/build.gradle | 3 +- settings.gradle.kts| 1 - 7 files changed, 62 insertions(+), 80 deletions(-) delete mode 100644 sdks/java/fn-execution/build.gradle
(beam) 01/01: Merge pull request #30193: Remove sdks/java/fn-execution
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8ec51f181a02c8e1163e6fa0239e2361f30d16d8 Merge: 30a778b9cbe 00f07864784 Author: Kenneth Knowles AuthorDate: Fri Feb 2 15:32:22 2024 -0500 Merge pull request #30193: Remove sdks/java/fn-execution build.gradle.kts | 1 - runners/direct-java/build.gradle | 2 - .../worker/DataflowWorkUnitClientTest.java | 99 +- sdks/java/expansion-service/build.gradle | 2 - sdks/java/fn-execution/build.gradle| 34 sdks/java/harness/build.gradle | 3 +- settings.gradle.kts| 1 - 7 files changed, 62 insertions(+), 80 deletions(-)
(beam) branch master updated (574e7bfbc1d -> e8df60381a3)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 574e7bfbc1d Fix script typos for validating docker licenses. (#30176) add 7fb05cbb5e1 Move the code from sdks-java-fn-execution to sdks-java-core new e8df60381a3 Merge pull request #30170: Move the code from sdks-java-fn-execution to sdks-java-core The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/core-construction-java/build.gradle| 1 - runners/core-java/build.gradle | 1 - runners/direct-java/build.gradle | 1 - runners/flink/flink_runner.gradle | 1 - runners/google-cloud-dataflow-java/worker/build.gradle | 5 - runners/java-fn-execution/build.gradle | 1 - runners/java-job-service/build.gradle | 1 - runners/samza/build.gradle | 1 - runners/spark/spark_runner.gradle | 1 - sdks/java/core/build.gradle| 2 ++ .../main/java/org/apache/beam/sdk/fn/CancellableQueue.java | 0 .../src/main/java/org/apache/beam/sdk/fn/IdGenerator.java | 0 .../src/main/java/org/apache/beam/sdk/fn/IdGenerators.java | 0 .../main/java/org/apache/beam/sdk/fn/JvmInitializers.java | 0 .../beam/sdk/fn/channel/AddHarnessIdInterceptor.java | 0 .../apache/beam/sdk/fn/channel/ManagedChannelFactory.java | 0 .../apache/beam/sdk/fn/channel/SocketAddressFactory.java | 0 .../java/org/apache/beam/sdk/fn/channel/package-info.java | 0 .../apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 0 .../apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java | 0 .../beam/sdk/fn/data/BeamFnDataOutboundAggregator.java | 0 .../apache/beam/sdk/fn/data/CloseableFnDataReceiver.java | 0 .../java/org/apache/beam/sdk/fn/data/DataEndpoint.java | 0 .../apache/beam/sdk/fn/data/DecodingFnDataReceiver.java| 0 .../java/org/apache/beam/sdk/fn/data/FnDataReceiver.java | 0 .../java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java | 0 .../org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java| 0 .../org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java | 0 .../java/org/apache/beam/sdk/fn/data/TimerEndpoint.java| 0 .../java/org/apache/beam/sdk/fn/data/WeightedList.java | 0 .../java/org/apache/beam/sdk/fn/data/package-info.java | 0 .../src/main/java/org/apache/beam/sdk/fn/package-info.java | 0 .../main/java/org/apache/beam/sdk/fn/server/FnService.java | 0 .../sdk/fn/server/GrpcContextHeaderAccessorProvider.java | 0 .../java/org/apache/beam/sdk/fn/server/GrpcFnServer.java | 0 .../java/org/apache/beam/sdk/fn/server/HeaderAccessor.java | 0 .../apache/beam/sdk/fn/server/InProcessServerFactory.java | 0 .../java/org/apache/beam/sdk/fn/server/ServerFactory.java | 0 .../java/org/apache/beam/sdk/fn/server/package-info.java | 0 .../beam/sdk/fn/splittabledofn/RestrictionTrackers.java| 0 .../beam/sdk/fn/splittabledofn/WatermarkEstimators.java| 0 .../apache/beam/sdk/fn/splittabledofn/package-info.java| 0 .../org/apache/beam/sdk/fn/stream/AdvancingPhaser.java | 0 .../apache/beam/sdk/fn/stream/BufferingStreamObserver.java | 0 .../java/org/apache/beam/sdk/fn/stream/DataStreams.java| 0 .../apache/beam/sdk/fn/stream/DirectStreamObserver.java| 0 .../sdk/fn/stream/ForwardingClientResponseObserver.java| 0 .../apache/beam/sdk/fn/stream/OutboundObserverFactory.java | 0 .../apache/beam/sdk/fn/stream/PrefetchableIterable.java| 0 .../apache/beam/sdk/fn/stream/PrefetchableIterables.java | 0 .../apache/beam/sdk/fn/stream/PrefetchableIterator.java| 0 .../apache/beam/sdk/fn/stream/PrefetchableIterators.java | 0 .../beam/sdk/fn/stream/SynchronizedStreamObserver.java | 0 .../java/org/apache/beam/sdk/fn/stream/package-info.java | 0 .../java/org/apache/beam/sdk/fn/test/TestExecutors.java| 0 .../main/java/org/apache/beam/sdk/fn/test/TestStreams.java | 0 .../java/org/apache/beam/sdk/fn/test/package-info.java | 0 .../apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java | 0 .../org/apache/beam/sdk/fn/windowing/package-info.java | 0 .../java/org/apache/beam/sdk/fn/CancellableQueueTest.java | 0 .../test/java/org/apache/beam/sdk/fn/IdGeneratorsTest.java | 0 .../java/org/apache/beam/sdk/fn/JvmInitializersTest.java | 5 +++-- .../beam/sdk/fn/channel/ManagedChannelFactoryTest.java | 0 .../beam/sdk/fn/channel/SocketAddressFactoryTest.java | 0 .../beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java| 0
(beam) 01/01: Merge pull request #30170: Move the code from sdks-java-fn-execution to sdks-java-core
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit e8df60381a39327200ae70f38b569b9126e3ae5e Merge: 574e7bfbc1d 7fb05cbb5e1 Author: Kenneth Knowles AuthorDate: Thu Feb 1 21:39:14 2024 -0500 Merge pull request #30170: Move the code from sdks-java-fn-execution to sdks-java-core runners/core-construction-java/build.gradle| 1 - runners/core-java/build.gradle | 1 - runners/direct-java/build.gradle | 1 - runners/flink/flink_runner.gradle | 1 - runners/google-cloud-dataflow-java/worker/build.gradle | 5 - runners/java-fn-execution/build.gradle | 1 - runners/java-job-service/build.gradle | 1 - runners/samza/build.gradle | 1 - runners/spark/spark_runner.gradle | 1 - sdks/java/core/build.gradle| 2 ++ .../main/java/org/apache/beam/sdk/fn/CancellableQueue.java | 0 .../src/main/java/org/apache/beam/sdk/fn/IdGenerator.java | 0 .../src/main/java/org/apache/beam/sdk/fn/IdGenerators.java | 0 .../main/java/org/apache/beam/sdk/fn/JvmInitializers.java | 0 .../beam/sdk/fn/channel/AddHarnessIdInterceptor.java | 0 .../apache/beam/sdk/fn/channel/ManagedChannelFactory.java | 0 .../apache/beam/sdk/fn/channel/SocketAddressFactory.java | 0 .../java/org/apache/beam/sdk/fn/channel/package-info.java | 0 .../apache/beam/sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 0 .../apache/beam/sdk/fn/data/BeamFnDataInboundObserver.java | 0 .../beam/sdk/fn/data/BeamFnDataOutboundAggregator.java | 0 .../apache/beam/sdk/fn/data/CloseableFnDataReceiver.java | 0 .../java/org/apache/beam/sdk/fn/data/DataEndpoint.java | 0 .../apache/beam/sdk/fn/data/DecodingFnDataReceiver.java| 0 .../java/org/apache/beam/sdk/fn/data/FnDataReceiver.java | 0 .../java/org/apache/beam/sdk/fn/data/LogicalEndpoint.java | 0 .../org/apache/beam/sdk/fn/data/RemoteGrpcPortRead.java| 0 .../org/apache/beam/sdk/fn/data/RemoteGrpcPortWrite.java | 0 .../java/org/apache/beam/sdk/fn/data/TimerEndpoint.java| 0 .../java/org/apache/beam/sdk/fn/data/WeightedList.java | 0 .../java/org/apache/beam/sdk/fn/data/package-info.java | 0 .../src/main/java/org/apache/beam/sdk/fn/package-info.java | 0 .../main/java/org/apache/beam/sdk/fn/server/FnService.java | 0 .../sdk/fn/server/GrpcContextHeaderAccessorProvider.java | 0 .../java/org/apache/beam/sdk/fn/server/GrpcFnServer.java | 0 .../java/org/apache/beam/sdk/fn/server/HeaderAccessor.java | 0 .../apache/beam/sdk/fn/server/InProcessServerFactory.java | 0 .../java/org/apache/beam/sdk/fn/server/ServerFactory.java | 0 .../java/org/apache/beam/sdk/fn/server/package-info.java | 0 .../beam/sdk/fn/splittabledofn/RestrictionTrackers.java| 0 .../beam/sdk/fn/splittabledofn/WatermarkEstimators.java| 0 .../apache/beam/sdk/fn/splittabledofn/package-info.java| 0 .../org/apache/beam/sdk/fn/stream/AdvancingPhaser.java | 0 .../apache/beam/sdk/fn/stream/BufferingStreamObserver.java | 0 .../java/org/apache/beam/sdk/fn/stream/DataStreams.java| 0 .../apache/beam/sdk/fn/stream/DirectStreamObserver.java| 0 .../sdk/fn/stream/ForwardingClientResponseObserver.java| 0 .../apache/beam/sdk/fn/stream/OutboundObserverFactory.java | 0 .../apache/beam/sdk/fn/stream/PrefetchableIterable.java| 0 .../apache/beam/sdk/fn/stream/PrefetchableIterables.java | 0 .../apache/beam/sdk/fn/stream/PrefetchableIterator.java| 0 .../apache/beam/sdk/fn/stream/PrefetchableIterators.java | 0 .../beam/sdk/fn/stream/SynchronizedStreamObserver.java | 0 .../java/org/apache/beam/sdk/fn/stream/package-info.java | 0 .../java/org/apache/beam/sdk/fn/test/TestExecutors.java| 0 .../main/java/org/apache/beam/sdk/fn/test/TestStreams.java | 0 .../java/org/apache/beam/sdk/fn/test/package-info.java | 0 .../apache/beam/sdk/fn/windowing/EncodedBoundedWindow.java | 0 .../org/apache/beam/sdk/fn/windowing/package-info.java | 0 .../java/org/apache/beam/sdk/fn/CancellableQueueTest.java | 0 .../test/java/org/apache/beam/sdk/fn/IdGeneratorsTest.java | 0 .../java/org/apache/beam/sdk/fn/JvmInitializersTest.java | 5 +++-- .../beam/sdk/fn/channel/ManagedChannelFactoryTest.java | 0 .../beam/sdk/fn/channel/SocketAddressFactoryTest.java | 0 .../beam/sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java| 0 .../beam/sdk/fn/data/BeamFnDataInboundObserverTest.java| 14 +- .../beam/sdk/fn/data/BeamFnDataOutboundAggregatorTest.java | 0 .../apache/beam/sdk/fn/data/RemoteGrpcPortReadTest.java| 0 .../apache/beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 0 .../sdk/fn/splittabledofn/RestrictionTrackersTest.java
(beam) 01/01: Merge pull request #29961: Upgrade checkerframework to 3.42.0
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 8772389d0c2d77c3121e0bd6f27a9f5208de0dc2 Merge: b4c23b32f2b 65811eed6ea Author: Kenneth Knowles AuthorDate: Thu Jan 18 14:42:52 2024 -0500 Merge pull request #29961: Upgrade checkerframework to 3.42.0 .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../it/gcp/dataflow/AbstractPipelineLauncher.java | 17 ++- .../core/metrics/ExecutionStateSampler.java| 3 +- .../apache/beam/runners/flink/metrics/Metrics.java | 3 +- .../beam/runners/dataflow/DataflowMetrics.java | 8 +- runners/spark/spark_runner.gradle | 2 + sdks/java/core/build.gradle| 1 + .../src/main/java/org/apache/beam/sdk/io/Read.java | 131 - .../org/apache/beam/sdk/values/RowWithGetters.java | 21 +++- .../extensions/protobuf/ProtoSchemaTranslator.java | 9 +- .../extensions/sorter/BufferedExternalSorter.java | 6 +- .../apache/beam/sdk/io/aws2/schemas/AwsTypes.java | 1 + .../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 4 +- .../beam/sdk/io/gcp/bigquery/RetryManager.java | 2 +- .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 27 +++-- .../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 10 +- .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 2 +- .../singlestore/SingleStoreDefaultRowMapper.java | 5 +- .../beam/sdk/tpcds/TableSchemaJSONLoader.java | 14 ++- 19 files changed, 166 insertions(+), 102 deletions(-)
(beam) branch master updated (b4c23b32f2b -> 8772389d0c2)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b4c23b32f2b Correct Examples of utilizing `UploadToDicomStore` (#30008) add 65811eed6ea Upgrade checkerframework to 3.42.0 new 8772389d0c2 Merge pull request #29961: Upgrade checkerframework to 3.42.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- .../it/gcp/dataflow/AbstractPipelineLauncher.java | 17 ++- .../core/metrics/ExecutionStateSampler.java| 3 +- .../apache/beam/runners/flink/metrics/Metrics.java | 3 +- .../beam/runners/dataflow/DataflowMetrics.java | 8 +- runners/spark/spark_runner.gradle | 2 + sdks/java/core/build.gradle| 1 + .../src/main/java/org/apache/beam/sdk/io/Read.java | 131 - .../org/apache/beam/sdk/values/RowWithGetters.java | 21 +++- .../extensions/protobuf/ProtoSchemaTranslator.java | 9 +- .../extensions/sorter/BufferedExternalSorter.java | 6 +- .../apache/beam/sdk/io/aws2/schemas/AwsTypes.java | 1 + .../org/apache/beam/sdk/io/aws2/sqs/SqsIO.java | 4 +- .../beam/sdk/io/gcp/bigquery/RetryManager.java | 2 +- .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 27 +++-- .../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 10 +- .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 2 +- .../singlestore/SingleStoreDefaultRowMapper.java | 5 +- .../beam/sdk/tpcds/TableSchemaJSONLoader.java | 14 ++- 19 files changed, 166 insertions(+), 102 deletions(-)
(beam) 01/01: Merge pull request #30021: typo in docs
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit f1c154778f5e338c02eed0fac9a9f908aca71bd1 Merge: c62c24dda4c fc626b8b280 Author: Kenneth Knowles AuthorDate: Tue Jan 16 19:26:46 2024 -0500 Merge pull request #30021: typo in docs website/www/site/content/en/documentation/programming-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated (c62c24dda4c -> f1c154778f5)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c62c24dda4c [yaml] Revise yaml user label (#29989) add fc626b8b280 typo in docs new f1c154778f5 Merge pull request #30021: typo in docs The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/content/en/documentation/programming-guide.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated (c11b7c1f7cb -> 96a9830ac27)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c11b7c1f7cb Update scaling-streaming-workload.md (#29921) add cf201198df9 sickbay new reshuffle test on runners that do not support it new 96a9830ac27 Merge pull request #29994: sickbay new reshuffle test on runners that do not support it The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/portability/java/build.gradle| 3 +++ runners/samza/job-server/build.gradle| 2 ++ runners/spark/job-server/spark_job_server.gradle | 2 ++ runners/spark/spark_runner.gradle| 2 ++ 4 files changed, 9 insertions(+)
(beam) 01/01: Merge pull request #29994: sickbay new reshuffle test on runners that do not support it
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 96a9830ac2759f1586009d3c72a0fd0732e69398 Merge: c11b7c1f7cb cf201198df9 Author: Kenneth Knowles AuthorDate: Tue Jan 16 16:27:25 2024 -0500 Merge pull request #29994: sickbay new reshuffle test on runners that do not support it runners/portability/java/build.gradle| 3 +++ runners/samza/job-server/build.gradle| 2 ++ runners/spark/job-server/spark_job_server.gradle | 2 ++ runners/spark/spark_runner.gradle| 2 ++ 4 files changed, 9 insertions(+)
(beam) 01/01: Merge pull request #29811: Bump org.checkerframework:checkerframework-gradle-plugin from 0.6.34 to 0.6.37
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit bb0a2037970ac80e98766abb081768fd531abfad Merge: 97ca4430d97 f311d327aac Author: Kenneth Knowles AuthorDate: Tue Jan 9 12:34:17 2024 -0500 Merge pull request #29811: Bump org.checkerframework:checkerframework-gradle-plugin from 0.6.34 to 0.6.37 buildSrc/build.gradle.kts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch dependabot/gradle/org.checkerframework-checkerframework-gradle-plugin-0.6.37 deleted (was f311d327aac)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch dependabot/gradle/org.checkerframework-checkerframework-gradle-plugin-0.6.37 in repository https://gitbox.apache.org/repos/asf/beam.git was f311d327aac Bump org.checkerframework:checkerframework-gradle-plugin The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
(beam) branch master updated (97ca4430d97 -> bb0a2037970)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 97ca4430d97 remove threadlocal for DESERIALIZATION_CONTEXT and use copy instead (#29952) add f311d327aac Bump org.checkerframework:checkerframework-gradle-plugin new bb0a2037970 Merge pull request #29811: Bump org.checkerframework:checkerframework-gradle-plugin from 0.6.34 to 0.6.37 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: buildSrc/build.gradle.kts | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
(beam) branch master updated (36bdff42e8e -> 8bb66105689)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 36bdff42e8e Support key iteration in Python ULR. (#29839) add 7827d1d1809 Fix Reshuffle implementation in Java SDK add 8bb66105689 Merge pull request #28853: Fix Reshuffle implementation in Java SDK No new revisions were added by this update. Summary of changes: runners/flink/flink_runner.gradle | 2 + runners/flink/job-server/flink_job_server.gradle | 2 + runners/samza/src/test/resources/ExpectedDag.json | 68 -- .../org/apache/beam/sdk/transforms/Reshuffle.java | 89 + .../apache/beam/sdk/transforms/ReshuffleTest.java | 139 + 5 files changed, 253 insertions(+), 47 deletions(-)
(beam) branch master updated (c49e9ec7dc7 -> ef0ee769e8f)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from c49e9ec7dc7 test revert _beam_schema_proto (#29754) add db0bb446aa0 Add outputWindowedValue capability to Java SDK new ef0ee769e8f Merge pull request #29616: Add outputWindowedValue capability to Java SDK The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runners/core/construction/SplittableParDo.java | 10 + .../construction/SplittableParDoNaiveBounded.java | 38 +++ ...TimeBoundedSplittableProcessElementInvoker.java | 25 +- .../apache/beam/runners/core/SimpleDoFnRunner.java | 79 +- .../org/apache/beam/sdk/transforms/Create.java | 191 + .../java/org/apache/beam/sdk/transforms/DoFn.java | 58 .../beam/sdk/transforms/DoFnOutputReceivers.java | 27 ++ .../org/apache/beam/sdk/transforms/DoFnTester.java | 22 ++ .../sdk/transforms/windowing/GlobalWindow.java | 5 + .../org/apache/beam/sdk/transforms/CreateTest.java | 81 ++ .../sdk/transforms/reflect/DoFnInvokersTest.java | 22 ++ .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 12 + .../apache/beam/fn/harness/FnApiDoFnRunner.java| 295 - .../bigquery/StorageApiWriteUnshardedRecords.java | 21 ++ .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 + .../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 11 + .../beam/sdk/io/pulsar/ReadFromPulsarDoFnTest.java | 13 + .../ReadFromSparkReceiverWithOffsetDoFnTest.java | 13 + 18 files changed, 923 insertions(+), 12 deletions(-)
(beam) 01/01: Merge pull request #29616: Add outputWindowedValue capability to Java SDK
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ef0ee769e8f99be62ea575fcea40da4d4584da8f Merge: c49e9ec7dc7 db0bb446aa0 Author: Kenneth Knowles AuthorDate: Wed Dec 13 15:56:22 2023 -0500 Merge pull request #29616: Add outputWindowedValue capability to Java SDK .../runners/core/construction/SplittableParDo.java | 10 + .../construction/SplittableParDoNaiveBounded.java | 38 +++ ...TimeBoundedSplittableProcessElementInvoker.java | 25 +- .../apache/beam/runners/core/SimpleDoFnRunner.java | 79 +- .../org/apache/beam/sdk/transforms/Create.java | 191 + .../java/org/apache/beam/sdk/transforms/DoFn.java | 58 .../beam/sdk/transforms/DoFnOutputReceivers.java | 27 ++ .../org/apache/beam/sdk/transforms/DoFnTester.java | 22 ++ .../sdk/transforms/windowing/GlobalWindow.java | 5 + .../org/apache/beam/sdk/transforms/CreateTest.java | 81 ++ .../sdk/transforms/reflect/DoFnInvokersTest.java | 22 ++ .../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 12 + .../apache/beam/fn/harness/FnApiDoFnRunner.java| 295 - .../bigquery/StorageApiWriteUnshardedRecords.java | 21 ++ .../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 + .../beam/sdk/io/kafka/ReadFromKafkaDoFnTest.java | 11 + .../beam/sdk/io/pulsar/ReadFromPulsarDoFnTest.java | 13 + .../ReadFromSparkReceiverWithOffsetDoFnTest.java | 13 + 18 files changed, 923 insertions(+), 12 deletions(-)
(beam) branch master updated (59930cd2584 -> 5ebd5d823d8)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 59930cd2584 Bump github.com/aws/aws-sdk-go-v2 from 1.23.4 to 1.23.5 in /sdks (#29611) add 4b4cc5d8088 Esacpe strings in integrationTestPipelineOptions in Java Tests new 5ebd5d823d8 Merge pull request #29620: Escape strings in integrationTestPipelineOptions in Java Tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/java_tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
(beam) 01/01: Merge pull request #29620: Escape strings in integrationTestPipelineOptions in Java Tests
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5ebd5d823d82ba7a03846f177a4981f7a3bfb7ce Merge: 59930cd2584 4b4cc5d8088 Author: Kenneth Knowles AuthorDate: Tue Dec 5 13:39:39 2023 -0500 Merge pull request #29620: Escape strings in integrationTestPipelineOptions in Java Tests .github/workflows/java_tests.yml | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam-site] branch updates_release_2.51.0 deleted (was a41442d7fe)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch updates_release_2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-site.git was a41442d7fe Update beam-site for release 2.51.0. The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
svn commit: r64613 - /dev/beam/2.51.0/ /release/beam/2.51.0/
Author: kenn Date: Wed Oct 18 14:44:13 2023 New Revision: 64613 Log: Beam release 2.51.0 Added: release/beam/2.51.0/ - copied from r64612, dev/beam/2.51.0/ Removed: dev/beam/2.51.0/
[beam] 01/01: Merge pull request #28656: Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit fddab443cca87ed5ceed7534d538edbf9fb9ea40 Merge: 6ca14e5ce46 7ac5c175871 Author: Kenneth Knowles AuthorDate: Mon Oct 16 13:40:26 2023 -0400 Merge pull request #28656: Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 14 +++--- sdks/java/container/license_scripts/dep_urls_java.yaml | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-)
[beam] branch master updated (6ca14e5ce46 -> fddab443cca)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6ca14e5ce46 Bump google.golang.org/api from 0.146.0 to 0.147.0 in /sdks (#28975) add 7ac5c175871 Update Google Cloud Java Libraries BOM new fddab443cca Merge pull request #28656: Update Google Cloud Java Libraries BOM from 26.22.0 to 26.23.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 14 +++--- sdks/java/container/license_scripts/dep_urls_java.yaml | 2 +- 2 files changed, 8 insertions(+), 8 deletions(-)
[beam] branch master updated (6758aeb0eb4 -> 41fb7e25163)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6758aeb0eb4 Support metric_name as list for perf alert tool (#28902) add 5e06f54371b Update .htaccess to redirect https://beam.apache.org/contribute/release-guide/ new 41fb7e25163 Merge pull request #28998: Update .htaccess to redirect the release guide The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: website/www/site/static/.htaccess | 2 ++ 1 file changed, 2 insertions(+)
[beam] 01/01: Merge pull request #28998: Update .htaccess to redirect the release guide
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 41fb7e25163b688112a9973f2807b331759c7926 Merge: 6758aeb0eb4 5e06f54371b Author: Kenneth Knowles AuthorDate: Mon Oct 16 12:48:21 2023 -0400 Merge pull request #28998: Update .htaccess to redirect the release guide website/www/site/static/.htaccess | 2 ++ 1 file changed, 2 insertions(+)
[beam] 01/01: Merge pull request #28800: [Website] Announce Beam 2.51.0
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 0ff6795e268e7deb5611000b068221b424d2b92e Merge: 6b3b3eb603f fb99630d1e0 Author: Kenneth Knowles AuthorDate: Wed Oct 11 22:46:01 2023 -0400 Merge pull request #28800: [Website] Announce Beam 2.51.0 CHANGES.md | 18 +- website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.51.0.md| 210 + .../www/site/content/en/get-started/downloads.md | 14 +- 4 files changed, 226 insertions(+), 18 deletions(-)
[beam] branch master updated (6b3b3eb603f -> 0ff6795e268)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 6b3b3eb603f Update go_tests.yml to cache (#28954) add fb99630d1e0 [Website] Announce Beam 2.51.0 new 0ff6795e268 Merge pull request #28800: [Website] Announce Beam 2.51.0 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: CHANGES.md | 18 +- website/www/site/config.toml | 2 +- website/www/site/content/en/blog/beam-2.51.0.md| 210 + .../www/site/content/en/get-started/downloads.md | 14 +- 4 files changed, 226 insertions(+), 18 deletions(-) create mode 100644 website/www/site/content/en/blog/beam-2.51.0.md
[beam] branch master updated: [#20970] Fix gRPC leak by closing ResidualSource at BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new aedfa461fb3 [#20970] Fix gRPC leak by closing ResidualSource at
BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
aedfa461fb3 is described below
commit aedfa461fb354e0e97d75694d2a117d9ad505602
Author: Minbo Bae <49642083+baemi...@users.noreply.github.com>
AuthorDate: Wed Oct 11 10:57:24 2023 -0700
[#20970] Fix gRPC leak by closing ResidualSource at
BoundedToUnboundedSourceAdapter.Reader#init() in Dataflow worker (#28548)
---
.../UnboundedReadFromBoundedSource.java| 14 +-
.../UnboundedReadFromBoundedSourceTest.java| 180 +
.../dataflow/worker/WorkerCustomSources.java | 7 +-
3 files changed, 198 insertions(+), 3 deletions(-)
diff --git
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
index 67697636a36..53fad782da9 100644
---
a/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
+++
b/runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSource.java
@@ -17,8 +17,8 @@
*/
package org.apache.beam.runners.core.construction;
-import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkNotNull;
+import static
org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkState;
import java.io.IOException;
import java.io.InputStream;
@@ -288,6 +288,15 @@ public class UnboundedReadFromBoundedSource extends
PTransform extends
PTransform extends
PTransform getCheckpointMark() {
+checkState(!closed, "getCheckpointMark() call on closed %s",
getClass().getName());
if (reader == null) {
// Reader hasn't started, checkpoint the residualSource.
return new Checkpoint<>(null /* residualElements */, residualSource);
diff --git
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
index cd4b49262fc..31f6842a42b 100644
---
a/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
+++
b/runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/UnboundedReadFromBoundedSourceTest.java
@@ -26,9 +26,15 @@ import java.io.File;
import java.io.FileOutputStream;
import java.io.IOException;
import java.io.OutputStream;
+import java.lang.ref.Reference;
+import java.lang.ref.ReferenceQueue;
+import java.lang.ref.WeakReference;
import java.nio.ByteBuffer;
import java.nio.channels.ReadableByteChannel;
+import java.util.Collections;
+import java.util.HashMap;
import java.util.List;
+import java.util.Map;
import java.util.NoSuchElementException;
import java.util.Random;
import
org.apache.beam.runners.core.construction.UnboundedReadFromBoundedSource.BoundedToUnboundedSourceAdapter;
@@ -69,10 +75,14 @@ import org.junit.rules.ExpectedException;
import org.junit.rules.TemporaryFolder;
import org.junit.runner.RunWith;
import org.junit.runners.JUnit4;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
/** Unit tests for {@link UnboundedReadFromBoundedSource}. */
@RunWith(JUnit4.class)
public class UnboundedReadFromBoundedSourceTest {
+ private static final Logger LOG =
+ LoggerFactory.getLogger(UnboundedReadFromBoundedSourceTest.class);
@Rule public TemporaryFolder tmpFolder = new TemporaryFolder();
@@ -280,6 +290,38 @@ public class UnboundedReadFromBoundedSourceTest {
unboundedSource.createReader(options, checkpoint).getCurrent();
}
+ @Test
+ public void testReadersClosedProperly() throws IOException {
+ManagedReaderBoundedSource boundedSource = new
ManagedReaderBoundedSource(0, 10);
+BoundedToUnboundedSourceAdapter unboundedSource =
+new BoundedToUnboundedSourceAdapter<>(boundedSource);
+PipelineOptions options = PipelineOptionsFactory.create();
+
+BoundedToUnboundedSourceAdapter.Reader reader =
+unboundedSource.createReader(options, new Checkpoint(null,
boundedSource));
+
+for (int i = 0; i < 3; ++i) {
+ if (i == 0) {
+assertTrue(reader.start());
+ } else {
+assertTrue(reader.advance());
+ }
+ assertEq
[beam] annotated tag v2.51.0 updated (cd653e33b34 -> 1ab140a97a2)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag v2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag v2.51.0 was modified! *** from cd653e33b34 (commit) to 1ab140a97a2 (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Wed Oct 11 12:37:30 2023 -0400 - Log - Release 2.51.0 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEA9uj5qvdBL/RVY3BbtVRqK4CRhwFAmUmz08ACgkQbtVRqK4C Rhx78xAAigWUs83oGVebIIz+eQV8SCSj8kuocCFb6PkQW3O97eZK0XqiKoAqA4jb pud6mNQmsycfidA/XPkiycWGIUZQFEfAIOsNkFtBq77FPT4UZvd1hAEYppKGwNlR KQ4tLV4TTmF1PfO3QYlJucmVH21NbHxCPbCuTCYqV/EE8D9tJqIgRjvZpGABs1Mv 8DjH8+7RhGyzy5di/vz1CE4oCZUKZ9Tv0VML7VjrndTid9gOxN7lw5dPLsv8A8kG xfxd+32sucOtu4oxg/a+AIqZsiZKEQkpgncKYl5UUV0nf9jVfNZ5qFZTRvKi+INl suFh4ETYXZJI/dF+YVbQ3IBDIBPEaSMffTRNudQD/H9fRLKpylX4WA592iHi5S2M QNdt+ydoZooNExMAbNaK0/1woDJb8eLIvU2uYdZg4oZ7171R4NIBrfsfACMjKv4q T0GdJQRvnBCPwl0heKSkGlDMLcpzoeebhM2o50waDOoMgXqUNE68S5q7a+98iGyU RxmBlMM4dGofA2lmYNS+40PHPLJadProhZgAG4yN9SZzhsifxSPPCDYIo3VMIDGh VaFktcpTmlmYpueU4WF3bOn3a4fCG3/2EB1gZQ2XBapfeE0Qw1jsqnIKQ2U/VIFy H1Hi8YGH1c/CkXiojjKh09LkkjI5m/45l2hrG5EtwVr94tY2oaI= =UQPu -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] annotated tag sdks/v2.51.0 updated (cd653e33b34 -> adf666acc0a)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag sdks/v2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag sdks/v2.51.0 was modified! *** from cd653e33b34 (commit) to adf666acc0a (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Wed Oct 11 12:37:01 2023 -0400 - Log - Release 2.51.0 -BEGIN PGP SIGNATURE- iQIzBAABCAAdFiEEA9uj5qvdBL/RVY3BbtVRqK4CRhwFAmUmzzQACgkQbtVRqK4C Rhy6+w/8ChFv+chOqD6LXqAQQSKM0CYe838xLO8oFtkwuqugh09XwNmuy1uM+SRe LUj2GTefQkjB3fff+XeC+iDh20TZyiy4+TK/q99a6i6o4l3ss7JFPZlKeOiuuHh0 P1MRdn3Oki4u+6D2j5ZtobBadLWcfIROHQ+fenjuhxQaRPRzIpU9PqRJwv3whM5u GJUR+tj84Pk7Sv7sEFB8gPf+FZgCQ+Gdxs0zLDHzEwLln8oiE0wrHqT2HGIwLukR 7Uske1AooC8OxRHoVbNeIQldMRbR8x009CzjIzwQzZBYK+vv5y/8UMWKrIy7VGjT 9QUWgDGj0kSdocGV5Xj/PO0jC5bQ2GDzv0wZ2AJNbtfxE53eazA9aoDZ7iTjy28e kHgZsTSKI6TAtI4k14uowrk/FMANBFVVim3fpJ4kemNpvyyLcF7iUQJSuJVP1bq2 mvwnPSSma7zxU8XJNJdMgYq7tOZIagIUNyf3b3yzLV/MtMB8mvnHApP61uIaZdU7 vXZ1Rq6vE4uF4OHH+sX1NhkD3xSC5ozbvjiw75KKF3hoU8M8hwCXftmQDM6qcDt4 MDE9TmP5NSMaAufqHBcBeZUJzygz5zDNCSRucfJmAr+mawRQTyUZehRKy9R2YrG+ 9933sbKTNHJV/bsHQ6Ukfdj4b3JPa6b//RQgdGnSBNwRkJfXsOs= =5Yez -END PGP SIGNATURE- --- No new revisions were added by this update. Summary of changes:
[beam] branch release-2.51.0 updated (e600ea854a1 -> 33b41689c7b)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git from e600ea854a1 Cherry-pick #28665 to 2.51.0 release branch: Add GitHub Action workflows to mass_commit add b49296a11f4 Fix Jpms test (#28726) new 33b41689c7b Cherrypick #28726 to 2.51.0 release branch: fix JPMS tests The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/testing/jpms-tests/build.gradle | 40 +-- 1 file changed, 16 insertions(+), 24 deletions(-)
[beam] 01/01: Cherrypick #28726 to 2.51.0 release branch: fix JPMS tests
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 33b41689c7b88a7246c4be2c1740f2f2d5cccec1 Merge: e600ea854a1 b49296a11f4 Author: Kenneth Knowles AuthorDate: Fri Oct 6 18:21:05 2023 -0400 Cherrypick #28726 to 2.51.0 release branch: fix JPMS tests sdks/java/testing/jpms-tests/build.gradle | 40 +-- 1 file changed, 16 insertions(+), 24 deletions(-)
[beam] branch release-2.51.0 updated (2420c904f66 -> e600ea854a1)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git from 2420c904f66 Cherry picking PR #28618 into 2.51.0 (setting numShards for Python BigQuery xlang) (#28631) add 0f3e801325c Merge pull request #28665: Add GitHub Action workflows to mass_commit add e600ea854a1 Cherry-pick #28665 to 2.51.0 release branch: Add GitHub Action workflows to mass_commit No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python.yml | 2 +- release/src/main/scripts/github_actions_jobs.txt | 33 ++-- release/src/main/scripts/mass_comment.py | 3 ++- 3 files changed, 28 insertions(+), 10 deletions(-)
[beam] 01/01: Merge pull request #28822: Add python wheel staging to GitHub Actions build_release_candidate
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 795d366c395bf465b49e8f5765b8fc18ada64f3f Merge: ce217d36b13 57821c191d3 Author: Kenneth Knowles AuthorDate: Fri Oct 6 16:19:16 2023 -0400 Merge pull request #28822: Add python wheel staging to GitHub Actions build_release_candidate .github/workflows/build_release_candidate.yml | 76 +- .../scripts/download_github_actions_artifacts.py | 15 +++-- 2 files changed, 81 insertions(+), 10 deletions(-)
[beam] branch master updated (ce217d36b13 -> 795d366c395)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from ce217d36b13 Merge pull request #28855 [YAML] Schemify avroio and add a yaml provider. add 57821c191d3 Add python wheel staging to GitHub Actions build_release_candidate new 795d366c395 Merge pull request #28822: Add python wheel staging to GitHub Actions build_release_candidate The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/build_release_candidate.yml | 76 +- .../scripts/download_github_actions_artifacts.py | 15 +++-- 2 files changed, 81 insertions(+), 10 deletions(-)
[beam] branch master updated (af9da3151d9 -> 84b9336cffc)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from af9da3151d9 Merge pull request #28836: Update release guide somewhat according to 2.51.0 release experience add 84b9336cffc Merge pull request #28665: Add GitHub Action workflows to mass_commit No new revisions were added by this update. Summary of changes: .github/workflows/beam_PostCommit_Python.yml | 2 +- release/src/main/scripts/github_actions_jobs.txt | 33 ++-- release/src/main/scripts/mass_comment.py | 3 ++- 3 files changed, 28 insertions(+), 10 deletions(-)
[beam] branch master updated (b9398e0e3e9 -> af9da3151d9)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from b9398e0e3e9 Add Java JMH workflow (#28860) add 14950aab794 Update release guide somewhat according to 2.51.0 release experience add af9da3151d9 Merge pull request #28836: Update release guide somewhat according to 2.51.0 release experience No new revisions were added by this update. Summary of changes: contributor-docs/release-guide.md | 1092 +++-- 1 file changed, 574 insertions(+), 518 deletions(-)
[beam] branch master updated (13e4da3c2d7 -> 5d49733bf15)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 13e4da3c2d7 Bump google.golang.org/api from 0.143.0 to 0.144.0 in /sdks (#28839) add 27a9f92d92f Do not rewrite project version in BeamModulePlugin new 5d49733bf15 Merge pull request #28843: Do not rewrite project version in BeamModulePlugin The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 7 --- 1 file changed, 7 deletions(-)
[beam] 01/01: Merge pull request #28843: Do not rewrite project version in BeamModulePlugin
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5d49733bf1585889ddadd33ae21d498c36fbd9a8 Merge: 13e4da3c2d7 27a9f92d92f Author: Kenneth Knowles AuthorDate: Thu Oct 5 17:15:14 2023 -0400 Merge pull request #28843: Do not rewrite project version in BeamModulePlugin .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 7 --- 1 file changed, 7 deletions(-)
svn commit: r64320 - /dev/beam/2.51.0/python/
Author: kenn Date: Tue Oct 3 15:50:01 2023 New Revision: 64320 Log: Staging Python artifacts for Apache Beam 2.51.0 RC1 Added: dev/beam/2.51.0/python/ dev/beam/2.51.0/python/apache-beam-2.51.0.zip (with props) dev/beam/2.51.0/python/apache-beam-2.51.0.zip.asc dev/beam/2.51.0/python/apache-beam-2.51.0.zip.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-macosx_10_9_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-macosx_10_9_x86_64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-macosx_10_9_x86_64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win32.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win32.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win32.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win_amd64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win_amd64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp310-cp310-win_amd64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-macosx_10_9_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-macosx_10_9_x86_64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-macosx_10_9_x86_64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win32.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win32.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win32.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win_amd64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win_amd64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp311-cp311-win_amd64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-macosx_10_9_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-macosx_10_9_x86_64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-macosx_10_9_x86_64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_aarch64.manylinux2014_aarch64.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl.asc dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_i686.manylinux2014_i686.whl.sha512 dev/beam/2.51.0/python/apache_beam-2.51.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (with props) dev/beam/2.51.0/python/apache_beam-2.51.0-cp38
[beam] branch damccorm-patch-1 deleted (was 5fd870e1023)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch damccorm-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git was 5fd870e1023 Fix dockerhub task The revisions that were on this branch are still contained in other references; therefore, this change does not discard any commits from the repository.
[beam] 01/01: Merge pull request #28766: Fix dockerhub task
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 282d027f16214c9581b1d19073252425cff53ee9 Merge: bf1e8297582 5fd870e1023 Author: Kenneth Knowles AuthorDate: Mon Oct 2 11:21:57 2023 -0400 Merge pull request #28766: Fix dockerhub task .github/workflows/build_release_candidate.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (bf1e8297582 -> 282d027f162)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from bf1e8297582 Report total active threads (#28513) add 5fd870e1023 Fix dockerhub task new 282d027f162 Merge pull request #28766: Fix dockerhub task The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .github/workflows/build_release_candidate.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam-site] branch updates_release_2.51.0 created (now a41442d7fe)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch updates_release_2.51.0 in repository https://gitbox.apache.org/repos/asf/beam-site.git at a41442d7fe Update beam-site for release 2.51.0. No new revisions were added by this update.
svn commit: r64296 - in /dev/beam/2.51.0: ./ apache-beam-2.51.0-source-release.zip apache-beam-2.51.0-source-release.zip.asc apache-beam-2.51.0-source-release.zip.sha512
Author: kenn Date: Mon Oct 2 14:35:30 2023 New Revision: 64296 Log: Staging Java artifacts for Apache Beam 2.51.0 RC1 Added: dev/beam/2.51.0/ dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip (with props) dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.asc dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.sha512 Added: dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip == Binary file - no diff available. Propchange: dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.asc == --- dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.asc (added) +++ dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.asc Mon Oct 2 14:35:30 2023 @@ -0,0 +1,17 @@ +-BEGIN PGP SIGNATURE- + +iQJMBAABCgA2FiEEkTwzkqdwx4HtxN2r0gMW9xIhNCIFAmUa1S8YHHByaXZhdGVA +YmVhbS5hcGFjaGUub3JnAAoJENIDFvcSITQiGwAQALWrW6ArqAXCclsfG+YuecZa +DDRQWAF15x/CwqnkUSesUtYrBr0DZbuob/zPJAch88k9EtfZV8cEFapL+3iuD2IL ++9lRUWGo4NzEE//QW+8ONYHd+9pNHB3gRXt44VCFx7TEHYznoIW12hjp30IGELBf +LErNrA5vMqw4ZU3Ab66p4+RjdrQJCJnzqti50hTdD/H7vB/KZ+8Mzw8UTUKtL1w/ +aGgROgG0rMH+Nv/DORCRtf/HxFzzAKZNtCzhQLEl0c5yD0KE5e/Seaf4h4VeGNeW +mWyD1PH6vqOt38C1A8NJ1DZRqP1Te1s7Z3lQTIGYfYjScKjfk0fIATrMbtG+HaZA +zKBoToe5jxhlG2pTI+hxayYg+VaWXOSEW4EH9KcmMmXneamud5Mx1B86nlPvP35j +xslE9POo06j4KsG2u/aUjpxfMb8nKb6kYLLvsnthfMfkGP2mzyWJkANxnEheqQYL +V/o34rqM4+/65FI51BfuF37gg68pOdWmQX+V8MqN1vD5opJK04BVZ6MUPhV49xER +1wak4Phg/hppnuTlJ4g5QLSZl6l151t2Vw5O4L3Mn+VfFb94/4Lw+8f8GbonIzrV +PEumxnM3ELE2g6aY6rnfkDSflkySY2jT7CabG7aAsVdUdybA/Z4VX8mQW3t0q6S/ +PxNdGv48mwVglsMU9tka +=8oBm +-END PGP SIGNATURE- Added: dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.sha512 == --- dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.sha512 (added) +++ dev/beam/2.51.0/apache-beam-2.51.0-source-release.zip.sha512 Mon Oct 2 14:35:30 2023 @@ -0,0 +1 @@ +5310634e7cf8e7c9eb9510f208a8f5a1bae7c189d542d72eefa4711f38d9951c3958afa48f3d8a19dc945d96970fc08d09aaeb4a50063514c06cb70d28c15120 apache-beam-2.51.0-source-release.zip
[beam] branch master updated (2b20e26bb3b -> f2194f6b4c8)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 2b20e26bb3b Restrict build_runner_image to pull_request instead of pull_request_target (#28745) add f2194f6b4c8 Temporarily Ignore 2 FhirIO Integration Tests (#28717) No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 3 +++ 1 file changed, 3 insertions(+)
[beam] annotated tag v2.51.0-RC1 updated (cd653e33b34 -> 91d4283aa53)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag v2.51.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag v2.51.0-RC1 was modified! *** from cd653e33b34 (commit) to 91d4283aa53 (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Fri Sep 29 10:59:32 2023 -0400 - Log - v2.51.0-RC1 --- No new revisions were added by this update. Summary of changes:
[beam] annotated tag sdks/v2.51.0-RC1 updated (cd653e33b34 -> f3c3babdcae)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to annotated tag sdks/v2.51.0-RC1 in repository https://gitbox.apache.org/repos/asf/beam.git *** WARNING: tag sdks/v2.51.0-RC1 was modified! *** from cd653e33b34 (commit) to f3c3babdcae (tag) tagging cd653e33b342bd09c76c2bbaca12597fec5b4a2c (commit) replaces jupyterlab-sidepanel-v3.0.0 by Kenneth Knowles on Fri Sep 29 10:59:32 2023 -0400 - Log - Go SDK v2.51.0-RC1 --- No new revisions were added by this update. Summary of changes:
[beam] branch release-2.51.0 updated: Cherry picking PR #28618 into 2.51.0 (setting numShards for Python BigQuery xlang) (#28631)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch release-2.51.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.51.0 by this push:
new 2420c904f66 Cherry picking PR #28618 into 2.51.0 (setting numShards
for Python BigQuery xlang) (#28631)
2420c904f66 is described below
commit 2420c904f66a8882eb454416f4029604c9160502
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Wed Sep 27 17:43:54 2023 -0400
Cherry picking PR #28618 into 2.51.0 (setting numShards for Python BigQuery
xlang) (#28631)
---
...Commit_Python_CrossLanguage_Gcp_Dataflow.groovy | 2 +-
...stCommit_Python_CrossLanguage_Gcp_Direct.groovy | 2 +-
...ueryStorageWriteApiSchemaTransformProvider.java | 37 ++
.../io/external/xlang_bigqueryio_it_test.py| 44 ++
sdks/python/apache_beam/io/gcp/bigquery.py | 9 +
.../documentation/io/built-in/google-bigquery.md | 2 +-
6 files changed, 69 insertions(+), 27 deletions(-)
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
index d1ee27088c7..1280fcb4e23 100644
---
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
+++
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Dataflow.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Dataflow',
'Run Python_Xlang_Gcp_Dataflow PostCommit', 'Python_Xlang_Gcp_Dataflow
(\"Run Python_Xlang_Gcp_Dataflow PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Dataflow
runner. \"Run Python_Xlang_Gcp_Dataflow PostCommit\"')
// Set common parameters.
diff --git
a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
index 438b735fba7..e4bf771be1a 100644
--- a/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
+++ b/.test-infra/jenkins/job_PostCommit_Python_CrossLanguage_Gcp_Direct.groovy
@@ -28,7 +28,7 @@ import static
PythonTestProperties.CROSS_LANGUAGE_VALIDATES_RUNNER_PYTHON_VERSIO
// Collects tests with the @pytest.mark.uses_gcp_java_expansion_service
decorator
PostcommitJobBuilder.postCommitJob('beam_PostCommit_Python_Xlang_Gcp_Direct',
'Run Python_Xlang_Gcp_Direct PostCommit', 'Python_Xlang_Gcp_Direct (\"Run
Python_Xlang_Gcp_Direct PostCommit\")', this) {
- description('Runs end-to-end cross language GCP IO tests on the Direct
runner.')
+ description('Runs end-to-end cross language GCP IO tests on the Direct
runner. \"Run Python_Xlang_Gcp_Direct PostCommit\"')
// Set common parameters.
commonJobProperties.setTopLevelMainJobProperties(delegate)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
index e4461793011..1b9eb309ec4 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/providers/BigQueryStorageWriteApiSchemaTransformProvider.java
@@ -176,6 +176,13 @@ public class BigQueryStorageWriteApiSchemaTransformProvider
!Strings.isNullOrEmpty(this.getErrorHandling().getOutput()),
invalidConfigMessage + "Output must not be empty if error handling
specified.");
}
+
+ if (this.getAutoSharding() != null && this.getAutoSharding()) {
+checkArgument(
+this.getNumStreams() == 0,
+invalidConfigMessage
++ "Cannot set a fixed number of streams when auto-sharding is
enabled. Please pick only one of the two options.");
+ }
}
/**
@@ -218,11 +225,17 @@ public class
BigQueryStorageWriteApiSchemaTransformProvider
public abstract Boolean getUseAtLeastOnceSemantics();
@SchemaFieldDescription(
-"This option enables using a dynamically determined number of shards
to write to "
+"This option enables using a dynamically determined number of Storage
Write API streams to write to "
+ "BigQuer
[beam] branch release-2.51.0 updated: CP for #28624 into release 2.51.0 (Bigtable Python timestamp bug fix) (#28634)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a commit to branch release-2.51.0
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/release-2.51.0 by this push:
new 70f4a1ae26d CP for #28624 into release 2.51.0 (Bigtable Python
timestamp bug fix) (#28634)
70f4a1ae26d is described below
commit 70f4a1ae26d71fc89c3f5a99cae4e417cd1ee446
Author: Ahmed Abualsaud <65791736+ahmedab...@users.noreply.github.com>
AuthorDate: Wed Sep 27 17:42:18 2023 -0400
CP for #28624 into release 2.51.0 (Bigtable Python timestamp bug fix)
(#28634)
---
.../BigtableWriteSchemaTransformProvider.java | 13 +++--
.../BigtableWriteSchemaTransformProviderIT.java | 19 +++
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +++
sdks/python/apache_beam/io/gcp/bigtableio_it_test.py | 18 ++
4 files changed, 39 insertions(+), 18 deletions(-)
diff --git
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
index d38bdae2f09..b99b69621a8 100644
---
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
+++
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProvider.java
@@ -179,12 +179,13 @@ public class BigtableWriteSchemaTransformProvider
.setColumnQualifier(
ByteString.copyFrom(ofNullable(mutation.get("column_qualifier")).get()))
.setFamilyNameBytes(
-
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()));
-if (mutation.containsKey("timestamp_micros")) {
- setMutation =
- setMutation.setTimestampMicros(
-
Longs.fromByteArray(ofNullable(mutation.get("timestamp_micros")).get()));
-}
+
ByteString.copyFrom(ofNullable(mutation.get("family_name")).get()))
+// Use timestamp if provided, else default to -1 (current
Bigtable server time)
+.setTimestampMicros(
+mutation.containsKey("timestamp_micros")
+? Longs.fromByteArray(
+
ofNullable(mutation.get("timestamp_micros")).get())
+: -1);
bigtableMutation =
Mutation.newBuilder().setSetCell(setMutation.build()).build();
break;
case "DeleteFromColumn":
diff --git
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
index 14bb04b0315..1a60fe661b5 100644
---
a/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
+++
b/sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteSchemaTransformProviderIT.java
@@ -154,8 +154,8 @@ public class BigtableWriteSchemaTransformProviderIT {
public void testSetMutationsExistingColumn() {
RowMutation rowMutation =
RowMutation.create(tableId, "key-1")
-.setCell(COLUMN_FAMILY_NAME_1, "col_a", "val-1-a")
-.setCell(COLUMN_FAMILY_NAME_2, "col_c", "val-1-c");
+.setCell(COLUMN_FAMILY_NAME_1, "col_a", 1000, "val-1-a")
+.setCell(COLUMN_FAMILY_NAME_2, "col_c", 1000, "val-1-c");
dataClient.mutateRow(rowMutation);
List> mutations = new ArrayList<>();
@@ -165,13 +165,15 @@ public class BigtableWriteSchemaTransformProviderIT {
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-a".getBytes(StandardCharsets.UTF_8),
"column_qualifier", "col_a".getBytes(StandardCharsets.UTF_8),
-"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8)));
+"family_name",
COLUMN_FAMILY_NAME_1.getBytes(StandardCharsets.UTF_8),
+"timestamp_micros", Longs.toByteArray(2000)));
mutations.add(
ImmutableMap.of(
"type", "SetCell".getBytes(StandardCharsets.UTF_8),
"value", "new-val-1-c".getBytes(StandardCharsets.UTF_8),
"column_qualifier&
[beam] branch master updated (a7e12db9b59 -> 70979bfa361)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
from a7e12db9b59 Filter out some warnings that aren't useful in pytest
(#28622)
add 70979bfa361 Move committer onboarding and committer guide to
contributor-docs (#28643)
No new revisions were added by this update.
Summary of changes:
.../committer-guide.md | 90 --
contributor-docs/committer-onboarding.md | 37 +
.../partials/section-menu/en/contribute.html | 1 -
3 files changed, 88 insertions(+), 40 deletions(-)
rename {website/www/site/content/en/contribute =>
contributor-docs}/committer-guide.md (58%)
create mode 100644 contributor-docs/committer-onboarding.md
[beam] branch master updated (d52b077ad50 -> c86a9bb5dbf)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from d52b077ad50 Bump github.com/aws/aws-sdk-go-v2/feature/s3/manager in /sdks (#28641) add c86a9bb5dbf Move Python dep upgrade instructions into release guide (#28648) No new revisions were added by this update. Summary of changes: contributor-docs/release-guide.md | 11 --- 1 file changed, 8 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #28658: [release-2.51.0] Cherrypick #28571 to release branch.
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git commit 34ff286c2b3bd6a3b2fdfd0ca7b683ca0dda547f Merge: 22abcde2aa5 94adacdcfb1 Author: Kenneth Knowles AuthorDate: Tue Sep 26 11:54:41 2023 -0400 Merge pull request #28658: [release-2.51.0] Cherrypick #28571 to release branch. sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py | 3 +-- sdks/python/pytest.ini | 2 +- sdks/python/test-suites/dataflow/common.gradle | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-)
[beam] branch release-2.51.0 updated (22abcde2aa5 -> 34ff286c2b3)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch release-2.51.0 in repository https://gitbox.apache.org/repos/asf/beam.git from 22abcde2aa5 Merge pull request #28628: [Cherry-pick #28625 for 2.51.0] Update Python base container deps add 94adacdcfb1 Use a single marker for Vertex AI tests to not run them twice. new 34ff286c2b3 Merge pull request #28658: [release-2.51.0] Cherrypick #28571 to release branch. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/ml/inference/vertex_ai_inference_it_test.py | 3 +-- sdks/python/pytest.ini | 2 +- sdks/python/test-suites/dataflow/common.gradle | 2 +- 3 files changed, 3 insertions(+), 4 deletions(-)
[beam] branch master updated (27f589e34f1 -> cf0a1259509)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 27f589e34f1 Fix Vertex AI postcommit suite to run the tests only once. (#28571) add ca9b44d6923 Move instructions for upgrading Go version to release guide new cf0a1259509 Merge pull request #28647: Move instructions for upgrading Go version to release guide The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: contributor-docs/release-guide.md | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #28647: Move instructions for upgrading Go version to release guide
This is an automated email from the ASF dual-hosted git repository. kenn pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit cf0a1259509e0530ceae029e308d7e2e400f4c17 Merge: 27f589e34f1 ca9b44d6923 Author: Kenneth Knowles AuthorDate: Tue Sep 26 08:50:25 2023 -0400 Merge pull request #28647: Move instructions for upgrading Go version to release guide contributor-docs/release-guide.md | 10 +++--- 1 file changed, 7 insertions(+), 3 deletions(-)
