This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from b39089870e7 Remove unused ToRow, GroupByKey, mark WithSchema as
experimental. (#28504)
add 4184f5ea521 Update HuggingFace api doc and add text2audio pipeline
task (#28474)
add 79d0a8d1109 tooltips, footer, humanize (#28538)
add 635f9ef5a0e Bump github.com/testcontainers/testcontainers-go in /sdks
(#28550)
add a70a5845230 Add schema-aware text file reading and writing. (#28486)
add 275b177fa41 Bump google.golang.org/api from 0.141.0 to 0.142.0 in
/sdks (#28549)
add 93c3224be6d Enable remote gradle cache and build scan for GitHub
Actions workflows (#28539)
add a7335cb0b3c Remove deprecated week and weekofyear for Pandas 2 (#28492)
add 932744801e6 Change pd.core.strings.StringMethods for Pandas 2
compatability. (#28455)
add 840c85883fc [#28187][prism] Basic cross language support. (#28545)
add 7dfc0c03b1b Sync it framework (#28541)
add 93de970ea66 add setJoinSubsetType to inject joinSubsetType to
BeamSqlSeekableTable (#28477)
add f676d93030c When comparing Series, sort the values in Dataframe tests
(#28557)
add ef0d8d4041c Label Python external SchemaTransform with its URN (#28540)
add 955cd8920f4 Bump actions/checkout from 3 to 4 (#28552)
add 3b7e1dcc4e6 Typescript changes for Beam 2.51.0 (#28553)
add 534f93acd18 Introduce PeriodicImpulse.stopAfter() (#28503)
add 0b131c9ae7c Change handling of copy=None defaults for Pandas 2 (#28523)
add 5fb13e05d54 Make launcher method public in LoadTestBase. (#28568)
add 17db78c10dc [prism] Auto remove containers after stop. (#28570)
add 1d94f5ffb84 Fix test for new group keys behavior for Pandas 2 (#28566)
add c5e6c7962e6 Refactor and cleanup yaml MapToFields. (#28462)
add 451561784c6 Moving to 2.52.0-SNAPSHOT on master branch.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 1 +
.github/actions/setup-action/action.yml | 2 +
.../action.yml | 53 ++-
.../workflows/beam_PostCommit_Go_Dataflow_ARM.yml | 16 +-
.../beam_PostCommit_Java_Avro_Versions.yml | 5 +
.../beam_PostCommit_Java_Examples_Dataflow_ARM.yml | 15 +-
.../beam_PostCommit_Java_Examples_Flink.yml | 16 +-
.../beam_PostCommit_Java_IO_Performance_Tests.yml | 11 +-
.../beam_PostCommit_Java_Nexmark_Dataflow.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Dataflow_V2.yml | 2 +-
...am_PostCommit_Java_Nexmark_Dataflow_V2_Java.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Direct.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Flink.yml | 2 +-
.../beam_PostCommit_Java_Nexmark_Spark.yml | 2 +-
.../beam_PostCommit_Java_ValidatesRunner_Flink.yml | 16 +-
.github/workflows/beam_PostCommit_Javadoc.yml | 2 +-
.../beam_PostCommit_Python_Examples_Dataflow.yml | 14 +-
.../beam_PostCommit_Python_Examples_Direct.yml | 14 +-
.../beam_PostCommit_Python_Examples_Flink.yml | 14 +-
.../beam_PostCommit_Python_Examples_Spark.yml | 14 +-
.../beam_PostCommit_Python_MongoDBIO_IT.yml | 14 +-
.../beam_PostCommit_Python_Nexmark_Direct.yml | 2 +-
...stCommit_Python_ValidatesContainer_Dataflow.yml | 14 +-
..._Python_ValidatesContainer_Dataflow_With_RC.yml | 16 +-
..._PostCommit_Python_ValidatesRunner_Dataflow.yml | 14 +-
...eam_PostCommit_Python_ValidatesRunner_Flink.yml | 14 +-
...eam_PostCommit_Python_ValidatesRunner_Samza.yml | 14 +-
...eam_PostCommit_Python_ValidatesRunner_Spark.yml | 14 +-
.../workflows/beam_PreCommit_CommunityMetrics.yml | 11 +-
.github/workflows/beam_PreCommit_Go.yml | 16 +-
.github/workflows/beam_PreCommit_GoPortable.yml | 8 +-
.github/workflows/beam_PreCommit_GoPrism.yml | 8 +-
.github/workflows/beam_PreCommit_ItFramework.yml | 12 +-
.github/workflows/beam_PreCommit_Java.yml | 5 +
.../beam_PreCommit_Java_Debezium_IO_Direct.yml | 5 +
...beam_PreCommit_Java_ElasticSearch_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Examples_Dataflow.yml | 12 +-
...eam_PreCommit_Java_Examples_Dataflow_Java17.yml | 5 +
...Commit_Java_File-schema-transform_IO_Direct.yml | 5 +
.../beam_PreCommit_Java_Flink_Versions.yml | 16 +-
.../beam_PreCommit_Java_PVR_Flink_Batch.yml | 5 +
.github/workflows/beam_PreCommit_Python.yml | 14 +-
.github/workflows/beam_PreCommit_PythonDocker.yml | 19 +-
.github/workflows/beam_PreCommit_PythonDocs.yml | 16 +-
.../workflows/beam_PreCommit_PythonFormatter.yml | 16 +-
.github/workflows/beam_PreCommit_PythonLint.yml | 21 +-
.../workflows/beam_PreCommit_Python_Coverage.yml | 16 +-
.../workflows/beam_PreCommit_Python_Dataframes.yml | 14 +-
.../workflows/beam_PreCommit_Python_Examples.yml | 14 +-
.../beam_PreCommit_Python_Integration.yml | 14 +-
.../workflows/beam_PreCommit_Python_PVR_Flink.yml | 5 +
.../workflows/beam_PreCommit_Python_Runners.yml | 14 +-
.../workflows/beam_PreCommit_Python_Transforms.yml | 14 +-
.github/workflows/beam_PreCommit_RAT.yml | 11 +-
.github/workflows/beam_PreCommit_Spotless.yml | 5 +
.github/workflows/beam_PreCommit_Typescript.yml | 12 +-
.github/workflows/beam_PreCommit_Website.yml | 11 +-
.../workflows/beam_PreCommit_Website_Stage_GCS.yml | 12 +-
.github/workflows/beam_PreCommit_Whitespace.yml | 16 +-
...beam_Python_ValidatesContainer_Dataflow_ARM.yml | 8 +-
.github/workflows/beam_Release_NightlySnapshot.yml | 11 +-
.../beam_Release_Python_NightlySnapshot.yml | 16 +-
.github/workflows/java_tests.yml | 27 +-
.github/workflows/playground_backend_precommit.yml | 15 +-
.../workflows/tour_of_beam_backend_integration.yml | 9 +-
.github/workflows/update_python_dependencies.yml | 8 +-
CHANGES.md | 1 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
gradle.properties | 4 +-
.../it/cassandra/matchers/CassandraAsserts.java | 2 +-
.../apache/beam/it/common/utils/PipelineUtils.java | 21 +-
.../beam/it/common/utils/PipelineUtilsTest.java | 8 +-
.../it/elasticsearch/ElasticsearchUtilsTest.java | 2 +-
.../org/apache/beam/it/gcp/IOLoadTestBase.java | 2 +-
.../java/org/apache/beam/it/gcp/LoadTestBase.java | 55 ++-
.../it/gcp/bigquery/BigQueryResourceManager.java | 1 +
.../it/gcp/bigtable/BigtableResourceManager.java | 33 ++
.../gcp/bigtable/BigtableResourceManagerUtils.java | 2 +-
.../it/gcp/dataflow/AbstractPipelineLauncher.java | 13 +-
.../it/gcp/dataflow/DefaultPipelineLauncher.java | 6 +-
.../beam/it/gcp/dataflow/DirectRunnerClient.java | 6 +-
.../beam/it/gcp/datagenerator/DataGenerator.java | 54 +--
.../gcp/datastore/matchers/DatastoreAsserts.java | 3 +-
.../apache/beam/it/gcp/dlp/DlpResourceManager.java | 5 +-
.../apache/beam/it/gcp/kms/KMSResourceManager.java | 5 +-
.../beam/it/gcp/monitoring/MonitoringClient.java | 12 +-
.../beam/it/gcp/pubsub/PubsubResourceManager.java | 45 ++-
.../src/main/resources/test-artifact.json | 1 +
.../apache/beam/it/gcp/bigquery/BigQueryIOLT.java | 18 +-
.../apache/beam/it/gcp/bigtable/BigTableIOLT.java | 22 +-
.../gcp/bigtable/BigtableResourceManagerTest.java | 1 +
.../it/gcp/dataflow/ClassicTemplateClientTest.java | 14 +-
.../it/gcp/dataflow/FlexTemplateClientTest.java | 14 +-
.../apache/beam/it/gcp/storage/FileBasedIOLT.java | 14 +-
.../it/gcp/storage/GcsResourceManagerTest.java | 2 +-
.../src/test/resources/test-artifact.txt | 1 -
.../apache/beam/it/jdbc/MSSQLResourceManager.java | 7 +-
.../apache/beam/it/jdbc/MySQLResourceManager.java | 4 +-
.../apache/beam/it/jdbc/OracleResourceManager.java | 8 +-
.../beam/it/jdbc/PostgresResourceManager.java | 9 +-
.../apache/beam/it/kafka/KafkaResourceManager.java | 11 +-
.../java/org/apache/beam/it/kafka/KafkaIOLT.java | 4 +-
.../beam/it/mongodb/MongoDBResourceManager.java | 10 +-
.../beam/it/mongodb/matchers/MongoDBAsserts.java | 2 +-
.../apache/beam/it/neo4j/Neo4jResourceManager.java | 11 +-
.../beam/it/splunk/SplunkResourceManager.java | 10 +-
.../TestContainerResourceManager.java | 15 +-
.../beam/it/truthmatchers/LaunchInfoSubject.java | 2 +-
.../beam/it/truthmatchers/RecordsSubject.java | 2 +-
release/src/main/scripts/jenkins_jobs.txt | 16 +-
.../beam/runners/dataflow/DataflowRunner.java | 2 +-
sdks/go.mod | 23 +-
sdks/go.sum | 49 ++-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 6 +
.../beam/runners/prism/internal/environments.go | 3 +
sdks/go/pkg/beam/runners/prism/internal/execute.go | 52 +--
.../prism/internal/jobservices/management.go | 20 +-
sdks/go/pkg/beam/runners/prism/internal/stage.go | 16 +-
.../pkg/beam/runners/prism/internal/urns/urns.go | 2 +
.../runners/prism/internal/web/assets/style.css | 58 +++-
.../pkg/beam/runners/prism/internal/web/debugz.go | 25 +-
.../beam/runners/prism/internal/web/debugz.html | 10 +-
.../pkg/beam/runners/prism/internal/web/index.html | 52 +--
.../beam/runners/prism/internal/worker/bundle.go | 2 +-
.../beam/runners/prism/internal/worker/worker.go | 24 +-
.../runners/prism/internal/worker/worker_test.go | 14 +-
sdks/go/test/integration/integration.go | 4 +-
.../beam/sdk/transforms/PeriodicImpulse.java | 98 +++++-
.../beam/sdk/transforms/PeriodicSequence.java | 3 +
.../sdk/extensions/sql/BeamSqlSeekableTable.java | 9 +-
.../sql/impl/transform/BeamJoinTransforms.java | 2 +-
.../impl/rel/BeamSideInputLookupJoinRelTest.java | 9 +
.../io/gcp/bigquery/BigQueryIOStorageWriteIT.java | 42 ++-
sdks/python/apache_beam/dataframe/frame_base.py | 8 +
.../apache_beam/dataframe/frame_base_test.py | 15 +
sdks/python/apache_beam/dataframe/frames.py | 39 +--
sdks/python/apache_beam/dataframe/frames_test.py | 9 +-
.../ml/inference/huggingface_inference.py | 18 +-
sdks/python/apache_beam/transforms/core.py | 8 +
sdks/python/apache_beam/transforms/external.py | 12 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/apache_beam/yaml/readme_test.py | 23 +-
sdks/python/apache_beam/yaml/standard_io.yaml | 2 +
sdks/python/apache_beam/yaml/yaml_io.py | 26 ++
sdks/python/apache_beam/yaml/yaml_mapping.md | 35 +-
sdks/python/apache_beam/yaml/yaml_mapping.py | 367 ++++++++++++---------
sdks/python/apache_beam/yaml/yaml_mapping_test.py | 32 +-
sdks/python/apache_beam/yaml/yaml_provider.py | 7 +-
sdks/python/apache_beam/yaml/yaml_transform.py | 15 +
.../python/apache_beam/yaml/yaml_transform_test.py | 14 +-
sdks/python/apache_beam/yaml/yaml_udf_test.py | 38 ++-
sdks/typescript/package-lock.json | 238 +------------
sdks/typescript/package.json | 6 +-
.../typescript/src/apache_beam/runners/dataflow.ts | 2 +
155 files changed, 1374 insertions(+), 1307 deletions(-)
rename .github/actions/{setup-self-hosted-action =>
setup-environment-action}/action.yml (50%)
create mode 100644
it/google-cloud-platform/src/main/resources/test-artifact.json
delete mode 100644
it/google-cloud-platform/src/test/resources/test-artifact.txt
