This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from 775d371762d Bump Legacy dataflow container image tag (#23625)
add 5440bae895f Align translation logging for Spark dataset runner with
rdd runner for better comparison
add 7e7cbd6cfd3 Merge pull request #23623: Align translation logging for
Spark dataset runner with rdd runner
add 5ecfeabcd7d Fix SparkSessionFactory to not fail when using Spark
master local[*]
add c96e8cd5f31 Merge pull request #23622: Fix SparkSessionFactory to not
fail when using Spark master local[*]
add a079fd1eee7 Minor improvements to the tpcds gradle build for Spark
add b784c988643 Merge pull request #23621: Minor improvements to the tpcds
gradle build for Spark
add 92587e8d301 [Spark dataset runner] Add direct translation of Reshuffle
and Reshuffle.ViaRandomKey transforms.
add 45cc0854be5 Merge pull request #23638: [Spark dataset runner] Add
direct translation of Reshuffle and Reshuffle.ViaRandomKey transforms.
add fdb12230556 Make GCP OAuth scopes configurable via pipeline options.
(#23644)
add 8e2431c0e55 Update BQIO to a single scheduled executor service reduce
threads (#23234)
No new revisions were added by this update.
Summary of changes:
.../translation/PipelineTranslator.java | 29 +----------
.../translation/SparkSessionFactory.java | 1 +
.../translation/batch/PipelineTranslatorBatch.java | 6 ++-
...torBatch.java => ReshuffleTranslatorBatch.java} | 32 +++++++-----
.../apache/beam/sdk/options/ExecutorOptions.java | 59 ++++++++++++++++++++++
.../extensions/gcp/auth/GcpCredentialFactory.java | 35 ++++---------
.../sdk/extensions/gcp/options/GcpOptions.java | 41 ++++++++++++++-
.../sdk/extensions/gcp/options/GcsOptions.java | 29 +++++------
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 27 ++++++++++
sdks/java/testing/tpcds/build.gradle | 29 +++++------
sdks/python/apache_beam/internal/gcp/auth.py | 28 +++-------
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 3 +-
.../python/apache_beam/options/pipeline_options.py | 18 +++++++
.../apache_beam/runners/interactive/utils.py | 3 +-
14 files changed, 216 insertions(+), 124 deletions(-)
copy
runners/spark/3/src/main/java/org/apache/beam/runners/spark/structuredstreaming/translation/batch/{CreatePCollectionViewTranslatorBatch.java
=> ReshuffleTranslatorBatch.java} (53%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/options/ExecutorOptions.java
