This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 6dc1b6f Merge pull request #15997: [BEAM-8688] Upgrading gcsio
library to latest along with flogger
add 18fd949 [BEAM-3293] Add check for error on NewIterable call (#16006)
add 725db64 [BEAM-13268] Change autosharding big query inserts to happen
in parallel for a bundle.
add 5656e26 Merge pull request #16003 from [BEAM-13268] Change
autosharding big query inserts to happen in parallel for a bundle.
add c26edbc [BEAM-12818] Write windowed file output to a consistent
subdirectory of the temporary directory and with hash prefix. This ensures
that files have distributed prefix that is common across pipeline executions
and thus can reuse GCS autoscaling of the file metadata.
add d3544f0 [BEAM-12818] Write windowed temporary files to consistent
temporary directory and with hash prefix #15416
add b7f9ba9 Update dev container tags used by Dataflow runner with
unreleased SDKs.
add faa390d Update dev container tags used by Dataflow runner with
unreleased SDKs. #16007
add 1ccaba3 [BEAM-13265] Add withDeterministicRecordIdFn which allows for
BigQueryIO.Write without added checkpointing (#15998)
add 99f489f remove pkg_resources in run_generate_requirements.sh
add 7be74ef Run generatePythonRequirementsAll for pyarrow 6
add 2335ec6 [BEAM-13624] Run generatePythonRequirementsAll for pyarrow 6
#16000
No new revisions were added by this update.
Summary of changes:
sdks/go/pkg/beam/core/runtime/exec/fn.go | 3 +
.../java/org/apache/beam/sdk/io/FileBasedSink.java | 100 ++++++++++++-------
.../java/org/apache/beam/sdk/io/WriteFiles.java | 19 ++--
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 11 ++-
.../java/org/apache/beam/sdk/io/SimpleSink.java | 4 -
.../sdk/io/gcp/bigquery/BatchedStreamingWrite.java | 107 ++++++---------------
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 34 ++++++-
.../beam/sdk/io/gcp/bigquery/StreamingInserts.java | 34 ++++++-
.../sdk/io/gcp/bigquery/StreamingWriteTables.java | 65 ++++++++++---
.../beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java | 23 ++++-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../container/py36/base_image_requirements.txt | 8 +-
.../container/py37/base_image_requirements.txt | 10 +-
.../container/py38/base_image_requirements.txt | 12 ++-
sdks/python/container/run_generate_requirements.sh | 4 +-
15 files changed, 276 insertions(+), 162 deletions(-)
