This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from 56048f2 (commit)
to f841d60 (commit)
from 56048f2 Move Java function implementations to a sub-module (#12247)
add 8250a7b Reducing how much we call BigQuery Get Table API. The BQ API
is queried once per bundle. With this change, it will be queried once per
worker thread. This will help with throughput for BQ streaming inserts
add e34fef8 Merge pull request #12125 from pabloem/BQ_API_quota
add 053ac16 [BEAM-10310] Adding thread-safe restriction tracker wrapper.
add 04e5cd3 [BEAM-10310] Adding GetRestriction method for RTrackers.
add 535c0f6 Merge pull request #12215: [BEAM-10310] Adding thread-safe
restriction tracker wrapper.
add 2003985 [BEAM-10471] Change assertion method to improve
expressiveness and increase error margin on non deterministic size test to
avoid flakiness
add bedae33 Merge pull request #12276 from
echauchot/BEAM-10471-cassandraIO-size-flakiness
add 188f89d [BEAM-10185] Building python wheels for Windows (on Github
Actions) (#12172)
add e0a16e0 Merge pull request #12147 from [BEAM 7624] Added Null
Behavior in JsonToRow
add ad7ee50 Disable flaky test:
org.apache.beam.runners.samza.adapter.UnboundedSourceSystemTest.testMultipleAdvanceWatermark
add b0b8ece Disable flaky test:
org.apache.beam.runners.samza.adapter.UnboundedSourceSystemTest.testMultipleAdvanceWatermark
(#12291)
add 235c905 Ignore NameErrors right after WontImplementErrors.
add 5538d9e Add doctest skipping framework.
add e324b5c Add tests for all Pandas DataFrame and Series tests.
add 063a102 Merge pull request #12259 [BEAM-9561] Run Pandas tests on
Beam.
add ed6bbfd [BEAM-10494] PubsubSchemaCapableIOProvider config inner class
rather than row (#12266)
add cd6d904 [BEAM-10514] Restrict cache file path length
add 7e4e501 Merge pull request #12283 from KevinGG/fix_path_length
add a1c17cd [BEAM-8244] Split FlinkRunnerTest.test_external_transforms.
add 60d84dc Merge pull request #12286 from ibzib/BEAM-8244
add f509f78 [BEAM-10487] Link release blog posts on the Downloads page
add c8d2162 Merge pull request #12256 from ibzib/BEAM-10487
add 698401c Insert TruncateSizedRestriction when pipeline starts to drain.
add 0240471 Address latest comments.
add 047bed7 SpotlessApply
add 280e33d Fix java build.
add cf02dad spotlessApply
add 9b66b72 Only forward split/progress when the only consumer is
splittable.
add ed39236 Merge pull request #12287 from boyuanzz/drain_java
add b7dffe3 [BEAM-8244] Don't run external transform tests with
pre_optimize=all.
add f7b23ec Merge pull request #12303 from ibzib/BEAM-8244
add 373859a [BEAM-8454] Increase timeout and also enable thread stuckness
detector. (#12293)
add cc27255 [BEAM-10490] Support read/write ZetaSQL DATE/TIME types
from/to BigQuery (#12261)
add 8039b56 [BEAM-10526] Use GrpcCleanupRule to use consistent
methodology on server shutdown. (#12301)
add 177445f [BEAM-7672] dynamically setup acceptable wheel specs
according to installed python version
add 03296b3 fixup: update error message when parse failed
add 31a2657 fixup: not to abort when wheel spec setup failed
add 86456be fixup: not to use m flag at wheel name with python 3.8
add a03ebd6 fixup: simplified wheel name setting
add 0642073 [BEAM-7672] dynamically setup acceptable wheel specs
according to installed python version #12115
add e688b28 [BEAM-9968] Guarantee that outstanding split/progress
requests are handled before bundle completion (and before any final
progress/checkpoint data sent to handlers). (#12305)
add 39bc4d3 Relax to matchers to match display data from the specific
tests, not from all options.
add abf63e2 Merge pull request #12288 from aaltay/matchers
add ac6d80e [BEAM-10420] Add support for per window invocation of
beam:transform:sdf_process_sized_element_and_restrictions:v1 (#12275)
add 8589741 use debian:buster for go SDK container
add 98e9922 Merge pull request #12290: Use debian:buster for go SDK
container
add 2734080 Fix abi suffix tags for Python 3.8.
add af2d6b0 [BEAM-10434] Fix abi suffix tags for Python 3.8.
add 4decedd Insert TruncateSizedRestriction when pipeline starts to drain.
add f841d60 Merge pull request #12289 from boyuanzz/drain_py
No new revisions were added by this update.
Summary of changes:
.github/workflows/build_wheels.yml | 21 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 10 +
.../runners/core/construction/Environments.java | 2 +
.../core/construction/PTransformTranslation.java | 5 +
.../core/construction/ParDoTranslation.java | 3 +
.../graph/SplittableParDoExpander.java | 232 +++++++++
.../core/construction/EnvironmentsTest.java | 7 +
.../core/construction/SplittableParDoTest.java | 5 +
.../graph/SplittableParDoExpanderTest.java | 56 ++
.../runners/core/SplittableParDoProcessFnTest.java | 5 +
.../worker/fn/BeamFnControlServiceTest.java | 20 +-
.../runners/fnexecution/control/RemoteBundle.java | 17 +-
.../fnexecution/control/SdkHarnessClient.java | 75 ++-
.../GrpcContextHeaderAccessorProviderTest.java | 55 +-
.../runners/fnexecution/ServerFactoryTest.java | 24 +-
.../fnexecution/control/RemoteExecutionTest.java | 6 +-
.../fnexecution/control/SdkHarnessClientTest.java | 232 +++++++++
.../runners/portability/PortableRunnerTest.java | 77 ++-
.../samza/adapter/UnboundedSourceSystemTest.java | 2 +
sdks/go/container/Dockerfile | 2 +-
sdks/go/examples/stringsplit/stringsplit.go | 15 +-
sdks/go/pkg/beam/core/graph/fn_test.go | 3 +
.../beam/core/runtime/exec/sdf_invokers_test.go | 3 +-
.../pkg/beam/core/runtime/graphx/translate_test.go | 15 +-
sdks/go/pkg/beam/core/sdf/lock.go | 82 +++
sdks/go/pkg/beam/core/sdf/sdf.go | 4 +
.../beam/io/rtrackers/offsetrange/offsetrange.go | 67 +--
sdks/go/pkg/beam/io/synthetic/source.go | 11 +-
sdks/go/pkg/beam/io/synthetic/step.go | 15 +-
.../src/main/java/org/apache/beam/sdk/io/Read.java | 15 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 57 +++
.../org/apache/beam/sdk/transforms/JsonToRow.java | 53 +-
.../beam/sdk/transforms/PeriodicSequence.java | 5 +
.../java/org/apache/beam/sdk/transforms/Watch.java | 8 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 30 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 6 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 32 ++
.../sdk/transforms/reflect/DoFnSignatures.java | 82 +++
.../splittabledofn/ByteKeyRangeTracker.java | 5 +
.../splittabledofn/GrowableOffsetRangeTracker.java | 9 +
.../splittabledofn/OffsetRangeTracker.java | 5 +
.../splittabledofn/RestrictionTracker.java | 30 ++
.../apache/beam/sdk/transforms/JsonToRowTest.java | 99 ++++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 240 ++++++++-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 131 +++++
.../GrowableOffsetRangeTrackerTest.java | 20 +
...amFnDataSizeBasedBufferingOutboundObserver.java | 2 +-
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 5 +
...DataSizeBasedBufferingOutboundObserverTest.java | 9 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 10 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 353 +++++++++----
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 570 +++++++++++++++++++++
.../org/apache/beam/fn/harness/FnHarnessTest.java | 9 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 9 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 133 +++--
.../gcp/pubsub/PubsubSchemaCapableIOProvider.java | 59 ++-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 14 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 86 +---
sdks/python/apache_beam/dataframe/doctests.py | 39 +-
sdks/python/apache_beam/dataframe/doctests_test.py | 20 +
.../apache_beam/dataframe/pandas_doctests_test.py | 186 +++++++
sdks/python/apache_beam/io/gcp/bigquery.py | 10 +-
sdks/python/apache_beam/io/iobase.py | 33 +-
sdks/python/apache_beam/io/restriction_trackers.py | 6 +
sdks/python/apache_beam/runners/common.pxd | 5 +
sdks/python/apache_beam/runners/common.py | 378 +++++++++++---
sdks/python/apache_beam/runners/common_test.py | 438 ++++++++++++++++
.../runners/interactive/cache_manager.py | 13 +-
.../runners/interactive/interactive_environment.py | 2 +-
.../runners/interactive/pipeline_instrument.py | 5 +-
.../runners/portability/flink_runner_test.py | 12 +-
.../runners/portability/fn_api_runner/execution.py | 14 +
.../runners/portability/fn_api_runner/fn_runner.py | 8 +-
.../portability/fn_api_runner/fn_runner_test.py | 214 ++++++--
.../portability/fn_api_runner/translations.py | 66 ++-
.../runners/portability/portable_runner_test.py | 14 +-
.../apache_beam/runners/portability/stager.py | 9 +-
sdks/python/apache_beam/runners/sdf_utils.py | 6 +
.../apache_beam/runners/worker/bundle_processor.py | 60 ++-
.../runners/worker/bundle_processor_test.py | 17 +-
.../runners/worker/crossplatform_time.h | 63 +++
.../runners/worker/crossplatform_unistd.h | 37 +-
.../apache_beam/runners/worker/operations.pxd | 2 +
.../apache_beam/runners/worker/operations.py | 31 +-
.../runners/worker/statesampler_fast.pyx | 4 +-
sdks/python/apache_beam/transforms/core.py | 28 +
sdks/python/apache_beam/transforms/display_test.py | 10 +-
sdks/python/apache_beam/transforms/environments.py | 2 +
.../apache_beam/transforms/environments_test.py | 4 +
sdks/python/container/boot.go | 34 +-
sdks/python/setup.py | 18 +-
.../www/site/content/en/get-started/downloads.md | 20 +
92 files changed, 4293 insertions(+), 672 deletions(-)
create mode 100644 sdks/go/pkg/beam/core/sdf/lock.go
create mode 100644 sdks/python/apache_beam/dataframe/pandas_doctests_test.py
create mode 100644 sdks/python/apache_beam/runners/worker/crossplatform_time.h
copy
runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkIdentityFunction.java
=> sdks/python/apache_beam/runners/worker/crossplatform_unistd.h (57%)
