This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from 378e6b8 (commit)
to 9035a92 (commit)
from 378e6b8 Puts more expensive BQ empty table check to the right of the
'and' condition (#14094)
add f6d25c4 [BEAM-11730] Reduce context switching overhead for appliance
reads by issuing reads directly from calling threads in the case that there is
no reads being queued.
add 248104e Merge pull request #13878:[BEAM-11730] Reduce context
switching overhead for appliance reads by
add 7583375 [BEAM-11531] Use pandas 1.2 for python>=3.7 (#14099)
add 07e1e02 [BEAM-11861] Add methods to explicitly provide coder for
ParquetIO's Parse and ParseFiles (#14078)
add a31e004 Do not stage dataflow worker jar when use runner_v2.
add 11be7aa Merge pull request #14100 from Do not stage dataflow worker
jar when use runner_v2.
add 0f79092 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/schemaio-expansion-service
add e1863cd [BEAM-10961] add reference to BEAM-11761
add 913e08f Merge pull request #14090 from
sonam-vend/sdks-extensions-sch-io-exp-svc
add b728c80 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/sorter
add 527648c Merge pull request #14092 from
sonam-vend/sdks-extensions-sorter
add 43706f5 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/ml
add 1d87462 [BEAM-10961] add explicit compile for auto_value_annotations
in sdks/extensions/ml/build.gradle
add c2290b3 Merge pull request #14088 from sonam-vend/sdks-extensions-ml
add 2f7578d [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/sketching
add 4502376 Merge pull request #14091 from
sonam-vend/sdks-extensions-sketching
add 4092b72 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/protobuf
add 162c7f6 Merge pull request #14089 from
sonam-vend/sdks-extensions-protobuf
add 8d0065c [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/kyro
add 55a15d4 Merge pull request #14087 from sonam-vend/sdks-extensions-kyro
add 13d584c [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/join-library
add 5f99ef4 Merge pull request #14086 from
sonam-vend/sdks-extensions-join-library
add d729c91 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/jackson
add 5d9a7fa [BEAM-10961] undo line moves (originally intended for
alphabeticization)
add abb304e [BEAM-10961] fix stray reordering of lines
add 26feefe Merge pull request #14085 from
sonam-vend/sdks-extensions-jackson
add c93f937 Recognize JOB_STATE_PENDING from Dataflow and map to RUNNING
add 4191500 Merge pull request #14097: Recognize JOB_STATE_PENDING from
Dataflow and map to RUNNING
add 4afa738 Attempting improvements on DirectRunner Python dash
add 8152b98 Merge pull request #14102 from pabloem/try-dash
add 6308ef9 Merge pull request #13802: [BEAM-1474]. Adding MapState and
SetState support for the Dataflow runner
add b7125d0 [BEAM-10961] enable strict dependency checking for
sdks/java/extensions/google-cloud-platform-core (#14084)
add 3f01950 [BEAM-11707] Change WindmillStateCache cache invalidation to
be based upon reference invalidation instead of expensive set management.
Reduce operations of shared cache by caching per-key object sets locally and
flushing as groups to shared cache. Remove byte tracking which could be racy
based upon background evictions in favor of just iterating for rendering the
status page. This also lets us capture more stats.
add 31825af Merge pull request #13862: [BEAM-11707] Change
WindmillStateCache cache invalidation to be based…
add f48af32 Add validate runner test for testing custom merging windows
fn without custom window types
add 9f5ded6 Fix up! formatting
add 3a8993b Merge pull request #14096 from Add validate runner test for
testing custom merging windows fn without custom window types
add 2b17316 Initial watermark latency benchmark
add 4d9a7ab Merge pull request #13770: Initial watermark latency benchmark
add 03b883b [BEAM-10961] Strict dependency checking for sdks/java/io/gcp
(#13791)
add 6d806fa Remove use of model SYNCHRONIZED_PROCESSING_TIME
add 020e13b Remove SYNCHRONIZED_PROCESSING_TIME from model proto
add 5df6b9b Merge pull request #14044: [BEAM-11867] Remove use of model
SYNCHRONIZED_PROCESSING_TIME
add 0c42740 Attempting improvements on DirectRunner Python dash
add dbdf4ad Merge pull request #14106 from pabloem/try-dash
add 7a27c79 Revert "Revert "[BEAM-2914] Add portable merging window
support to Python. (#12995)""
add 9839525 Use the windowing strategy of the input, not output,
PCollection of GBK.
add 31368b54 Improve test, error on ALREADY_MERGED.
add 9035a92 Merge pull request #14095 from robertwb/merging-windows
No new revisions were added by this update.
Summary of changes:
.../Python_FnApiRunner_ubenchmarks.json | 62 ++-
.test-infra/metrics/influxdb/Dockerfile | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +
.../examples/cookbook/MapClassIntegrationIT.java | 71 +++
.../pipeline/src/main/proto/beam_runner_api.proto | 9 +-
.../core/construction/ParDoTranslation.java | 5 +-
.../core/construction/TriggerTranslation.java | 5 +-
.../beam/runners/core/InMemoryStateInternals.java | 44 +-
.../org/apache/beam/runners/core/StateTags.java | 6 +
.../beam/runners/core/StateInternalsTest.java | 2 +-
.../streaming/state/FlinkStateInternals.java | 63 ++-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../dataflow/BatchStatefulParDoOverrides.java | 16 +-
.../dataflow/DataflowPipelineTranslator.java | 7 +-
.../beam/runners/dataflow/DataflowRunner.java | 58 ++-
.../beam/runners/dataflow/util/MonitoringUtil.java | 56 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 21 +-
.../worker/MetricTrackingWindmillServerStub.java | 168 +++---
.../dataflow/worker/WindmillStateCache.java | 361 ++++++-------
.../dataflow/worker/WindmillStateInternals.java | 568 ++++++++++++++++++++-
.../dataflow/worker/WindmillStateReader.java | 188 +++++--
.../dataflow/worker/WindmillStateCacheTest.java | 84 +--
.../worker/WindmillStateInternalsTest.java | 419 ++++++++++++++-
.../dataflow/worker/WindmillStateReaderTest.java | 138 +++++
.../worker/windmill/src/main/proto/windmill.proto | 26 +
.../control/ProcessBundleDescriptors.java | 6 +-
runners/portability/java/build.gradle | 1 +
.../samza/runtime/SamzaStoreStateInternals.java | 48 +-
.../java/org/apache/beam/sdk/state/MapState.java | 52 +-
.../java/org/apache/beam/sdk/state/StateSpecs.java | 24 +
.../beam/sdk/transforms/windowing/WindowTest.java | 91 ++++
sdks/java/extensions/euphoria/build.gradle | 6 +-
.../google-cloud-platform-core/build.gradle | 6 +-
sdks/java/extensions/jackson/build.gradle | 4 +-
sdks/java/extensions/join-library/build.gradle | 7 +-
sdks/java/extensions/kryo/build.gradle | 4 +
sdks/java/extensions/ml/build.gradle | 18 +-
sdks/java/extensions/protobuf/build.gradle | 5 +-
.../schemaio-expansion-service/build.gradle | 10 +-
sdks/java/extensions/sketching/build.gradle | 4 +-
sdks/java/extensions/sorter/build.gradle | 4 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 8 +-
sdks/java/io/google-cloud-platform/build.gradle | 36 +-
.../expansion-service/build.gradle | 3 +
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 38 +-
.../apache/beam/sdk/io/parquet/ParquetIOTest.java | 31 ++
.../{load-tests => watermarks}/build.gradle | 12 +-
.../sdk/testing/watermarks/WatermarkLatency.java | 240 +++++++++
.../runners/portability/flink_runner_test.py | 3 +
.../runners/portability/fn_api_runner/execution.py | 248 ++++++++-
.../portability/fn_api_runner/fn_runner_test.py | 37 ++
.../runners/portability/spark_runner_test.py | 3 +
.../apache_beam/runners/worker/bundle_processor.py | 43 ++
sdks/python/apache_beam/transforms/timeutil.py | 2 -
sdks/python/container/base_image_requirements.txt | 3 +-
settings.gradle.kts | 1 +
56 files changed, 2869 insertions(+), 517 deletions(-)
create mode 100644
examples/java/src/test/java/org/apache/beam/examples/cookbook/MapClassIntegrationIT.java
copy sdks/java/testing/{load-tests => watermarks}/build.gradle (92%)
create mode 100644
sdks/java/testing/watermarks/src/main/java/org/apache/beam/sdk/testing/watermarks/WatermarkLatency.java
