[beam] tag nightly-master updated (2f02771 -> 0302dcd)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag nightly-master was modified! *** from 2f02771 (commit) to 0302dcd (commit) from 2f02771 Merge pull request #14491 from ihji/BEAM-12140 add 7b51b51 [BEAM-7372] cleanup following codes from apache_beam/runners/dataflow and apache_beam/runners/interactive * usage of future package * usage of past package * unnecessary builtins import add 35881d7 [BEAM-7372] cleanup py2 codepath from runners/dataflow and runners/interactive #14564 add 50fcf55 Merge pull request #14639: [BEAM-10277][BEAM-12198] Populate encoding positions to all matching schemas add 1a5f90e Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" add 87e1164 Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" add b87f1f0 [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals add be906f3 [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils add 7418c84 Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations add e9df62b Moving to 2.31.0-SNAPSHOT on master branch. add cdef5bb [BEAM-12238] Make StateBackedIterable Serializable. (#14660) add e284374 [BEAM-9615] Use UUIDs in Go SDK Schemas (#14665) add 3eb3dc9 [BEAM-12174] Propertly close ExecutableStageContext in DoFnOp#close (#14659) add 400b1d7 [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering failed precondition. (#14668) add 8867610 [BEAM-9547] WontImplement a few more DeferredGroupBy operations (#14675) add a60546e [BEAM-12227] Update ExternalWorkerServiceTest add f5cf55d Merge pull request #14677 from [BEAM-12227] Update ExternalWorkerServiceTest add f3fab5c [BEAM-8706] Bump antlr4 to 4.9.2 add 0ec9084 Merge pull request #14669 from [BEAM-8706] Bump antlr4 to 4.9.2 add 9c5f08f Fix kenn at apache email address in blog author list add 21fca9c Add Beam 2.29.0 release blog post add 99cb746 Merge pull request #14562: Add Beam 2.29.0 release blog post add bb3a769 Update website for 2.29.0 release add fd33f16 Merge pull request #14556: Update website for 2.29.0 release add b6ee454 [BEAM-1] Update Dataflow container for side input changes add ce4c192 Merge pull request #14687: [BEAM-1] Update Dataflow container for side input changes add 637cf50 Increase timeout of Dataflow V1 streaming ValidatesRunner add f716b2f Merge pull request #14688: [BEAM-12250] Increase timeout of Dataflow V1 streaming ValidatesRunner add 22e3ce3 [BEAM-12158] Regen go protos & rename internal proto package (#14612) add 17fef39 bugfix for cut_release_branch.sh add a2849e0 Merge pull request #14682 from ihji/bugfix_cut_release_branch_script add 5ecbe17 Update golock instructions (#14678) add a5296ff adding 2.31.0 section to CHANGES.md add 0302dcd Merge pull request #14681 from ihji/changes_2_31_0 No new revisions were added by this update. Summary of changes: ...Java_ValidatesRunner_Dataflow_Streaming.groovy} |2 +- .test-infra/jenkins/job_PostCommit_Python.groovy |2 +- ...ommit_Python_ValidatesContainer_Dataflow.groovy |2 +- ...stCommit_Python_ValidatesRunner_Dataflow.groovy |2 +- ...ommit_Python_ValidatesRunner_Dataflow_V2.groovy |2 +- ..._PostCommit_Python_ValidatesRunner_Flink.groovy |4 - CHANGES.md | 31 + .../org/apache/beam/gradle/BeamModulePlugin.groovy |7 +- gradle.properties |4 +- model/pipeline/src/main/proto/schema.proto |4 + release/src/main/scripts/cut_release_branch.sh |5 +- .../beam/runners/core/InMemoryTimerInternals.java | 46 +- runners/google-cloud-dataflow-java/build.gradle|2 +- .../util/RowCoderCloudObjectTranslator.java|6 + .../util/SchemaCoderCloudObjectTranslator.java |6 + .../portability/ExternalWorkerServiceTest.java |4 +- .../apache/beam/runners/samza/runtime/DoFnOp.java |7 +- sdks/go/README.md |3 +- sdks/go/gogradle.lock |7 + sdks/go/pkg/beam/core/core.go |2 +- .../pkg/beam/core/runtime/graphx/schema/schema.go | 20 +- sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go |2 +- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 1954 --- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto |4 +- .../model/fnexecution_v1/beam_provision_api.pb.go |4 +- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 3595 +++- sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 855 ++---
[beam] branch master updated (0302dcd -> b7d8569)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0302dcd Merge pull request #14681 from ihji/changes_2_31_0 add 871dbbc [BEAM-12147][BEAM-9919] Add Go SDK x-lang prefix transform add b7d8569 Merge pull request #14666: [BEAM-12147][BEAM-9919] Add Go SDK x-lang prefix transform No new revisions were added by this update. Summary of changes: sdks/go/examples/xlang/transforms.go| 31 sdks/go/pkg/beam/core/runtime/xlangx/payload.go | 67 + sdks/go/pkg/beam/xlang.go | 11 sdks/go/test/integration/integration.go | 1 + sdks/go/test/integration/xlang/xlang_test.go| 15 ++ sdks/go/test/run_validatesrunner_tests.sh | 8 +-- 6 files changed, 129 insertions(+), 4 deletions(-) create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/payload.go
[beam] branch master updated: adding 2.31.0 section to CHANGES.md
This is an automated email from the ASF dual-hosted git repository. heejong pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new a5296ff adding 2.31.0 section to CHANGES.md new 0302dcd Merge pull request #14681 from ihji/changes_2_31_0 a5296ff is described below commit a5296ff3e7b8bd2e0dd7929ec6d0b8e2d59030e9 Author: Heejong Lee AuthorDate: Thu Apr 29 00:35:05 2021 -0700 adding 2.31.0 section to CHANGES.md --- CHANGES.md | 31 +++ 1 file changed, 31 insertions(+) diff --git a/CHANGES.md b/CHANGES.md index 695fc0e..0fae45a 100644 --- a/CHANGES.md +++ b/CHANGES.md @@ -50,6 +50,37 @@ * Fixed X (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). --> +# [2.31.0] - Unreleased + +## Highlights + +* New highly anticipated feature X added to Python SDK ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). +* New highly anticipated feature Y added to Java SDK ([BEAM-Y](https://issues.apache.org/jira/browse/BEAM-Y)). + +## I/Os + +* Support for X source added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). + +## New Features / Improvements + +* X feature added (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). + +## Breaking Changes + +* X behavior was changed ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). +* Python Row objects are now sensitive to field order. So `Row(x=3, y=4)` is no + longer considered equal to `Row(y=4, x=3)` (BEAM-11929). +* Kafka Beam SQL tables now ascribe meaning to the LOCATION field; previously + it was ignored if provided. + +## Deprecations + +* X behavior is deprecated and will be removed in X versions ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). + +## Known Issues + +* Fixed X (Java/Python) ([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)). + # [2.30.0] - Unreleased ## Highlights
[beam] branch master updated (a2849e0 -> 5ecbe17)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a2849e0 Merge pull request #14682 from ihji/bugfix_cut_release_branch_script add 5ecbe17 Update golock instructions (#14678) No new revisions were added by this update. Summary of changes: sdks/go/README.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated: bugfix for cut_release_branch.sh
This is an automated email from the ASF dual-hosted git repository.
heejong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 17fef39 bugfix for cut_release_branch.sh
new a2849e0 Merge pull request #14682 from
ihji/bugfix_cut_release_branch_script
17fef39 is described below
commit 17fef39613e15165acf70ebb322673d69e1d4b7d
Author: Heejong Lee
AuthorDate: Thu Apr 29 00:36:18 2021 -0700
bugfix for cut_release_branch.sh
---
release/src/main/scripts/cut_release_branch.sh | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git a/release/src/main/scripts/cut_release_branch.sh
b/release/src/main/scripts/cut_release_branch.sh
index 221d8fa..0a2b025 100755
--- a/release/src/main/scripts/cut_release_branch.sh
+++ b/release/src/main/scripts/cut_release_branch.sh
@@ -55,7 +55,8 @@ if [[ -z "$RELEASE" || -z "$NEXT_VERSION_IN_BASE_BRANCH" ]];
then
exit
fi
-SCRIPT_DIR=$(dirname $0)
+SCRIPT=$(readlink -f $0)
+SCRIPT_DIR=$(dirname $SCRIPT)
MASTER_BRANCH=master
DEV=${RELEASE}.dev
RELEASE_BRANCH=release-${RELEASE}
@@ -91,7 +92,7 @@ echo ${MASTER_BRANCH}
echo "==="
# Update master branch
-sh "$SCRIPT_DIR"/set_version.sh "$NEXT_VERSION_IN_BASE_BRANCH""
+sh "$SCRIPT_DIR"/set_version.sh "$NEXT_VERSION_IN_BASE_BRANCH"
echo "==Update master branch as following"
git diff
[beam] branch master updated (f716b2f -> 22e3ce3)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f716b2f Merge pull request #14688: [BEAM-12250] Increase timeout of Dataflow V1 streaming ValidatesRunner add 22e3ce3 [BEAM-12158] Regen go protos & rename internal proto package (#14612) No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go |2 +- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 1954 --- sdks/go/pkg/beam/core/runtime/graphx/v1/v1.proto |4 +- .../model/fnexecution_v1/beam_provision_api.pb.go |4 +- .../beam/model/pipeline_v1/beam_runner_api.pb.go | 3595 +++- sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 855 ++--- sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go| 401 +-- 7 files changed, 4155 insertions(+), 2660 deletions(-)
[beam] branch master updated (ce4c192 -> f716b2f)
This is an automated email from the ASF dual-hosted git repository.
kenn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from ce4c192 Merge pull request #14687: [BEAM-1] Update Dataflow
container for side input changes
add 637cf50 Increase timeout of Dataflow V1 streaming ValidatesRunner
add f716b2f Merge pull request #14688: [BEAM-12250] Increase timeout of
Dataflow V1 streaming ValidatesRunner
No new revisions were added by this update.
Summary of changes:
...vy => job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy} | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
rename
.test-infra/jenkins/{job_PostCommit_Java_ValidatesRunner_Dataflow_streaming.groovy
=> job_PostCommit_Java_ValidatesRunner_Dataflow_Streaming.groovy} (99%)
[beam] branch youngoli-patch-1 updated (cbd0b5c -> f2ee179)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch youngoli-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. from cbd0b5c Update golock instructions add f2ee179 Fixup No new revisions were added by this update. Summary of changes: sdks/go/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (fd33f16 -> ce4c192)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fd33f16 Merge pull request #14556: Update website for 2.29.0 release add b6ee454 [BEAM-1] Update Dataflow container for side input changes add ce4c192 Merge pull request #14687: [BEAM-1] Update Dataflow container for side input changes No new revisions were added by this update. Summary of changes: runners/google-cloud-dataflow-java/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (99cb746 -> fd33f16)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 99cb746 Merge pull request #14562: Add Beam 2.29.0 release blog post add bb3a769 Update website for 2.29.0 release add fd33f16 Merge pull request #14556: Update website for 2.29.0 release No new revisions were added by this update. Summary of changes: website/www/site/config.toml | 2 +- website/www/site/content/en/get-started/downloads.md | 8 website/www/site/static/.htaccess| 2 +- 3 files changed, 10 insertions(+), 2 deletions(-)
[beam] branch master updated (0ec9084 -> 99cb746)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0ec9084 Merge pull request #14669 from [BEAM-8706] Bump antlr4 to 4.9.2 add 9c5f08f Fix kenn at apache email address in blog author list add 21fca9c Add Beam 2.29.0 release blog post add 99cb746 Merge pull request #14562: Add Beam 2.29.0 release blog post No new revisions were added by this update. Summary of changes: website/www/site/content/en/blog/beam-2.29.0.md | 84 + website/www/site/data/authors.yml | 2 +- 2 files changed, 85 insertions(+), 1 deletion(-) create mode 100644 website/www/site/content/en/blog/beam-2.29.0.md
[beam] branch master updated: [BEAM-8706] Bump antlr4 to 4.9.2
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new f3fab5c [BEAM-8706] Bump antlr4 to 4.9.2
new 0ec9084 Merge pull request #14669 from [BEAM-8706] Bump antlr4 to
4.9.2
f3fab5c is described below
commit f3fab5c18009d13427e1dfbe0091364ede9ed7a9
Author: masahitojp
AuthorDate: Wed Apr 28 20:23:31 2021 +0900
[BEAM-8706] Bump antlr4 to 4.9.2
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +++--
1 file changed, 3 insertions(+), 2 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 17dcaf5..eb8ed64 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -417,6 +417,7 @@ class BeamModulePlugin implements Plugin {
// a dependency version which should match across multiple
// Maven artifacts.
def activemq_version = "5.14.5"
+def antlr_version = "4.9.2"
def autovalue_version = "1.8.1"
def aws_java_sdk_version = "1.11.974"
def aws_java_sdk2_version = "2.15.31"
@@ -466,8 +467,8 @@ class BeamModulePlugin implements Plugin {
activemq_junit :
"org.apache.activemq.tooling:activemq-junit:$activemq_version",
activemq_kahadb_store :
"org.apache.activemq:activemq-kahadb-store:$activemq_version",
activemq_mqtt :
"org.apache.activemq:activemq-mqtt:$activemq_version",
-antlr : "org.antlr:antlr4:4.7",
-antlr_runtime :
"org.antlr:antlr4-runtime:4.7",
+antlr :
"org.antlr:antlr4:$antlr_version",
+antlr_runtime :
"org.antlr:antlr4-runtime:$antlr_version",
args4j : "args4j:args4j:2.33",
auto_value_annotations :
"com.google.auto.value:auto-value-annotations:$autovalue_version",
avro:
"org.apache.avro:avro:1.8.2",
[beam] branch master updated: [BEAM-12227] Update ExternalWorkerServiceTest
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new a60546e [BEAM-12227] Update ExternalWorkerServiceTest
new f5cf55d Merge pull request #14677 from [BEAM-12227] Update
ExternalWorkerServiceTest
a60546e is described below
commit a60546eec1054c70c1c7c7438754c69fb41c56b2
Author: Ke Wu
AuthorDate: Wed Apr 28 15:23:12 2021 -0700
[BEAM-12227] Update ExternalWorkerServiceTest
Update ExternalWorkerServiceTest and switch from #newBuilder().build() to
#getDefaultInstance
---
.../apache/beam/runners/portability/ExternalWorkerServiceTest.java| 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git
a/runners/portability/java/src/test/java/org/apache/beam/runners/portability/ExternalWorkerServiceTest.java
b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/ExternalWorkerServiceTest.java
index 8997cd2..ee2c5e9 100644
---
a/runners/portability/java/src/test/java/org/apache/beam/runners/portability/ExternalWorkerServiceTest.java
+++
b/runners/portability/java/src/test/java/org/apache/beam/runners/portability/ExternalWorkerServiceTest.java
@@ -38,7 +38,7 @@ public class ExternalWorkerServiceTest {
@Test
public void startWorker() {
PipelineOptions options = PipelineOptionsFactory.create();
-StartWorkerRequest request = StartWorkerRequest.newBuilder().build();
+StartWorkerRequest request = StartWorkerRequest.getDefaultInstance();
StreamObserver responseObserver =
mock(StreamObserver.class);
ExternalWorkerService service = new ExternalWorkerService(options);
service.startWorker(request, responseObserver);
@@ -50,7 +50,7 @@ public class ExternalWorkerServiceTest {
@Test
public void stopWorker() {
PipelineOptions options = PipelineOptionsFactory.create();
-StopWorkerRequest request = StopWorkerRequest.newBuilder().build();
+StopWorkerRequest request = StopWorkerRequest.getDefaultInstance();
StreamObserver responseObserver =
mock(StreamObserver.class);
ExternalWorkerService service = new ExternalWorkerService(options);
service.stopWorker(request, responseObserver);
[beam] branch master updated (400b1d7 -> 8867610)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 400b1d7 [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering failed precondition. (#14668) add 8867610 [BEAM-9547] WontImplement a few more DeferredGroupBy operations (#14675) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/frames.py | 10 -- 1 file changed, 8 insertions(+), 2 deletions(-)
[beam] branch master updated: [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering failed precondition. (#14668)
This is an automated email from the ASF dual-hosted git repository.
amaliujia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 400b1d7 [BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient
triggering failed precondition. (#14668)
400b1d7 is described below
commit 400b1d73a74b88c3cb7d7f4ac9a20580a7fcd5fd
Author: scwhittle
AuthorDate: Thu Apr 29 09:30:08 2021 -0700
[BEAM-12118] Fix race introduced in QueeuingBeamFnDataClient triggering
failed precondition. (#14668)
---
.../beam/fn/harness/data/QueueingBeamFnDataClient.java | 14 --
1 file changed, 12 insertions(+), 2 deletions(-)
diff --git
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
index 21d0b14..11dea66 100644
---
a/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
+++
b/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/QueueingBeamFnDataClient.java
@@ -83,12 +83,22 @@ public class QueueingBeamFnDataClient implements
BeamFnDataClient {
// empty in which case it returns null.
@Nullable
ConsumerAndData take() throws InterruptedException {
+ // We first poll without blocking to optimize for the case there is data.
+ // If there is no data we end up blocking on take() and thus the extra
+ // poll doesn't matter.
@Nullable ConsumerAndData result = queue.poll();
if (result == null) {
if (closed.get()) {
- return null;
+ // Poll again to ensure that there is nothing in the queue. Once we
observe closed as true
+ // we are guaranteed no additional elements other than the POISON
will be added. However
+ // we can't rely on the previous poll result as it could race with
additional offers and
+ // close.
+ result = queue.poll();
+} else {
+ // We are not closed so we perform a blocking take. We are
guaranteed that additional
+ // elements will be offered or the POISON will be added by close to
unblock this thread.
+ result = queue.take();
}
-result = queue.take();
}
if (result == POISON) {
return null;
[beam] branch master updated (e284374 -> 3eb3dc9)
This is an automated email from the ASF dual-hosted git repository. xinyu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e284374 [BEAM-9615] Use UUIDs in Go SDK Schemas (#14665) add 3eb3dc9 [BEAM-12174] Propertly close ExecutableStageContext in DoFnOp#close (#14659) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/runners/samza/runtime/DoFnOp.java| 7 --- 1 file changed, 4 insertions(+), 3 deletions(-)
[beam] branch master updated (cdef5bb -> e284374)
This is an automated email from the ASF dual-hosted git repository. lostluck pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cdef5bb [BEAM-12238] Make StateBackedIterable Serializable. (#14660) add e284374 [BEAM-9615] Use UUIDs in Go SDK Schemas (#14665) No new revisions were added by this update. Summary of changes: sdks/go/gogradle.lock| 7 +++ .../go/pkg/beam/core/runtime/graphx/schema/schema.go | 20 +++- 2 files changed, 14 insertions(+), 13 deletions(-)
[beam] branch master updated (e9df62b -> cdef5bb)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e9df62b Moving to 2.31.0-SNAPSHOT on master branch. add cdef5bb [BEAM-12238] Make StateBackedIterable Serializable. (#14660) No new revisions were added by this update. Summary of changes: .../beam/fn/harness/state/StateBackedIterable.java | 11 +-- .../fn/harness/state/StateBackedIterableTest.java | 34 ++ 2 files changed, 43 insertions(+), 2 deletions(-)
[beam] branch release-2.30.0 created (now f4b1402)
This is an automated email from the ASF dual-hosted git repository. heejong pushed a change to branch release-2.30.0 in repository https://gitbox.apache.org/repos/asf/beam.git. at f4b1402 Set Dataflow container to release version. This branch includes the following new commits: new f4b1402 Set Dataflow container to release version. The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] 01/01: Set Dataflow container to release version.
This is an automated email from the ASF dual-hosted git repository.
heejong pushed a commit to branch release-2.30.0
in repository https://gitbox.apache.org/repos/asf/beam.git
commit f4b1402ee242c125f19f6d346f0fcd1d86e1e136
Author: Heejong Lee
AuthorDate: Thu Apr 29 00:18:54 2021 -0700
Set Dataflow container to release version.
---
runners/google-cloud-dataflow-java/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/runners/google-cloud-dataflow-java/build.gradle
b/runners/google-cloud-dataflow-java/build.gradle
index eafc6e3..079650b 100644
--- a/runners/google-cloud-dataflow-java/build.gradle
+++ b/runners/google-cloud-dataflow-java/build.gradle
@@ -45,7 +45,7 @@ processResources {
filter org.apache.tools.ant.filters.ReplaceTokens, tokens: [
'dataflow.legacy_environment_major_version' : '8',
'dataflow.fnapi_environment_major_version' : '8',
-'dataflow.container_version' : 'beam-master-20210427',
+'dataflow.container_version' : 'beam-2.30.0',
'dataflow.container_base_repository' : 'gcr.io/cloud-dataflow/v1beta3',
]
}
[beam] branch master updated: Moving to 2.31.0-SNAPSHOT on master branch.
This is an automated email from the ASF dual-hosted git repository.
heejong pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new e9df62b Moving to 2.31.0-SNAPSHOT on master branch.
e9df62b is described below
commit e9df62bba672571e82cbef45970166d42d07bc06
Author: Heejong Lee
AuthorDate: Thu Apr 29 00:16:18 2021 -0700
Moving to 2.31.0-SNAPSHOT on master branch.
---
.../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 2 +-
gradle.properties | 4 ++--
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/python/apache_beam/version.py| 2 +-
4 files changed, 5 insertions(+), 5 deletions(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index f601908..8c1d183 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -352,7 +352,7 @@ class BeamModulePlugin implements Plugin {
// Automatically use the official release version if we are performing a
release
// otherwise append '-SNAPSHOT'
-project.version = '2.30.0'
+project.version = '2.31.0'
if (!isRelease(project)) {
project.version += '-SNAPSHOT'
}
diff --git a/gradle.properties b/gradle.properties
index 4255ef8..a5c5195 100644
--- a/gradle.properties
+++ b/gradle.properties
@@ -24,8 +24,8 @@ offlineRepositoryRoot=offline-repository
signing.gnupg.executable=gpg
signing.gnupg.useLegacyGpg=true
-version=2.30.0-SNAPSHOT
-sdk_version=2.30.0.dev
+version=2.31.0-SNAPSHOT
+sdk_version=2.31.0.dev
javaVersion=1.8
diff --git a/sdks/go/pkg/beam/core/core.go b/sdks/go/pkg/beam/core/core.go
index baed22a..9b2c713 100644
--- a/sdks/go/pkg/beam/core/core.go
+++ b/sdks/go/pkg/beam/core/core.go
@@ -27,5 +27,5 @@ const (
// SdkName is the human readable name of the SDK for UserAgents.
SdkName = "Apache Beam SDK for Go"
// SdkVersion is the current version of the SDK.
- SdkVersion = "2.30.0.dev"
+ SdkVersion = "2.31.0.dev"
)
diff --git a/sdks/python/apache_beam/version.py
b/sdks/python/apache_beam/version.py
index 91b417e..ceda28a 100644
--- a/sdks/python/apache_beam/version.py
+++ b/sdks/python/apache_beam/version.py
@@ -17,4 +17,4 @@
"""Apache Beam SDK version information and utilities."""
-__version__ = '2.30.0.dev'
+__version__ = '2.31.0.dev'
[beam] 03/03: Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7418c84de2b68b06e0318766ca6c69f4d2144390 Merge: 87e1164 be906f3 Author: Ismaël Mejía AuthorDate: Thu Apr 29 08:13:58 2021 +0200 Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations .../beam/runners/core/InMemoryTimerInternals.java | 46 -- .../main/java/org/apache/beam/sdk/values/Row.java | 28 ++--- .../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++ 3 files changed, 41 insertions(+), 46 deletions(-)
[beam] 02/03: [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit be906f38c5cac14c465332b8179022085ed76c13
Author: Daniel Kulp
AuthorDate: Wed Apr 28 14:21:13 2021 -0400
[BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
---
.../main/java/org/apache/beam/sdk/values/Row.java | 28 ++
.../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++
2 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index a19d9b3..9aa9277 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -847,23 +847,19 @@ public abstract class Row implements Serializable {
+ " fields.");
}
- FieldOverrides fieldOverrides = new FieldOverrides(schema);
- fieldOverrides.setOverrides(this.values);
-
- Row row;
- if (!fieldOverrides.isEmpty()) {
-row =
-(Row)
-new RowFieldMatcher()
-.match(
-new CapturingRowCases(schema, fieldOverrides),
-FieldType.row(schema),
-new RowPosition(FieldAccessDescriptor.create()),
-null);
- } else {
-row = new RowWithStorage(schema, Collections.emptyList());
+ if (!values.isEmpty()) {
+FieldOverrides fieldOverrides = new FieldOverrides(schema,
this.values);
+if (!fieldOverrides.isEmpty()) {
+ return (Row)
+ new RowFieldMatcher()
+ .match(
+ new CapturingRowCases(schema, fieldOverrides),
+ FieldType.row(schema),
+ new RowPosition(FieldAccessDescriptor.create()),
+ null);
+}
}
- return row;
+ return new RowWithStorage(schema, Collections.emptyList());
}
}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
index d3b1c21..a5f6210 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
@@ -236,6 +236,11 @@ class RowUtils {
this.rootSchema = rootSchema;
}
+FieldOverrides(Schema rootSchema, List overrides) {
+ this.topNode = new FieldAccessNode(rootSchema, overrides);
+ this.rootSchema = rootSchema;
+}
+
boolean isEmpty() {
return topNode.isEmpty();
}
@@ -270,6 +275,14 @@ class RowUtils {
nestedAccess = Lists.newArrayList();
}
+ FieldAccessNode(Schema schema, List overrides) {
+fieldOverrides =
Lists.newArrayListWithExpectedSize(schema.getFieldCount());
+for (Object value : overrides) {
+ fieldOverrides.add(new FieldOverride(value));
+}
+nestedAccess = Lists.newArrayList();
+ }
+
boolean isEmpty() {
return fieldOverrides.isEmpty() && nestedAccess.isEmpty();
}
[beam] branch master updated (87e1164 -> 7418c84)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 87e1164 Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" new b87f1f0 [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals new be906f3 [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils new 7418c84 Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/core/InMemoryTimerInternals.java | 46 -- .../main/java/org/apache/beam/sdk/values/Row.java | 28 ++--- .../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++ 3 files changed, 41 insertions(+), 46 deletions(-)
[beam] 01/03: [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b87f1f01197a78d60fb08a0ee7a4ef96bc4e08eb
Author: Daniel Kulp
AuthorDate: Wed Apr 28 14:16:31 2021 -0400
[BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals
---
.../beam/runners/core/InMemoryTimerInternals.java | 46 --
1 file changed, 16 insertions(+), 30 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
index 8be9081..d0b3bed 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -63,6 +63,9 @@ public class InMemoryTimerInternals implements TimerInternals
{
/** Current synchronized processing time. */
private Instant synchronizedProcessingTime =
BoundedWindow.TIMESTAMP_MIN_VALUE;
+ /** Class.getSimpleName() cached to avoid allocations for tracing. */
+ private static final String SIMPLE_NAME =
InMemoryTimerInternals.class.getSimpleName();
+
@Override
public @Nullable Instant currentOutputWatermarkTime() {
return outputWatermarkTime;
@@ -125,17 +128,12 @@ public class InMemoryTimerInternals implements
TimerInternals {
@Deprecated
@Override
public void setTimer(TimerData timerData) {
-WindowTracing.trace("{}.setTimer: {}", getClass().getSimpleName(),
timerData);
+WindowTracing.trace("{}.setTimer: {}", SIMPLE_NAME, timerData);
-@Nullable
-TimerData existing =
-existingTimers.get(
-timerData.getNamespace(), timerData.getTimerId() + '+' +
timerData.getTimerFamilyId());
+@Nullable String colKey = timerData.getTimerId() + '+' +
timerData.getTimerFamilyId();
+TimerData existing = existingTimers.get(timerData.getNamespace(), colKey);
if (existing == null) {
- existingTimers.put(
- timerData.getNamespace(),
- timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
- timerData);
+ existingTimers.put(timerData.getNamespace(), colKey, timerData);
timersForDomain(timerData.getDomain()).add(timerData);
} else {
checkArgument(
@@ -149,10 +147,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
NavigableSet timers =
timersForDomain(timerData.getDomain());
timers.remove(existing);
timers.add(timerData);
-existingTimers.put(
-timerData.getNamespace(),
-timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
-timerData);
+existingTimers.put(timerData.getNamespace(), colKey, timerData);
}
}
}
@@ -216,7 +211,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
newInputWatermark);
WindowTracing.trace(
"{}.advanceInputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
inputWatermarkTime,
newInputWatermark);
inputWatermarkTime = newInputWatermark;
@@ -229,7 +224,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
if (newOutputWatermark.isAfter(inputWatermarkTime)) {
WindowTracing.trace(
"{}.advanceOutputWatermark: clipping output watermark from {} to {}",
- getClass().getSimpleName(),
+ SIMPLE_NAME,
newOutputWatermark,
inputWatermarkTime);
adjustedOutputWatermark = inputWatermarkTime;
@@ -244,7 +239,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
adjustedOutputWatermark);
WindowTracing.trace(
"{}.advanceOutputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
outputWatermarkTime,
adjustedOutputWatermark);
outputWatermarkTime = adjustedOutputWatermark;
@@ -259,10 +254,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
processingTime,
newProcessingTime);
WindowTracing.trace(
-"{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
-processingTime,
-newProcessingTime);
+"{}.advanceProcessingTime: from {} to {}", SIMPLE_NAME,
processingTime, newProcessingTime);
processingTime = newProcessingTime;
}
@@ -277,7 +269,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
newSynchronizedProcessingTime);
WindowTracing.trace(
"{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
synchronizedProcessingTime,
newSynchronizedProcessingTime);
synchronizedProcessingTime = newSynchronizedProcessingTime;
@@ -288,10 +280,7 @@ public class InMemoryTimerInternals
