[beam] branch master updated (1117508 -> a537ae6)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1117508 [BEAM-9359] Switch to Data Catalog client (#10917)
add a537ae6 [BEAM-9344] Add support for bundle finalization execution to
the Beam Java SDK. (#10910)
No new revisions were added by this update.
Summary of changes:
runners/apex/build.gradle | 1 +
.../core/construction/ParDoTranslation.java| 22 +
.../runners/core/construction/SplittableParDo.java | 19 +
.../construction/SplittableParDoNaiveBounded.java | 61 +-
.../core/construction/ParDoTranslationTest.java| 77 +++
...TimeBoundedSplittableProcessElementInvoker.java | 7 +
.../apache/beam/runners/core/SimpleDoFnRunner.java | 25 +
.../core/SplittableParDoViaKeyedWorkItems.java | 70 +-
runners/direct-java/build.gradle | 2 +
runners/flink/flink_runner.gradle | 1 +
runners/flink/job-server/flink_job_server.gradle | 1 +
runners/gearpump/build.gradle | 1 +
runners/google-cloud-dataflow-java/build.gradle| 1 +
.../dataflow/PrimitiveParDoSingleFactory.java | 20 +
runners/jet/build.gradle | 2 +
runners/portability/java/build.gradle | 5 -
runners/samza/build.gradle | 1 +
runners/spark/build.gradle | 2 +
runners/spark/job-server/build.gradle | 1 +
.../{NeedsRunner.java => UsesBundleFinalizer.java} | 8 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 4 +-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 63 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 13 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 24 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 52 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 96 ++-
.../beam/sdk/transforms/SplittableDoFnTest.java| 80 ++-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 41 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 56 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 4 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 4 +-
.../beam/fn/harness/BoundedSourceRunner.java | 4 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 4 +-
.../org/apache/beam/fn/harness/FlattenRunner.java | 4 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java| 751 +++--
.../java/org/apache/beam/fn/harness/FnHarness.java | 13 +-
.../org/apache/beam/fn/harness/MapFnRunners.java | 4 +-
.../beam/fn/harness/PTransformRunnerFactory.java | 7 +-
.../fn/harness/control/FinalizeBundleHandler.java | 161 +
.../fn/harness/control/ProcessBundleHandler.java | 57 +-
.../beam/fn/harness/AssignWindowsRunnerTest.java | 3 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 6 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 3 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 3 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 4 +
.../apache/beam/fn/harness/FlattenRunnerTest.java | 6 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 24 +-
.../apache/beam/fn/harness/MapFnRunnersTest.java | 9 +-
.../harness/control/FinalizeBundleHandlerTest.java | 115
.../harness/control/ProcessBundleHandlerTest.java | 118 +++-
50 files changed, 1530 insertions(+), 530 deletions(-)
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/testing/{NeedsRunner.java =>
UsesBundleFinalizer.java} (76%)
create mode 100644
sdks/java/harness/src/main/java/org/apache/beam/fn/harness/control/FinalizeBundleHandler.java
create mode 100644
sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/FinalizeBundleHandlerTest.java
[beam] branch master updated (91125d1 -> 1117508)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 91125d1 [BEAM-9252] Exclude jboss's Main and module-info.java (#10930) add 1117508 [BEAM-9359] Switch to Data Catalog client (#10917) No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 5 +-- sdks/java/extensions/sql/datacatalog/build.gradle | 11 ++ .../provider/datacatalog/BigQueryTableFactory.java | 2 +- .../provider/datacatalog/ChainedTableFactory.java | 2 +- .../datacatalog/DataCatalogPipelineOptions.java| 2 +- .../datacatalog/DataCatalogTableProvider.java | 46 +++--- .../meta/provider/datacatalog/GcsTableFactory.java | 4 +- .../provider/datacatalog/PubsubTableFactory.java | 2 +- .../sql/meta/provider/datacatalog/SchemaUtils.java | 4 +- .../meta/provider/datacatalog/TableFactory.java| 2 +- 10 files changed, 36 insertions(+), 44 deletions(-)
svn commit: r38213 - in /dev/beam/vendor: ./ beam-vendor-grpc-1_26_0/ beam-vendor-grpc-1_26_0/0.2/
Author: lcwik Date: Sat Feb 22 00:29:00 2020 New Revision: 38213 Log: Apache Beam vendored gRPC 1.26.0 v0.2 Added: dev/beam/vendor/ dev/beam/vendor/beam-vendor-grpc-1_26_0/ dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/ dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip (with props) dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.asc dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.sha512 Added: dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip == Binary file - no diff available. Propchange: dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.asc == --- dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.asc (added) +++ dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.asc Sat Feb 22 00:29:00 2020 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEE6tXeKT9KA90ud1ZVieaKVuNxzKIFAl5QdXMACgkQieaKVuNx +zKLXNhAArm1ektQ5azj7Yh6OBZg5NHf8yIFk/8LjNS9XGG86arxBa3KQNp1h+dto +cllS+U9Cze7Zti2ohGlC6hSEEJ5z7HMSbeqJ4Pw5cYklceGg0XysavKYuaGX4kj7 +6AOVV9mmJbM93j7yKof4pppieE8ZV3E6RtUnnY9VDWOjzXHHcoMEoStJQ3v/C/Y6 +sWk3pxIKc5dr7H58wlXJmTqP1Ze/lRDI5eSYCYvI7siQawjJXW/tJJPtq8BixjO7 +WvVeKsViEQaaH6n/pulySIMSEc8FB71wvlEORFErEOVs0dsqVsQgteNpv3wtX1Ct +hsM7KhqeHtvhWgWH9FRFqKFi61rBZpJt2RYwx4FxhOTD3BTIOVQEUbmZln6WgvIt +aiOLaooKtYWGZGTKysUayhoAYTrj951ooRU59gTjJmPyBuPPP0OEhlwkdoYSX213 +wziI+2hN95CJQyRPvCLhcRwjk5E0iAFj61kQ5RnFANAI0y1pV7zNwemvV7Uvy1HF +p5PuSpECFFvc5+J/qBV+uGjrVq8DAqx4xgfGvmtIMPvU26U2e+TPUTMB6TP55vel +TFK1FQFs5zOPrr+CCCWHnWrO67fODXmJmyHWsMqqZN3zuhcypPKvowpT8PraWcCg +tfwZrFMNNZGZ1ECZoAncaBHq74MrXUznANpG7uflaYbkXzO9cjk= +=O+SB +-END PGP SIGNATURE- Added: dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.sha512 == --- dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.sha512 (added) +++ dev/beam/vendor/beam-vendor-grpc-1_26_0/0.2/apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip.sha512 Sat Feb 22 00:29:00 2020 @@ -0,0 +1 @@ +813d30d83f25eb4fe98e96f1fd5aeaff6776f35af2a35331d4814eede93d2a96c2048ad74870fbf64c10acb6e3d1e42ada2dde45c22f1de0900a85f7e4c39b10 apache-beam-91125d1d1fc1fe8c5684a486c9b6163c4ec41549-source-release.zip
[beam] branch master updated (d616863 -> 91125d1)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d616863 Merge pull request #10902 from iemejia/minor-fixes add 91125d1 [BEAM-9252] Exclude jboss's Main and module-info.java (#10930) No new revisions were added by this update. Summary of changes: .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 2 +- .../groovy/org/apache/beam/gradle/GrpcVendoring_1_26_0.groovy | 3 +++ .../main/groovy/org/apache/beam/gradle/VendorJavaPlugin.groovy| 8 vendor/grpc-1_26_0/build.gradle | 2 +- 4 files changed, 9 insertions(+), 6 deletions(-)
[beam] branch master updated: Mark Test categories as internal and improve categorization
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 8ec216b Mark Test categories as internal and improve categorization
new d616863 Merge pull request #10902 from iemejia/minor-fixes
8ec216b is described below
commit 8ec216b3a2215512737f89b35ad0e328315f115d
Author: Ismaël Mejía
AuthorDate: Wed Feb 19 17:38:19 2020 +0100
Mark Test categories as internal and improve categorization
---
.../core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java| 3 +++
.../core/src/main/java/org/apache/beam/sdk/testing/NeedsRunner.java | 2 ++
.../java/org/apache/beam/sdk/testing/UsesBoundedSplittableParDo.java | 2 ++
.../main/java/org/apache/beam/sdk/testing/UsesCommittedMetrics.java | 2 ++
.../main/java/org/apache/beam/sdk/testing/UsesCounterMetrics.java| 3 +++
.../org/apache/beam/sdk/testing/UsesCrossLanguageTransforms.java | 3 +++
.../java/org/apache/beam/sdk/testing/UsesCustomWindowMerging.java| 3 +++
.../java/org/apache/beam/sdk/testing/UsesDistributionMetrics.java| 3 +++
.../main/java/org/apache/beam/sdk/testing/UsesFailureMessage.java| 3 +++
.../src/main/java/org/apache/beam/sdk/testing/UsesGaugeMetrics.java | 3 +++
.../core/src/main/java/org/apache/beam/sdk/testing/UsesImpulse.java | 3 +++
.../java/core/src/main/java/org/apache/beam/sdk/testing/UsesKms.java | 3 +++
.../core/src/main/java/org/apache/beam/sdk/testing/UsesMapState.java | 4 +++-
.../src/main/java/org/apache/beam/sdk/testing/UsesMetricsPusher.java | 3 +++
.../main/java/org/apache/beam/sdk/testing/UsesParDoLifecycle.java| 3 +++
.../org/apache/beam/sdk/testing/UsesRequiresTimeSortedInput.java | 2 ++
.../core/src/main/java/org/apache/beam/sdk/testing/UsesSchema.java | 3 +++
.../core/src/main/java/org/apache/beam/sdk/testing/UsesSetState.java | 4 +++-
.../src/main/java/org/apache/beam/sdk/testing/UsesSideInputs.java| 3 +++
.../apache/beam/sdk/testing/UsesSideInputsWithDifferentCoders.java | 5 -
.../beam/sdk/testing/UsesSplittableParDoWithWindowedSideInputs.java | 2 ++
.../src/main/java/org/apache/beam/sdk/testing/UsesStatefulParDo.java | 2 ++
.../java/org/apache/beam/sdk/testing/UsesStrictTimerOrdering.java| 3 +++
.../src/main/java/org/apache/beam/sdk/testing/UsesSystemMetrics.java | 3 +++
.../src/main/java/org/apache/beam/sdk/testing/UsesTestStream.java| 3 +++
.../apache/beam/sdk/testing/UsesTestStreamWithMultipleStages.java| 3 +++
.../apache/beam/sdk/testing/UsesTestStreamWithOutputTimestamp.java | 5 -
.../apache/beam/sdk/testing/UsesTestStreamWithProcessingTime.java| 3 +++
.../core/src/main/java/org/apache/beam/sdk/testing/UsesTimerMap.java | 4
.../src/main/java/org/apache/beam/sdk/testing/UsesTimersInParDo.java | 2 ++
.../java/org/apache/beam/sdk/testing/UsesUnboundedPCollections.java | 3 +++
.../org/apache/beam/sdk/testing/UsesUnboundedSplittableParDo.java| 2 ++
.../src/main/java/org/apache/beam/sdk/testing/ValidatesRunner.java | 3 +++
.../test/java/org/apache/beam/sdk/runners/PipelineRunnerTest.java| 3 ++-
34 files changed, 96 insertions(+), 5 deletions(-)
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
index 922004f..4def3c7 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/LargeKeys.java
@@ -17,7 +17,10 @@
*/
package org.apache.beam.sdk.testing;
+import org.apache.beam.sdk.annotations.Internal;
+
/** Category tags for tests which validate that a Beam runner can handle keys
up to a given size. */
+@Internal
public interface LargeKeys {
/** Tests if a runner supports 10KB keys. */
interface Above10KB {}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/NeedsRunner.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/NeedsRunner.java
index 248ee9a..9ba23fa 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/NeedsRunner.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/NeedsRunner.java
@@ -18,9 +18,11 @@
package org.apache.beam.sdk.testing;
import org.apache.beam.sdk.PipelineRunner;
+import org.apache.beam.sdk.annotations.Internal;
/**
* Category tag for validation tests which utilize {@link TestPipeline} for
execution and expect to
* be executed by a {@link PipelineRunner}.
*/
+@Internal
public interface NeedsRunner {}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesBoundedSplittableParDo.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesBoundedSplittableParDo.java
index 28a7060..5dceed4 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesBoundedSplitt
[beam] branch asf-site updated: Publishing website 2020/02/21 22:01:47 at commit 00ed8a8
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new dab84fc Publishing website 2020/02/21 22:01:47 at commit 00ed8a8
dab84fc is described below
commit dab84fc814af7764b8f155a7b56aff4b0211e988
Author: jenkins
AuthorDate: Fri Feb 21 22:01:48 2020 +
Publishing website 2020/02/21 22:01:47 at commit 00ed8a8
---
.../contribute/release-guide/index.html| 22 +++---
.../documentation/runners/flink/index.html | 14 +++---
.../documentation/runtime/environments/index.html | 18 +-
.../runtime/sdk-harness-config/index.html | 2 +-
.../get-started/quickstart-go/index.html | 2 +-
.../get-started/wordcount-example/index.html | 2 +-
6 files changed, 30 insertions(+), 30 deletions(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 9bcbd15..275f47f 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1087,7 +1087,7 @@ Make sure they have blocker JIRAs attached and are OK to
get into the release by
Run gradle publish to push java artifacts into Maven staging
repo.
Stage source release into dist.apache.org dev https://dist.apache.org/repos/dist/dev/beam/";>repo.
Stage,sign and hash python binaries into dist.apache.ord dev repo
python dir
- Stage SDK docker images to https://hub.docker.com/u/apachebeam";>https://hub.docker.com/u/apachebeam.
+ Stage SDK docker images to https://hub.docker.com/search?q=apache%2Fbeam&type=image";>docker hub
Apache organization.
Create a PR to update beam and beam-site, changes includes:
Copy python doc into beam-site
@@ -1224,7 +1224,7 @@ svn commit
PYTHON_VER=("python2.7" "python3.5" "python3.6" "python3.7")
for ver in "${PYTHON_VER[@]}"; do
- docker push apachebeam/${ver}_sdk:${RELEASE}_rc{RC_NUM} &
+ docker push apache/beam_${ver}_sdk:${RELEASE}_rc{RC_NUM} &
done
@@ -1262,20 +1262,20 @@ done
Clean up images from local
for ver in "${PYTHON_VER[@]}"; do
- docker rmi -f apachebeam/${ver}_sdk:${RELEASE}_rc{RC_NUM}
+ docker rmi -f apache/beam_${ver}_sdk:${RELEASE}_rc{RC_NUM}
done
-docker rmi -f apachebeam/java_sdk:${RELEASE}_rc{RC_NUM}
-docker rmi -f apachebeam/go_sdk:${RELEASE}_rc{RC_NUM}
+docker rmi -f apache/beam_java_sdk:${RELEASE}_rc{RC_NUM}
+docker rmi -f apache/beam_go_sdk:${RELEASE}_rc{RC_NUM}
for ver in "${FLINK_VER[@]}"; do
- docker rmi -f "apachebeam/flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
+ docker rmi -f "apache/beam_flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
done
-docker rmi -f "apachebeam/spark_job_server:${RELEASE}_rc${RC_NUM}"
+docker rmi -f "apache/beam_spark_job_server:${RELEASE}_rc${RC_NUM}"
How to find images:
- Visit https://hub.docker.com/u/apachebeam";>https://hub.docker.com/u/apachebeam
+ Visit https://hub.docker.com/search?q=apache%2Fbeam&type=image";>https://hub.docker.com/u/apache
Visit each repository and navigate to tags tab.
Verify images are pushed with tags: ${RELEASE}_rc{RC_NUM}
@@ -1446,7 +1446,7 @@ all major features and bug fixes, and all known
issues.
Maven artifacts deployed to the staging repository of https://repository.apache.org/content/repositories/";>repository.apache.org
Source distribution deployed to the dev repository of https://dist.apache.org/repos/dist/dev/beam/";>dist.apache.org
Website pull request proposed to list the release, publish the https://beam.apache.org/releases/javadoc/";>Java API reference manual,
and publish the https://beam.apache.org/releases/pydoc/";>Python API
reference manual.
- Docker images are published to https://hub.docker.com/u/apachebeam";>DockerHub with tags:
{RELEASE}_rc{RC_NUM}.
+ Docker images are published to https://hub.docker.com/search?q=apache%2Fbeam&type=image";>DockerHub
with tags: {RELEASE}_rc{RC_NUM}.
You can (optionally) also do additional verification by:
@@ -1458,7 +1458,7 @@ all major features and bug fixes, and all known
issues.
Run all jenkins suites and include links to passing tests in the voting
email. (Select “Run with parameters”)
Pull docker images to make sure they are pullable.
docker pull {image_name}
-docker pull apachebeam/python3.5_sdk:2.16.0_rc1
+docker pull apache/beam_python3.5_sdk:2.16.0_rc1
@@ -1506,7 +1506,7 @@ Release Manager
[7] https://github.com/apache/beam-site/pull/...
[8] https://github.com/apache/beam/pull/...
[9]
https://docs.google.com/spreadsheets/d/1qk-N5vjXvbcEk68GjbkSZTR8AGqyNUM-oLFo_ZXBpJw/edit#gid=...
-[10] https://hub.docker.com/u/apachebeam
+[10] https://hub.docker.com/search?q=apache%2Fbeam&type=image
[beam] branch master updated (3aece70 -> 00ed8a8)
This is an automated email from the ASF dual-hosted git repository. hannahjiang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3aece70 Merge pull request #10908 [BEAM-9339] Declare capabilities for Python SDK. add 00ed8a8 [BEAM-9063] migrate docker images to apache (#10612) No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 4 +- gradle.properties | 3 ++ .../src/main/scripts/build_release_candidate.sh| 15 +++--- release/src/main/scripts/publish_docker_images.sh | 55 -- .../runners/core/construction/Environments.java| 6 ++- .../flink_job_server_container.gradle | 5 +- runners/flink/job-server/flink_job_server.gradle | 8 +++- runners/google-cloud-dataflow-java/build.gradle| 4 +- runners/spark/job-server/build.gradle | 4 +- runners/spark/job-server/container/build.gradle| 5 +- sdks/go/container/build.gradle | 5 +- sdks/go/pkg/beam/options/jobopts/options.go| 2 +- sdks/java/container/build.gradle | 5 +- sdks/java/core/build.gradle| 2 + .../java/org/apache/beam/sdk/util/ReleaseInfo.java | 18 +++ .../apache_beam/runners/portability/job_server.py | 2 +- sdks/python/apache_beam/transforms/environments.py | 2 +- sdks/python/container/py2/build.gradle | 5 +- sdks/python/container/py35/build.gradle| 5 +- sdks/python/container/py36/build.gradle| 5 +- sdks/python/container/py37/build.gradle| 5 +- website/src/contribute/release-guide.md| 22 - website/src/documentation/runners/flink.md | 14 +++--- website/src/documentation/runtime/environments.md | 18 +++ .../documentation/runtime/sdk-harness-config.md| 2 +- website/src/get-started/quickstart-go.md | 2 +- website/src/get-started/wordcount-example.md | 2 +- 27 files changed, 136 insertions(+), 89 deletions(-)
[beam] branch master updated (1133f7b -> 3aece70)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1133f7b Make sure calling try_claim(0) more than once also trows exception. (#10922) add 71e2d5d Add standard protocol capabilities to protos. add 4be0a85 [BEAM-9339] Declare capabilities for Python SDK. add 3aece70 Merge pull request #10908 [BEAM-9339] Declare capabilities for Python SDK. No new revisions were added by this update. Summary of changes: .../pipeline/src/main/proto/beam_runner_api.proto | 24 + sdks/python/apache_beam/portability/common_urns.py | 3 + sdks/python/apache_beam/transforms/environments.py | 106 +++-- .../apache_beam/transforms/environments_test.py| 31 +- 4 files changed, 131 insertions(+), 33 deletions(-)
[beam] branch master updated (c6f812f -> 1133f7b)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c6f812f [BEAM-8537] Provide WatermarkEstimator to track watermark (#10375) add 1133f7b Make sure calling try_claim(0) more than once also trows exception. (#10922) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/restriction_trackers.py | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (fdaf5d8 -> c6f812f)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fdaf5d8 Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)" (#10924) add c6f812f [BEAM-8537] Provide WatermarkEstimator to track watermark (#10375) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/io/iobase.py | 35 + sdks/python/apache_beam/io/watermark_estimators.py | 150 + .../apache_beam/io/watermark_estimators_test.py| 106 +++ sdks/python/apache_beam/runners/common.pxd | 17 +-- sdks/python/apache_beam/runners/common.py | 119 ++-- .../runners/direct/sdf_direct_runner.py| 7 +- .../runners/portability/fn_api_runner_test.py | 11 +- sdks/python/apache_beam/runners/sdf_utils.py | 133 +- sdks/python/apache_beam/runners/sdf_utils_test.py | 26 .../apache_beam/runners/worker/bundle_processor.py | 27 +++- sdks/python/apache_beam/transforms/core.py | 77 --- sdks/python/apache_beam/transforms/core_test.py| 55 12 files changed, 563 insertions(+), 200 deletions(-) create mode 100644 sdks/python/apache_beam/io/watermark_estimators.py create mode 100644 sdks/python/apache_beam/io/watermark_estimators_test.py delete mode 100644 sdks/python/apache_beam/transforms/core_test.py
[beam] branch master updated (9c0e29b -> fdaf5d8)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9c0e29b Merge pull request #10920 from lukecwik/splittabledofn2 add fdaf5d8 Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)" (#10924) No new revisions were added by this update. Summary of changes: .../apache_beam/testing/synthetic_pipeline.py | 25 +++--- 1 file changed, 8 insertions(+), 17 deletions(-)
[beam] branch master updated (439641c -> 9c0e29b)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 439641c Merge pull request #10916 from ibzib/flink-stuck new f5841b9 [BEAM-5605] Eagerly close the BoundedReader once we have read everything or have failed. new fc42d94 fixup! Fix SpotBugs failure new 9c0e29b Merge pull request #10920 from lukecwik/splittabledofn2 The 25675 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../src/main/java/org/apache/beam/sdk/io/Read.java | 39 ++ 1 file changed, 39 insertions(+)
[beam] branch asf-site updated: Publishing website 2020/02/21 18:14:50 at commit a24bc3b
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new a0d2416 Publishing website 2020/02/21 18:14:50 at commit a24bc3b
a0d2416 is described below
commit a0d241688fddfece05a3014710b60c2e8296f91a
Author: jenkins
AuthorDate: Fri Feb 21 18:14:51 2020 +
Publishing website 2020/02/21 18:14:50 at commit a24bc3b
---
website/generated-content/contribute/release-guide/index.html | 8
1 file changed, 8 insertions(+)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 977700a..9bcbd15 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1252,6 +1252,13 @@ for ver in "${FLINK_VER[@]}"; do
done
+
+ Build Spark job server image and push to DockerHub.
+
+
+./gradlew
":runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
+
+
Clean up images from local
for ver in "${PYTHON_VER[@]}"; do
@@ -1262,6 +1269,7 @@ docker rmi -f apachebeam/go_sdk:${RELEASE}_rc{RC_NUM}
for ver in "${FLINK_VER[@]}"; do
docker rmi -f "apachebeam/flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
done
+docker rmi -f "apachebeam/spark_job_server:${RELEASE}_rc${RC_NUM}"
[beam] branch master updated (a24bc3b -> 439641c)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a24bc3b Merge pull request #10921 from ibzib/spark-container-publish new 949609d [BEAM-9225] Fix Flink uberjar job termination bug. new 69bcea2 Reuse get_state method. new 439641c Merge pull request #10916 from ibzib/flink-stuck The 25672 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/flink/job-server/test_flink_uber_jar.sh | 3 +++ .../apache_beam/runners/portability/flink_uber_jar_job_server.py | 5 + .../runners/portability/flink_uber_jar_job_server_test.py| 1 + 3 files changed, 5 insertions(+), 4 deletions(-)
[beam] branch master updated: [BEAM-9022] publish Spark job server Docker image
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3b4befd [BEAM-9022] publish Spark job server Docker image
new a24bc3b Merge pull request #10921 from ibzib/spark-container-publish
3b4befd is described below
commit 3b4befd1541dd3e9a0408dfb8f11363216e02753
Author: Kyle Weaver
AuthorDate: Thu Feb 20 16:38:40 2020 -0800
[BEAM-9022] publish Spark job server Docker image
---
release/src/main/scripts/build_release_candidate.sh | 4
release/src/main/scripts/publish_docker_images.sh | 19 +++
runners/spark/job-server/container/build.gradle | 2 +-
website/src/contribute/release-guide.md | 7 +++
4 files changed, 31 insertions(+), 1 deletion(-)
diff --git a/release/src/main/scripts/build_release_candidate.sh
b/release/src/main/scripts/build_release_candidate.sh
index a367b60..fd9fd0f 100755
--- a/release/src/main/scripts/build_release_candidate.sh
+++ b/release/src/main/scripts/build_release_candidate.sh
@@ -236,6 +236,9 @@ if [[ $confirmation = "y" ]]; then
./gradlew ":runners:flink:${ver}:job-server-container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
done
+ echo '-Generating and Pushing Spark job server
image-'
+ ./gradlew ":runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
+
rm -rf ~/${PYTHON_ARTIFACTS_DIR}
echo '---Clean up images at local-'
@@ -247,6 +250,7 @@ if [[ $confirmation = "y" ]]; then
for ver in "${FLINK_VER[@]}"; do
docker rmi -f "apachebeam/flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
done
+ docker rmi -f "apachebeam/spark_job_server:${RELEASE}_rc${RC_NUM}"
fi
echo "[Current Step]: Update beam-site"
diff --git a/release/src/main/scripts/publish_docker_images.sh
b/release/src/main/scripts/publish_docker_images.sh
index e80c446..c4d5e16 100755
--- a/release/src/main/scripts/publish_docker_images.sh
+++ b/release/src/main/scripts/publish_docker_images.sh
@@ -119,4 +119,23 @@ if [[ $confirmation = "y" ]]; then
docker rmi -f "${FLINK_IMAGE_NAME}:${RELEASE}"
docker rmi -f "${FLINK_IMAGE_NAME}:latest"
done
+
+ echo '-Generating and Pushing Spark job server
image-'
+ SPARK_IMAGE_NAME="apachebeam/spark_job_server"
+
+ # Pull verified RC from dockerhub.
+ docker pull "${SPARK_IMAGE_NAME}:${RELEASE}_${RC_VERSION}"
+
+ # Tag with ${RELEASE} and push to dockerhub.
+ docker tag "${SPARK_IMAGE_NAME}:${RELEASE}_${RC_VERSION}"
"${SPARK_IMAGE_NAME}:${RELEASE}"
+ docker push "${SPARK_IMAGE_NAME}:${RELEASE}"
+
+ # Tag with latest and push to dockerhub.
+ docker tag "${SPARK_IMAGE_NAME}:${RELEASE}_${RC_VERSION}"
"${SPARK_IMAGE_NAME}:latest"
+ docker push "${SPARK_IMAGE_NAME}:latest"
+
+ # Cleanup images from local
+ docker rmi -f "${SPARK_IMAGE_NAME}:${RELEASE}_${RC_VERSION}"
+ docker rmi -f "${SPARK_IMAGE_NAME}:${RELEASE}"
+ docker rmi -f "${SPARK_IMAGE_NAME}:latest"
fi
diff --git a/runners/spark/job-server/container/build.gradle
b/runners/spark/job-server/container/build.gradle
index c7244a3..c68ffed 100644
--- a/runners/spark/job-server/container/build.gradle
+++ b/runners/spark/job-server/container/build.gradle
@@ -54,7 +54,7 @@ task copyDockerfileDependencies(type: Copy) {
}
docker {
- name containerImageName(name: 'spark-job-server',
+ name containerImageName(name: 'spark_job_server',
root:
project.rootProject.hasProperty(["docker-repository-root"]) ?
project.rootProject["docker-repository-root"] : "apachebeam")
files "./build/"
diff --git a/website/src/contribute/release-guide.md
b/website/src/contribute/release-guide.md
index 137e691..10a6ac6 100644
--- a/website/src/contribute/release-guide.md
+++ b/website/src/contribute/release-guide.md
@@ -717,6 +717,12 @@ for ver in "${FLINK_VER[@]}"; do
done
```
+* Build Spark job server image and push to DockerHub.
+
+```
+./gradlew ":runners:spark:job-server:container:dockerPush"
-Pdocker-tag="${RELEASE}_rc${RC_NUM}"
+```
+
Clean up images from local
```
@@ -728,6 +734,7 @@ docker rmi -f apachebeam/go_sdk:${RELEASE}_rc{RC_NUM}
for ver in "${FLINK_VER[@]}"; do
docker rmi -f "apachebeam/flink${ver}_job_server:${RELEASE}_rc${RC_NUM}"
done
+docker rmi -f "apachebeam/spark_job_server:${RELEASE}_rc${RC_NUM}"
```
[beam] branch master updated (bdf0be1 -> 6ab6b12)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bdf0be1 Merge pull request #10643 from iemejia/BEAM-9162-jackson-update add 41bf2de Update go beam runner generated protos. add 5f46b67 [BEAM-9339] Declare capabilities for Go SDK. add 6ab6b12 Merge pull request #10911 [BEAM-9339] Declare capabilities for Go SDK. No new revisions were added by this update. Summary of changes: sdks/go/pkg/beam/core/runtime/graphx/coder.go | 16 + sdks/go/pkg/beam/core/runtime/graphx/translate.go | 37 + .../beam/model/pipeline_v1/beam_runner_api.pb.go | 841 - sdks/go/pkg/beam/runners/dataflow/dataflow.go | 28 +- sdks/go/pkg/beam/runners/universal/universal.go| 28 +- 5 files changed, 523 insertions(+), 427 deletions(-)
[beam] branch master updated: [BEAM-9162] Upgrade Jackson to version 2.10.2
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 63bef93 [BEAM-9162] Upgrade Jackson to version 2.10.2
new bdf0be1 Merge pull request #10643 from
iemejia/BEAM-9162-jackson-update
63bef93 is described below
commit 63bef93859700d66f5dbbb39a507bc893ad9f054
Author: Ismaël Mejía
AuthorDate: Tue Jan 21 17:21:41 2020 +0100
[BEAM-9162] Upgrade Jackson to version 2.10.2
---
buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git
a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
index 8939bf2..48d14c8 100644
--- a/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
+++ b/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy
@@ -378,7 +378,7 @@ class BeamModulePlugin implements Plugin {
def guava_version = "25.1-jre"
def hadoop_version = "2.8.5"
def hamcrest_version = "2.1"
-def jackson_version = "2.9.10"
+def jackson_version = "2.10.2"
def jaxb_api_version = "2.2.12"
def kafka_version = "1.0.0"
def nemo_version = "0.1"
[beam] 01/01: Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)"
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a commit to branch revert-10885-synthetic-source-perf-fix
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 683d03f384789b141445c5c8ca172853e320a26a
Author: Michal Walenia <32354134+mwale...@users.noreply.github.com>
AuthorDate: Fri Feb 21 14:14:34 2020 +0100
Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)"
This reverts commit b4668a1a1e234c071c3a7b182a76f8f4cf6bfe64.
---
.../apache_beam/testing/synthetic_pipeline.py | 25 +++---
1 file changed, 8 insertions(+), 17 deletions(-)
diff --git a/sdks/python/apache_beam/testing/synthetic_pipeline.py
b/sdks/python/apache_beam/testing/synthetic_pipeline.py
index af21e6e..6dd41c8 100644
--- a/sdks/python/apache_beam/testing/synthetic_pipeline.py
+++ b/sdks/python/apache_beam/testing/synthetic_pipeline.py
@@ -41,10 +41,7 @@ import argparse
import json
import logging
import math
-import random
-import struct
import time
-from builtins import range
import apache_beam as beam
from apache_beam.io import WriteToText
@@ -418,31 +415,25 @@ class SyntheticSource(iobase.BoundedSource):
tracker = range_trackers.UnsplittableRangeTracker(tracker)
return tracker
- @staticmethod
- def random_bytes(length, generator):
-"""Return random bytes."""
-return b''.join(
-(struct.pack('B', generator.getrandbits(8)) for _ in range(length)))
-
- def _gen_kv_pair(self, generator, index):
-generator.seed(index)
-rand = generator.random()
+ def _gen_kv_pair(self, index):
+r = np.random.RandomState(index)
+rand = r.random_sample()
# Determines whether to generate hot key or not.
if rand < self._hot_key_fraction:
# Generate hot key.
# An integer is randomly selected from the range [0, numHotKeys-1]
# with equal probability.
- generator.seed(index % self._num_hot_keys)
-return self.random_bytes(self._key_size, generator), self.random_bytes(
- self._value_size, generator)
+ r_hot = np.random.RandomState(index % self._num_hot_keys)
+ return r_hot.bytes(self._key_size), r.bytes(self._value_size)
+else:
+ return r.bytes(self._key_size), r.bytes(self._value_size)
def read(self, range_tracker):
index = range_tracker.start_position()
-generator = random.Random()
while range_tracker.try_claim(index):
time.sleep(self._sleep_per_input_record_sec)
- yield self._gen_kv_pair(generator, index)
+ yield self._gen_kv_pair(index)
index += 1
def default_output_coder(self):
[beam] branch revert-10885-synthetic-source-perf-fix created (now 683d03f)
This is an automated email from the ASF dual-hosted git repository. mwalenia pushed a change to branch revert-10885-synthetic-source-perf-fix in repository https://gitbox.apache.org/repos/asf/beam.git. at 683d03f Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)" This branch includes the following new commits: new 683d03f Revert "[BEAM-9085] Fix performance regression in SyntheticSource (#10885)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (76501ab -> 7392322)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 76501ab [BEAM-9229] Adding dependency information to Environment proto (#10733) add 0cbf8ef [BEAM-9349] Update joda-time version add 7392322 Merge pull request #10923: [BEAM-9349] Update joda-time version to 2.10.5 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
