[beam] branch asf-site updated: Publishing website 2020/02/12 07:25:15 at commit cc151c3
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new b8939f0 Publishing website 2020/02/12 07:25:15 at commit cc151c3 b8939f0 is described below commit b8939f09507e251530b6d0f7a79b3f6f4285003e Author: jenkins AuthorDate: Wed Feb 12 07:25:15 2020 + Publishing website 2020/02/12 07:25:15 at commit cc151c3 --- website/generated-content/roadmap/java-sdk/index.html | 5 +++-- website/generated-content/roadmap/spark-runner/index.html | 12 +--- 2 files changed, 12 insertions(+), 5 deletions(-) diff --git a/website/generated-content/roadmap/java-sdk/index.html b/website/generated-content/roadmap/java-sdk/index.html index 829239d..2075d15 100644 --- a/website/generated-content/roadmap/java-sdk/index.html +++ b/website/generated-content/roadmap/java-sdk/index.html @@ -252,11 +252,12 @@ limitations under the License. Next Java LTS version support (Java 11 / 18.9) -Work to support the next LTS release of Java is in progress. For more details about the scope and info on the various tasks please see the JIRA ticket. +Work to support the next LTS release of Java is in progress. For more details +about the scope and info on the various tasks please see the JIRA ticket. JIRA: https://issues.apache.org/jira/browse/BEAM-2530";>BEAM-2530 - Contact: mailto:ieme...@gmail.com";>Ismaël Mejía + Contact: mailto:mwale...@apache.org";>Michał Walenia diff --git a/website/generated-content/roadmap/spark-runner/index.html b/website/generated-content/roadmap/spark-runner/index.html index be003d3..7805fb9 100644 --- a/website/generated-content/roadmap/spark-runner/index.html +++ b/website/generated-content/roadmap/spark-runner/index.html @@ -226,6 +226,9 @@ + + Spark 3 + @@ -247,11 +250,14 @@ limitations under the License. Apache Spark Runner Roadmap -This roadmap is in progress. In the meantime, here are available resources: +Spark 3 + +Support for Spark 3 in Beam’s Spark runner is ongoing. For info on the various +tasks please refer to the JIRA ticket. - Runner documentation - JIRA: https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-spark";>runner-spark + JIRA: https://issues.apache.org/jira/browse/BEAM-7093";>BEAM-7093 + Contact: mailto:ieme...@apache.org";>Ismaël Mejía
[beam] branch master updated: [website] Update Java 11 and Spark roadmap
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 94fc1ad [website] Update Java 11 and Spark roadmap
new cc151c3 Merge pull request #10819 from iemejia/website-roadmap
94fc1ad is described below
commit 94fc1ad5039bb1d6a74e4782b5835451bd238a2c
Author: Ismaël Mejía
AuthorDate: Mon Feb 10 17:21:07 2020 +0100
[website] Update Java 11 and Spark roadmap
---
website/src/roadmap/java-sdk.md | 5 +++--
website/src/roadmap/spark-runner.md | 9 ++---
2 files changed, 9 insertions(+), 5 deletions(-)
diff --git a/website/src/roadmap/java-sdk.md b/website/src/roadmap/java-sdk.md
index e8f8470..50f6754 100644
--- a/website/src/roadmap/java-sdk.md
+++ b/website/src/roadmap/java-sdk.md
@@ -22,8 +22,9 @@ limitations under the License.
## Next Java LTS version support (Java 11 / 18.9)
-Work to support the next LTS release of Java is in progress. For more details
about the scope and info on the various tasks please see the JIRA ticket.
+Work to support the next LTS release of Java is in progress. For more details
+about the scope and info on the various tasks please see the JIRA ticket.
- JIRA: [BEAM-2530](https://issues.apache.org/jira/browse/BEAM-2530)
-- Contact: [Ismaël Mejía](mailto:ieme...@gmail.com)
+- Contact: [Michał Walenia](mailto:mwale...@apache.org)
diff --git a/website/src/roadmap/spark-runner.md
b/website/src/roadmap/spark-runner.md
index 63adb55..e7d60ca 100644
--- a/website/src/roadmap/spark-runner.md
+++ b/website/src/roadmap/spark-runner.md
@@ -20,7 +20,10 @@ limitations under the License.
# Apache Spark Runner Roadmap
-This roadmap is in progress. In the meantime, here are available resources:
+## Spark 3
- - [Runner documentation]({{site.baseurl}}/documentation/runners/spark)
- - JIRA:
[runner-spark](https://issues.apache.org/jira/issues/?jql=project%20%3D%20BEAM%20AND%20component%20%3D%20runner-spark)
+Support for Spark 3 in Beam's Spark runner is ongoing. For info on the various
+tasks please refer to the JIRA ticket.
+
+- JIRA: [BEAM-7093](https://issues.apache.org/jira/browse/BEAM-7093)
+- Contact: [Ismaël Mejía](mailto:ieme...@apache.org)
[beam] branch master updated (07aab1f -> f6633d8)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 07aab1f Merge pull request #10837 [BEAM-7284] Cleanup MappingProxy reducer since dill supports it now. new f72841d [BEAM-8095] Remove no_xdist for test new c7daa7c fix lint new f6633d8 Merge pull request #10759 from udim/xdist-fixup The 25519 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/python/apache_beam/runners/dataflow/dataflow_runner_test.py | 3 --- 1 file changed, 3 deletions(-)
[beam] branch master updated (9165828 -> 07aab1f)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9165828 Remove tables and refer to dependency locations in code (#10745) add 6df1c74 Cleanup MappingProxy reducer since dill supports it natively now. add 07aab1f Merge pull request #10837 [BEAM-7284] Cleanup MappingProxy reducer since dill supports it now. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/internal/pickler.py | 8 1 file changed, 8 deletions(-)
[beam] branch asf-site updated: Publishing website 2020/02/12 00:10:30 at commit 9165828
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new ac5d331 Publishing website 2020/02/12 00:10:30 at commit 9165828 ac5d331 is described below commit ac5d331725200126d9c1a83541ed1a0946d13336 Author: jenkins AuthorDate: Wed Feb 12 00:10:31 2020 + Publishing website 2020/02/12 00:10:30 at commit 9165828 --- .../sdks/java-dependencies/index.html | 386 +++- .../sdks/python-dependencies/index.html| 501 ++--- 2 files changed, 98 insertions(+), 789 deletions(-) diff --git a/website/generated-content/documentation/sdks/java-dependencies/index.html b/website/generated-content/documentation/sdks/java-dependencies/index.html index e7523d0..09005a0 100644 --- a/website/generated-content/documentation/sdks/java-dependencies/index.html +++ b/website/generated-content/documentation/sdks/java-dependencies/index.html @@ -313,338 +313,60 @@ behavior in the service. If you are using any of these packages in your code, be aware that some libraries are not forward-compatible and you may need to pin to the listed versions that will be in scope during execution. -To see the compile and runtime dependencies for your Beam SDK version, expand -the relevant section below. - -2.9.0 - -Beam SDK for Java 2.9.0 has the following compile and runtime dependencies. - - - GroupIdArtifactIdVersion - org.apache.activemqactivemq-amqp5.13.1 - org.apache.activemqactivemq-broker5.13.1 - org.apache.activemqactivemq-client5.13.1 - org.apache.activemqactivemq-jaas5.13.1 - org.apache.activemq.toolingactivemq-junit5.13.1 - org.apache.activemqactivemq-kahadb-store5.13.1 - org.apache.activemqactivemq-mqtt5.13.1 - org.apache.apexapex-common3.7.0 - org.apache.apexapex-engine3.7.0 - args4jargs4j2.33 - org.apache.avroavro1.8.2 - com.google.cloud.bigdataossgcsio1.9.0 - com.google.cloud.bigdataossutil1.9.0 - com.google.cloud.bigtablebigtable-client-core1.4.0 - net.bytebuddybyte-buddy1.9.3 - org.apache.commonscommons-compress1.16.1 - org.apache.commonscommons-csv1.4 - commons-iocommons-io1.3.2 - commons-iocommons-io2.5 - org.apache.commonscommons-lang33.6 - org.apache.commonscommons-math33.6.1 - com.google.cloud.datastoredatastore-v1-proto-client1.6.0 - com.google.errorproneerror_prone_annotations2.0.15 - com.google.apigax-grpc1.29.0 - com.google.api-clientgoogle-api-client1.27.0 - com.google.api-clientgoogle-api-client-jackson21.27.0 - com.google.api-clientgoogle-api-client-java61.27.0 - com.google.apisgoogle-api-services-bigqueryv2-rev20181104-1.27.0 - com.google.apisgoogle-api-services-clouddebuggerv2-rev20180801-1.27.0 - com.google.apisgoogle-api-services-cloudresourcemanagerv1-rev20181015-1.27.0 - com.google.apisgoogle-api-services-dataflowv1b3-rev20181107-1.27.0 - com.google.apisgoogle-api-services-pubsubv1-rev20181105-1.27.0 - com.google.apisgoogle-api-services-storagev1-rev20181013-1.27.0 - com.google.authgoogle-auth-library-credentials0.10.0 - com.google.authgoogle-auth-library-oauth2-http0.10.0 - com.google.cloudgoogle-cloud-bigquery1.27.0 - com.google.cloudgoogle-cloud-core1.36.0 - com.google.cloudgoogle-cloud-core-grpc1.36.0 - com.google.cloud.dataflowgoogle-cloud-dataflow-java-proto-library-all0.5.160304 - com.google.cloudgoogle-cloud-spanner0.54.0-beta - com.google.http-clientgoogle-http-client1.27.0 - com.google.http-clientgoogle-http-client-jackson1.27.0 - com.google.http-clientgoogle-http-client-jackson21.27.0 - com.google.http-clientgoogle-http-client-protobuf1.27.0 - com.google.oauth-clientgoogle-oauth-client1.27.0 - com.google.oauth-clientgoogle-oauth-client-java61.27.0 - io.grpcgrpc-all1.13.1 - io.grpcgrpc-auth1.13.1 - io.grpcgrpc-core1.13.1 - com.google.api.grpcgrpc-google-cloud-pubsub-v11.18.0 - io.grpcgrpc-netty1.13.1 - io.grpcgrpc-protobuf-lite1.13.1 - io.grpcgrpc-protobuf1.13.1 - io.grpcgrpc-stub1.13.1 - com.google.guavaguava20.0 - com.google.guavaguava-testlib20.0 - org.apache.hadoophadoop-client2.7.3 - org.apache.hadoophadoop-common2.7.3 - org.apache.hadoophadoop-hdfs2.7.3 - org.apache.hadoophadoop-mapreduce-client-core2.7.3 - org.apache.hadoophadoop-minicluster2.7.3 - org.hamcresthamcrest-core1.3 - org.hamcresthamcrest-library1.3 - com.fasterxml.jackson.corejackson-annotations2.9.5 - com.fasterxml.jackson.corejackson-core2.9.5 - com.fasterxml.jackson.corejackson-databind2.9.5 - com.fasterxml.jackson.dataformatjackson-dataformat-cbor2.9.5 - com.fasterxml.jackson.dataformatjackson-dataformat-yaml2.9.5 - com.fasterxml.jackson.datatypejackson-datatype-joda2.9.5 - com.fasterxml.jackson.modulejackson-module-scala_2.112.9.5 - javax.xml.bindjaxb-api2.2.12 - joda-timejoda-time2.4 - junitjunit4.12 - org.apache.kafkakafka_2.111.0.0 - org.apache.kafkaka
[beam] branch master updated (bcc3e13 -> 9165828)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bcc3e13 Merge pull request #10802 from boyuanzz/refactor add 9165828 Remove tables and refer to dependency locations in code (#10745) No new revisions were added by this update. Summary of changes: .../src/documentation/sdks/java-dependencies.md| 355 ++-- .../src/documentation/sdks/python-dependencies.md | 473 ++--- 2 files changed, 62 insertions(+), 766 deletions(-)
[beam] branch master updated: Move ThreadsafeRestrictionTracker and RestrictionTrackerView out from iobase.py
This is an automated email from the ASF dual-hosted git repository. boyuanz pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new cd6e54b Move ThreadsafeRestrictionTracker and RestrictionTrackerView out from iobase.py new bcc3e13 Merge pull request #10802 from boyuanzz/refactor cd6e54b is described below commit cd6e54bc19f5e69cab49d22f5044b1c869b9ec69 Author: Boyuan Zhang AuthorDate: Fri Feb 7 16:07:20 2020 -0800 Move ThreadsafeRestrictionTracker and RestrictionTrackerView out from iobase.py --- sdks/python/apache_beam/io/iobase.py | 128 --- sdks/python/apache_beam/io/iobase_test.py | 86 +- sdks/python/apache_beam/runners/common.py | 69 .../runners/portability/fn_api_runner_test.py | 8 +- sdks/python/apache_beam/runners/sdf_utils.py | 176 + sdks/python/apache_beam/runners/sdf_utils_test.py | 114 + .../apache_beam/runners/worker/bundle_processor.py | 16 +- 7 files changed, 337 insertions(+), 260 deletions(-) diff --git a/sdks/python/apache_beam/io/iobase.py b/sdks/python/apache_beam/io/iobase.py index 10d2933..1302c38 100644 --- a/sdks/python/apache_beam/io/iobase.py +++ b/sdks/python/apache_beam/io/iobase.py @@ -37,7 +37,6 @@ from __future__ import division import logging import math import random -import threading import uuid from builtins import object from builtins import range @@ -65,9 +64,7 @@ from apache_beam.utils import urns from apache_beam.utils.windowed_value import WindowedValue if TYPE_CHECKING: - from apache_beam.io import restriction_trackers from apache_beam.runners.pipeline_context import PipelineContext - from apache_beam.utils.timestamp import Timestamp __all__ = [ 'BoundedSource', @@ -1246,131 +1243,6 @@ class RestrictionTracker(object): raise NotImplementedError -class ThreadsafeRestrictionTracker(object): - """A thread-safe wrapper which wraps a `RestritionTracker`. - - This wrapper guarantees synchronization of modifying restrictions across - multi-thread. - """ - def __init__(self, restriction_tracker): -# type: (RestrictionTracker) -> None -if not isinstance(restriction_tracker, RestrictionTracker): - raise ValueError( - 'Initialize ThreadsafeRestrictionTracker requires' - 'RestrictionTracker.') -self._restriction_tracker = restriction_tracker -# Records an absolute timestamp when defer_remainder is called. -self._deferred_timestamp = None -self._lock = threading.RLock() -self._deferred_residual = None -self._deferred_watermark = None - - def current_restriction(self): -with self._lock: - return self._restriction_tracker.current_restriction() - - def try_claim(self, position): -with self._lock: - return self._restriction_tracker.try_claim(position) - - def defer_remainder(self, deferred_time=None): -"""Performs self-checkpoint on current processing restriction with an -expected resuming time. - -Self-checkpoint could happen during processing elements. When executing an -DoFn.process(), you may want to stop processing an element and resuming -later if current element has been processed quit a long time or you also -want to have some outputs from other elements. ``defer_remainder()`` can be -called on per element if needed. - -Args: - deferred_time: A relative ``timestamp.Duration`` that indicates the ideal - time gap between now and resuming, or an absolute ``timestamp.Timestamp`` - for resuming execution time. If the time_delay is None, the deferred work - will be executed as soon as possible. -""" - -# Record current time for calculating deferred_time later. -self._deferred_timestamp = timestamp.Timestamp.now() -if (deferred_time and not isinstance(deferred_time, timestamp.Duration) and -not isinstance(deferred_time, timestamp.Timestamp)): - raise ValueError( - 'The timestamp of deter_remainder() should be a ' - 'Duration or a Timestamp, or None.') -self._deferred_watermark = deferred_time -checkpoint = self.try_split(0) -if checkpoint: - _, self._deferred_residual = checkpoint - - def check_done(self): -with self._lock: - return self._restriction_tracker.check_done() - - def current_progress(self): -with self._lock: - return self._restriction_tracker.current_progress() - - def try_split(self, fraction_of_remainder): -with self._lock: - return self._restriction_tracker.try_split(fraction_of_remainder) - - def deferred_status(self): -# type: () -> Optional[Tuple[Any, Timestamp]] - -"""Returns deferred work which is produced by ``defer_remainder()``. - -When there is a self-checkpoint performed, the system needs to fulfill the -DelayedBundleApplication with def
[beam] branch master updated (930b5ef -> 6d721c1)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 930b5ef Merge pull request #10825: [BEAM-9160] Update AWS SDK to support Pod Level Identity add 0a15bca Move external PubsubIO hooks outside of PubsubIO. add 6d721c1 Merge pull request #10476: [BEAM-8932][Cleanup] Move external PubsubIO hooks outside of PubsubIO. No new revisions were added by this update. Summary of changes: .../beam/sdk/io/gcp/pubsub/ExternalRead.java | 138 + .../beam/sdk/io/gcp/pubsub/ExternalWrite.java | 102 .../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java| 172 + 3 files changed, 242 insertions(+), 170 deletions(-) create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalRead.java create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/ExternalWrite.java
[beam] branch master updated (e24d1e5 -> 930b5ef)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e24d1e5 Merge pull request #10820: [BEAM-8630] Validate prepared expression on expand add 1fee64c [BEAM-9160] Update AWS SDK to support Pod Level Identity add 930b5ef Merge pull request #10825: [BEAM-9160] Update AWS SDK to support Pod Level Identity No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 4 +- .../apache/beam/sdk/io/aws/options/AwsModule.java | 4 + .../beam/sdk/io/aws/options/AwsModuleTest.java | 7 + .../apache/beam/sdk/io/aws/sns/AmazonSNSMock.java | 385 - .../beam/sdk/io/aws/sns/AmazonSNSMockErrors.java | 30 -- .../beam/sdk/io/aws/sns/AmazonSNSMockSuccess.java | 39 --- .../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 43 ++- .../apache/beam/sdk/io/aws2/options/AwsModule.java | 4 + .../beam/sdk/io/aws2/options/AwsModuleTest.java| 7 + 9 files changed, 65 insertions(+), 458 deletions(-) delete mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMock.java delete mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMockErrors.java delete mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/sns/AmazonSNSMockSuccess.java
