[beam] branch master updated (50fcf55 -> 87e1164)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 50fcf55 Merge pull request #14639: [BEAM-10277][BEAM-12198] Populate encoding positions to all matching schemas add 1a5f90e Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" new 87e1164 Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .test-infra/jenkins/job_PostCommit_Python.groovy | 2 +- ...ommit_Python_ValidatesContainer_Dataflow.groovy | 2 +- ...stCommit_Python_ValidatesRunner_Dataflow.groovy | 2 +- ...ommit_Python_ValidatesRunner_Dataflow_V2.groovy | 2 +- ..._PostCommit_Python_ValidatesRunner_Flink.groovy | 4 -- .../examples/complete/autocomplete_test.py | 4 +- .../examples/complete/game/game_stats_it_test.py | 6 +- .../complete/game/hourly_team_score_it_test.py | 6 +- .../examples/complete/game/leader_board_it_test.py | 6 +- .../examples/complete/game/user_score_it_test.py | 6 +- .../complete/juliaset/juliaset/juliaset_test_it.py | 4 +- .../cookbook/bigquery_tornadoes_it_test.py | 4 +- .../cookbook/datastore_wordcount_it_test.py| 4 +- .../apache_beam/examples/fastavro_it_test.py | 8 +-- .../streaming_wordcount_debugging_it_test.py | 4 +- .../examples/streaming_wordcount_it_test.py| 4 +- .../apache_beam/examples/wordcount_it_test.py | 11 ++-- sdks/python/apache_beam/io/fileio_test.py | 4 +- .../io/gcp/big_query_query_to_table_it_test.py | 14 ++-- .../apache_beam/io/gcp/bigquery_file_loads_test.py | 8 +-- .../apache_beam/io/gcp/bigquery_io_read_it_test.py | 6 +- .../apache_beam/io/gcp/bigquery_read_it_test.py| 12 ++-- sdks/python/apache_beam/io/gcp/bigquery_test.py| 12 ++-- .../apache_beam/io/gcp/bigquery_write_it_test.py | 12 ++-- .../gcp/datastore/v1new/datastore_write_it_test.py | 4 +- .../apache_beam/io/gcp/dicomio_integration_test.py | 6 +- .../io/gcp/experimental/spannerio_read_it_test.py | 6 +- .../io/gcp/experimental/spannerio_write_it_test.py | 8 +-- .../apache_beam/io/gcp/gcsio_integration_test.py | 14 ++-- .../apache_beam/io/gcp/pubsub_integration_test.py | 6 +- sdks/python/apache_beam/io/parquetio_it_test.py| 4 +- sdks/python/apache_beam/metrics/metric_test.py | 4 +- .../python/apache_beam/ml/gcp/cloud_dlp_it_test.py | 6 +- .../ml/gcp/naturallanguageml_test_it.py| 4 +- .../ml/gcp/videointelligenceml_test_it.py | 4 +- sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 4 +- sdks/python/apache_beam/pipeline_test.py | 9 +-- .../dataflow_exercise_metrics_pipeline_test.py | 7 +- ...low_exercise_streaming_metrics_pipeline_test.py | 6 +- .../portability/fn_api_runner/fn_runner_test.py| 4 +- .../portability/fn_api_runner/translations_test.py | 6 +- sdks/python/apache_beam/testing/test_pipeline.py | 13 ++-- .../apache_beam/testing/test_stream_it_test.py | 8 +-- .../transforms/combinefn_lifecycle_test.py | 4 +- .../apache_beam/transforms/dofn_lifecycle_test.py | 4 +- .../apache_beam/transforms/external_it_test.py | 4 +- .../apache_beam/transforms/ptransform_test.py | 26 .../apache_beam/transforms/sideinputs_test.py | 29 - sdks/python/apache_beam/transforms/util_test.py| 4 +- sdks/python/conftest.py| 7 -- sdks/python/container/run_validatescontainer.sh| 17 ++--- sdks/python/pytest.ini | 5 -- sdks/python/scripts/run_integration_test.sh| 39 +++ sdks/python/test-suites/dataflow/common.gradle | 52 +++ sdks/python/test-suites/direct/common.gradle | 76 ++ sdks/python/test-suites/portable/common.gradle | 19 +++--- 56 files changed, 246 insertions(+), 310 deletions(-)
[beam] 01/01: Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)"
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 87e11644c44a4c677ec2faa78f50cdffbb33605a Merge: 50fcf55 1a5f90e Author: Daniel Oliveira AuthorDate: Wed Apr 28 22:55:14 2021 -0700 Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" .test-infra/jenkins/job_PostCommit_Python.groovy | 2 +- ...ommit_Python_ValidatesContainer_Dataflow.groovy | 2 +- ...stCommit_Python_ValidatesRunner_Dataflow.groovy | 2 +- ...ommit_Python_ValidatesRunner_Dataflow_V2.groovy | 2 +- ..._PostCommit_Python_ValidatesRunner_Flink.groovy | 4 -- .../examples/complete/autocomplete_test.py | 4 +- .../examples/complete/game/game_stats_it_test.py | 6 +- .../complete/game/hourly_team_score_it_test.py | 6 +- .../examples/complete/game/leader_board_it_test.py | 6 +- .../examples/complete/game/user_score_it_test.py | 6 +- .../complete/juliaset/juliaset/juliaset_test_it.py | 4 +- .../cookbook/bigquery_tornadoes_it_test.py | 4 +- .../cookbook/datastore_wordcount_it_test.py| 4 +- .../apache_beam/examples/fastavro_it_test.py | 8 +-- .../streaming_wordcount_debugging_it_test.py | 4 +- .../examples/streaming_wordcount_it_test.py| 4 +- .../apache_beam/examples/wordcount_it_test.py | 11 ++-- sdks/python/apache_beam/io/fileio_test.py | 4 +- .../io/gcp/big_query_query_to_table_it_test.py | 14 ++-- .../apache_beam/io/gcp/bigquery_file_loads_test.py | 8 +-- .../apache_beam/io/gcp/bigquery_io_read_it_test.py | 6 +- .../apache_beam/io/gcp/bigquery_read_it_test.py| 12 ++-- sdks/python/apache_beam/io/gcp/bigquery_test.py| 12 ++-- .../apache_beam/io/gcp/bigquery_write_it_test.py | 12 ++-- .../gcp/datastore/v1new/datastore_write_it_test.py | 4 +- .../apache_beam/io/gcp/dicomio_integration_test.py | 6 +- .../io/gcp/experimental/spannerio_read_it_test.py | 6 +- .../io/gcp/experimental/spannerio_write_it_test.py | 8 +-- .../apache_beam/io/gcp/gcsio_integration_test.py | 14 ++-- .../apache_beam/io/gcp/pubsub_integration_test.py | 6 +- sdks/python/apache_beam/io/parquetio_it_test.py| 4 +- sdks/python/apache_beam/metrics/metric_test.py | 4 +- .../python/apache_beam/ml/gcp/cloud_dlp_it_test.py | 6 +- .../ml/gcp/naturallanguageml_test_it.py| 4 +- .../ml/gcp/videointelligenceml_test_it.py | 4 +- sdks/python/apache_beam/ml/gcp/visionml_test_it.py | 4 +- sdks/python/apache_beam/pipeline_test.py | 9 +-- .../dataflow_exercise_metrics_pipeline_test.py | 7 +- ...low_exercise_streaming_metrics_pipeline_test.py | 6 +- .../portability/fn_api_runner/fn_runner_test.py| 4 +- .../portability/fn_api_runner/translations_test.py | 6 +- sdks/python/apache_beam/testing/test_pipeline.py | 13 ++-- .../apache_beam/testing/test_stream_it_test.py | 8 +-- .../transforms/combinefn_lifecycle_test.py | 4 +- .../apache_beam/transforms/dofn_lifecycle_test.py | 4 +- .../apache_beam/transforms/external_it_test.py | 4 +- .../apache_beam/transforms/ptransform_test.py | 26 .../apache_beam/transforms/sideinputs_test.py | 29 - sdks/python/apache_beam/transforms/util_test.py| 4 +- sdks/python/conftest.py| 7 -- sdks/python/container/run_validatescontainer.sh| 17 ++--- sdks/python/pytest.ini | 5 -- sdks/python/scripts/run_integration_test.sh| 39 +++ sdks/python/test-suites/dataflow/common.gradle | 52 +++ sdks/python/test-suites/direct/common.gradle | 76 ++ sdks/python/test-suites/portable/common.gradle | 19 +++--- 56 files changed, 246 insertions(+), 310 deletions(-)
[beam] branch master updated: Merge pull request #14639: [BEAM-10277][BEAM-12198] Populate encoding positions to all matching schemas
This is an automated email from the ASF dual-hosted git repository.
reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 50fcf55 Merge pull request #14639: [BEAM-10277][BEAM-12198] Populate
encoding positions to all matching schemas
50fcf55 is described below
commit 50fcf55d60e8fa7a8399d63e030c930a2d45402a
Author: reuvenlax
AuthorDate: Wed Apr 28 21:13:57 2021 -0700
Merge pull request #14639: [BEAM-10277][BEAM-12198] Populate encoding
positions to all matching schemas
---
model/pipeline/src/main/proto/schema.proto | 4 ++
.../util/RowCoderCloudObjectTranslator.java| 6 +++
.../util/SchemaCoderCloudObjectTranslator.java | 6 +++
.../java/org/apache/beam/sdk/coders/RowCoder.java | 7 +++
.../apache/beam/sdk/coders/RowCoderGenerator.java | 9 +++-
.../java/org/apache/beam/sdk/schemas/Schema.java | 8
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 6 +++
.../apache/beam/sdk/schemas/SchemaTranslation.java | 2 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 56 +-
.../impl/transform/BeamSqlOutputToConsoleFn.java | 2 +-
.../apache/beam/sdk/extensions/sql/TestUtils.java | 6 +--
11 files changed, 104 insertions(+), 8 deletions(-)
diff --git a/model/pipeline/src/main/proto/schema.proto
b/model/pipeline/src/main/proto/schema.proto
index a40087c..837689f 100644
--- a/model/pipeline/src/main/proto/schema.proto
+++ b/model/pipeline/src/main/proto/schema.proto
@@ -37,6 +37,8 @@ message Schema {
// REQUIRED. An RFC 4122 UUID.
string id = 2;
repeated Option options = 3;
+ // Indicates that encoding positions have been overridden.
+ bool encoding_positions_set = 4;
}
message Field {
@@ -52,6 +54,8 @@ message Field {
// or all of them are. Used to support backwards compatibility with schema
// changes.
// If no fields have encoding position populated the order of encoding is
the same as the order in the Schema.
+ // If this Field is part of a Schema where encoding_positions_set is True
then encoding_position must be
+ // defined, otherwise this field is ignored.
int32 encoding_position = 5;
repeated Option options = 6;
}
diff --git
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RowCoderCloudObjectTranslator.java
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RowCoderCloudObjectTranslator.java
index 6de8321..e84d89b 100644
---
a/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RowCoderCloudObjectTranslator.java
+++
b/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/util/RowCoderCloudObjectTranslator.java
@@ -18,6 +18,8 @@
package org.apache.beam.runners.dataflow.util;
import java.io.IOException;
+import java.util.UUID;
+import javax.annotation.Nullable;
import org.apache.beam.model.pipeline.v1.SchemaApi;
import org.apache.beam.runners.core.construction.SdkComponents;
import org.apache.beam.sdk.annotations.Experimental;
@@ -59,6 +61,10 @@ public class RowCoderCloudObjectTranslator implements
CloudObjectTranslator {
return new RowCoder(schema);
}
+ /** Override encoding positions for the given schema. */
+ public static void overrideEncodingPositions(UUID uuid, Map
encodingPositions) {
+SchemaCoder.overrideEncodingPositions(uuid, encodingPositions);
+ }
+
private RowCoder(Schema schema) {
super(
schema,
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
index 46c7758..a8d45b7 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java
@@ -112,9 +112,15 @@ public abstract class RowCoderGenerator {
// Cache for Coder class that are already generated.
private static final Map> GENERATED_CODERS =
Maps.newConcurrentMap();
+ private static final Map>
ENCODING_POSITION_OVERRIDES =
+ Maps.newConcurrentMap();
private static final Logger LOG =
LoggerFactory.getLogger(RowCoderGenerator.class);
+ public static void overrideEncodingPositions(UUID uuid, Map
encodingPositions) {
+ENCODING_POSITION_OVERRIDES.put(uuid, encodingPositions);
+ }
+
@SuppressWarnings("unchecked")
public static Coder generate(Schema schema) {
// Using ConcurrentHashMap::computeIfAbsent here would deadlock in case of
nested
@@ -128,7 +134,8 @@ public abstract class RowCoderGenerator {
builder = implementMethods(schema, builder);
int[] encodingPosToRowIndex = new int[schema.getFieldCount()];
- Map encodingPositions = schema.getEncodingPositions();
+ Map encodingPositions =
+
[beam] branch master updated (2f02771 -> 35881d7)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2f02771 Merge pull request #14491 from ihji/BEAM-12140 add 7b51b51 [BEAM-7372] cleanup following codes from apache_beam/runners/dataflow and apache_beam/runners/interactive * usage of future package * usage of past package * unnecessary builtins import add 35881d7 [BEAM-7372] cleanup py2 codepath from runners/dataflow and runners/interactive #14564 No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/dataflow/__init__.py | 2 -- .../runners/dataflow/dataflow_exercise_metrics_pipeline.py | 2 -- .../dataflow/dataflow_exercise_metrics_pipeline_test.py| 2 -- .../dataflow/dataflow_exercise_streaming_metrics_pipeline.py | 2 -- .../dataflow_exercise_streaming_metrics_pipeline_test.py | 2 -- sdks/python/apache_beam/runners/dataflow/dataflow_metrics.py | 6 +- .../apache_beam/runners/dataflow/dataflow_metrics_test.py | 3 --- sdks/python/apache_beam/runners/dataflow/dataflow_runner.py| 10 ++ .../apache_beam/runners/dataflow/dataflow_runner_test.py | 4 sdks/python/apache_beam/runners/dataflow/internal/__init__.py | 1 - sdks/python/apache_beam/runners/dataflow/internal/apiclient.py | 7 +-- .../apache_beam/runners/dataflow/internal/apiclient_test.py| 2 -- .../apache_beam/runners/dataflow/internal/clients/__init__.py | 1 - .../runners/dataflow/internal/clients/cloudbuild/__init__.py | 2 -- .../runners/dataflow/internal/clients/dataflow/__init__.py | 2 -- .../dataflow/internal/clients/dataflow/dataflow_v1b3_client.py | 2 -- .../internal/clients/dataflow/dataflow_v1b3_messages.py| 2 -- .../dataflow/internal/clients/dataflow/message_matchers.py | 5 + .../internal/clients/dataflow/message_matchers_test.py | 2 -- sdks/python/apache_beam/runners/dataflow/internal/names.py | 5 - sdks/python/apache_beam/runners/dataflow/native_io/__init__.py | 1 - sdks/python/apache_beam/runners/dataflow/native_io/iobase.py | 3 --- .../apache_beam/runners/dataflow/native_io/iobase_test.py | 2 -- .../apache_beam/runners/dataflow/ptransform_overrides.py | 2 -- .../apache_beam/runners/dataflow/template_runner_test.py | 2 -- .../apache_beam/runners/dataflow/test_dataflow_runner.py | 3 --- .../apache_beam/runners/interactive/augmented_pipeline.py | 2 -- .../apache_beam/runners/interactive/augmented_pipeline_test.py | 2 -- .../apache_beam/runners/interactive/background_caching_job.py | 2 -- .../runners/interactive/background_caching_job_test.py | 2 -- sdks/python/apache_beam/runners/interactive/cache_manager.py | 4 .../apache_beam/runners/interactive/cache_manager_test.py | 4 .../python/apache_beam/runners/interactive/caching/__init__.py | 2 -- .../apache_beam/runners/interactive/caching/cacheable.py | 2 -- .../apache_beam/runners/interactive/caching/read_cache.py | 2 -- .../apache_beam/runners/interactive/caching/read_cache_test.py | 2 -- .../apache_beam/runners/interactive/caching/streaming_cache.py | 2 -- .../runners/interactive/caching/streaming_cache_test.py| 2 -- .../apache_beam/runners/interactive/caching/write_cache.py | 2 -- .../runners/interactive/caching/write_cache_test.py| 2 -- .../apache_beam/runners/interactive/display/display_manager.py | 4 .../runners/interactive/display/interactive_pipeline_graph.py | 4 .../runners/interactive/display/pcoll_visualization.py | 3 --- .../runners/interactive/display/pcoll_visualization_test.py| 2 -- .../apache_beam/runners/interactive/display/pipeline_graph.py | 4 .../runners/interactive/display/pipeline_graph_renderer.py | 8 +--- .../runners/interactive/display/pipeline_graph_test.py | 2 -- .../python/apache_beam/runners/interactive/interactive_beam.py | 2 -- .../apache_beam/runners/interactive/interactive_beam_test.py | 2 -- .../apache_beam/runners/interactive/interactive_environment.py | 2 -- .../runners/interactive/interactive_environment_test.py| 2 -- .../apache_beam/runners/interactive/interactive_runner.py | 4 .../apache_beam/runners/interactive/interactive_runner_test.py | 4 .../interactive/messaging/interactive_environment_inspector.py | 2 -- .../messaging/interactive_environment_inspector_test.py| 2 -- .../apache_beam/runners/interactive/options/capture_control.py | 2 -- .../runners/interactive/options/capture_control_test.py| 2 -- .../runners/interactive/options/capture_limiters.py| 2 -- .../runners/interactive/options/capture_limiters_test.py | 2 -- .../runners/interactive/options/interactive_options.py | 2 --
[beam] tag nightly-master updated (4202fb8 -> 2f02771)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag nightly-master was modified! *** from 4202fb8 (commit) to 2f02771 (commit) from 4202fb8 Merge pull request #14661 - Fix worker_pool plumbing to use a mutable List. add 62b9c0f [BEAM-9283] Disable caching ValidatesRunner tests tasks add ea107d0 Merge pull request #14640: [BEAM-9283] Disable caching ValidatesRunner tests tasks add 729a5ed [BEAM-2085] Add resource hints support in Java SDK. add d15c210 Translate Resource Hints into v1beta3 API in DF Runner and add test coverage. add 270903b Support passing resource hints through pipeline options. add 9a9fb44 Add resource hints supported by Dataflow. add b1e4ece [BEAM-2085] Support resource hints on java transforms. #14607 add d284d07 Enable dual job submission for Dataflow add a25261a Merge pull request #14633: [BEAM-12213] Enable dual job submission for Dataflow add 8a24eb8 [BEAM-11055] Update log4j to version 2.14.1 add cbc67c8 Merge pull request #13073: [BEAM-11055] Update log4j to version 2.14.1 add da03439 Remove WindowFn.getOutputTime add defca4d Merge pull request #14650: [BEAM-12011] Remove WindowFn.getOutputTime add 9f6a8c6 Update run_rc_validation.sh to pull RC tag add 2d0f36c Merge pull request #14613: Update run_rc_validation.sh to pull RC tag add 2020555 [BEAM-12227] Implement ExternalWorkerService#stopWorker to support LOOPBACK add a5cb30d Add ExternalWorkerServiceTest add 0bb5fd8 Merge pull request #14658 from kw2542/BEAM-12227 add d871b5e Refinements to the portability page, reflecting its state of completeness. add 1fe677b Merge pull request #14662 Refinements to the portability page, reflecting its state of completeness. add 73d9356 Merge pull request #14615 from [BEAM-12206] Update BigQuery IO documentation with quota and sharding information add 6f6715e [BEAM-12188] Bump snakeyaml to 1.28 add bc2fb2a Merge pull request #14645 from [BEAM-12188] Bump snakeyaml to 1.28 add 6ebadec [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" add b1a67ef Merge pull request #14670: [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" add 331ac5d [BEAM-12229] StreamingModeExecutionContext invalidated WindmillStateCache keys repeatedly for fused stages with more than one stage. add faf1542 Merge pull request #14649: [BEAM-12229] StreamingModeExecutionContext invalidated WindmillStateCache keys repeatedly for fused stages with more than one stage. add 4872cf7 Revert "[BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable." add 5e4d632 Merge pull request #14673: Revert "[BEAM-11994] Refactor BigQueryTornadoes to make more options … add 7e59751 Use try-with-resources in ZipFiles add d4e7697 Allow overwriting zip file in packageDirectoriesToStage add a2e440e [BEAM-12220] Use try-with-resources in ZipFiles (#14654) add f5ebfad Add scikit-learn to additional user packages (#14664) add e941e77 [BEAM-12140] Update artifact information when staging artifacts to gcs for Dataflow add 217b8c1 use get_scheme add 627468a fix typo, add optional sha256 to url payload add 8d45785 fix formatting add 2f02771 Merge pull request #14491 from ihji/BEAM-12140 No new revisions were added by this update. Summary of changes: .test-infra/validate-runner/build.gradle | 2 +- CHANGES.md | 3 +- .../org/apache/beam/gradle/BeamModulePlugin.groovy | 3 + .../beam/examples/cookbook/BigQueryTornadoes.java | 91 +++ .../pipeline/src/main/proto/beam_runner_api.proto | 3 + release/src/main/scripts/run_rc_validation.sh | 11 +- .../beam/runners/core/construction/External.java | 3 + .../core/construction/PTransformTranslation.java | 9 +- .../core/construction/ParDoTranslation.java| 2 +- .../runners/core/construction/SdkComponents.java | 41 ++- .../construction/WindowingStrategyTranslation.java | 3 +- .../construction/resources/PipelineResources.java | 15 +- .../CreatePCollectionViewTranslationTest.java | 3 + .../EmptyFlattenAsCreateFactoryTest.java | 4 + .../GroupIntoBatchesTranslationTest.java | 15 +- .../core/construction/PTransformMatchersTest.java | 21 +- .../construction/PTransformReplacementsTest.java | 5 + .../construction/PTransformTranslationTest.java| 11 +- .../core/construction/ParDoTranslationTest.java| 3 +- .../core/construction/PipelineTranslationTest.java | 52 .../core/construction/SdkComponentsTest.java | 53 +++- .../core/construction/SplittableParDoTest.java | 2 +
[beam] branch master updated (f5ebfad -> 2f02771)
This is an automated email from the ASF dual-hosted git repository. heejong pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f5ebfad Add scikit-learn to additional user packages (#14664) new e941e77 [BEAM-12140] Update artifact information when staging artifacts to gcs for Dataflow new 217b8c1 use get_scheme new 627468a fix typo, add optional sha256 to url payload new 8d45785 fix formatting new 2f02771 Merge pull request #14491 from ihji/BEAM-12140 The 31832 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../pipeline/src/main/proto/beam_runner_api.proto | 3 ++ .../runners/dataflow/internal/apiclient.py | 32 +++ .../runners/dataflow/internal/apiclient_test.py| 48 ++ 3 files changed, 75 insertions(+), 8 deletions(-)
[beam] branch youngoli-patch-1 created (now cbd0b5c)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch youngoli-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at cbd0b5c Update golock instructions No new revisions were added by this update.
[beam] branch master updated (a2e440e -> f5ebfad)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a2e440e [BEAM-12220] Use try-with-resources in ZipFiles (#14654) add f5ebfad Add scikit-learn to additional user packages (#14664) No new revisions were added by this update. Summary of changes: sdks/python/container/base_image_requirements.txt | 1 + 1 file changed, 1 insertion(+)
[beam] branch asf-site updated: Publishing website 2021/04/29 00:02:24 at commit a2e440e
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 9c89c89 Publishing website 2021/04/29 00:02:24 at commit a2e440e 9c89c89 is described below commit 9c89c899b3919cf2e85f5d60e4aed9cf2eb042fd Author: jenkins AuthorDate: Thu Apr 29 00:02:24 2021 + Publishing website 2021/04/29 00:02:24 at commit a2e440e --- website/generated-content/documentation/index.xml | 73 -- .../io/built-in/google-bigquery/index.html | 48 +- website/generated-content/sitemap.xml | 2 +- 3 files changed, 89 insertions(+), 34 deletions(-) diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml index 6b29bf0..039472b 100644 --- a/website/generated-content/documentation/index.xml +++ b/website/generated-content/documentation/index.xml @@ -15264,36 +15264,71 @@ as the previous example./p> p>BigQueryIO supports two methods of inserting data into BigQuery: load jobs and streaming inserts. Each insertion method provides different tradeoffs of cost, quota, and data consistency. See the BigQuery documentation for -a href="https://cloud.google.com/bigquery/loading-data;>load jobs/a> and -a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery;>streaming inserts/a> +a href="https://cloud.google.com/bigquery/loading-data;>different data ingestion options/a> +(specifically, a href="https://cloud.google.com/bigquery/docs/batch-loading-data;>load jobs/a> +and a href="https://cloud.google.com/bigquery/streaming-data-into-bigquery;>streaming inserts/a>) for more information about these tradeoffs./p> -p>BigQueryIO chooses a default insertion method based on the input code>PCollection/code>./p> -p class="language-py">BigQueryIO uses load jobs when you apply a BigQueryIO write transform to a -bounded code>PCollection/code>./p> -p class="language-java">BigQueryIO uses load jobs in the following situations:/p> +p class="language-java">BigQueryIO chooses a default insertion method based on the input code>PCollection/code>. +You can use code>withMethod/code> to specify the desired insertion method. See +a href="https://beam.apache.org/releases/javadoc/2.28.0/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.Write.Method.html;>code>Write.Method/code>/a> +for the list of the available methods and their restrictions./p> +p class="language-py">BigQueryIO chooses a default insertion method based on the input code>PCollection/code>. +You can use code>method/code> to specify the desired insertion method. See +a href="https://beam.apache.org/releases/pydoc/2.28.0/apache_beam.io.gcp.bigquery.html#apache_beam.io.gcp.bigquery.WriteToBigQuery;>code>WriteToBigQuery/code>/a> +for the list of the available methods and their restrictions./p> +p>BigQueryIO uses load jobs in the following situations:/p> span class="language-java">ul> li>When you apply a BigQueryIO write transform to a bounded code>PCollection/code>./li> -li>When you apply a BigQueryIO write transform to an unbounded code>PCollection/code> and -use code>BigQueryIO.write().withTriggeringFrequency()/code> to set the triggering -frequency./li> li>When you specify load jobs as the insertion method using code>BigQueryIO.write().withMethod(FILE_LOADS)/code>./li> /ul>/span> -p class="language-py">BigQueryIO uses streaming inserts when you apply a BigQueryIO write transform to -an unbounded code>PCollection/code>./p> -p class="language-java">BigQueryIO uses streaming inserts in the following situations:/p> +span class="language-py">ul> +li>When you apply a BigQueryIO write transform to a bounded code>PCollection/code>./li> +li>When you specify load jobs as the insertion method using +code>WriteToBigQuery(method='FILE_LOADS')/code>./li> +/ul>/span> +p>em>strong>Note:/strong>/em> If you use batch loads in a streaming pipeline:/p> +p class="language-java">You must use code>withTriggeringFrequency/code> to specify a triggering frequency for +initiating load jobs. Be careful about setting the frequency such that your +pipeline doesnrsquo;t exceed the BigQuery load job a href="https://cloud.google.com/bigquery/quotas#load_jobs;>quota limit/a>./p> +p class="language-java">You can either use code>withNumFileShards/code> to explicitly set the number of file +shards written, or use code>withAutoSharding/code> to enable dynamic sharding (starting +2.29.0 release) and the number of shards may be determined and changed at +runtime. The sharding behavior depends on the runners./p> +p class="language-py">You must use code>triggering_frequency/code> to specify a triggering frequency for +initiating load jobs. Be careful about setting the frequency such that your +pipeline doesnrsquo;t exceed the BigQuery load job a
[beam] branch master updated (5e4d632 -> a2e440e)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5e4d632 Merge pull request #14673: Revert "[BEAM-11994] Refactor BigQueryTornadoes to make more options … add 7e59751 Use try-with-resources in ZipFiles add d4e7697 Allow overwriting zip file in packageDirectoriesToStage add a2e440e [BEAM-12220] Use try-with-resources in ZipFiles (#14654) No new revisions were added by this update. Summary of changes: .../construction/resources/PipelineResources.java | 15 +++ .../java/org/apache/beam/sdk/util/ZipFiles.java| 50 ++ 2 files changed, 37 insertions(+), 28 deletions(-)
[beam] branch master updated (faf1542 -> 5e4d632)
This is an automated email from the ASF dual-hosted git repository. danoliveira pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from faf1542 Merge pull request #14649: [BEAM-12229] StreamingModeExecutionContext invalidated WindmillStateCache keys repeatedly for fused stages with more than one stage. add 4872cf7 Revert "[BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable." add 5e4d632 Merge pull request #14673: Revert "[BEAM-11994] Refactor BigQueryTornadoes to make more options … No new revisions were added by this update. Summary of changes: .../beam/examples/cookbook/BigQueryTornadoes.java | 91 +++--- 1 file changed, 44 insertions(+), 47 deletions(-)
[beam] branch master updated (b1a67ef -> faf1542)
This is an automated email from the ASF dual-hosted git repository. reuvenlax pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b1a67ef Merge pull request #14670: [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" add 331ac5d [BEAM-12229] StreamingModeExecutionContext invalidated WindmillStateCache keys repeatedly for fused stages with more than one stage. add faf1542 Merge pull request #14649: [BEAM-12229] StreamingModeExecutionContext invalidated WindmillStateCache keys repeatedly for fused stages with more than one stage. No new revisions were added by this update. Summary of changes: .../dataflow/worker/StreamingDataflowWorker.java | 2 +- .../worker/StreamingModeExecutionContext.java | 27 +- .../dataflow/worker/WindmillStateCache.java| 41 +-- .../worker/StreamingDataflowWorkerTest.java| 303 + .../dataflow/worker/WindmillStateCacheTest.java| 133 ++--- .../worker/WindmillStateInternalsTest.java | 8 +- 6 files changed, 449 insertions(+), 65 deletions(-)
[beam] branch aaltay-patch-1 updated (3897c005 -> 94a1874)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch aaltay-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. from 3897c005 Add scikit-learn to additional user packages add 94a1874 Remove python version requirement No new revisions were added by this update. Summary of changes: sdks/python/container/base_image_requirements.txt | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (bc2fb2a -> b1a67ef)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bc2fb2a Merge pull request #14645 from [BEAM-12188] Bump snakeyaml to 1.28 add 6ebadec [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" add b1a67ef Merge pull request #14670: [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" No new revisions were added by this update. Summary of changes: sdks/java/testing/tpcds/src/main/resources/queries/query15.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query19.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query23a.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query23b.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query45.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query62.sql | 6 +++--- sdks/java/testing/tpcds/src/main/resources/queries/query79.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query8.sql | 8 sdks/java/testing/tpcds/src/main/resources/queries/query85.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query99.sql | 6 +++--- 10 files changed, 21 insertions(+), 21 deletions(-)
[beam] branch master updated: [BEAM-12188] Bump snakeyaml to 1.28
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 6f6715e [BEAM-12188] Bump snakeyaml to 1.28
new bc2fb2a Merge pull request #14645 from [BEAM-12188] Bump snakeyaml to
1.28
6f6715e is described below
commit 6f6715e271d45de2d206afff3dafd77b9decbc35
Author: masahitojp
AuthorDate: Tue Apr 27 00:01:16 2021 +0900
[BEAM-12188] Bump snakeyaml to 1.28
---
.test-infra/validate-runner/build.gradle | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/.test-infra/validate-runner/build.gradle
b/.test-infra/validate-runner/build.gradle
index cd062f5..6e8cb49 100644
--- a/.test-infra/validate-runner/build.gradle
+++ b/.test-infra/validate-runner/build.gradle
@@ -39,7 +39,7 @@ dependencies {
compile 'com.offbytwo.jenkins:jenkins-client:0.3.8'
compile group: 'com.fasterxml.jackson.core', name: 'jackson-databind',
version: '2.0.1'
compile group: 'org.jenkins-ci.plugins', name: 'junit', version: '1.34'
-compile group: 'org.yaml', name: 'snakeyaml', version: '1.8'
+compile group: 'org.yaml', name: 'snakeyaml', version: '1.28'
compile group: 'com.fasterxml.jackson.dataformat', name:
'jackson-dataformat-yaml', version: '2.11.3'
compile 'org.slf4j:slf4j-simple:1.7.9'
compile group: 'junit', name: 'junit', version: '4.12'
[beam] branch master updated (1fe677b -> 73d9356)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1fe677b Merge pull request #14662 Refinements to the portability page, reflecting its state of completeness. add 73d9356 Merge pull request #14615 from [BEAM-12206] Update BigQuery IO documentation with quota and sharding information No new revisions were added by this update. Summary of changes: .../documentation/io/built-in/google-bigquery.md | 93 -- 1 file changed, 70 insertions(+), 23 deletions(-)
[beam] branch asf-site updated: Publishing website 2021/04/28 18:03:38 at commit 1fe677b
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 6763dbb Publishing website 2021/04/28 18:03:38 at commit 1fe677b 6763dbb is described below commit 6763dbb3ff3b58df3150d99014235575c72d954f Author: jenkins AuthorDate: Wed Apr 28 18:03:38 2021 + Publishing website 2021/04/28 18:03:38 at commit 1fe677b --- website/generated-content/roadmap/index.xml| 81 -- .../roadmap/portability/index.html | 68 +- website/generated-content/sitemap.xml | 2 +- 3 files changed, 31 insertions(+), 120 deletions(-) diff --git a/website/generated-content/roadmap/index.xml b/website/generated-content/roadmap/index.xml index 9250c48..50f2c0a 100644 --- a/website/generated-content/roadmap/index.xml +++ b/website/generated-content/roadmap/index.xml @@ -355,11 +355,11 @@ limitations under the License. h1 id="portability-framework-roadmap">Portability Framework Roadmap/h1> h2 id="overview">Overview/h2> p>Interoperability between SDKs and runners is a key aspect of Apache -Beam. So far, however, the reality is that most runners support the -Java SDK only, because each SDK-runner combination requires non-trivial -work on both sides. All runners are also currently written in Java, +Beam. Previously, the reality was that most runners supported the +Java SDK only, because each SDK-runner combination required non-trivial +work on both sides. Most runners are also currently written in Java, which makes support of non-Java SDKs far more expensive. The -em>portability framework/em> aims to rectify this situation and provide +em>portability framework/em> rectified this situation and provided full interoperability across the Beam ecosystem./p> p>The portability framework introduces well-defined, language-neutral data structures and protocols between the SDK and runner. This interop @@ -410,60 +410,16 @@ the portability API, perhaps exclusively./p> p>If you are interested in digging in to the designs, you can find them on the a href="https://cwiki.apache.org/confluence/display/BEAM/Design+Documents;>Beam developersrsquo; wiki/a>. Another overview can be found a href="https://docs.google.com/presentation/d/1Yg8Xm4fb-oRjiLQjwLt5153hpwwTLclZrVOKP2hQifo/edit#slide=id.g42e4c9aad6_1_3070;>here/a>./p> -h2 id="milestones">Milestones/h2> -p>The portability framework is a substantial effort that touches every -Beam component. In addition to the sheer magnitude, a major challenge -is engineering an interop layer that does not significantly compromise -performance due to the additional serialization overhead of a -language-neutral protocol./p> -p>The proposed project phases are roughly as follows and are not -strictly sequential, as various components will likely move at -different speeds. Additionally, there have been (and continues to be) -supporting refactorings that are not always tracked as part of the -portability effort. Work already done is not tracked here either./p> -ul> -li> -p>strong>P1 [MVP]/strong>: Implement the fundamental plumbing for portable SDKs -and runners for batch and streaming, including containers and the -ULR -[a href="https://issues.apache.org/jira/browse/BEAM-2899;>BEAM-2899/a>]. Each -SDK and runner should use the portability framework at least to the -extent that wordcount -[a href="https://issues.apache.org/jira/browse/BEAM-2896;>BEAM-2896/a>] and -windowed wordcount -[a href="https://issues.apache.org/jira/browse/BEAM-2941;>BEAM-2941/a>] run -portably./p> -/li> -li> -p>strong>P2 [Feature complete]/strong>: Design and implement portability support -for remaining execution-side features, so that any pipeline from -any SDK can run portably on any runner. These features include side -inputs -[a href="https://issues.apache.org/jira/browse/BEAM-2863;>BEAM-2863/a>], User state [a href="https://issues.apache.org/jira/browse/BEAM-2862;>BEAM-2862/a>], User -timers -[a href="https://issues.apache.org/jira/browse/BEAM-2925;>BEAM-2925/a>], -Splittable DoFn -[a href="https://issues.apache.org/jira/browse/BEAM-2896;>BEAM-2896/a>] and -more. Each SDK and runner should use the portability framework at -least to the extent that the mobile gaming examples -[a href="https://issues.apache.org/jira/browse/BEAM-2940;>BEAM-2940/a>] run -portably./p> -/li> -li> -p>strong>P3 [Performance]/strong>: Measure and tune performance of portable -pipelines using benchmarks such as Nexmark. Features such as -progress reporting -[a href="https://issues.apache.org/jira/browse/BEAM-2940;>BEAM-2940/a>], -combiner lifting -[a href="https://issues.apache.org/jira/browse/BEAM-2937;>BEAM-2937/a>] and -fusion are expected to be needed./p> -/li> -li> -p>strong>P4 [Cross language]/strong>: Design and implement cross-language -pipeline support,
[beam] branch master updated (0bb5fd8 -> 1fe677b)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0bb5fd8 Merge pull request #14658 from kw2542/BEAM-12227 add d871b5e Refinements to the portability page, reflecting its state of completeness. add 1fe677b Merge pull request #14662 Refinements to the portability page, reflecting its state of completeness. No new revisions were added by this update. Summary of changes: website/www/site/content/en/roadmap/portability.md | 81 +- 1 file changed, 16 insertions(+), 65 deletions(-)
[beam] branch master updated (2d0f36c -> 0bb5fd8)
This is an automated email from the ASF dual-hosted git repository. ibzib pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2d0f36c Merge pull request #14613: Update run_rc_validation.sh to pull RC tag new 2020555 [BEAM-12227] Implement ExternalWorkerService#stopWorker to support LOOPBACK new a5cb30d Add ExternalWorkerServiceTest new 0bb5fd8 Merge pull request #14658 from kw2542/BEAM-12227 The 31812 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../runners/portability/ExternalWorkerService.java | 10 .../portability/ExternalWorkerServiceTest.java | 61 ++ 2 files changed, 71 insertions(+) create mode 100644 runners/portability/java/src/test/java/org/apache/beam/runners/portability/ExternalWorkerServiceTest.java
[beam] branch master updated (defca4d -> 2d0f36c)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from defca4d Merge pull request #14650: [BEAM-12011] Remove WindowFn.getOutputTime add 9f6a8c6 Update run_rc_validation.sh to pull RC tag add 2d0f36c Merge pull request #14613: Update run_rc_validation.sh to pull RC tag No new revisions were added by this update. Summary of changes: release/src/main/scripts/run_rc_validation.sh | 11 ++- 1 file changed, 6 insertions(+), 5 deletions(-)
[beam] branch master updated (cbc67c8 -> defca4d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cbc67c8 Merge pull request #13073: [BEAM-11055] Update log4j to version 2.14.1 add da03439 Remove WindowFn.getOutputTime add defca4d Merge pull request #14650: [BEAM-12011] Remove WindowFn.getOutputTime No new revisions were added by this update. Summary of changes: .../beam/sdk/transforms/windowing/WindowFn.java| 23 -- 1 file changed, 23 deletions(-)
[beam] branch master updated (a25261a -> cbc67c8)
This is an automated email from the ASF dual-hosted git repository. aromanenko pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a25261a Merge pull request #14633: [BEAM-12213] Enable dual job submission for Dataflow add 8a24eb8 [BEAM-11055] Update log4j to version 2.14.1 add cbc67c8 Merge pull request #13073: [BEAM-11055] Update log4j to version 2.14.1 No new revisions were added by this update. Summary of changes: .../groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 3 +++ .../io/elasticsearch-tests/elasticsearch-tests-2/build.gradle | 5 ++--- .../io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 9 - .../io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 9 - .../io/elasticsearch-tests/elasticsearch-tests-7/build.gradle | 9 - .../elasticsearch-tests-common/build.gradle | 9 - sdks/java/io/hadoop-format/build.gradle | 1 + sdks/java/io/hcatalog/build.gradle| 11 +++ 8 files changed, 33 insertions(+), 23 deletions(-)
