[beam] branch master updated (b1e4ece -> a25261a)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b1e4ece [BEAM-2085] Support resource hints on java transforms. #14607 add d284d07 Enable dual job submission for Dataflow add a25261a Merge pull request #14633: [BEAM-12213] Enable dual job submission for Dataflow No new revisions were added by this update. Summary of changes: .../dataflow/DataflowPipelineTranslator.java | 7 +- .../dataflow/DataflowPipelineTranslatorTest.java | 153 ++--- 2 files changed, 77 insertions(+), 83 deletions(-)
[beam] branch master updated (ea107d0 -> b1e4ece)
This is an automated email from the ASF dual-hosted git repository.
tvalentyn pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from ea107d0 Merge pull request #14640: [BEAM-9283] Disable caching
ValidatesRunner tests tasks
add 729a5ed [BEAM-2085] Add resource hints support in Java SDK.
add d15c210 Translate Resource Hints into v1beta3 API in DF Runner and
add test coverage.
add 270903b Support passing resource hints through pipeline options.
add 9a9fb44 Add resource hints supported by Dataflow.
add b1e4ece [BEAM-2085] Support resource hints on java transforms. #14607
No new revisions were added by this update.
Summary of changes:
CHANGES.md | 3 +-
.../beam/runners/core/construction/External.java | 3 +
.../core/construction/PTransformTranslation.java | 9 +-
.../core/construction/ParDoTranslation.java| 2 +-
.../runners/core/construction/SdkComponents.java | 41 ++-
.../construction/WindowingStrategyTranslation.java | 3 +-
.../CreatePCollectionViewTranslationTest.java | 3 +
.../EmptyFlattenAsCreateFactoryTest.java | 4 +
.../GroupIntoBatchesTranslationTest.java | 15 +-
.../core/construction/PTransformMatchersTest.java | 21 +-
.../construction/PTransformReplacementsTest.java | 5 +
.../construction/PTransformTranslationTest.java| 11 +-
.../core/construction/ParDoTranslationTest.java| 3 +-
.../core/construction/PipelineTranslationTest.java | 52
.../core/construction/SdkComponentsTest.java | 53 +++-
.../core/construction/SplittableParDoTest.java | 2 +
.../construction/TestStreamTranslationTest.java| 2 +
.../core/construction/TransformInputsTest.java | 12 +-
.../construction/WriteFilesTranslationTest.java| 8 +-
.../beam/runners/direct/CommittedResultTest.java | 2 +
.../direct/UnboundedReadEvaluatorFactoryTest.java | 2 +
.../direct/WriteWithShardingFactoryTest.java | 8 +-
.../FlinkStreamingTransformTranslatorsTest.java| 2 +
.../runners/flink/FlinkTransformOverridesTest.java | 8 +-
runners/google-cloud-dataflow-java/build.gradle| 1 +
.../dataflow/DataflowPipelineTranslator.java | 37 +++
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../runners/dataflow/DataflowPipelineJobTest.java | 8 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 112
.../beam/runners/dataflow/DataflowRunnerTest.java | 8 +-
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 4 +
.../main/java/org/apache/beam/sdk/Pipeline.java| 3 +-
.../apache/beam/sdk/runners/AppliedPTransform.java | 6 +-
.../beam/sdk/runners/TransformHierarchy.java | 14 +-
.../org/apache/beam/sdk/transforms/PTransform.java | 28 ++
.../sdk/transforms/resourcehints/ResourceHint.java | 32 +--
.../transforms/resourcehints/ResourceHints.java| 291 +
.../resourcehints/ResourceHintsOptions.java} | 27 +-
.../transforms/resourcehints}/package-info.java| 10 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 3 +-
.../resourcehints/ResourceHintsTest.java | 101 +++
.../pubsub/PubSubReadPayloadTranslationTest.java | 2 +
.../pubsub/PubSubWritePayloadTranslationTest.java | 15 +-
43 files changed, 909 insertions(+), 68 deletions(-)
copy
runners/spark/2/src/main/java/org/apache/beam/runners/spark/structuredstreaming/metrics/SparkMetricsContainerStepMap.java
=>
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/resourcehints/ResourceHint.java
(54%)
create mode 100644
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/resourcehints/ResourceHints.java
copy
sdks/java/core/src/main/java/org/apache/beam/sdk/{options/RemoteEnvironmentOptions.java
=> transforms/resourcehints/ResourceHintsOptions.java} (65%)
copy {runners/direct-java/src/main/java/org/apache/beam/runners/direct =>
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/resourcehints}/package-info.java
(70%)
create mode 100644
sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/resourcehints/ResourceHintsTest.java
[beam] tag nightly-master updated (e13e548 -> 4202fb8)
This is an automated email from the ASF dual-hosted git repository. github-bot pushed a change to tag nightly-master in repository https://gitbox.apache.org/repos/asf/beam.git. *** WARNING: tag nightly-master was modified! *** from e13e548 (commit) to 4202fb8 (commit) from e13e548 [BEAM-10029] Adds Grafana config for Spanner IO Perf Metrics (#14643) add ef654f6 Close ZipOutputStream add 9505f4e Merge pull request #14631: [BEAM-12220] Close ZipOutputStream to avoid native memory leak add 93fe942 [BEAM-12226] JdbcIO DefaultRetryStrategy: retry on PostgreSQL-specific 40P01 error code add d152c9c Merge pull request #14646: [BEAM-12226] JdbcIO DefaultRetryStrategy: retry on PostgreSQL-specific 40P01 error code add 46b704a Clarify when to use `@DefaultCoder` (#14326) add 422db52 [BEAM-12231] Update Dataflow container to adjust for API changes add a2230e0 Merge pull request #14651: [BEAM-12231] Update Dataflow container to adjust for API changes add 4774c1c [BEAM-12232] Expose public factory methods in SparkJobServerDriver (#14652) add 729129e Add View.toList pipeline to translator tests add 1c2b9bc Temporarily disable FnApiDoFnRunnerTest.testUsingMetrics add 4492fa3 Resolve beam_fn_api experiment for side inputs, using portable expansion always add 72fc94d Merge #14641: [BEAM-1] Use portable side input expansion always add 3981686 [BEAM-12221] Add experimental pipeline option to control the execution state sampling period. add 8532acb Merge pull request #14632 from y1chi/sampler add 8821163 [BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable. add 33fe359 Merge pull request #14489: [BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable add c046d7f [BEAM-12009] Enable Calc splitting rule by default. add def1ea6 Merge pull request #14655 from ibzib/enable-calc-splitting add ebdfee9 [BEAM-12016] Initial implementation for add_suffix, add_prefix (#14623) add eca1426 [BEAM-12045] Install google cloud profiler agent in java sdk container image add fe2fc9d Merge pull request #14598 from y1chi/profiler add b379725 [BEAM-12237] Fix logging in Samza Runner (#14657) add 62d6d41 Fix worker_pool plumbing to use a mutable List. add 4202fb8 Merge pull request #14661 - Fix worker_pool plumbing to use a mutable List. No new revisions were added by this update. Summary of changes: .../beam/examples/cookbook/BigQueryTornadoes.java | 91 +++-- .../src/main/proto/beam_provision_api.proto| 4 + .../core/construction/MorePipelineTest.java| 21 ++- .../core/metrics/ExecutionStateSampler.java| 14 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../dataflow/DataflowPipelineTranslatorTest.java | 101 ++ .../dataflow/worker/DataflowWorkUnitClient.java| 8 +- .../dataflow/worker/IsmSideInputReader.java| 35 +++-- runners/samza/build.gradle | 4 + runners/samza/src/main/resources/log4j.properties | 3 +- .../beam/runners/spark/SparkJobServerDriver.java | 4 +- .../model/fnexecution_v1/beam_provision_api.pb.go | 104 -- sdks/java/container/Dockerfile | 3 + sdks/java/container/boot.go| 20 ++- sdks/java/container/build.gradle | 6 + sdks/java/container/common.gradle | 1 + .../beam/sdk/options/ExperimentalOptions.java | 2 + .../org/apache/beam/sdk/transforms/Combine.java| 36 + .../java/org/apache/beam/sdk/transforms/View.java | 150 + .../java/org/apache/beam/sdk/util/ZipFiles.java| 1 + .../apache/beam/sdk/values/PCollectionViews.java | 60 - .../beam/sdk/testing/PCollectionViewTesting.java | 108 +-- .../sql/zetasql/ZetaSQLQueryPlanner.java | 3 +- .../extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 15 --- .../sql/zetasql/ZetaSqlJavaUdfTypeTest.java| 12 +- .../java/org/apache/beam/fn/harness/FnHarness.java | 6 + .../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 + .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 13 +- .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 11 +- sdks/python/apache_beam/dataframe/frames.py| 13 ++ sdks/python/apache_beam/dataframe/frames_test.py | 14 ++ .../apache_beam/dataframe/pandas_doctests_test.py | 2 - .../content/en/documentation/programming-guide.md | 17 ++- 33 files changed, 512 insertions(+), 374 deletions(-)
[beam] branch master updated (4202fb8 -> ea107d0)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4202fb8 Merge pull request #14661 - Fix worker_pool plumbing to use a mutable List. add 62b9c0f [BEAM-9283] Disable caching ValidatesRunner tests tasks add ea107d0 Merge pull request #14640: [BEAM-9283] Disable caching ValidatesRunner tests tasks No new revisions were added by this update. Summary of changes: runners/flink/flink_runner.gradle | 2 ++ runners/spark/spark_runner.gradle | 6 ++ 2 files changed, 8 insertions(+)
[beam] 01/01: Add scikit-learn to additional user packages
This is an automated email from the ASF dual-hosted git repository. altay pushed a commit to branch aaltay-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git commit 3897c005f2b77786f7d36133f9fbda4a42c9e0d3 Author: Ahmet Altay AuthorDate: Tue Apr 27 18:29:24 2021 -0700 Add scikit-learn to additional user packages --- sdks/python/container/base_image_requirements.txt | 1 + 1 file changed, 1 insertion(+) diff --git a/sdks/python/container/base_image_requirements.txt b/sdks/python/container/base_image_requirements.txt index 061507a..4b7b5b1 100644 --- a/sdks/python/container/base_image_requirements.txt +++ b/sdks/python/container/base_image_requirements.txt @@ -57,6 +57,7 @@ mmh3==2.5.1 # These are additional packages likely to be used by customers. numpy==1.19.5 scipy==1.4.1 +scikit-learn==0.24.1 ; python_version>="3.6" pandas==1.1.5 ; python_version<"3.7" pandas==1.2.4 ; python_version>="3.7" protorpc==0.12.0
[beam] branch aaltay-patch-1 created (now 3897c005)
This is an automated email from the ASF dual-hosted git repository. altay pushed a change to branch aaltay-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at 3897c005 Add scikit-learn to additional user packages This branch includes the following new commits: new 3897c005 Add scikit-learn to additional user packages The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference.
[beam] branch master updated (b379725 -> 4202fb8)
This is an automated email from the ASF dual-hosted git repository. tvalentyn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b379725 [BEAM-12237] Fix logging in Samza Runner (#14657) add 62d6d41 Fix worker_pool plumbing to use a mutable List. add 4202fb8 Merge pull request #14661 - Fix worker_pool plumbing to use a mutable List. No new revisions were added by this update. Summary of changes: .../beam/runners/dataflow/worker/DataflowWorkUnitClient.java | 8 ++-- 1 file changed, 6 insertions(+), 2 deletions(-)
[beam] branch master updated (fe2fc9d -> b379725)
This is an automated email from the ASF dual-hosted git repository. xinyu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fe2fc9d Merge pull request #14598 from y1chi/profiler add b379725 [BEAM-12237] Fix logging in Samza Runner (#14657) No new revisions were added by this update. Summary of changes: runners/samza/build.gradle| 4 runners/samza/src/main/resources/log4j.properties | 3 ++- 2 files changed, 6 insertions(+), 1 deletion(-)
[beam] branch master updated: [BEAM-12009] Enable Calc splitting rule by default.
This is an automated email from the ASF dual-hosted git repository.
ibzib pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new c046d7f [BEAM-12009] Enable Calc splitting rule by default.
new def1ea6 Merge pull request #14655 from ibzib/enable-calc-splitting
c046d7f is described below
commit c046d7fff3746d7431d6fed09526b365cfd0d9a1
Author: Kyle Weaver
AuthorDate: Tue Apr 27 13:43:15 2021 -0700
[BEAM-12009] Enable Calc splitting rule by default.
---
.../sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java | 3 +--
.../sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java| 15 ---
.../extensions/sql/zetasql/ZetaSqlJavaUdfTypeTest.java| 12 +---
3 files changed, 2 insertions(+), 28 deletions(-)
diff --git
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
index 918f7ce..fb90a26 100644
---
a/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
+++
b/sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
@@ -75,10 +75,9 @@ import org.slf4j.LoggerFactory;
"nullness" // TODO(https://issues.apache.org/jira/browse/BEAM-10402)
})
public class ZetaSQLQueryPlanner implements QueryPlanner {
- // TODO(BEAM-11747) Re-enable BeamJavaUdfCalcRule by default when it is safe
to do so.
public static final Collection DEFAULT_CALC =
ImmutableList.builder()
- .add(BeamZetaSqlCalcRule.INSTANCE, BeamZetaSqlCalcMergeRule.INSTANCE)
+ .add(BeamZetaSqlCalcSplittingRule.INSTANCE,
BeamZetaSqlCalcMergeRule.INSTANCE)
.build();
private static final Logger LOG =
LoggerFactory.getLogger(ZetaSQLQueryPlanner.class);
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
index 22b18f9..57b5f4c 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
@@ -42,14 +42,11 @@ import org.apache.beam.sdk.transforms.Sum;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import
org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.Frameworks;
-import org.apache.beam.vendor.calcite.v1_20_0.org.apache.calcite.tools.RuleSet;
-import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableMap;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.codehaus.commons.compiler.CompileException;
import org.joda.time.Duration;
import org.junit.Before;
-import org.junit.Ignore;
import org.junit.Rule;
import org.junit.Test;
import org.junit.rules.ExpectedException;
@@ -88,15 +85,6 @@ public class ZetaSqlJavaUdfTest extends ZetaSqlTestBase {
emptyJarPathProperty, ZetaSqlJavaUdfTest.class.getSimpleName()));
}
initialize();
-
-// Add BeamZetaSqlCalcSplittingRule to planner to enable UDFs.
-this.config =
-Frameworks.newConfigBuilder(config)
-.ruleSets(
-ZetaSQLQueryPlanner.getZetaSqlRuleSets(
-
ImmutableList.of(BeamZetaSqlCalcSplittingRule.INSTANCE))
-.toArray(new RuleSet[0]))
-.build();
}
@Test
@@ -221,9 +209,6 @@ public class ZetaSqlJavaUdfTest extends ZetaSqlTestBase {
}
@Test
- @Ignore(
- "Re-enable when ZetaSQLQueryPlanner has UDFs enabled by default. "
- + "Until then, this test fails because we can't pass a
ZetaSQLQueryPlanner instance to SqlTransform.")
public void testSqlTransformRegisterUdf() {
String sql = "SELECT increment(0);";
PCollection stream =
diff --git
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTypeTest.java
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTypeTest.java
index 1481fc80..498e1b8 100644
---
a/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTypeTest.java
+++
b/sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTypeTest.java
@@ -33,8 +33,6 @@ import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
import
org.apache.beam.vendor.calcite.v1_20_0.
[beam] branch master updated: [BEAM-12045] Install google cloud profiler agent in java sdk container image
This is an automated email from the ASF dual-hosted git repository.
yichi pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new eca1426 [BEAM-12045] Install google cloud profiler agent in java sdk
container image
new fe2fc9d Merge pull request #14598 from y1chi/profiler
eca1426 is described below
commit eca1426eebd4928c8c74f697084bed04164ce94a
Author: Yichi Zhang
AuthorDate: Tue Apr 20 19:12:58 2021 -0700
[BEAM-12045] Install google cloud profiler agent in java sdk container image
---
.../src/main/proto/beam_provision_api.proto| 4 +
.../model/fnexecution_v1/beam_provision_api.pb.go | 104 +
sdks/java/container/Dockerfile | 3 +
sdks/java/container/boot.go| 20 +++-
sdks/java/container/build.gradle | 6 ++
sdks/java/container/common.gradle | 1 +
6 files changed, 96 insertions(+), 42 deletions(-)
diff --git a/model/fn-execution/src/main/proto/beam_provision_api.proto
b/model/fn-execution/src/main/proto/beam_provision_api.proto
index 0481326..6e1fa7a 100644
--- a/model/fn-execution/src/main/proto/beam_provision_api.proto
+++ b/model/fn-execution/src/main/proto/beam_provision_api.proto
@@ -81,4 +81,8 @@ message ProvisionInfo {
// (optional) A set of capabilities that this SDK is allowed to use in its
// interactions with this runner.
repeated string runner_capabilities = 12;
+
+// (optional) Runtime environment metadata that are static throughout the
+// pipeline execution.
+map metadata = 13;
}
diff --git a/sdks/go/pkg/beam/model/fnexecution_v1/beam_provision_api.pb.go
b/sdks/go/pkg/beam/model/fnexecution_v1/beam_provision_api.pb.go
index e6c5322..7348315 100644
--- a/sdks/go/pkg/beam/model/fnexecution_v1/beam_provision_api.pb.go
+++ b/sdks/go/pkg/beam/model/fnexecution_v1/beam_provision_api.pb.go
@@ -21,8 +21,8 @@
// Code generated by protoc-gen-go. DO NOT EDIT.
// versions:
-// protoc-gen-go v1.25.0-devel
-// protocv3.13.0
+// protoc-gen-go v1.26.0
+// protocv3.10.1
// source: beam_provision_api.proto
package fnexecution_v1
@@ -163,6 +163,9 @@ type ProvisionInfo struct {
// (optional) A set of capabilities that this SDK is allowed to use in
its
// interactions with this runner.
RunnerCapabilities []string
`protobuf:"bytes,12,rep,name=runner_capabilities,json=runnerCapabilities,proto3"
json:"runner_capabilities,omitempty"`
+ // (optional) Runtime environment metadata that are static throughout
the
+ // pipeline execution.
+ Metadata map[string]string
`protobuf:"bytes,13,rep,name=metadata,proto3" json:"metadata,omitempty"
protobuf_key:"bytes,1,opt,name=key,proto3"
protobuf_val:"bytes,2,opt,name=value,proto3"`
}
func (x *ProvisionInfo) Reset() {
@@ -253,6 +256,13 @@ func (x *ProvisionInfo) GetRunnerCapabilities() []string {
return nil
}
+func (x *ProvisionInfo) GetMetadata() map[string]string {
+ if x != nil {
+ return x.Metadata
+ }
+ return nil
+}
+
var File_beam_provision_api_proto protoreflect.FileDescriptor
var file_beam_provision_api_proto_rawDesc = []byte{
@@ -272,7 +282,7 @@ var file_beam_provision_api_proto_rawDesc = []byte{
0x72, 0x67, 0x2e, 0x61, 0x70, 0x61, 0x63, 0x68, 0x65, 0x2e, 0x62, 0x65,
0x61, 0x6d, 0x2e, 0x6d,
0x6f, 0x64, 0x65, 0x6c, 0x2e, 0x66, 0x6e, 0x5f, 0x65, 0x78, 0x65, 0x63,
0x75, 0x74, 0x69, 0x6f,
0x6e, 0x2e, 0x76, 0x31, 0x2e, 0x50, 0x72, 0x6f, 0x76, 0x69, 0x73, 0x69,
0x6f, 0x6e, 0x49, 0x6e,
- 0x66, 0x6f, 0x52, 0x04, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0x99, 0x05, 0x0a,
0x0d, 0x50, 0x72, 0x6f,
+ 0x66, 0x6f, 0x52, 0x04, 0x69, 0x6e, 0x66, 0x6f, 0x22, 0xb6, 0x06, 0x0a,
0x0d, 0x50, 0x72, 0x6f,
0x76, 0x69, 0x73, 0x69, 0x6f, 0x6e, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x42,
0x0a, 0x10, 0x70, 0x69,
0x70, 0x65, 0x6c, 0x69, 0x6e, 0x65, 0x5f, 0x6f, 0x70, 0x74, 0x69, 0x6f,
0x6e, 0x73, 0x18, 0x03,
0x20, 0x01, 0x28, 0x0b, 0x32, 0x17, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c,
0x65, 0x2e, 0x70, 0x72,
@@ -314,26 +324,36 @@ var file_beam_provision_api_proto_rawDesc = []byte{
0x69, 0x65, 0x73, 0x12, 0x2f, 0x0a, 0x13, 0x72, 0x75, 0x6e, 0x6e, 0x65,
0x72, 0x5f, 0x63, 0x61,
0x70, 0x61, 0x62, 0x69, 0x6c, 0x69, 0x74, 0x69, 0x65, 0x73, 0x18, 0x0c,
0x20, 0x03, 0x28, 0x09,
0x52, 0x12, 0x72, 0x75, 0x6e, 0x6e, 0x65, 0x72, 0x43, 0x61, 0x70, 0x61,
0x62, 0x69, 0x6c, 0x69,
- 0x74, 0x69, 0x65, 0x73, 0x32, 0xa8, 0x01, 0x0a, 0x10, 0x50, 0x72, 0x6f,
0x76, 0x69, 0x73, 0x69,
- 0x6f, 0x6e, 0x53, 0x65, 0x72, 0x76, 0x69, 0x63, 0x65, 0x12, 0x93, 0x01,
0x0a, 0x10, 0x47, 0x65,
- 0x74, 0x50, 0x72, 0x6f, 0x76, 0x69, 0x73, 0x69, 0x6f, 0x6e, 0x49, 0x6e,
0x66, 0x6f, 0x12, 0x3e,
- 0x2e, 0x6f, 0x72, 0x67, 0x2e, 0x61, 0
[beam] branch master updated (def1ea6 -> ebdfee9)
This is an automated email from the ASF dual-hosted git repository. bhulette pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from def1ea6 Merge pull request #14655 from ibzib/enable-calc-splitting add ebdfee9 [BEAM-12016] Initial implementation for add_suffix, add_prefix (#14623) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/dataframe/frames.py | 13 + sdks/python/apache_beam/dataframe/frames_test.py | 14 ++ sdks/python/apache_beam/dataframe/pandas_doctests_test.py | 2 -- 3 files changed, 27 insertions(+), 2 deletions(-)
[beam] branch master updated (8532acb -> 33fe359)
This is an automated email from the ASF dual-hosted git repository. chamikara pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8532acb Merge pull request #14632 from y1chi/sampler add 8821163 [BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable. add 33fe359 Merge pull request #14489: [BEAM-11994] Refactor BigQueryTornadoes to make more options parameterizable No new revisions were added by this update. Summary of changes: .../beam/examples/cookbook/BigQueryTornadoes.java | 91 +++--- 1 file changed, 47 insertions(+), 44 deletions(-)
[beam] branch master updated (72fc94d -> 8532acb)
This is an automated email from the ASF dual-hosted git repository. yichi pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 72fc94d Merge #14641: [BEAM-1] Use portable side input expansion always new 3981686 [BEAM-12221] Add experimental pipeline option to control the execution state sampling period. new 8532acb Merge pull request #14632 from y1chi/sampler The 31784 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/core/metrics/ExecutionStateSampler.java | 14 ++ .../org/apache/beam/sdk/options/ExperimentalOptions.java | 2 ++ .../main/java/org/apache/beam/fn/harness/FnHarness.java| 6 ++ 3 files changed, 18 insertions(+), 4 deletions(-)
[beam] branch master updated (4774c1c -> 72fc94d)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4774c1c [BEAM-12232] Expose public factory methods in SparkJobServerDriver (#14652) add 729129e Add View.toList pipeline to translator tests add 1c2b9bc Temporarily disable FnApiDoFnRunnerTest.testUsingMetrics add 4492fa3 Resolve beam_fn_api experiment for side inputs, using portable expansion always add 72fc94d Merge #14641: [BEAM-1] Use portable side input expansion always No new revisions were added by this update. Summary of changes: .../core/construction/MorePipelineTest.java| 21 ++- .../dataflow/DataflowPipelineTranslatorTest.java | 101 ++ .../dataflow/worker/IsmSideInputReader.java| 35 +++-- .../org/apache/beam/sdk/transforms/Combine.java| 36 + .../java/org/apache/beam/sdk/transforms/View.java | 150 + .../apache/beam/sdk/values/PCollectionViews.java | 60 - .../beam/sdk/testing/PCollectionViewTesting.java | 108 +-- .../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 + 8 files changed, 272 insertions(+), 241 deletions(-)
[beam] branch master updated (a2230e0 -> 4774c1c)
This is an automated email from the ASF dual-hosted git repository. xinyu pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a2230e0 Merge pull request #14651: [BEAM-12231] Update Dataflow container to adjust for API changes add 4774c1c [BEAM-12232] Expose public factory methods in SparkJobServerDriver (#14652) No new revisions were added by this update. Summary of changes: .../main/java/org/apache/beam/runners/spark/SparkJobServerDriver.java | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch asf-site updated: Publishing website 2021/04/27 18:03:10 at commit a2230e0
This is an automated email from the ASF dual-hosted git repository. git-site-role pushed a commit to branch asf-site in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/asf-site by this push: new 54a6958 Publishing website 2021/04/27 18:03:10 at commit a2230e0 54a6958 is described below commit 54a6958f843f95081e5166d057c393e6b9514cd9 Author: jenkins AuthorDate: Tue Apr 27 18:03:11 2021 + Publishing website 2021/04/27 18:03:10 at commit a2230e0 --- website/generated-content/documentation/index.xml | 23 ++ .../documentation/programming-guide/index.html | 14 - website/generated-content/sitemap.xml | 2 +- 3 files changed, 29 insertions(+), 10 deletions(-) diff --git a/website/generated-content/documentation/index.xml b/website/generated-content/documentation/index.xml index 0b5c467..6b29bf0 100644 --- a/website/generated-content/documentation/index.xml +++ b/website/generated-content/documentation/index.xml @@ -7029,10 +7029,25 @@ values for a pipeline.
