[beam] branch master updated (762edd7f3a6 -> 483a0c95734)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git from 762edd7f3a6 Improved pipeline translation in SparkStructuredStreamingRunner (#22446) add a6cda1370b3 use avro DataFileReader to read avro container files new 483a0c95734 Merge pull request #23214: Use avro DataFileReader to read avro container files The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../java/org/apache/beam/sdk/io/AvroSource.java| 343 ++--- .../org/apache/beam/sdk/io/AvroSourceTest.java | 166 -- 2 files changed, 86 insertions(+), 423 deletions(-)
[beam] 01/01: Merge pull request #23214: Use avro DataFileReader to read avro container files
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 483a0c95734c528aa45419596e9f27e9e650c5d7 Merge: 762edd7f3a6 a6cda1370b3 Author: Ismaël Mejía AuthorDate: Thu Sep 22 18:58:10 2022 +0200 Merge pull request #23214: Use avro DataFileReader to read avro container files .../java/org/apache/beam/sdk/io/AvroSource.java| 343 ++--- .../org/apache/beam/sdk/io/AvroSourceTest.java | 166 -- 2 files changed, 86 insertions(+), 423 deletions(-)
[beam] branch master updated: Fix #22466 Add github actions dependency updates with dependabot
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 3031a3d2aca Fix #22466 Add github actions dependency updates with dependabot 3031a3d2aca is described below commit 3031a3d2aca8e81b219364ca43cbf811abd68445 Author: Ismaël Mejía AuthorDate: Wed Aug 10 22:02:55 2022 +0200 Fix #22466 Add github actions dependency updates with dependabot --- .github/dependabot.yml | 8 1 file changed, 8 insertions(+) diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 334414df9db..248e8d6a69b 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -42,3 +42,11 @@ updates: - dependency-name: "com.google.api.grpc:grpc-*" - dependency-name: "com.google.http-client:*" - dependency-name: "com.google.apis:google-api-services-*" + - package-ecosystem: "github-actions" +directory: "/" +schedule: + interval: "daily" +allow: + # Allow only automatic updates for official github actions + # Other github-actions require approval from INFRA + - dependency-name: "actions/*"
[beam] branch master updated (d352d60 -> 0a68801)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d352d60 [BEAM-14071] Enabling Flink on Dataproc for Interactive Beam (#17044) add 7257e37 [BEAM-4106] Remove filesToStage from Flink pipeline option list. add 0a68801 Merge pull request #17143: [BEAM-4106] Remove filesToStage from Flink pipeline option list. No new revisions were added by this update. Summary of changes: website/www/site/layouts/shortcodes/flink_java_pipeline_options.html | 5 - .../www/site/layouts/shortcodes/flink_python_pipeline_options.html | 5 - 2 files changed, 10 deletions(-)
[beam] branch master updated (9e0aa6b -> eea81f4)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9e0aa6b Merge pull request #17056 from [BEAM-14076] [SnowflakeIO] Add support for GEOGRAPHY column add 1aba87d [BEAM-13981] Remove Spark Runner specific code for event logging add eea81f4 Merge pull request #17073: [BEAM-13981] Remove Spark Runner specific code for event logging No new revisions were added by this update. Summary of changes: .../beam/runners/spark/SparkPipelineRunner.java| 17 - .../org/apache/beam/runners/spark/SparkRunner.java | 22 +- .../runners/spark/metrics/SparkBeamMetric.java | 10 --- .../beam/runners/spark/util/SparkCommon.java | 79 -- .../beam/runners/spark/util/SparkCompat.java | 61 - 5 files changed, 1 insertion(+), 188 deletions(-) delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCommon.java
[beam] branch master updated (3cd1f7f -> 75c25f0)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3cd1f7f [BEAM-13960] Add support for more types when converting from between row and proto (#16875) add fa32292 Bump org.mongodb:mongo-java-driver to 3.12.10 add 75c25f0 Merge pull request #16989: [BEAM-5577] Bump org.mongodb:mongo-java-driver to 3.12.10 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (b2f2128 -> 6e98dd4)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b2f2128 Merge pull request #16865: Create test category for UsesProcessingTimeTimers are exclude from Samza add add7bbc [BEAM-13202] Fix typos on tests names for VarianceFnTest add 1105c34 [BEAM-13202] Add Coder to CountIfFn.Accum add cf75357 [BEAM-13202] Reuse Count transform code since CountIf is a specific case add 6e98dd4 Merge pull request #16856: [BEAM-13202] Add Coder to CountIfFn.Accum No new revisions were added by this update. Summary of changes: .../extensions/sql/impl/transform/agg/CountIf.java | 53 +++ .../sql/impl/transform/agg/CountIfTest.java| 78 ++ .../sql/impl/transform/agg/VarianceFnTest.java | 4 +- 3 files changed, 104 insertions(+), 31 deletions(-) create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CountIfTest.java
[beam] branch master updated (7fa5387 -> c73066c)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7fa5387 Regenerate python container base_image_requirements.txt (#16832) add ed693e5 [BEAM-9195] Bump org.testcontainers to 1.16.3 add c73066c Merge pull request #16661: [BEAM-9195] Bump org.testcontainers to 1.16.3 No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- sdks/java/io/debezium/build.gradle | 6 +++--- sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 2 +- sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/build.gradle | 2 +- .../io/elasticsearch-tests/elasticsearch-tests-common/build.gradle | 2 +- 6 files changed, 8 insertions(+), 8 deletions(-)
[beam] branch master updated (5beae2a -> 9434c4d)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5beae2a [BEAM-13605] Add support for pandas 1.4.0 (#16590) add 99b903f Change links to Books from Amazon to Publisher add 9434c4d Merge pull request #16718: [website] Change links to Books from Amazon to Publisher No new revisions were added by this update. Summary of changes: .../www/site/content/en/documentation/resources/learning-resources.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch master updated (cbbebcd -> 50d8a85)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from cbbebcd Merge pull request #15335 from [BEAM-12751] Set clientRequestId for Dataflow python job creation add 41d515d [BEAM-12270] TPC-DS: Add schema projection for Parquet source add 50d8a85 Merge pull request #15361: [BEAM-12270] TPC-DS: Add schema projection for Parquet source No new revisions were added by this update. Summary of changes: .../org/apache/beam/sdk/tpcds/QueryReader.java | 20 ++ .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 72 +++--- .../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 28 + 3 files changed, 111 insertions(+), 9 deletions(-)
[beam] branch master updated (98747fd -> 3537f7e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 98747fd Bump Python FnAPI beam-master container #15283 add e5c25a0 [BEAM-12628] Add Avro reflect-based Coder option add 3537f7e Merge pull request #15292: [BEAM-12628] Add Avro reflect-based Coder option No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/coders/AvroCoder.java | 32 +++--- .../org/apache/beam/sdk/coders/AvroCoderTest.java | 21 ++ 2 files changed, 49 insertions(+), 4 deletions(-)
[beam] branch master updated (f9a4bfc -> 68c56f8)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f9a4bfc [BEAM-11289] [Python] Integrate Google Cloud Recommendations AI functionality (#14806) add 5690af7 [BEAM-12422] Removing unnecessary log4j-api dependency add 68c56f8 Merge pull request #15098: [BEAM-12422] Vendored gRPC: Removing unnecessary log4j-api dependency No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_36_0.groovy | 3 --- 1 file changed, 3 deletions(-)
[beam] branch master updated (225fcd0 -> 90c854e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 225fcd0 [BEAM-9547] Add support for xs on DataFrame and Series (#15078) add 37e3fdf [BEAM-12094] Add Spark 3 to Python. add 90c854e Merge pull request #15063: [BEAM-12094] Add Spark 3 to Python (spark_runner.py) No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/options/pipeline_options.py | 8 +++- .../apache_beam/runners/portability/spark_runner.py | 3 +++ .../runners/portability/spark_uber_jar_job_server.py | 16 ++-- 3 files changed, 20 insertions(+), 7 deletions(-)
[beam] branch master updated (67164a1 -> 815e2c9)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 67164a1 Merge pull request #14964: [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord add 525d0f8 [BEAM-12471] Fixes NumberFormatException new 815e2c9 Merge pull request #14982: [BEAM-12471] Fixes NumberFormatException The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java | 9 +++-- 1 file changed, 3 insertions(+), 6 deletions(-)
[beam] 01/01: Merge pull request #14982: [BEAM-12471] Fixes NumberFormatException
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 815e2c94bee24ef0c15c78876d6266618c238ba5 Merge: 67164a1 525d0f8 Author: Ismaël Mejía AuthorDate: Thu Jun 10 15:43:16 2021 +0200 Merge pull request #14982: [BEAM-12471] Fixes NumberFormatException .../beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java | 9 +++-- 1 file changed, 3 insertions(+), 6 deletions(-)
[beam] 01/01: Merge pull request #14964: [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 67164a1d9a3715528e03507c792d1c89c4d5efac Merge: 9cd7479 c13a0cc Author: Ismaël Mejía AuthorDate: Thu Jun 10 15:03:46 2021 +0200 Merge pull request #14964: [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java| 8 .../java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java | 7 +++ 2 files changed, 15 insertions(+)
[beam] branch master updated (9cd7479 -> 67164a1)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9cd7479 Merge pull request #14888: [BEAM-12411] Update Tensorflow to version 2.5.0 & grpcio to 1.34.0 add c13a0cc [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord. new 67164a1 Merge pull request #14964: [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java| 8 .../java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java | 7 +++ 2 files changed, 15 insertions(+)
[beam] branch master updated (0b369a5 -> 9cd7479)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0b369a5 [BEAM-10982] Add linux aarch64 wheel build support #14832 add ce0040d Update tensorflow to the latest version add a12bf7a Update grpcio add 81808d1 Add license info for keras-nightly package. add 9cd7479 Merge pull request #14888: [BEAM-12411] Update Tensorflow to version 2.5.0 & grpcio to 1.34.0 No new revisions were added by this update. Summary of changes: sdks/python/container/base_image_requirements.txt | 4 ++-- sdks/python/container/license_scripts/dep_urls_py.yaml | 2 ++ 2 files changed, 4 insertions(+), 2 deletions(-)
[beam] branch master updated: [BEAM-12385] Handle VARCHAR and other SQL specific logical types in AvroUtils
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 88d4712 [BEAM-12385] Handle VARCHAR and other SQL specific logical
types in AvroUtils
new 62e8f84 Merge pull request #14858: [BEAM-12385] Handle VARCHAR and
other SQL specific logical types in AvroUtils
88d4712 is described below
commit 88d4712147911744761cb385b8226c81e283d1fe
Author: Anant Damle
AuthorDate: Fri May 21 23:47:53 2021 +0800
[BEAM-12385] Handle VARCHAR and other SQL specific logical types in
AvroUtils
---
CHANGES.md | 3 +
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 43
.../beam/sdk/schemas/utils/AvroUtilsTest.java | 244 +
.../apache/beam/sdk/io/jdbc/SchemaUtilTest.java| 60 +
4 files changed, 350 insertions(+)
diff --git a/CHANGES.md b/CHANGES.md
index 7029cd2..62b8e71 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -66,6 +66,9 @@
* `CREATE FUNCTION` DDL statement added to Calcite SQL syntax. `JAR` and
`AGGREGATE` are now reserved keywords.
([BEAM-12339](https://issues.apache.org/jira/browse/BEAM-12339)).
* Flink 1.13 is now supported by the Flink runner
([BEAM-12277](https://issues.apache.org/jira/browse/BEAM-12277)).
* X feature added (Java/Python)
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+* Add support to convert Beam Schema to Avro Schema for JDBC LogicalTypes:
+ `VARCHAR`, `NVARCHAR`, `LONGVARCHAR`, `LONGNVARCHAR`, `DATE`, `TIME`
+ (Java)([BEAM-12385](https://issues.apache.org/jira/browse/BEAM-12385)).
## Breaking Changes
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
index 77b5445..0835f9b 100644
---
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
+++
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
@@ -906,6 +906,26 @@ public class AvroUtils {
.map(x -> getFieldSchema(x.getType(), x.getName(),
namespace))
.collect(Collectors.toList()));
break;
+ case "CHAR":
+ case "NCHAR":
+baseType =
+buildHiveLogicalTypeSchema("char", (int)
fieldType.getLogicalType().getArgument());
+break;
+ case "NVARCHAR":
+ case "VARCHAR":
+ case "LONGNVARCHAR":
+ case "LONGVARCHAR":
+baseType =
+buildHiveLogicalTypeSchema(
+"varchar", (int) fieldType.getLogicalType().getArgument());
+break;
+ case "DATE":
+baseType =
LogicalTypes.date().addToSchema(org.apache.avro.Schema.create(Type.INT));
+break;
+ case "TIME":
+baseType =
+
LogicalTypes.timeMillis().addToSchema(org.apache.avro.Schema.create(Type.INT));
+break;
default:
throw new RuntimeException(
"Unhandled logical type " +
fieldType.getLogicalType().getIdentifier());
@@ -1017,6 +1037,15 @@ public class AvroUtils {
typeWithNullability.type.getTypes().get(oneOfValue.getCaseType().getValue()),
oneOfValue.getValue());
}
+ case "NVARCHAR":
+ case "VARCHAR":
+ case "LONGNVARCHAR":
+ case "LONGVARCHAR":
+return new Utf8((String) value);
+ case "DATE":
+return Days.daysBetween(Instant.EPOCH, (Instant) value).getDays();
+ case "TIME":
+return (int) ((Instant) value).getMillis();
default:
throw new RuntimeException(
"Unhandled logical type " +
fieldType.getLogicalType().getIdentifier());
@@ -1277,4 +1306,18 @@ public class AvroUtils {
checkArgument(
got.equals(expected), "Can't convert '%s' to %s, expected: %s", label,
got, expected);
}
+
+ /**
+ * Helper factory to build Avro Logical types schemas for SQL *CHAR types.
This method https://github.com/apache/hive/blob/5d268834a5f5278ea76399f8af0d0ab043ae0b45/serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java#L110-L121;>represents
+ * the logical as Hive does.
+ */
+ private static org.apache.avro.Schema buildHiveLogicalTypeSchema(
+ String hiveLogicalType, int size) {
+String schemaJson =
+String.format(
+"{\"type\": \"string\", \"logicalType\": \"%s\", \"maxLength\":
%s}",
+hiveL
[beam] 01/01: Merge pull request #14906: [BEAM-12281] Drop support for Flink 1.10
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 511fb05e5925fe48947efdd872433d621c85e52d
Merge: 4f0ea5f 22575c7
Author: Ismaël Mejía
AuthorDate: Wed Jun 2 11:08:53 2021 +0200
Merge pull request #14906: [BEAM-12281] Drop support for Flink 1.10
.test-infra/dataproc/flink_cluster.sh | 2 +-
.test-infra/jenkins/Flink.groovy | 4 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 2 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 2 +-
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 2 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 2 +-
.../jenkins/job_LoadTests_ParDo_Flink_Go.groovy| 2 +-
.../job_LoadTests_ParDo_Flink_Python.groovy| 2 +-
.../job_LoadTests_SideInput_Flink_Go.groovy| 2 +-
.../jenkins/job_LoadTests_coGBK_Flink_Go.groovy| 2 +-
.../job_LoadTests_coGBK_Flink_Python.groovy| 2 +-
...Commit_Python_Chicago_Taxi_Example_Flink.groovy | 2 +-
CHANGES.md | 1 +
gradle.properties | 2 +-
release/build.gradle.kts | 2 +-
release/src/main/scripts/run_rc_validation.sh | 2 +-
runners/flink/1.10/build.gradle| 33 --
.../flink/1.10/job-server-container/build.gradle | 26
runners/flink/1.10/job-server/build.gradle | 31 --
.../streaming/AbstractStreamOperatorCompat.java| 30 -
.../beam/runners/flink/FlinkRunnerTestCompat.java | 42 -
.../beam/runners/flink/RemoteMiniClusterImpl.java | 71 --
.../runners/flink/SourceTransformationCompat.java | 28 -
runners/flink/1.11/build.gradle| 4 +-
.../runners/flink/streaming/StreamSources.java | 0
runners/flink/1.12/build.gradle| 4 +-
runners/flink/1.13/build.gradle| 4 +-
runners/flink/flink_runner.gradle | 2 +-
.../beam/runners/flink/FlinkPipelineRunner.java| 7 +--
.../org/apache/beam/runners/flink/FlinkRunner.java | 7 +--
.../flink/FlinkExecutionEnvironmentsTest.java | 34 +++
.../FlinkPipelineExecutionEnvironmentTest.java | 22 +++
sdks/go/examples/stringsplit/stringsplit.go| 2 +-
sdks/java/testing/nexmark/build.gradle | 2 +-
sdks/java/testing/tpcds/build.gradle | 2 +-
sdks/python/apache_beam/transforms/sql_test.py | 4 +-
settings.gradle.kts| 4 --
.../site/content/en/contribute/release-guide.md| 2 +-
.../content/en/documentation/dsls/sql/shell.md | 6 +-
.../site/content/en/documentation/runners/flink.md | 6 +-
.../en/documentation/sdks/java/testing/nexmark.md | 6 +-
41 files changed, 57 insertions(+), 355 deletions(-)
diff --cc runners/flink/1.12/build.gradle
index 1096d80,d38d873..99dd015
--- a/runners/flink/1.12/build.gradle
+++ b/runners/flink/1.12/build.gradle
@@@ -20,10 -20,10 +20,10 @@@ def basePath = '..
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.12.3'
+ flink_version = '1.12.4'
// Version specific code overrides.
- main_source_overrides = ["${basePath}/1.10/src/main/java",
"${basePath}/1.11/src/main/java", './src/main/java']
- test_source_overrides = ["${basePath}/1.10/src/test/java",
"${basePath}/1.11/src/test/java", './src/test/java']
+ main_source_overrides = ["${basePath}/1.11/src/main/java",
'./src/main/java']
+ test_source_overrides = ["${basePath}/1.11/src/test/java",
'./src/test/java']
main_resources_overrides = []
test_resources_overrides = []
archives_base_name = 'beam-runners-flink-1.12'
diff --cc runners/flink/1.13/build.gradle
index 1799ffc,0484c29..067933f
--- a/runners/flink/1.13/build.gradle
+++ b/runners/flink/1.13/build.gradle
@@@ -20,10 -20,10 +20,10 @@@ def basePath = '..
/* All properties required for loading the Flink build script */
project.ext {
// Set the version of all Flink-related dependencies here.
- flink_version = '1.13.0'
+ flink_version = '1.13.1'
// Version specific code overrides.
- main_source_overrides = ["${basePath}/1.10/src/main/java",
"${basePath}/1.11/src/main/java", "${basePath}/1.12/src/main/java",
'./src/main/java']
- test_source_overrides = ["${basePath}/1.10/src/test/java",
"${basePath}/1.11/src/test/java", "${basePath}/1.12/src/test/java",
'./src/test/java']
+ main_source_overrides = ["${basePath}/1.11/src/main/java",
"${basePath}/1.12/src/main/java", './src/main/java']
+ test_source_overrides = ["${ba
[beam] branch master updated (4f0ea5f -> 511fb05)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 4f0ea5f Merge pull request #14911 from
TheNeuralBit/dataframe-changes-230
add bed7982 [BEAM-12281] Drop support for Flink 1.10
add 22575c7 [BEAM-12281] Update Flink Jenkins jobs to use Flink 1.12
new 511fb05 Merge pull request #14906: [BEAM-12281] Drop support for
Flink 1.10
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.test-infra/dataproc/flink_cluster.sh | 2 +-
.test-infra/jenkins/Flink.groovy | 4 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 2 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 2 +-
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 2 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 2 +-
.../jenkins/job_LoadTests_ParDo_Flink_Go.groovy| 2 +-
.../job_LoadTests_ParDo_Flink_Python.groovy| 2 +-
.../job_LoadTests_SideInput_Flink_Go.groovy| 2 +-
.../jenkins/job_LoadTests_coGBK_Flink_Go.groovy| 2 +-
.../job_LoadTests_coGBK_Flink_Python.groovy| 2 +-
...Commit_Python_Chicago_Taxi_Example_Flink.groovy | 2 +-
CHANGES.md | 1 +
gradle.properties | 2 +-
release/build.gradle.kts | 2 +-
release/src/main/scripts/run_rc_validation.sh | 2 +-
runners/flink/1.10/build.gradle| 33 --
.../flink/1.10/job-server-container/build.gradle | 26
runners/flink/1.10/job-server/build.gradle | 31 --
.../streaming/AbstractStreamOperatorCompat.java| 30 -
.../beam/runners/flink/FlinkRunnerTestCompat.java | 42 -
.../beam/runners/flink/RemoteMiniClusterImpl.java | 71 --
.../runners/flink/SourceTransformationCompat.java | 28 -
runners/flink/1.11/build.gradle| 4 +-
.../runners/flink/streaming/StreamSources.java | 0
runners/flink/1.12/build.gradle| 4 +-
runners/flink/1.13/build.gradle| 4 +-
runners/flink/flink_runner.gradle | 2 +-
.../beam/runners/flink/FlinkPipelineRunner.java| 7 +--
.../org/apache/beam/runners/flink/FlinkRunner.java | 7 +--
.../flink/FlinkExecutionEnvironmentsTest.java | 34 +++
.../FlinkPipelineExecutionEnvironmentTest.java | 22 +++
sdks/go/examples/stringsplit/stringsplit.go| 2 +-
sdks/java/testing/nexmark/build.gradle | 2 +-
sdks/java/testing/tpcds/build.gradle | 2 +-
sdks/python/apache_beam/transforms/sql_test.py | 4 +-
settings.gradle.kts| 4 --
.../site/content/en/contribute/release-guide.md| 2 +-
.../content/en/documentation/dsls/sql/shell.md | 6 +-
.../site/content/en/documentation/runners/flink.md | 6 +-
.../en/documentation/sdks/java/testing/nexmark.md | 6 +-
41 files changed, 57 insertions(+), 355 deletions(-)
delete mode 100644 runners/flink/1.10/build.gradle
delete mode 100644 runners/flink/1.10/job-server-container/build.gradle
delete mode 100644 runners/flink/1.10/job-server/build.gradle
delete mode 100644
runners/flink/1.10/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
delete mode 100644
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkRunnerTestCompat.java
delete mode 100644
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java
delete mode 100644
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/SourceTransformationCompat.java
rename runners/flink/{1.10 =>
1.11}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
(100%)
[beam] branch master updated (b39ff90 -> ae82018)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b39ff90 Merge pull request #14899 from y1chi/prebuild_timeout add 10b6f34 [BEAM-12427] Ignore the AutoValue_* classes generated in "generated" folder from build process. Appears for sdks>java>io module. new ae82018 Merge pull request #14913: [BEAM-12427] Ignore the AutoValue_* classes generated in "generated" in sdks>java>io The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .gitignore | 1 + 1 file changed, 1 insertion(+)
[beam] 01/01: Merge pull request #14913: [BEAM-12427] Ignore the AutoValue_* classes generated in "generated" in sdks>java>io
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit ae820187ad5f8fc4c27504aa708068476e299b8f Merge: b39ff90 10b6f34 Author: Ismaël Mejía AuthorDate: Tue Jun 1 21:35:29 2021 +0200 Merge pull request #14913: [BEAM-12427] Ignore the AutoValue_* classes generated in "generated" in sdks>java>io .gitignore | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (441dc8b -> 3876ddb)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 441dc8b Merge pull request #14905: [BEAM-12277] Update Flink 1.13 version to 1.13.1 add 780e9ca [BEAM-12424] Update Flink 1.12 to version 1.12.4 add 3876ddb Merge pull request #14907: [BEAM-12424] Update Flink 1.12 to version 1.12.4 No new revisions were added by this update. Summary of changes: runners/flink/1.12/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (d8c5da8 -> 441dc8b)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d8c5da8 [BEAM-11608] Development environment set up automation (#14584) add 4031432 [BEAM-12277] Update Flink 1.13 version to 1.13.1 add 441dc8b Merge pull request #14905: [BEAM-12277] Update Flink 1.13 version to 1.13.1 No new revisions were added by this update. Summary of changes: runners/flink/1.13/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build target.
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 00eb4207506720e7367098ad904bb37ba6de88a9 Merge: 2516cac 881da20 Author: Ismaël Mejía AuthorDate: Fri May 28 15:38:17 2021 +0200 Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build target. .test-infra/jenkins/CommonTestProperties.groovy| 2 +- CHANGES.md | 1 + gradle.properties | 2 +- runners/flink/1.13/build.gradle| 33 +++ .../flink/1.13/job-server-container/build.gradle | 26 runners/flink/1.13/job-server/build.gradle | 31 ++ .../runners/flink/streaming/StreamSources.java | 69 ++ .../beam/runners/flink/metrics/FileReporter.java | 8 ++- .../python/apache_beam/options/pipeline_options.py | 2 +- .../runners/portability/flink_runner_test.py | 49 +++ settings.gradle.kts| 4 ++ .../site/content/en/documentation/runners/flink.md | 24 +++- 12 files changed, 220 insertions(+), 31 deletions(-)
[beam] branch master updated (2516cac -> 00eb420)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 2516cac Merge pull request #14897: [BEAM-12415] Update Spark 3
version to 3.1.2
add b342805 [BEAM-12277] Add flink 1.13 build target.
add 8ad8965 [BEAM-12277] Make Flink 1.13 the default Flink version to run
the tests
add 8be1cb1 [BEAM-12277] Update expected metric name formatting.
add 881da20 [BEAM-12277] fix mistake in source overrides
new 00eb420 Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build
target.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.test-infra/jenkins/CommonTestProperties.groovy| 2 +-
CHANGES.md | 1 +
gradle.properties | 2 +-
runners/flink/{1.12 => 1.13}/build.gradle | 8 ++--
.../job-server-container/build.gradle | 0
.../flink/{1.10 => 1.13}/job-server/build.gradle | 2 +-
.../runners/flink/streaming/StreamSources.java | 0
.../beam/runners/flink/metrics/FileReporter.java | 8 +++-
.../python/apache_beam/options/pipeline_options.py | 2 +-
.../runners/portability/flink_runner_test.py | 49 +++---
settings.gradle.kts| 4 ++
.../site/content/en/documentation/runners/flink.md | 24 +--
12 files changed, 66 insertions(+), 36 deletions(-)
copy runners/flink/{1.12 => 1.13}/build.gradle (84%)
copy runners/flink/{1.12 => 1.13}/job-server-container/build.gradle (100%)
copy runners/flink/{1.10 => 1.13}/job-server/build.gradle (95%)
copy runners/flink/{1.12 =>
1.13}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
(100%)
[beam] branch master updated (b03e429 -> 2516cac)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b03e429 Merge pull request #14854: [BEAM-12384] Set output typeDescriptor explictly in Read.Bounded transform add b6fca2b [BEAM-12415] Update Spark 3 version to 3.1.2 add 2516cac Merge pull request #14897: [BEAM-12415] Update Spark 3 version to 3.1.2 No new revisions were added by this update. Summary of changes: runners/spark/3/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (31988c8 -> b03e429)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 31988c8 Merge pull request #14898 from boyuanzz/fix add 331c67c [BEAM-12384] Refine generic types on Read.Bounded internals add 3b705c1 [BEAM-12384] Set output typeDescriptor explictly in Read.Bounded transform add 83bccf9 [BEAM-12384] Infer typeDescriptor from coder if typeDescriptor is not set explicitly. add b03e429 Merge pull request #14854: [BEAM-12384] Set output typeDescriptor explictly in Read.Bounded transform No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/io/Read.java | 56 -- .../org/apache/beam/sdk/values/PCollection.java| 8 +++- .../test/java/org/apache/beam/sdk/io/ReadTest.java | 43 + 3 files changed, 80 insertions(+), 27 deletions(-)
[beam] branch master updated (1c63759 -> 2abbd37)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 1c63759 [BEAM-9547] Add additional unliftable grouped aggregations, add describe to DataFrame and Series (#14850) add 248dc26 [BEAM-12240] Add Java 17 as a valid environment add 2abbd37 Merge pull request #14880: [BEAM-12240] Add Java 17 as a valid environment No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/runners/core/construction/Environments.java | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-)
[beam] branch master updated (c18a06c -> df3b4d8)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c18a06c Merge pull request #14867 from ihji/BEAM-12390 add a4d535b [BEAM-12241] Update vendored bytebuddy to version 1.11.0 add df3b4d8 Merge pull request #14824: [BEAM-12241] Update vendored bytebuddy to version 1.11.0 No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- sdks/java/core/build.gradle| 2 +- .../apache/beam/sdk/coders/RowCoderGenerator.java | 40 ++-- .../beam/sdk/schemas/utils/AutoValueUtils.java | 44 ++--- .../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 26 .../apache/beam/sdk/schemas/utils/AvroUtils.java | 16 ++--- .../utils/ByteBuddyLocalVariableManager.java | 10 +-- .../beam/sdk/schemas/utils/ByteBuddyUtils.java | 72 +++--- .../beam/sdk/schemas/utils/ConvertHelpers.java | 28 - .../beam/sdk/schemas/utils/JavaBeanUtils.java | 34 +- .../apache/beam/sdk/schemas/utils/POJOUtils.java | 46 +++--- .../sdk/schemas/utils/SelectByteBuddyHelpers.java | 62 +-- .../reflect/ByteBuddyDoFnInvokerFactory.java | 68 ++-- .../reflect/ByteBuddyOnTimerInvokerFactory.java| 32 +- .../reflect/StableInvokerNamingStrategy.java | 4 +- sdks/java/extensions/protobuf/build.gradle | 2 +- .../extensions/protobuf/ProtoByteBuddyUtils.java | 62 +-- 17 files changed, 275 insertions(+), 275 deletions(-)
svn commit: r47844 - /dev/beam/2.29.0/
Author: iemejia Date: Thu May 20 21:28:29 2021 New Revision: 47844 Log: Move 2.29.0 to release repository Removed: dev/beam/2.29.0/
svn commit: r47843 - in /release/beam: ./ 2.29.0/ 2.29.0/python/
Author: iemejia Date: Thu May 20 21:16:14 2021 New Revision: 47843 Log: Move 2.29.0 to release repository Added: release/beam/2.29.0/ release/beam/2.29.0/apache-beam-2.29.0-source-release.zip (with props) release/beam/2.29.0/apache-beam-2.29.0-source-release.zip.asc release/beam/2.29.0/apache-beam-2.29.0-source-release.zip.sha512 release/beam/2.29.0/python/ release/beam/2.29.0/python/apache-beam-2.29.0.zip (with props) release/beam/2.29.0/python/apache-beam-2.29.0.zip.asc release/beam/2.29.0/python/apache-beam-2.29.0.zip.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl (with props) release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.asc release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.sha512 release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_x86_64.whl
svn commit: r47842 - in /dev/beam: extensions/ vendor/
Author: iemejia Date: Thu May 20 20:34:11 2021 New Revision: 47842 Log: Remove empty directories for extensions and vendor Removed: dev/beam/extensions/ dev/beam/vendor/
svn commit: r47841 - in /release/beam/vendor/beam-vendor-bytebuddy-1_11_0: ./ 0.1/
Author: iemejia Date: Thu May 20 20:32:25 2021 New Revision: 47841 Log: Move Apache Beam vendored Byte Buddy 1.11.0 v0.1 to release branch Added: release/beam/vendor/beam-vendor-bytebuddy-1_11_0/ release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/ release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip (with props) release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 Added: release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip == Binary file - no diff available. Propchange: release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip -- svn:mime-type = application/octet-stream Added: release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc == --- release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc (added) +++ release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc Thu May 20 20:32:25 2021 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEENBVjFynhWzMFGttnCp2vZxO4Y0kFAmCiiIUACgkQCp2vZxO4 +Y0kxwg//Rk4VH2BFbhQg//kc8KGjmjyT22exx1M55ttHmDaYmRlGR8YYBoLg6nfK +uybzgQVHlAlMLqv3CuHyVX0YrA+iBc9AtvbwuIvHDhkwV5asyXbBhkUJ6nOzy6WY +wUWOA7TTpFEOZUJiVDfOLYICo7QEuStJSVaEFome1JSFyK18cIy7uAKQfYIC1n4S +Sl32RKYg4EXJACULO6sOnQq9D4sBeiqJ9IJcV/uGPwgN3ByJAk/GzsTnPsUrVA+B +8tsa0USllAhMB6jTr3fGIOMDHaJJnIMr+qi/0Wmuc1Lq07tSsUtK8jeTVrtYy6zr +FMQ6g2RKZq8ofQfPyPVLD7Glfb8COnJASgcGY6Iwlht9PRCmvUBaBuKN6Xz5pruS +QHbf+EsiWi4dU9VX/aWI8w7csKLh2mOiQf/8ARIqPdQTmGTsfmQyKfZQzjPgq+lg +Qw1j8lU/hMjCQmRU6RnDToJP70+THEK/PffiyycFsjSbkoGtu6CjROdjWVQQJALi +jtqxf0KxH+jaYmQ3DfwToj4CdeVoY1IpoqXv+Jt+qboWE+xiUJYpsTzDq2Agy8+d +kKTk/8dU/sxFNwwo2DgkryGw0sk+0PZfGAirBjVhn2CnAfsoaJvdWBqYnRCYLXYn +FrV5PKzsrTMe8SZK183KJudKpASGXj+t2dxRTSF3MMxk3HJJIlE= +=GL+y +-END PGP SIGNATURE- Added: release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 == --- release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 (added) +++ release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 Thu May 20 20:32:25 2021 @@ -0,0 +1 @@ +e368451efee117d44d1286e87566e8a085395dad0c4c0051cd7e15b3cf87c56cfb8f63168efc5e1b44ccf408f90b1a3242915abdc780b41bb12630a454573697 apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
svn commit: r47840 - /dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/
Author: iemejia Date: Thu May 20 20:21:17 2021 New Revision: 47840 Log: Move Apache Beam vendored Byte Buddy 1.11.0 v0.1 to release branch Removed: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/
svn commit: r47839 - in /release/beam: 2.25.0/ 2.26.0/ 2.27.0/ 2.28.0/ extensions/ vendor/beam-vendor-bytebuddy-1_10_8/ vendor/beam-vendor-bytebuddy-1_9_3/ vendor/beam-vendor-grpc-1_21_0/ vendor/beam-
Author: iemejia Date: Thu May 20 20:17:19 2021 New Revision: 47839 Log: Remove releases already moved to archive.apache.org Removed: release/beam/2.25.0/ release/beam/2.26.0/ release/beam/2.27.0/ release/beam/2.28.0/ release/beam/extensions/ release/beam/vendor/beam-vendor-bytebuddy-1_10_8/ release/beam/vendor/beam-vendor-bytebuddy-1_9_3/ release/beam/vendor/beam-vendor-grpc-1_21_0/ release/beam/vendor/beam-vendor-grpc-1_26_0/ release/beam/vendor/beam-vendor-guava-26_0-jre/ release/beam/vendor/calcite/
[beam] branch master updated (8463a05 -> e0e3432)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8463a05 [BEAM-11227] Upgrade vendored gRPC to 1.36.0 with test fix (#14833) add 922f7f6 [BEAM-12342] Upgrade Spark 2 to version 2.4.8 add e0e3432 Merge pull request #14823: [BEAM-12342] Upgrade Spark 2 to version 2.4.8 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- runners/spark/2/build.gradle| 2 +- 2 files changed, 2 insertions(+), 2 deletions(-)
svn commit: r47768 - in /dev/beam/vendor/beam-vendor-bytebuddy-1_11_0: ./ 0.1/
Author: iemejia Date: Mon May 17 15:29:42 2021 New Revision: 47768 Log: Add vendor bytebuddy 1.11.0 Added: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/ dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/ dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip (with props) dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 Added: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip == Binary file - no diff available. Propchange: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip -- svn:mime-type = application/octet-stream Added: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc == --- dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc (added) +++ dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc Mon May 17 15:29:42 2021 @@ -0,0 +1,16 @@ +-BEGIN PGP SIGNATURE- + +iQIzBAABCgAdFiEENBVjFynhWzMFGttnCp2vZxO4Y0kFAmCiiIUACgkQCp2vZxO4 +Y0kxwg//Rk4VH2BFbhQg//kc8KGjmjyT22exx1M55ttHmDaYmRlGR8YYBoLg6nfK +uybzgQVHlAlMLqv3CuHyVX0YrA+iBc9AtvbwuIvHDhkwV5asyXbBhkUJ6nOzy6WY +wUWOA7TTpFEOZUJiVDfOLYICo7QEuStJSVaEFome1JSFyK18cIy7uAKQfYIC1n4S +Sl32RKYg4EXJACULO6sOnQq9D4sBeiqJ9IJcV/uGPwgN3ByJAk/GzsTnPsUrVA+B +8tsa0USllAhMB6jTr3fGIOMDHaJJnIMr+qi/0Wmuc1Lq07tSsUtK8jeTVrtYy6zr +FMQ6g2RKZq8ofQfPyPVLD7Glfb8COnJASgcGY6Iwlht9PRCmvUBaBuKN6Xz5pruS +QHbf+EsiWi4dU9VX/aWI8w7csKLh2mOiQf/8ARIqPdQTmGTsfmQyKfZQzjPgq+lg +Qw1j8lU/hMjCQmRU6RnDToJP70+THEK/PffiyycFsjSbkoGtu6CjROdjWVQQJALi +jtqxf0KxH+jaYmQ3DfwToj4CdeVoY1IpoqXv+Jt+qboWE+xiUJYpsTzDq2Agy8+d +kKTk/8dU/sxFNwwo2DgkryGw0sk+0PZfGAirBjVhn2CnAfsoaJvdWBqYnRCYLXYn +FrV5PKzsrTMe8SZK183KJudKpASGXj+t2dxRTSF3MMxk3HJJIlE= +=GL+y +-END PGP SIGNATURE- Added: dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 == --- dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 (added) +++ dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512 Mon May 17 15:29:42 2021 @@ -0,0 +1 @@ +e368451efee117d44d1286e87566e8a085395dad0c4c0051cd7e15b3cf87c56cfb8f63168efc5e1b44ccf408f90b1a3242915abdc780b41bb12630a454573697 apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
[beam] branch master updated: [BEAM-12343] GroupByKeyTest for changing WindowFn from GlobalWindow after GBK
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 3fd92d0 [BEAM-12343] GroupByKeyTest for changing WindowFn from
GlobalWindow after GBK
new d93c591 Merge pull request #14667:[BEAM-12343] GroupByKeyTest for
changing WindowFn from GlobalWindow after GBK
3fd92d0 is described below
commit 3fd92d0df24f671a9f5705860250acc3afd11e47
Author: Jan Lukavsky
AuthorDate: Wed Apr 28 09:38:54 2021 +0200
[BEAM-12343] GroupByKeyTest for changing WindowFn from GlobalWindow after
GBK
---
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 40 ++
1 file changed, 40 insertions(+)
diff --git
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index 88d40ac..fd849a5 100644
---
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -70,8 +70,10 @@ import org.apache.beam.sdk.values.KV;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.TypeDescriptors;
import org.apache.beam.sdk.values.WindowingStrategy;
import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Streams;
import org.checkerframework.checker.nullness.qual.Nullable;
import org.hamcrest.Matcher;
import org.joda.time.Duration;
@@ -585,6 +587,44 @@ public class GroupByKeyTest implements Serializable {
}
@Test
+@Category(ValidatesRunner.class)
+public void testRewindowWithTimestampCombiner() {
+ PCollection> input =
+ p.apply(
+ Create.timestamped(
+ TimestampedValue.of(KV.of("foo", 1), new Instant(1)),
+ TimestampedValue.of(KV.of("foo", 4), new Instant(4)),
+ TimestampedValue.of(KV.of("bar", 3), new Instant(3)),
+ TimestampedValue.of(KV.of("foo", 9), new Instant(9
+ .apply(
+ "GlobalWindows",
+ Window.>configure()
+ .withTimestampCombiner(TimestampCombiner.LATEST));
+
+ PCollection> result =
+ input
+ .apply(GroupByKey.create())
+ .apply(
+ MapElements.into(
+ TypeDescriptors.kvs(
+ TypeDescriptors.strings(),
TypeDescriptors.integers()))
+ .via(kv -> KV.of(kv.getKey(), sum(kv.getValue()
+ .apply("FixedWindows",
Window.into(FixedWindows.of(Duration.millis(1;
+
+ PAssert.that(result)
+ .inWindow(new IntervalWindow(new Instant(9), new Instant(10)))
+ .containsInAnyOrder(KV.of("foo", 14))
+ .inWindow(new IntervalWindow(new Instant(3), new Instant(4)))
+ .containsInAnyOrder(KV.of("bar", 3));
+
+ p.run();
+}
+
+private static int sum(Iterable parts) {
+ return Streams.stream(parts).mapToInt(e -> e).sum();
+}
+
+@Test
@Category(NeedsRunner.class)
public void testIdentityWindowFnPropagation() {
[beam] branch master updated (beb0d16 -> d9c113e)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from beb0d16 Merge pull request #14746 from ibzib/BEAM-12302
add 7fc292b [BEAM-12241] Upgrade vendored bytebuddy to version 1.11.0
add d9c113e Merge pull request #14766: [BEAM-12241] Upgrade vendored
bytebuddy to version 1.11.0
No new revisions were added by this update.
Summary of changes:
settings.gradle.kts| 2 +-
vendor/{bytebuddy-1_10_8 => bytebuddy-1_11_0}/build.gradle.kts | 8
2 files changed, 5 insertions(+), 5 deletions(-)
rename vendor/{bytebuddy-1_10_8 => bytebuddy-1_11_0}/build.gradle.kts (89%)
[beam] branch master updated (c3c4933 -> 9dbc7b7)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c3c4933 Merge pull request #14783 [BEAM-12326] Preserve resource hints across transform substitution. add a1c8f4b [BEAM-10670] Use non-SDF based translation for Read by default on Spark Runner add 9dbc7b7 Merge pull request #14755: [BEAM-10670] Use non-SDF based translation for Read by default on Spark Runner No new revisions were added by this update. Summary of changes: .../structuredstreaming/SparkStructuredStreamingRunner.java| 8 ++-- .../main/java/org/apache/beam/runners/spark/SparkRunner.java | 8 ++-- .../org/apache/beam/runners/spark/SparkRunnerDebugger.java | 10 -- .../org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java | 2 +- 4 files changed, 9 insertions(+), 19 deletions(-)
[beam] 01/01: Merge pull request #14751: [BEAM-11055] Fix mistaken version of log4j in HCatalog
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit d6d6f30723fbfb1a0472fe814b419232e06952a5 Merge: 5da41e6 d589b62 Author: Ismaël Mejía AuthorDate: Sat May 8 06:45:53 2021 +0200 Merge pull request #14751: [BEAM-11055] Fix mistaken version of log4j in HCatalog sdks/java/io/hcatalog/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (5da41e6 -> d6d6f30)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 5da41e6 Merge pull request #14757 from iemejia/BEAM-12282-vendor-grpc-update-revert add d589b62 [BEAM-11055] Fix mistaken version of log4j in HCatalog new d6d6f30 Merge pull request #14751: [BEAM-11055] Fix mistaken version of log4j in HCatalog The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: sdks/java/io/hcatalog/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (7af241e -> 8787c13)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7af241e Merge pull request #14739 from kennknowles/actually-2.29.0-was-later add 9fef5b4 Only run one pipeline in testWriteViaSink add 8787c13 Merge pull request #14740: [BEAM-7320] Only run one pipeline in TextIOWriteTest.testWriteViaSink No new revisions were added by this update. Summary of changes: .../test/java/org/apache/beam/sdk/io/TextIOWriteTest.java | 14 -- 1 file changed, 14 deletions(-)
[beam] branch master updated (66caaa9 -> 7f0d11e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 66caaa9 Merge pull request #14708: [BEAM-12271] Make max width for code snippets adaptive with text blocks add 0547c64 [BEAM-12280] Upgrade Flink runner to Flink version 1.12.3 add 7f0d11e Merge pull request #14721: [BEAM-12280] Upgrade Flink runner to Flink version 1.12.3 No new revisions were added by this update. Summary of changes: runners/flink/1.12/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (b9e8aca -> 07e7613a)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from b9e8aca Merge pull request #14709 from pabloem/revert-pr13617 add 79ea6dd [BEAM-12253] Change Read.UnboundedSourceAsSDFRestrictionTracker to use the cache. add 07e7613a Merge pull request #14704: [BEAM-12253] Change Read.UnboundedSourceAsSDFRestrictionTracker.getSplitBacklog to use the reader cache No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/io/Read.java | 41 -- 1 file changed, 23 insertions(+), 18 deletions(-)
[beam] 03/03: Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 7418c84de2b68b06e0318766ca6c69f4d2144390 Merge: 87e1164 be906f3 Author: Ismaël Mejía AuthorDate: Thu Apr 29 08:13:58 2021 +0200 Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations .../beam/runners/core/InMemoryTimerInternals.java | 46 -- .../main/java/org/apache/beam/sdk/values/Row.java | 28 ++--- .../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++ 3 files changed, 41 insertions(+), 46 deletions(-)
[beam] 02/03: [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit be906f38c5cac14c465332b8179022085ed76c13
Author: Daniel Kulp
AuthorDate: Wed Apr 28 14:21:13 2021 -0400
[BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
---
.../main/java/org/apache/beam/sdk/values/Row.java | 28 ++
.../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++
2 files changed, 25 insertions(+), 16 deletions(-)
diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index a19d9b3..9aa9277 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -847,23 +847,19 @@ public abstract class Row implements Serializable {
+ " fields.");
}
- FieldOverrides fieldOverrides = new FieldOverrides(schema);
- fieldOverrides.setOverrides(this.values);
-
- Row row;
- if (!fieldOverrides.isEmpty()) {
-row =
-(Row)
-new RowFieldMatcher()
-.match(
-new CapturingRowCases(schema, fieldOverrides),
-FieldType.row(schema),
-new RowPosition(FieldAccessDescriptor.create()),
-null);
- } else {
-row = new RowWithStorage(schema, Collections.emptyList());
+ if (!values.isEmpty()) {
+FieldOverrides fieldOverrides = new FieldOverrides(schema,
this.values);
+if (!fieldOverrides.isEmpty()) {
+ return (Row)
+ new RowFieldMatcher()
+ .match(
+ new CapturingRowCases(schema, fieldOverrides),
+ FieldType.row(schema),
+ new RowPosition(FieldAccessDescriptor.create()),
+ null);
+}
}
- return row;
+ return new RowWithStorage(schema, Collections.emptyList());
}
}
diff --git
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
index d3b1c21..a5f6210 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
@@ -236,6 +236,11 @@ class RowUtils {
this.rootSchema = rootSchema;
}
+FieldOverrides(Schema rootSchema, List overrides) {
+ this.topNode = new FieldAccessNode(rootSchema, overrides);
+ this.rootSchema = rootSchema;
+}
+
boolean isEmpty() {
return topNode.isEmpty();
}
@@ -270,6 +275,14 @@ class RowUtils {
nestedAccess = Lists.newArrayList();
}
+ FieldAccessNode(Schema schema, List overrides) {
+fieldOverrides =
Lists.newArrayListWithExpectedSize(schema.getFieldCount());
+for (Object value : overrides) {
+ fieldOverrides.add(new FieldOverride(value));
+}
+nestedAccess = Lists.newArrayList();
+ }
+
boolean isEmpty() {
return fieldOverrides.isEmpty() && nestedAccess.isEmpty();
}
[beam] branch master updated (87e1164 -> 7418c84)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 87e1164 Merge pull request #14676: Revert "[BEAM-3713] Moving integration tests from nose to pytest (#14481)" new b87f1f0 [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals new be906f3 [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils new 7418c84 Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations The 3 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/runners/core/InMemoryTimerInternals.java | 46 -- .../main/java/org/apache/beam/sdk/values/Row.java | 28 ++--- .../java/org/apache/beam/sdk/values/RowUtils.java | 13 ++ 3 files changed, 41 insertions(+), 46 deletions(-)
[beam] 01/03: [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit b87f1f01197a78d60fb08a0ee7a4ef96bc4e08eb
Author: Daniel Kulp
AuthorDate: Wed Apr 28 14:16:31 2021 -0400
[BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals
---
.../beam/runners/core/InMemoryTimerInternals.java | 46 --
1 file changed, 16 insertions(+), 30 deletions(-)
diff --git
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
index 8be9081..d0b3bed 100644
---
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -63,6 +63,9 @@ public class InMemoryTimerInternals implements TimerInternals
{
/** Current synchronized processing time. */
private Instant synchronizedProcessingTime =
BoundedWindow.TIMESTAMP_MIN_VALUE;
+ /** Class.getSimpleName() cached to avoid allocations for tracing. */
+ private static final String SIMPLE_NAME =
InMemoryTimerInternals.class.getSimpleName();
+
@Override
public @Nullable Instant currentOutputWatermarkTime() {
return outputWatermarkTime;
@@ -125,17 +128,12 @@ public class InMemoryTimerInternals implements
TimerInternals {
@Deprecated
@Override
public void setTimer(TimerData timerData) {
-WindowTracing.trace("{}.setTimer: {}", getClass().getSimpleName(),
timerData);
+WindowTracing.trace("{}.setTimer: {}", SIMPLE_NAME, timerData);
-@Nullable
-TimerData existing =
-existingTimers.get(
-timerData.getNamespace(), timerData.getTimerId() + '+' +
timerData.getTimerFamilyId());
+@Nullable String colKey = timerData.getTimerId() + '+' +
timerData.getTimerFamilyId();
+TimerData existing = existingTimers.get(timerData.getNamespace(), colKey);
if (existing == null) {
- existingTimers.put(
- timerData.getNamespace(),
- timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
- timerData);
+ existingTimers.put(timerData.getNamespace(), colKey, timerData);
timersForDomain(timerData.getDomain()).add(timerData);
} else {
checkArgument(
@@ -149,10 +147,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
NavigableSet timers =
timersForDomain(timerData.getDomain());
timers.remove(existing);
timers.add(timerData);
-existingTimers.put(
-timerData.getNamespace(),
-timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
-timerData);
+existingTimers.put(timerData.getNamespace(), colKey, timerData);
}
}
}
@@ -216,7 +211,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
newInputWatermark);
WindowTracing.trace(
"{}.advanceInputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
inputWatermarkTime,
newInputWatermark);
inputWatermarkTime = newInputWatermark;
@@ -229,7 +224,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
if (newOutputWatermark.isAfter(inputWatermarkTime)) {
WindowTracing.trace(
"{}.advanceOutputWatermark: clipping output watermark from {} to {}",
- getClass().getSimpleName(),
+ SIMPLE_NAME,
newOutputWatermark,
inputWatermarkTime);
adjustedOutputWatermark = inputWatermarkTime;
@@ -244,7 +239,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
adjustedOutputWatermark);
WindowTracing.trace(
"{}.advanceOutputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
outputWatermarkTime,
adjustedOutputWatermark);
outputWatermarkTime = adjustedOutputWatermark;
@@ -259,10 +254,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
processingTime,
newProcessingTime);
WindowTracing.trace(
-"{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
-processingTime,
-newProcessingTime);
+"{}.advanceProcessingTime: from {} to {}", SIMPLE_NAME,
processingTime, newProcessingTime);
processingTime = newProcessingTime;
}
@@ -277,7 +269,7 @@ public class InMemoryTimerInternals implements
TimerInternals {
newSynchronizedProcessingTime);
WindowTracing.trace(
"{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
synchronizedProcessingTime,
newSynchronizedProcessingTime);
synchronizedProcessingTime = newSync
[beam] branch master updated (bc2fb2a -> b1a67ef)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from bc2fb2a Merge pull request #14645 from [BEAM-12188] Bump snakeyaml to 1.28 add 6ebadec [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" add b1a67ef Merge pull request #14670: [BEAM-12243] TPC-DS: use SQL "substring()" instead of "substr()" No new revisions were added by this update. Summary of changes: sdks/java/testing/tpcds/src/main/resources/queries/query15.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query19.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query23a.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query23b.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query45.sql | 2 +- sdks/java/testing/tpcds/src/main/resources/queries/query62.sql | 6 +++--- sdks/java/testing/tpcds/src/main/resources/queries/query79.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query8.sql | 8 sdks/java/testing/tpcds/src/main/resources/queries/query85.sql | 4 ++-- sdks/java/testing/tpcds/src/main/resources/queries/query99.sql | 6 +++--- 10 files changed, 21 insertions(+), 21 deletions(-)
[beam] branch master updated (4202fb8 -> ea107d0)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4202fb8 Merge pull request #14661 - Fix worker_pool plumbing to use a mutable List. add 62b9c0f [BEAM-9283] Disable caching ValidatesRunner tests tasks add ea107d0 Merge pull request #14640: [BEAM-9283] Disable caching ValidatesRunner tests tasks No new revisions were added by this update. Summary of changes: runners/flink/flink_runner.gradle | 2 ++ runners/spark/spark_runner.gradle | 6 ++ 2 files changed, 8 insertions(+)
[beam] branch master updated (e13e548 -> 9505f4e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e13e548 [BEAM-10029] Adds Grafana config for Spanner IO Perf Metrics (#14643) add ef654f6 Close ZipOutputStream add 9505f4e Merge pull request #14631: [BEAM-12220] Close ZipOutputStream to avoid native memory leak No new revisions were added by this update. Summary of changes: sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java | 1 + 1 file changed, 1 insertion(+)
[beam] branch master updated (618b060 -> f61d100)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 618b060 Merge pull request #14626: [BEAM-12214] RedisIO.readAll() was removed since it is deprecated since Beam 2.13.0 add 85a5560 [BEAM-12216] MqttIO.create() with clientId constructor removed because it was deprecated since Beam 2.13.0 add f61d100 Merge pull request #14627: [BEAM-12216] MqttIO.create() with clientId constructor removed as it is deprecated since Beam 2.13.0 No new revisions were added by this update. Summary of changes: CHANGES.md | 2 ++ .../main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java| 16 2 files changed, 2 insertions(+), 16 deletions(-)
[beam] branch master updated (018f5e7 -> 9209c75)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 018f5e7 Merge pull request #14575 from [BEAM-12212] Adds --sdkContainerImage as new Java Dataflow PipelineOption add bf17110 [BEAM-12217] MongoDbIO: Read.withFilter() and Read.withProjection() are removed add 9209c75 Merge pull request #14628: [BEAM-12217] MongoDbIO: Read.withFilter() and Read.withProjection() are removed since they are deprecated since Beam 2.12.0 No new revisions were added by this update. Summary of changes: CHANGES.md | 2 ++ .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 37 -- 2 files changed, 2 insertions(+), 37 deletions(-)
[beam] branch master updated (f15294a -> 3246690)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f15294a [BEAM-8691] Upgrading bigtable-client-core to 1.19.1 add 62e0f38 [BEAM-12210] Use formatting string for checkArgument to avoid excess String appends add fc45ad4 Fix spotless add 3246690 Merge pull request #14620: [BEAM-12210] Use formatting string for checkArgument to avoid excess … No new revisions were added by this update. Summary of changes: .../src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-)
[beam] branch master updated (2e9ee8c -> 93ecc1d)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2e9ee8c [BEAM-2085] Fixups for Python resource hints. (#14605) add 8e0ee41 [BEAM-12207] Remove log messages about files to stage. add 4e3decb [BEAM-12207] Enable nullness check on SparkPipelineRunner.java. add 93ecc1d Merge pull request #14616: [BEAM-12207] Remove log messages about files to stage. No new revisions were added by this update. Summary of changes: .../org/apache/beam/runners/spark/SparkPipelineRunner.java | 10 +++--- 1 file changed, 3 insertions(+), 7 deletions(-)
[beam] branch master updated (667ec40 -> 985e2f0)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 667ec40 Merge pull request #14504: Refactor PR template to separate test types and label test variants add 3ea492d [BEAM-12172] Bump gradle to 6.8.3 add 985e2f0 Merge pull request #14543: [BEAM-12172] Bump gradle to 6.8.3 No new revisions were added by this update. Summary of changes: gradle/wrapper/gradle-wrapper.properties | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (97af077 -> 8e66956)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 97af077 [BEAM-9547] DataFrame.corr cleanup (#14327) add f9d4805 [BEAM-12091] Make file staging uniform among runners add 8e66956 Merge pull request #14520: [BEAM-12091] Make file staging uniform among runners No new revisions were added by this update. Summary of changes: .../construction/resources/PipelineResources.java | 37 +- .../resources/PipelineResourcesTest.java | 78 +++--- .../flink/FlinkPipelineExecutionEnvironment.java | 18 + .../SparkStructuredStreamingRunner.java| 20 +- .../runners/spark/SparkCommonPipelineOptions.java | 27 +--- .../beam/runners/spark/SparkPipelineRunner.java| 8 --- .../org/apache/beam/runners/spark/SparkRunner.java | 18 + .../beam/runners/twister2/Twister2Runner.java | 33 + .../beam/sdk/util/common/ReflectHelpers.java | 2 +- 9 files changed, 114 insertions(+), 127 deletions(-)
[beam] branch master updated (a86dc06 -> b10ce99)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a86dc06 Bump container pandas version to 1.2.4 (#14524) add 992c378 [BEAM-2888] Added packages.confluent.io maven repo add b10ce99 Merge pull request #14545: [BEAM-2888] Added "packages.confluent.io" maven repo No new revisions were added by this update. Summary of changes: .test-infra/validate-runner/build.gradle | 3 +++ 1 file changed, 3 insertions(+)
[beam] 03/04: [BEAM-11712] Fix static analysis warnings and typos on TPC-DS module
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8fe0c5cee1bab62680ebd92a51aed8f3da80e190
Author: Ismaël Mejía
AuthorDate: Tue Apr 13 14:20:54 2021 +0200
[BEAM-11712] Fix static analysis warnings and typos on TPC-DS module
---
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 3 +-
.../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 4 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 3 +-
.../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 2 +-
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 2 +-
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 6 +-
.../beam/sdk/tpcds/TpcdsParametersReader.java | 4 +-
.../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 11 +--
.../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 102 ++---
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 3 +-
.../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 6 +-
11 files changed, 68 insertions(+), 78 deletions(-)
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 69e676f..304fdd2 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -211,8 +211,7 @@ public class BeamSqlEnvRunner {
// Transform the result from PCollection into
PCollection, and write it to the
// location where results are stored.
PCollection rowStrings =
-rows.apply(
-MapElements.into(TypeDescriptors.strings()).via((Row row) ->
row.toString()));
+
rows.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString));
rowStrings.apply(
TextIO.write()
.to(
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
index d66b128..d6c8ed8 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
@@ -33,8 +33,8 @@ import org.apache.commons.csv.CSVFormat;
*/
public class CsvToRow extends PTransform, PCollection>
implements Serializable {
- private Schema schema;
- private CSVFormat csvFormat;
+ private final Schema schema;
+ private final CSVFormat csvFormat;
public CSVFormat getCsvFormat() {
return csvFormat;
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
index 7b00a37..c6f3253 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
@@ -35,7 +35,6 @@ public class QueryReader {
*/
public static String readQuery(String queryFileName) throws Exception {
String path = "queries/" + queryFileName + ".sql";
-String query = Resources.toString(Resources.getResource(path),
Charsets.UTF_8);
-return query;
+return Resources.toString(Resources.getResource(path), Charsets.UTF_8);
}
}
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
index 40a8cc5..a087948 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
@@ -33,7 +33,7 @@ import org.apache.commons.csv.CSVFormat;
*/
public class RowToCsv extends PTransform, PCollection>
implements Serializable {
- private CSVFormat csvFormat;
+ private final CSVFormat csvFormat;
public RowToCsv(CSVFormat csvFormat) {
this.csvFormat = csvFormat;
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
index 4f56c1a..bea0261 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
@@ -189,7 +189,7 @@ public class SqlTransformRunner {
try {
tables
.apply(SqlTransform.query(queryString))
-.apply(MapElements.into(TypeDescriptors.strings()).via((Row row)
-> row.toString()))
+
.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString))
.apply(
TextIO.write()
[beam] 04/04: Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit b3ef2035abf9ca2dd94a11a1a6aa4440df28adb9 Merge: f805f1c 8fe0c5c Author: Ismaël Mejía AuthorDate: Tue Apr 13 14:22:12 2021 +0200 Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner sdks/java/testing/tpcds/README.md | 68 + sdks/java/testing/tpcds/build.gradle | 108 +- .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 327 +++-- .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 50 +- .../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 47 +- .../org/apache/beam/sdk/tpcds/QueryReader.java | 51 +- .../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 38 +- .../apache/beam/sdk/tpcds/SqlTransformRunner.java | 314 +++-- .../apache/beam/sdk/tpcds/SummaryGenerator.java| 219 ++-- .../beam/sdk/tpcds/TableSchemaJSONLoader.java | 162 +-- .../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 40 +- .../beam/sdk/tpcds/TpcdsOptionsRegistrar.java | 10 +- .../beam/sdk/tpcds/TpcdsParametersReader.java | 136 +- .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 54 +- .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 120 +- .../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++-- ...pcdsOptionsRegistrar.java => package-info.java} | 16 +- .../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 361 +++--- .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 261 ++-- .../beam/sdk/tpcds/TpcdsParametersReaderTest.java | 110 +- .../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 183 ++- 21 files changed, 2158 insertions(+), 1853 deletions(-)
[beam] 02/04: [BEAM-11712] Add options for input/output paths, make it run via SparkRunner
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit a407d79680d01c35760f3fe4e76cd4192e34edd1
Author: Alexey Romanenko
AuthorDate: Tue Mar 30 18:04:22 2021 +0200
[BEAM-11712] Add options for input/output paths, make it run via SparkRunner
---
sdks/java/testing/tpcds/README.md | 68 ++
sdks/java/testing/tpcds/build.gradle | 11 +++-
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 14 ++---
.../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 18 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 49 +---
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 9 +--
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 43 +-
.../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 14 +
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 4 +-
9 files changed, 123 insertions(+), 107 deletions(-)
diff --git a/sdks/java/testing/tpcds/README.md
b/sdks/java/testing/tpcds/README.md
new file mode 100644
index 000..89f8073
--- /dev/null
+++ b/sdks/java/testing/tpcds/README.md
@@ -0,0 +1,68 @@
+
+
+# TPC-DS Benchmark
+
+## Google Dataflow Runner
+
+To execute TPC-DS benchmark for 1Gb dataset on Google Dataflow, run the
following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+ --runner=DataflowRunner \
+ --queries=3,26,55 \
+ --tpcParallel=2 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --project=apache-beam-testing \
+ --stagingLocation=gs://beamsql_tpcds_1/staging \
+ --tempLocation=gs://beamsql_tpcds_2/temp \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --region=us-west1 \
+ --maxNumWorkers=10"
+```
+
+To run a query using ZetaSQL planner (currently Query96 can be run using
ZetaSQL), set the plannerName as below. If not specified, the default planner
is Calcite.
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+ --runner=DataflowRunner \
+ --queries=96 \
+ --tpcParallel=2 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --plannerName=org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner
\
+ --project=apache-beam-testing \
+ --stagingLocation=gs://beamsql_tpcds_1/staging \
+ --tempLocation=gs://beamsql_tpcds_2/temp \
+ --region=us-west1 \
+ --maxNumWorkers=10"
+```
+
+## Spark Runner
+
+To execute TPC-DS benchmark with Query3 for 1Gb dataset on Apache Spark 2.x,
run the following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.runner=":runners:spark:2"
-Ptpcds.args=" \
+ --runner=SparkRunner \
+ --queries=3 \
+ --tpcParallel=1 \
+ --dataDirectory=/path/to/tpcds_data/ \
+ --dataSize=1G \
+ --resultsDirectory=/path/to/tpcds_results/"
+```
diff --git a/sdks/java/testing/tpcds/build.gradle
b/sdks/java/testing/tpcds/build.gradle
index 6237776..79fb1e8 100644
--- a/sdks/java/testing/tpcds/build.gradle
+++ b/sdks/java/testing/tpcds/build.gradle
@@ -33,7 +33,7 @@ def tpcdsArgsProperty = "tpcds.args"
def tpcdsRunnerProperty = "tpcds.runner"
def tpcdsRunnerDependency = project.findProperty(tpcdsRunnerProperty)
?: ":runners:direct-java"
-def shouldProvideSpark = ":runners:spark".equals(tpcdsRunnerDependency)
+def shouldProvideSpark = ":runners:spark:2".equals(tpcdsRunnerDependency)
def isDataflowRunner =
":runners:google-cloud-dataflow-java".equals(tpcdsRunnerDependency)
def runnerConfiguration = ":runners:direct-java".equals(tpcdsRunnerDependency)
? "shadow" : null
@@ -88,6 +88,15 @@ if (shouldProvideSpark) {
}
}
+// Execute the TPC-DS queries or suites via Gradle.
+//
+// Parameters:
+// -Ptpcds.runner
+// Specify a runner subproject, such as ":runners:spark:2" or
":runners:flink:1.10"
+// Defaults to ":runners:direct-java"
+//
+// -Ptpcds.args
+// Specify the command line for invoking
org.apache.beam.sdk.tpcds.BeamTpcds
task run(type: JavaExec) {
def tpcdsArgsStr = project.findProperty(tpcdsArgsProperty) ?: ""
def tpcdsArgsList = new ArrayList()
diff --git
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 43b97d2..69e676f 100644
---
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -17,6 +17,8 @@
*/
package org.apache.beam.sdk.tpcds;
+import static org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull;
+
import com.alibaba.fastjson.JSONObject;
import java.util.ArrayList;
import java.util.Arrays;
@@ -6
[beam] branch master updated (f805f1c -> b3ef203)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from f805f1c Merge pull request #14499 from [BEAM-11408, BEAM-11772] Add
explicit output typehints to ensure coder determinism for BQ with auto-sharding
new 28eec3f [BEAM-11712] Make up-to-date build file and codestyle
new a407d79 [BEAM-11712] Add options for input/output paths, make it run
via SparkRunner
new 8fe0c5c [BEAM-11712] Fix static analysis warnings and typos on TPC-DS
module
new b3ef203 Merge pull request #14373: [BEAM-11712] Run TPC-DS via
BeamSQL and Spark runner
The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
sdks/java/testing/tpcds/README.md | 68 +
sdks/java/testing/tpcds/build.gradle | 108 +-
.../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 327 +++--
.../java/org/apache/beam/sdk/tpcds/BeamTpcds.java | 50 +-
.../java/org/apache/beam/sdk/tpcds/CsvToRow.java | 47 +-
.../org/apache/beam/sdk/tpcds/QueryReader.java | 51 +-
.../java/org/apache/beam/sdk/tpcds/RowToCsv.java | 38 +-
.../apache/beam/sdk/tpcds/SqlTransformRunner.java | 314 +++--
.../apache/beam/sdk/tpcds/SummaryGenerator.java| 219 ++--
.../beam/sdk/tpcds/TableSchemaJSONLoader.java | 162 +--
.../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 40 +-
.../beam/sdk/tpcds/TpcdsOptionsRegistrar.java | 10 +-
.../beam/sdk/tpcds/TpcdsParametersReader.java | 136 +-
.../java/org/apache/beam/sdk/tpcds/TpcdsRun.java | 54 +-
.../org/apache/beam/sdk/tpcds/TpcdsRunResult.java | 120 +-
.../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++--
.../org/apache/beam/sdk/tpcds}/package-info.java |4 +-
.../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 361 +++---
.../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java | 261 ++--
.../beam/sdk/tpcds/TpcdsParametersReaderTest.java | 110 +-
.../apache/beam/sdk/tpcds/TpcdsSchemasTest.java| 183 ++-
21 files changed, 2159 insertions(+), 1840 deletions(-)
create mode 100644 sdks/java/testing/tpcds/README.md
copy sdks/java/testing/{load-tests/src/main/java/org/apache/beam/sdk/loadtests
=> tpcds/src/main/java/org/apache/beam/sdk/tpcds}/package-info.java (92%)
[beam] branch master updated (253bf38 -> b908f59)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 253bf38 Merge pull request #14464 from ibzib/BEAM-12123 add ca05c57 [BEAM-2303] Support SpecificData in AvroCoder add b908f59 Merge pull request #14410: [BEAM-2303] Support SpecificData in AvroCoder No new revisions were added by this update. Summary of changes: .../java/org/apache/beam/sdk/coders/AvroCoder.java | 23 .../org/apache/beam/sdk/coders/AvroCoderTest.java | 43 -- 2 files changed, 55 insertions(+), 11 deletions(-)
[beam] branch master updated (62ada38 -> 752798e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 62ada38 Merge pull request #14506: [BEAM-11903] Bump objenesis to 3.2 add f725953 [BEAM-12151] Bump Apache Parquet to 1.12.0 add 752798e Merge pull request #14509: [BEAM-12151] Bump Apache Parquet to 1.12.0 No new revisions were added by this update. Summary of changes: sdks/java/io/parquet/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (0416017 -> 62ada38)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0416017 Merge pull request #14450: [BEAM-12040, BEAM-11934] Remove the option withRunnerDeterminedShardingUnboundedInternal; add a check for merging windows add 0b35b27 [BEAM-11903] Bump objenesis to 3.2 add 62ada38 Merge pull request #14506: [BEAM-11903] Bump objenesis to 3.2 No new revisions were added by this update. Summary of changes: sdks/java/extensions/kryo/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (c18e3cf -> e398a16)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from c18e3cf Merge pull request #14503 from [BEAM-12143] Fix PubsubReader to populate message id correctly add e398a16 Merge pull request #14472: [BEAM-12148] Align Spark runner jackson dependency version with Beam's No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 ++ runners/spark/spark_runner.gradle | 6 +- 2 files changed, 7 insertions(+), 1 deletion(-)
[beam] branch master updated (9601bde -> e6767c1)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9601bde [BEAM-11227] Try reverting #14295: Moving from vendored gRPC 1.26 to 1.36 (#14466) add 2cca8f1 [BEAM-12092] Bump jedis to 3.5.2 add e6767c1 Merge pull request #14471: [BEAM-12092] Bump jedis to 3.5.2 No new revisions were added by this update. Summary of changes: sdks/java/io/redis/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #14203: [BEAM-11948] Drop support for Flink 1.8 and 1.9
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 572a99bab07e53e043887243e2b1e69120563be5 Merge: cb31b7b 05b3fd3 Author: Ismaël Mejía AuthorDate: Thu Apr 8 06:25:51 2021 +0200 Merge pull request #14203: [BEAM-11948] Drop support for Flink 1.8 and 1.9 CHANGES.md | 1 + gradle.properties | 2 +- runners/flink/1.10/build.gradle| 4 +- .../beam/runners/flink/FlinkCapabilities.java | 34 .../streaming/AbstractStreamOperatorCompat.java| 0 .../beam/runners/flink/RemoteMiniClusterImpl.java | 0 .../runners/flink/SourceTransformationCompat.java | 0 runners/flink/1.11/build.gradle| 4 +- runners/flink/1.12/build.gradle| 4 +- runners/flink/1.8/build.gradle | 34 .../flink/1.8/job-server-container/build.gradle| 26 - runners/flink/1.8/job-server/build.gradle | 31 --- .../beam/runners/flink/FlinkCapabilities.java | 34 .../streaming/io/BeamStoppableFunction.java| 29 -- .../beam/runners/flink/FlinkRunnerTestCompat.java | 43 --- .../runners/flink/streaming/StreamSources.java | 50 - runners/flink/1.9/build.gradle | 33 .../flink/1.9/job-server-container/build.gradle| 26 - runners/flink/1.9/job-server/build.gradle | 31 --- .../runners/flink/streaming/StreamSources.java | 62 -- .../flink/FlinkBatchTransformTranslators.java | 10 +--- .../beam/runners/flink/FlinkPipelineRunner.java| 8 --- .../org/apache/beam/runners/flink/FlinkRunner.java | 7 --- .../translation/functions/FlinkDoFnFunction.java | 12 + .../translation/types/CoderTypeSerializer.java | 0 .../translation/types/EncodedValueSerializer.java | 0 .../streaming/io/BeamStoppableFunction.java| 0 .../flink/batch/NonMergingGroupByKeyTest.java | 5 -- .../translation/types/CoderTypeSerializerTest.java | 0 settings.gradle.kts| 8 --- .../site/content/en/documentation/runners/flink.md | 25 ++--- 31 files changed, 29 insertions(+), 494 deletions(-) diff --cc runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java index 396000b,96b8781..b911567 --- a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java +++ b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java @@@ -43,10 -43,9 +43,9 @@@ import org.apache.beam.sdk.metrics.Metr import org.apache.beam.sdk.metrics.MetricsOptions; import org.apache.beam.sdk.options.PipelineOptions; import org.apache.beam.sdk.options.PipelineOptionsFactory; -import org.apache.beam.vendor.grpc.v1p26p0.com.google.protobuf.Struct; +import org.apache.beam.vendor.grpc.v1p36p0.com.google.protobuf.Struct; import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions; import org.apache.flink.api.common.JobExecutionResult; - import org.apache.flink.runtime.util.EnvironmentInformation; import org.checkerframework.checker.nullness.qual.Nullable; import org.kohsuke.args4j.CmdLineException; import org.kohsuke.args4j.CmdLineParser;
[beam] branch master updated (cb31b7b -> 572a99b)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from cb31b7b [BEAM-7372] cleanup codes for py2 compatibility from
apache_beam/examples/snippets/*.py and apache_beam/examples/*.py (#1)
add 05b3fd3 [BEAM-11948] Drop support for Flink 1.8 and 1.9
new 572a99b Merge pull request #14203: [BEAM-11948] Drop support for
Flink 1.8 and 1.9
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
CHANGES.md | 1 +
gradle.properties | 2 +-
runners/flink/1.10/build.gradle| 4 +-
.../beam/runners/flink/FlinkCapabilities.java | 34
.../streaming/AbstractStreamOperatorCompat.java| 0
.../beam/runners/flink/RemoteMiniClusterImpl.java | 0
.../runners/flink/SourceTransformationCompat.java | 0
runners/flink/1.11/build.gradle| 4 +-
runners/flink/1.12/build.gradle| 4 +-
runners/flink/1.8/build.gradle | 34
.../flink/1.8/job-server-container/build.gradle| 26 -
runners/flink/1.8/job-server/build.gradle | 31 ---
.../beam/runners/flink/FlinkCapabilities.java | 34
.../streaming/io/BeamStoppableFunction.java| 29 --
.../beam/runners/flink/FlinkRunnerTestCompat.java | 43 ---
.../runners/flink/streaming/StreamSources.java | 50 -
runners/flink/1.9/build.gradle | 33
.../flink/1.9/job-server-container/build.gradle| 26 -
runners/flink/1.9/job-server/build.gradle | 31 ---
.../runners/flink/streaming/StreamSources.java | 62 --
.../flink/FlinkBatchTransformTranslators.java | 10 +---
.../beam/runners/flink/FlinkPipelineRunner.java| 8 ---
.../org/apache/beam/runners/flink/FlinkRunner.java | 7 ---
.../translation/functions/FlinkDoFnFunction.java | 12 +
.../translation/types/CoderTypeSerializer.java | 0
.../translation/types/EncodedValueSerializer.java | 0
.../streaming/io/BeamStoppableFunction.java| 0
.../flink/batch/NonMergingGroupByKeyTest.java | 5 --
.../translation/types/CoderTypeSerializerTest.java | 0
settings.gradle.kts| 8 ---
.../site/content/en/documentation/runners/flink.md | 25 ++---
31 files changed, 29 insertions(+), 494 deletions(-)
delete mode 100644
runners/flink/1.10/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
rename runners/flink/{1.8 =>
1.10}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
(100%)
rename runners/flink/{1.8 =>
1.10}/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java
(100%)
rename runners/flink/{1.8 =>
1.10}/src/test/java/org/apache/beam/runners/flink/SourceTransformationCompat.java
(100%)
delete mode 100644 runners/flink/1.8/build.gradle
delete mode 100644 runners/flink/1.8/job-server-container/build.gradle
delete mode 100644 runners/flink/1.8/job-server/build.gradle
delete mode 100644
runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
delete mode 100644
runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
delete mode 100644
runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/FlinkRunnerTestCompat.java
delete mode 100644
runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
delete mode 100644 runners/flink/1.9/build.gradle
delete mode 100644 runners/flink/1.9/job-server-container/build.gradle
delete mode 100644 runners/flink/1.9/job-server/build.gradle
delete mode 100644
runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
rename runners/flink/{1.8 =>
}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
(100%)
rename runners/flink/{1.8 =>
}/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
(100%)
rename runners/flink/{1.9 =>
}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
(100%)
rename runners/flink/{1.8 =>
}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
(100%)
[beam] branch master updated (86ac487 -> 968abf4)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 86ac487 [BEAM-11961] InfluxDBIOIT failing with unauthorized error (#14215) add e3d2654 [BEAM-12066] Bump classgraph to 4.8.104 add 968abf4 Merge pull request #14443: [BEAM-12066] Bump classgraph to 4.8.104 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (3216fcb -> b183b9a)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 3216fcb Merge pull request #14412 from [BEAM-8696] protobuf-java 3.14.0 in line with libraries BOM 16.3.0 add b8ff868 [BEAM-12095] Add unit tests for path_to_beam_jar(artifact_id). add b183b9a Merge pull request #14448: [BEAM-12095] Add unit tests for path_to_beam_jar(artifact_id) No new revisions were added by this update. Summary of changes: .../apache_beam/utils/subprocess_server_test.py| 22 ++ 1 file changed, 22 insertions(+)
[beam] branch master updated (abbe14f -> 7134cfd)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from abbe14f [BEAM-12083] Nexmark Query 13. (#14404)
add 37db190 [BEAM-4106] Add FileStagingOptions and merge staging file
options between runners
add 7134cfd Merge pull request #14423: [BEAM-4106] Add FileStagingOptions
and merge staging file options between runners
No new revisions were added by this update.
Summary of changes:
.../beam/runners/flink/FlinkPipelineOptions.java | 18 ++---
.../options/DataflowPipelineWorkerPoolOptions.java | 19 ++
.../runners/spark/SparkCommonPipelineOptions.java | 18 +++--
.../runners/twister2/Twister2PipelineOptions.java | 11 +++-
.../beam/sdk/options/FileStagingOptions.java} | 30 +++---
.../beam/sdk/options/PortablePipelineOptions.java | 18 +
6 files changed, 26 insertions(+), 88 deletions(-)
copy
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
=> core/src/main/java/org/apache/beam/sdk/options/FileStagingOptions.java}
(58%)
[beam] branch master updated (e5b3333 -> 5f13eff)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e5b Merge pull request #14414 [BEAM-10663] Re-enable CrossLanguageKafkaIOTest add 32af8d2 [BEAM-12088] Make file staging uniform among Spark Runners add 5f13eff Merge pull request #14417: [BEAM-12088] Make file staging uniform among Spark Runners No new revisions were added by this update. Summary of changes: .../SparkStructuredStreamingRunner.java| 7 ++-- .../translation/PipelineTranslator.java| 21 .../runners/spark/SparkCommonPipelineOptions.java | 36 + .../beam/runners/spark/SparkPipelineOptions.java | 37 -- .../beam/runners/spark/SparkPipelineRunner.java| 2 +- .../org/apache/beam/runners/spark/SparkRunner.java | 6 ++-- 6 files changed, 44 insertions(+), 65 deletions(-)
[beam] branch master updated (2c619c8 -> 1e60f38)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2c619c8 [BEAM-7372] delete codes for compatibility of py2 from apache_beam/io except for io/gcp (#14352) add 9e0b378 [BEAM-11213] Display Beam Metrics in Spark History Server for Classic Runner too add 1e60f38 Merge pull request #14409: [BEAM-11213] Display Beam Metrics in Spark History Server for Classic Runner too No new revisions were added by this update. Summary of changes: .../beam/runners/spark/SparkPipelineOptions.java | 12 .../beam/runners/spark/SparkPipelineRunner.java| 39 ++- .../org/apache/beam/runners/spark/SparkRunner.java | 59 +++- .../beam/runners/spark/util/SparkCommon.java | 79 ++ .../beam/runners/spark/util/SparkCompat.java | 6 +- .../spark/metrics/SparkMetricsPusherTest.java | 2 +- 6 files changed, 131 insertions(+), 66 deletions(-) create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCommon.java
[beam] branch master updated (288d379 -> ec9e413)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 288d379 Merge pull request #14386 from iemejia/update-python-deps add 50031c6 Update website build dependencies add ec9e413 Merge pull request #14387: Update website build dependencies No new revisions were added by this update. Summary of changes: website/www/yarn.lock | 355 +++--- 1 file changed, 164 insertions(+), 191 deletions(-)
[beam] branch master updated (fd3075f -> 460e093)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from fd3075f Merge pull request #13995 from [BEAM-11322] Apache Beam Example to tokenize sensitive data add 6eb3c24 Update dependencies add eb8ceeb Update dependencies add 460e093 Merge pull request #14379: Update python dependencies No new revisions were added by this update. Summary of changes: .test-infra/jenkins/metrics_report/requirements.txt | 2 +- sdks/python/container/base_image_requirements.txt | 4 ++-- sdks/python/setup.py| 2 +- 3 files changed, 4 insertions(+), 4 deletions(-)
[beam] branch master updated (a16bbf7 -> 7eff49f)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a16bbf7 Merge pull request #14339: Roll forward PJS with Pubsub fix add 4a1b0aa [BEAM-7093] Add Spark 3 and Java 11 info into the CHANGES file add 7eff49f Merge pull request #14378: [BEAM-7093] Add Spark 3 and Java 11 info into the CHANGES file No new revisions were added by this update. Summary of changes: CHANGES.md | 2 ++ 1 file changed, 2 insertions(+)
[beam] branch master updated (475bbbc -> 776bf54)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 475bbbc Merge pull request #14346: [BEAM-12054] CassandraIO: flush mutator in @FinishBundle add d1f4808 [BEAM-12050] Sickbay portable Flink tests of event time timers with TestStream add 776bf54 Merge pull request #14333: [BEAM-12050] Sickbay portable Flink tests of event time timers with TestStream No new revisions were added by this update. Summary of changes: runners/flink/job-server/flink_job_server.gradle | 4 1 file changed, 4 insertions(+)
[beam] branch master updated (a2da64f -> 75460dc)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from a2da64f [BEAM-11935] Updates Dataflow SDK Harness map to set Environment ID (#14189) add 54c75db [BEAM-11815] AWS SDK v1, DynamoDBIO should read all data with limits add ce214c0 [BEAM-11815] AWS SDK v2, DynamoDBIO reads all data with limits add 75460dc Merge pull request #14314: [BEAM-11815] DynamoDBIO should read all data with limits No new revisions were added by this update. Summary of changes: .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 17 ++--- .../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 34 ++ .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 23 .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 41 ++ 4 files changed, 104 insertions(+), 11 deletions(-)
[beam] branch master updated (8ac1eb5 -> 5bfdc3b)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 8ac1eb5 Merge pull request #14282: [BEAM-7093] Update some out-of-date Gradle Spark instructions. add b661b87 [BEAM-7078] Bump com.amazonaws:amazon-kinesis-client to version 1.14.2 add 5bfdc3b Merge pull request #14283: [BEAM-7078] Bump com.amazonaws:amazon-kinesis-client to version 1.14.2 No new revisions were added by this update. Summary of changes: sdks/java/io/kinesis/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (d6b020a -> 8ac1eb5)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from d6b020a Merge pull request #14279 from apache/tvalentyn-patch-1 add 209fd5d [BEAM-7093] Update some out-of-date Gradle Spark instructions. add 8ac1eb5 Merge pull request #14282: [BEAM-7093] Update some out-of-date Gradle Spark instructions. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/runners/portability/spark_runner.py | 2 +- .../python/apache_beam/runners/portability/spark_uber_jar_job_server.py | 2 +- website/www/site/content/en/get-started/quickstart-go.md| 2 +- 3 files changed, 3 insertions(+), 3 deletions(-)
[beam] 01/01: Merge pull request #14269: [BEAM-11992] Run CrossLanguage ValidatesRunner for Spark 3
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 40eef355c8321bf6f376c68043db4c589bad78d9 Merge: 22a8b18 ba14ca2 Author: Ismaël Mejía AuthorDate: Fri Mar 19 18:42:30 2021 +0100 Merge pull request #14269: [BEAM-11992] Run CrossLanguage ValidatesRunner for Spark 3 ...mmit_CrossLanguageValidatesRunner_Spark3.groovy | 49 ++ README.md | 2 +- release/src/main/scripts/mass_comment.py | 1 + 3 files changed, 51 insertions(+), 1 deletion(-)
[beam] branch master updated (22a8b18 -> 40eef35)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 22a8b18 Add an option to create Dataflow piplines from a snapshot for
python sdk (#14278)
add ba14ca2 [BEAM-11992] Run CrossLanguage ValidatesRunner for Spark 3
new 40eef35 Merge pull request #14269: [BEAM-11992] Run CrossLanguage
ValidatesRunner for Spark 3
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...ob_PostCommit_CrossLanguageValidatesRunner_Spark3.groovy} | 12 ++--
README.md| 2 +-
release/src/main/scripts/mass_comment.py | 1 +
3 files changed, 8 insertions(+), 7 deletions(-)
copy
.test-infra/jenkins/{job_PostCommit_CrossLanguageValidatesRunner_Direct.groovy
=> job_PostCommit_CrossLanguageValidatesRunner_Spark3.groovy} (84%)
[beam] branch master updated (e96f69a -> 5c9c8c0)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from e96f69a Revert "Increase the timeout for ValidatesContainer suite" (#14219) add ff4366d [BEAM-11023] Fix testGroupByKeyWithBadEqualsHashCode failing on Spark Structured Streaming runner add 96bf00a [BEAM-11023] Fix testLargeKeys100MB on Spark Structured Streaming runner add 3ac902a [BEAM-11023] Change access level GroupByKeyTest new 5c9c8c0 Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark Structured Streaming runner The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: runners/spark/spark_runner.gradle | 2 ++ .../src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)
[beam] 01/01: Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark Structured Streaming runner
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 5c9c8c0d1ef0f8224fac77265873d5f6afc431d2 Merge: e96f69a 3ac902a Author: Ismaël Mejía AuthorDate: Thu Mar 18 21:53:03 2021 +0100 Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark Structured Streaming runner runners/spark/spark_runner.gradle | 2 ++ .../src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java | 4 ++-- 2 files changed, 4 insertions(+), 2 deletions(-)
[beam] branch master updated (3fc2ab1 -> aee24ad)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 3fc2ab1 [BEAM-9547] Add NDFrame to doctests, implement a few more
operations (#14236)
add 9113c6f [BEAM-9283] Add Java 11 Jpms compatibility tests for Spark
runner
new aee24ad Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms
compatibility tests for Spark runner
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
...Java11.groovy => job_PostCommit_Java_Jpms_Spark_Java11.groovy} | 8
sdks/java/testing/jpms-tests/build.gradle | 8
2 files changed, 12 insertions(+), 4 deletions(-)
copy .test-infra/jenkins/{job_PostCommit_Java_Jpms_Flink_Java11.groovy =>
job_PostCommit_Java_Jpms_Spark_Java11.groovy} (85%)
[beam] 01/01: Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms compatibility tests for Spark runner
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit aee24ad4aa78d5b1a5258a1003c08714808edcd5 Merge: 3fc2ab1 9113c6f Author: Ismaël Mejía AuthorDate: Thu Mar 18 19:13:05 2021 +0100 Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms compatibility tests for Spark runner .../job_PostCommit_Java_Jpms_Spark_Java11.groovy | 49 ++ sdks/java/testing/jpms-tests/build.gradle | 8 2 files changed, 57 insertions(+)
[beam] branch master updated (540957e -> c4f62ca)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 540957e Merge pull request #14253: [BEAM-11992] Run CrossLanguage
ValidatesRunner tests only for Spark 2
add 6db22c6 [BEAM-8778] Bump software.amazon.awssdk version to 2.15.31
new c4f62ca Merge pull request #14264: [BEAM-8778] Bump
software.amazon.awssdk version to 2.15.31
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 5 -
sdks/java/io/{kinesis => amazon-web-services}/OWNERS | 0
sdks/java/io/{kinesis => amazon-web-services2}/OWNERS | 0
sdks/java/io/amazon-web-services2/build.gradle | 10 --
4 files changed, 8 insertions(+), 7 deletions(-)
copy sdks/java/io/{kinesis => amazon-web-services}/OWNERS (100%)
copy sdks/java/io/{kinesis => amazon-web-services2}/OWNERS (100%)
[beam] 01/01: Merge pull request #14264: [BEAM-8778] Bump software.amazon.awssdk version to 2.15.31
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit c4f62ca933b570ec300f501dc02f360054d5634a Merge: 540957e 6db22c6 Author: Ismaël Mejía AuthorDate: Thu Mar 18 12:40:45 2021 +0100 Merge pull request #14264: [BEAM-8778] Bump software.amazon.awssdk version to 2.15.31 .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 5 - sdks/java/io/amazon-web-services/OWNERS| 4 sdks/java/io/amazon-web-services2/OWNERS | 4 sdks/java/io/amazon-web-services2/build.gradle | 10 -- 4 files changed, 16 insertions(+), 7 deletions(-)
[beam] 01/01: Merge pull request #14253: [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for Spark 2
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 540957effe63ebe093dd1da82ae7f0d8f9aa482a Merge: 9791ef9 abc7b20 Author: Ismaël Mejía AuthorDate: Thu Mar 18 07:12:57 2021 +0100 Merge pull request #14253: [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for Spark 2 .../jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (9791ef9 -> 540957e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 9791ef9 Merge pull request #14237 from [BEAM-11979] Ignore not serializable filter fields in python MongoDBI… add abc7b20 [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for Spark 2 new 540957e Merge pull request #14253: [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for Spark 2 The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy | 1 - 1 file changed, 1 deletion(-)
[beam] branch master updated (25bad0e -> 8d6fa737)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 25bad0e Merge pull request #14246: [BEAM-11726] Bump Clickhouse version to 0.2.6 add 7af3d13 [BEAM-11764] Bump com.amazonaws version to 1.11.974 add 8d6fa737 Merge pull request #14247: [BEAM-11764] Bump com.amazonaws version to 1.11.974 No new revisions were added by this update. Summary of changes: buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated (7ff7ceb -> 25bad0e)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 7ff7ceb Create 2.30.0 entry in CHANGES.md (#14250) add 3d4518f [BEAM-11726] Bump Clickhouse version to "0.2.6" add 25bad0e Merge pull request #14246: [BEAM-11726] Bump Clickhouse version to 0.2.6 No new revisions were added by this update. Summary of changes: sdks/java/io/clickhouse/build.gradle | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] 01/01: Merge pull request #14217: [BEAM-8221] Fix NPE while reading from non-existent Kafka topic
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 08a9d54b60cd4743afeb61e697fa6016279b1ac6 Merge: 88d687b b2ce15e Author: Ismaël Mejía AuthorDate: Mon Mar 15 11:39:55 2021 +0100 Merge pull request #14217: [BEAM-8221] Fix NPE while reading from non-existent Kafka topic .../beam/sdk/io/kafka/KafkaUnboundedSource.java| 7 +- .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 26 ++ 2 files changed, 32 insertions(+), 1 deletion(-)
[beam] branch master updated (88d687b -> 08a9d54)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 88d687b Merge pull request #14228: [BEAM-7092] Fix artifact names for Spark separated modules + upgrade to Spark 3.1.1 add b2ce15e [BEAM-8221] Fix NPE while reading from non-existent Kafka topic new 08a9d54 Merge pull request #14217: [BEAM-8221] Fix NPE while reading from non-existent Kafka topic The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../beam/sdk/io/kafka/KafkaUnboundedSource.java| 7 +- .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 26 ++ 2 files changed, 32 insertions(+), 1 deletion(-)
[beam] branch master updated (153876f -> 88d687b)
This is an automated email from the ASF dual-hosted git repository. iemejia pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 153876f Merge pull request #14216: [BEAM-7093] Support Spark 3 in Spark runner add b388776 [BEAM-7092] Upgrade Spark 3 to version 3.1.1 add f06ffcb [BEAM-7092] Fix artifact name for Spark 2 runner to be the same as before add 1fdf315 [BEAM-7092] Make Spark 3 runner name consistent with other modules '-$VERSION' add 88d687b Merge pull request #14228: [BEAM-7092] Fix artifact names for Spark separated modules + upgrade to Spark 3.1.1 No new revisions were added by this update. Summary of changes: runners/spark/3/build.gradle | 4 ++-- runners/spark/3/job-server/build.gradle | 2 +- runners/spark/job-server/spark_job_server.gradle | 1 + runners/spark/spark_runner.gradle| 2 ++ sdks/python/test-suites/portable/common.gradle | 2 +- 5 files changed, 7 insertions(+), 4 deletions(-)
[beam] branch master updated (c925ff8 -> 153876f)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from c925ff8 Merge pull request #14154: [BEAM-11910] Increase the bag page
limit for continuation pages
add 9c0f608 [BEAM-9282] Move structured streaming runner into Spark 2
specific module
add 47fd176 [BEAM-9282] Separate modules for Spark 2/3
add 17710e5 [BEAM-9282] Separate modules for Spark 2/3 job-server
add e86862c [BEAM-9282] Separate modules for Spark 2/3 job-server
container
add 44b7a87 [BEAM-7092] Run PostCommit tests for Spark 3 module too
add cc29b5b [BEAM-7092] Update tests invocation for Spark 2 module
add b234bb4 [BEAM-9283] Add Spark 3 test jobs to the CI (Java 11)
add 07e52fc [BEAM-11654] Publish Spark 2 and 3 specific Job-Server
containers
add d9f3157 [BEAM-7092] Add paranamer 2.8 license to container (Spark 3 /
Avro)
add 153876f Merge pull request #14216: [BEAM-7093] Support Spark 3 in
Spark runner
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/CommonTestProperties.groovy| 4 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 3 +-
.../job_PostCommit_Java_Nexmark_Spark.groovy | 8 +-
...Java_PortableValidatesRunner_Spark_Batch.groovy | 3 +-
..._PortableValidatesRunner_Spark_Streaming.groovy | 3 +-
...ob_PostCommit_Java_ValidatesRunner_Spark.groovy | 3 +-
...ValidatesRunner_SparkStructuredStreaming.groovy | 2 +-
...ommit_Java_ValidatesRunner_Spark_Java11.groovy} | 13 ++-
build.gradle.kts | 5 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 7 +-
examples/java/build.gradle | 2 +-
examples/kotlin/build.gradle | 2 +-
release/build.gradle.kts | 2 +-
release/src/main/scripts/run_rc_validation.sh | 2 +-
runners/core-construction-java/build.gradle| 3 +
runners/{flink/1.8 => spark/2}/build.gradle| 13 +--
.../{flink/1.9 => spark/2}/job-server/build.gradle | 4 +-
.../2/job-server/container}/build.gradle | 5 +-
.../SparkStructuredStreamingPipelineOptions.java | 0
.../SparkStructuredStreamingPipelineResult.java| 0
.../SparkStructuredStreamingRunner.java| 0
.../SparkStructuredStreamingRunnerRegistrar.java | 0
.../aggregators/AggregatorsAccumulator.java| 0
.../aggregators/NamedAggregators.java | 0
.../aggregators/NamedAggregatorsAccumulator.java | 0
.../aggregators/package-info.java | 0
.../structuredstreaming/examples/WordCount.java| 0
.../metrics/AggregatorMetric.java | 0
.../metrics/AggregatorMetricSource.java| 0
.../metrics/CompositeSource.java | 0
.../metrics/MetricsAccumulator.java| 0
.../MetricsContainerStepMapAccumulator.java| 0
.../metrics/SparkBeamMetric.java | 0
.../metrics/SparkBeamMetricSource.java | 0
.../metrics/SparkMetricsContainerStepMap.java | 0
.../metrics/WithMetricsSupport.java| 0
.../structuredstreaming/metrics/package-info.java | 0
.../metrics/sink/CodahaleCsvSink.java | 0
.../metrics/sink/CodahaleGraphiteSink.java | 0
.../metrics/sink/package-info.java | 0
.../spark/structuredstreaming/package-info.java| 0
.../translation/PipelineTranslator.java| 0
.../translation/SchemaHelpers.java | 0
.../translation/SparkTransformOverrides.java | 0
.../translation/TransformTranslator.java | 0
.../translation/TranslationContext.java| 0
.../translation/batch/AggregatorCombiner.java | 0
.../batch/CombinePerKeyTranslatorBatch.java| 0
.../CreatePCollectionViewTranslatorBatch.java | 0
.../translation/batch/DatasetSourceBatch.java | 0
.../translation/batch/DoFnFunction.java| 0
.../translation/batch/DoFnRunnerWithMetrics.java | 0
.../translation/batch/FlattenTranslatorBatch.java | 0
.../batch/GroupByKeyTranslatorBatch.java | 0
.../translation/batch/ImpulseTranslatorBatch.java | 0
.../translation/batch/ParDoTranslatorBatch.java| 0
.../translation/batch/PipelineTranslatorBatch.java | 0
.../translation/batch/ProcessContext.java | 0
.../batch/ReadSourceTranslatorBatch.java | 0
.../batch/ReshuffleTranslatorBatch.java| 0
.../batch/WindowAssignTranslatorBatch.java | 0
.../GroupAlsoByWindowViaOutputBufferFn.java| 0
.../batch/functions/NoOpStepContext.java | 0
.../batch/functions/SparkSideInputReader.java | 0
.../translation/batch/functions/package-info.java | 0
.../translation/batch/package-info.java| 0
.../translation/helpers/CoderHelper
