[beam] branch master updated (762edd7f3a6 -> 483a0c95734)

2022-09-22 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


from 762edd7f3a6 Improved pipeline translation in 
SparkStructuredStreamingRunner (#22446)
 add a6cda1370b3 use avro DataFileReader to read avro container files
 new 483a0c95734 Merge pull request #23214: Use avro DataFileReader to read 
avro container files

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../java/org/apache/beam/sdk/io/AvroSource.java| 343 ++---
 .../org/apache/beam/sdk/io/AvroSourceTest.java | 166 --
 2 files changed, 86 insertions(+), 423 deletions(-)



[beam] 01/01: Merge pull request #23214: Use avro DataFileReader to read avro container files

2022-09-22 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 483a0c95734c528aa45419596e9f27e9e650c5d7
Merge: 762edd7f3a6 a6cda1370b3
Author: Ismaël Mejía 
AuthorDate: Thu Sep 22 18:58:10 2022 +0200

Merge pull request #23214: Use avro DataFileReader to read avro container 
files

 .../java/org/apache/beam/sdk/io/AvroSource.java| 343 ++---
 .../org/apache/beam/sdk/io/AvroSourceTest.java | 166 --
 2 files changed, 86 insertions(+), 423 deletions(-)



[beam] branch master updated: Fix #22466 Add github actions dependency updates with dependabot

2022-08-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 3031a3d2aca Fix #22466 Add github actions dependency updates with 
dependabot
3031a3d2aca is described below

commit 3031a3d2aca8e81b219364ca43cbf811abd68445
Author: Ismaël Mejía 
AuthorDate: Wed Aug 10 22:02:55 2022 +0200

Fix #22466 Add github actions dependency updates with dependabot
---
 .github/dependabot.yml | 8 
 1 file changed, 8 insertions(+)

diff --git a/.github/dependabot.yml b/.github/dependabot.yml
index 334414df9db..248e8d6a69b 100644
--- a/.github/dependabot.yml
+++ b/.github/dependabot.yml
@@ -42,3 +42,11 @@ updates:
 - dependency-name: "com.google.api.grpc:grpc-*"
 - dependency-name: "com.google.http-client:*"
 - dependency-name: "com.google.apis:google-api-services-*"
+  - package-ecosystem: "github-actions"
+directory: "/"
+schedule:
+  interval: "daily"
+allow:
+  # Allow only automatic updates for official github actions
+  # Other github-actions require approval from INFRA
+  - dependency-name: "actions/*"



[beam] branch master updated (d352d60 -> 0a68801)

2022-03-21 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d352d60  [BEAM-14071] Enabling Flink on Dataproc for Interactive Beam 
(#17044)
 add 7257e37  [BEAM-4106] Remove filesToStage from Flink pipeline option 
list.
 add 0a68801  Merge pull request #17143: [BEAM-4106] Remove filesToStage 
from Flink pipeline option list.

No new revisions were added by this update.

Summary of changes:
 website/www/site/layouts/shortcodes/flink_java_pipeline_options.html | 5 -
 .../www/site/layouts/shortcodes/flink_python_pipeline_options.html   | 5 -
 2 files changed, 10 deletions(-)


[beam] branch master updated (9e0aa6b -> eea81f4)

2022-03-11 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9e0aa6b  Merge pull request #17056 from [BEAM-14076] [SnowflakeIO] Add 
support for GEOGRAPHY column
 add 1aba87d  [BEAM-13981] Remove Spark Runner specific code for event 
logging
 add eea81f4  Merge pull request #17073: [BEAM-13981] Remove Spark Runner 
specific code for event logging

No new revisions were added by this update.

Summary of changes:
 .../beam/runners/spark/SparkPipelineRunner.java| 17 -
 .../org/apache/beam/runners/spark/SparkRunner.java | 22 +-
 .../runners/spark/metrics/SparkBeamMetric.java | 10 ---
 .../beam/runners/spark/util/SparkCommon.java   | 79 --
 .../beam/runners/spark/util/SparkCompat.java   | 61 -
 5 files changed, 1 insertion(+), 188 deletions(-)
 delete mode 100644 
runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCommon.java


[beam] branch master updated (3cd1f7f -> 75c25f0)

2022-03-02 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3cd1f7f  [BEAM-13960] Add support for more types when converting from 
between row and proto (#16875)
 add fa32292  Bump org.mongodb:mongo-java-driver to 3.12.10
 add 75c25f0  Merge pull request #16989: [BEAM-5577] Bump 
org.mongodb:mongo-java-driver to 3.12.10

No new revisions were added by this update.

Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (b2f2128 -> 6e98dd4)

2022-02-16 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b2f2128  Merge pull request #16865: Create test category for 
UsesProcessingTimeTimers are exclude from Samza
 add add7bbc  [BEAM-13202] Fix typos on tests names for VarianceFnTest
 add 1105c34  [BEAM-13202] Add Coder to CountIfFn.Accum
 add cf75357  [BEAM-13202] Reuse Count transform code since CountIf is a 
specific case
 add 6e98dd4  Merge pull request #16856: [BEAM-13202] Add Coder to 
CountIfFn.Accum

No new revisions were added by this update.

Summary of changes:
 .../extensions/sql/impl/transform/agg/CountIf.java | 53 +++
 .../sql/impl/transform/agg/CountIfTest.java| 78 ++
 .../sql/impl/transform/agg/VarianceFnTest.java |  4 +-
 3 files changed, 104 insertions(+), 31 deletions(-)
 create mode 100644 
sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CountIfTest.java


[beam] branch master updated (7fa5387 -> c73066c)

2022-02-12 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7fa5387  Regenerate python container base_image_requirements.txt 
(#16832)
 add ed693e5  [BEAM-9195] Bump org.testcontainers to 1.16.3
 add c73066c  Merge pull request #16661: [BEAM-9195] Bump 
org.testcontainers to 1.16.3

No new revisions were added by this update.

Summary of changes:
 .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy  | 2 +-
 sdks/java/io/debezium/build.gradle  | 6 +++---
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/build.gradle | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-6/build.gradle | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-7/build.gradle | 2 +-
 .../io/elasticsearch-tests/elasticsearch-tests-common/build.gradle  | 2 +-
 6 files changed, 8 insertions(+), 8 deletions(-)


[beam] branch master updated (5beae2a -> 9434c4d)

2022-02-03 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5beae2a  [BEAM-13605] Add support for pandas 1.4.0 (#16590)
 add 99b903f   Change links to Books from Amazon to Publisher
 add 9434c4d  Merge pull request #16718: [website] Change links to Books 
from Amazon to Publisher

No new revisions were added by this update.

Summary of changes:
 .../www/site/content/en/documentation/resources/learning-resources.md | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


[beam] branch master updated (cbbebcd -> 50d8a85)

2021-08-27 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cbbebcd  Merge pull request #15335 from [BEAM-12751] Set 
clientRequestId for Dataflow python job creation
 add 41d515d  [BEAM-12270] TPC-DS: Add schema projection for Parquet source
 add 50d8a85  Merge pull request #15361: [BEAM-12270] TPC-DS: Add schema 
projection for Parquet source

No new revisions were added by this update.

Summary of changes:
 .../org/apache/beam/sdk/tpcds/QueryReader.java | 20 ++
 .../apache/beam/sdk/tpcds/SqlTransformRunner.java  | 72 +++---
 .../org/apache/beam/sdk/tpcds/QueryReaderTest.java | 28 +
 3 files changed, 111 insertions(+), 9 deletions(-)


[beam] branch master updated (98747fd -> 3537f7e)

2021-08-06 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 98747fd  Bump Python FnAPI beam-master container #15283
 add e5c25a0  [BEAM-12628] Add Avro reflect-based Coder option
 add 3537f7e  Merge pull request #15292: [BEAM-12628] Add Avro 
reflect-based Coder option

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/beam/sdk/coders/AvroCoder.java | 32 +++---
 .../org/apache/beam/sdk/coders/AvroCoderTest.java  | 21 ++
 2 files changed, 49 insertions(+), 4 deletions(-)


[beam] branch master updated (f9a4bfc -> 68c56f8)

2021-06-30 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f9a4bfc  [BEAM-11289] [Python] Integrate Google Cloud Recommendations 
AI functionality (#14806)
 add 5690af7  [BEAM-12422] Removing unnecessary log4j-api dependency
 add 68c56f8  Merge pull request #15098: [BEAM-12422] Vendored gRPC: 
Removing unnecessary log4j-api dependency

No new revisions were added by this update.

Summary of changes:
 .../src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_36_0.groovy | 3 ---
 1 file changed, 3 deletions(-)


[beam] branch master updated (225fcd0 -> 90c854e)

2021-06-27 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 225fcd0  [BEAM-9547] Add support for xs on DataFrame and Series 
(#15078)
 add 37e3fdf  [BEAM-12094] Add Spark 3 to Python.
 add 90c854e  Merge pull request #15063: [BEAM-12094] Add Spark 3 to Python 
(spark_runner.py)

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/options/pipeline_options.py  |  8 +++-
 .../apache_beam/runners/portability/spark_runner.py  |  3 +++
 .../runners/portability/spark_uber_jar_job_server.py | 16 ++--
 3 files changed, 20 insertions(+), 7 deletions(-)


[beam] branch master updated (67164a1 -> 815e2c9)

2021-06-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 67164a1  Merge pull request #14964: [BEAM-12460] Provide a simpler 
interface to convert Beam Row to GenericRecord
 add 525d0f8  [BEAM-12471] Fixes NumberFormatException
 new 815e2c9  Merge pull request #14982: [BEAM-12471] Fixes 
NumberFormatException

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java  | 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)


[beam] 01/01: Merge pull request #14982: [BEAM-12471] Fixes NumberFormatException

2021-06-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 815e2c94bee24ef0c15c78876d6266618c238ba5
Merge: 67164a1 525d0f8
Author: Ismaël Mejía 
AuthorDate: Thu Jun 10 15:43:16 2021 +0200

Merge pull request #14982: [BEAM-12471] Fixes NumberFormatException

 .../beam/sdk/extensions/sql/impl/schema/BeamTableUtils.java  | 9 +++--
 1 file changed, 3 insertions(+), 6 deletions(-)


[beam] 01/01: Merge pull request #14964: [BEAM-12460] Provide a simpler interface to convert Beam Row to GenericRecord

2021-06-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 67164a1d9a3715528e03507c792d1c89c4d5efac
Merge: 9cd7479 c13a0cc
Author: Ismaël Mejía 
AuthorDate: Thu Jun 10 15:03:46 2021 +0200

Merge pull request #14964: [BEAM-12460] Provide a simpler interface to 
convert Beam Row to GenericRecord

 .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java| 8 
 .../java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java | 7 +++
 2 files changed, 15 insertions(+)


[beam] branch master updated (9cd7479 -> 67164a1)

2021-06-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9cd7479  Merge pull request #14888: [BEAM-12411] Update Tensorflow to 
version 2.5.0 & grpcio to 1.34.0
 add c13a0cc  [BEAM-12460] Provide a simpler interface to convert Beam Row 
to GenericRecord.
 new 67164a1  Merge pull request #14964: [BEAM-12460] Provide a simpler 
interface to convert Beam Row to GenericRecord

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java| 8 
 .../java/org/apache/beam/sdk/schemas/utils/AvroUtilsTest.java | 7 +++
 2 files changed, 15 insertions(+)


[beam] branch master updated (0b369a5 -> 9cd7479)

2021-06-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0b369a5  [BEAM-10982] Add linux aarch64 wheel build support #14832
 add ce0040d  Update tensorflow to the latest version
 add a12bf7a  Update grpcio
 add 81808d1  Add license info for keras-nightly package.
 add 9cd7479  Merge pull request #14888: [BEAM-12411] Update Tensorflow to 
version 2.5.0 & grpcio to 1.34.0

No new revisions were added by this update.

Summary of changes:
 sdks/python/container/base_image_requirements.txt  | 4 ++--
 sdks/python/container/license_scripts/dep_urls_py.yaml | 2 ++
 2 files changed, 4 insertions(+), 2 deletions(-)


[beam] branch master updated: [BEAM-12385] Handle VARCHAR and other SQL specific logical types in AvroUtils

2021-06-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 88d4712  [BEAM-12385] Handle VARCHAR and other SQL specific logical 
types in AvroUtils
 new 62e8f84  Merge pull request #14858: [BEAM-12385] Handle VARCHAR and 
other SQL specific logical types in AvroUtils
88d4712 is described below

commit 88d4712147911744761cb385b8226c81e283d1fe
Author: Anant Damle 
AuthorDate: Fri May 21 23:47:53 2021 +0800

[BEAM-12385] Handle VARCHAR and other SQL specific logical types in 
AvroUtils
---
 CHANGES.md |   3 +
 .../apache/beam/sdk/schemas/utils/AvroUtils.java   |  43 
 .../beam/sdk/schemas/utils/AvroUtilsTest.java  | 244 +
 .../apache/beam/sdk/io/jdbc/SchemaUtilTest.java|  60 +
 4 files changed, 350 insertions(+)

diff --git a/CHANGES.md b/CHANGES.md
index 7029cd2..62b8e71 100644
--- a/CHANGES.md
+++ b/CHANGES.md
@@ -66,6 +66,9 @@
 * `CREATE FUNCTION` DDL statement added to Calcite SQL syntax. `JAR` and 
`AGGREGATE` are now reserved keywords. 
([BEAM-12339](https://issues.apache.org/jira/browse/BEAM-12339)).
 * Flink 1.13 is now supported by the Flink runner 
([BEAM-12277](https://issues.apache.org/jira/browse/BEAM-12277)).
 * X feature added (Java/Python) 
([BEAM-X](https://issues.apache.org/jira/browse/BEAM-X)).
+* Add support to convert Beam Schema to Avro Schema for JDBC LogicalTypes:
+  `VARCHAR`, `NVARCHAR`, `LONGVARCHAR`, `LONGNVARCHAR`, `DATE`, `TIME`
+  (Java)([BEAM-12385](https://issues.apache.org/jira/browse/BEAM-12385)).
 
 ## Breaking Changes
 
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
index 77b5445..0835f9b 100644
--- 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
+++ 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java
@@ -906,6 +906,26 @@ public class AvroUtils {
 .map(x -> getFieldSchema(x.getType(), x.getName(), 
namespace))
 .collect(Collectors.toList()));
 break;
+  case "CHAR":
+  case "NCHAR":
+baseType =
+buildHiveLogicalTypeSchema("char", (int) 
fieldType.getLogicalType().getArgument());
+break;
+  case "NVARCHAR":
+  case "VARCHAR":
+  case "LONGNVARCHAR":
+  case "LONGVARCHAR":
+baseType =
+buildHiveLogicalTypeSchema(
+"varchar", (int) fieldType.getLogicalType().getArgument());
+break;
+  case "DATE":
+baseType = 
LogicalTypes.date().addToSchema(org.apache.avro.Schema.create(Type.INT));
+break;
+  case "TIME":
+baseType =
+
LogicalTypes.timeMillis().addToSchema(org.apache.avro.Schema.create(Type.INT));
+break;
   default:
 throw new RuntimeException(
 "Unhandled logical type " + 
fieldType.getLogicalType().getIdentifier());
@@ -1017,6 +1037,15 @@ public class AvroUtils {
   
typeWithNullability.type.getTypes().get(oneOfValue.getCaseType().getValue()),
   oneOfValue.getValue());
 }
+  case "NVARCHAR":
+  case "VARCHAR":
+  case "LONGNVARCHAR":
+  case "LONGVARCHAR":
+return new Utf8((String) value);
+  case "DATE":
+return Days.daysBetween(Instant.EPOCH, (Instant) value).getDays();
+  case "TIME":
+return (int) ((Instant) value).getMillis();
   default:
 throw new RuntimeException(
 "Unhandled logical type " + 
fieldType.getLogicalType().getIdentifier());
@@ -1277,4 +1306,18 @@ public class AvroUtils {
 checkArgument(
 got.equals(expected), "Can't convert '%s' to %s, expected: %s", label, 
got, expected);
   }
+
+  /**
+   * Helper factory to build Avro Logical types schemas for SQL *CHAR types. 
This method https://github.com/apache/hive/blob/5d268834a5f5278ea76399f8af0d0ab043ae0b45/serde/src/java/org/apache/hadoop/hive/serde2/avro/TypeInfoToSchema.java#L110-L121";>represents
+   * the logical as Hive does.
+   */
+  private static org.apache.avro.Schema buildHiveLogicalTypeSchema(
+  String hiveLogicalType, int size) {
+String schemaJson =
+String.format(
+"{\"type\": \"string\", \"logicalType\": \"%s\", \"maxLength\": 
%s}"

[beam] 01/01: Merge pull request #14906: [BEAM-12281] Drop support for Flink 1.10

2021-06-02 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 511fb05e5925fe48947efdd872433d621c85e52d
Merge: 4f0ea5f 22575c7
Author: Ismaël Mejía 
AuthorDate: Wed Jun 2 11:08:53 2021 +0200

Merge pull request #14906: [BEAM-12281] Drop support for Flink 1.10

 .test-infra/dataproc/flink_cluster.sh  |  2 +-
 .test-infra/jenkins/Flink.groovy   |  4 +-
 .../jenkins/job_LoadTests_Combine_Flink_Go.groovy  |  2 +-
 .../job_LoadTests_Combine_Flink_Python.groovy  |  2 +-
 .../jenkins/job_LoadTests_GBK_Flink_Go.groovy  |  2 +-
 .../jenkins/job_LoadTests_GBK_Flink_Python.groovy  |  2 +-
 .../jenkins/job_LoadTests_ParDo_Flink_Go.groovy|  2 +-
 .../job_LoadTests_ParDo_Flink_Python.groovy|  2 +-
 .../job_LoadTests_SideInput_Flink_Go.groovy|  2 +-
 .../jenkins/job_LoadTests_coGBK_Flink_Go.groovy|  2 +-
 .../job_LoadTests_coGBK_Flink_Python.groovy|  2 +-
 ...Commit_Python_Chicago_Taxi_Example_Flink.groovy |  2 +-
 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 release/build.gradle.kts   |  2 +-
 release/src/main/scripts/run_rc_validation.sh  |  2 +-
 runners/flink/1.10/build.gradle| 33 --
 .../flink/1.10/job-server-container/build.gradle   | 26 
 runners/flink/1.10/job-server/build.gradle | 31 --
 .../streaming/AbstractStreamOperatorCompat.java| 30 -
 .../beam/runners/flink/FlinkRunnerTestCompat.java  | 42 -
 .../beam/runners/flink/RemoteMiniClusterImpl.java  | 71 --
 .../runners/flink/SourceTransformationCompat.java  | 28 -
 runners/flink/1.11/build.gradle|  4 +-
 .../runners/flink/streaming/StreamSources.java |  0
 runners/flink/1.12/build.gradle|  4 +-
 runners/flink/1.13/build.gradle|  4 +-
 runners/flink/flink_runner.gradle  |  2 +-
 .../beam/runners/flink/FlinkPipelineRunner.java|  7 +--
 .../org/apache/beam/runners/flink/FlinkRunner.java |  7 +--
 .../flink/FlinkExecutionEnvironmentsTest.java  | 34 +++
 .../FlinkPipelineExecutionEnvironmentTest.java | 22 +++
 sdks/go/examples/stringsplit/stringsplit.go|  2 +-
 sdks/java/testing/nexmark/build.gradle |  2 +-
 sdks/java/testing/tpcds/build.gradle   |  2 +-
 sdks/python/apache_beam/transforms/sql_test.py |  4 +-
 settings.gradle.kts|  4 --
 .../site/content/en/contribute/release-guide.md|  2 +-
 .../content/en/documentation/dsls/sql/shell.md |  6 +-
 .../site/content/en/documentation/runners/flink.md |  6 +-
 .../en/documentation/sdks/java/testing/nexmark.md  |  6 +-
 41 files changed, 57 insertions(+), 355 deletions(-)

diff --cc runners/flink/1.12/build.gradle
index 1096d80,d38d873..99dd015
--- a/runners/flink/1.12/build.gradle
+++ b/runners/flink/1.12/build.gradle
@@@ -20,10 -20,10 +20,10 @@@ def basePath = '..
  /* All properties required for loading the Flink build script */
  project.ext {
// Set the version of all Flink-related dependencies here.
 -  flink_version = '1.12.3'
 +  flink_version = '1.12.4'
// Version specific code overrides.
-   main_source_overrides = ["${basePath}/1.10/src/main/java", 
"${basePath}/1.11/src/main/java", './src/main/java']
-   test_source_overrides = ["${basePath}/1.10/src/test/java", 
"${basePath}/1.11/src/test/java", './src/test/java']
+   main_source_overrides = ["${basePath}/1.11/src/main/java", 
'./src/main/java']
+   test_source_overrides = ["${basePath}/1.11/src/test/java", 
'./src/test/java']
main_resources_overrides = []
test_resources_overrides = []
archives_base_name = 'beam-runners-flink-1.12'
diff --cc runners/flink/1.13/build.gradle
index 1799ffc,0484c29..067933f
--- a/runners/flink/1.13/build.gradle
+++ b/runners/flink/1.13/build.gradle
@@@ -20,10 -20,10 +20,10 @@@ def basePath = '..
  /* All properties required for loading the Flink build script */
  project.ext {
// Set the version of all Flink-related dependencies here.
 -  flink_version = '1.13.0'
 +  flink_version = '1.13.1'
// Version specific code overrides.
-   main_source_overrides = ["${basePath}/1.10/src/main/java", 
"${basePath}/1.11/src/main/java", "${basePath}/1.12/src/main/java", 
'./src/main/java']
-   test_source_overrides = ["${basePath}/1.10/src/test/java", 
"${basePath}/1.11/src/test/java", "${basePath}/1.12/src/test/java", 
'./src/test/java']
+   main_source_overrides = ["${basePath}/1.11/src/

[beam] branch master updated (4f0ea5f -> 511fb05)

2021-06-02 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4f0ea5f  Merge pull request #14911 from 
TheNeuralBit/dataframe-changes-230
 add bed7982  [BEAM-12281] Drop support for Flink 1.10
 add 22575c7  [BEAM-12281] Update Flink Jenkins jobs to use Flink 1.12
 new 511fb05  Merge pull request #14906: [BEAM-12281] Drop support for 
Flink 1.10

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/dataproc/flink_cluster.sh  |  2 +-
 .test-infra/jenkins/Flink.groovy   |  4 +-
 .../jenkins/job_LoadTests_Combine_Flink_Go.groovy  |  2 +-
 .../job_LoadTests_Combine_Flink_Python.groovy  |  2 +-
 .../jenkins/job_LoadTests_GBK_Flink_Go.groovy  |  2 +-
 .../jenkins/job_LoadTests_GBK_Flink_Python.groovy  |  2 +-
 .../jenkins/job_LoadTests_ParDo_Flink_Go.groovy|  2 +-
 .../job_LoadTests_ParDo_Flink_Python.groovy|  2 +-
 .../job_LoadTests_SideInput_Flink_Go.groovy|  2 +-
 .../jenkins/job_LoadTests_coGBK_Flink_Go.groovy|  2 +-
 .../job_LoadTests_coGBK_Flink_Python.groovy|  2 +-
 ...Commit_Python_Chicago_Taxi_Example_Flink.groovy |  2 +-
 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 release/build.gradle.kts   |  2 +-
 release/src/main/scripts/run_rc_validation.sh  |  2 +-
 runners/flink/1.10/build.gradle| 33 --
 .../flink/1.10/job-server-container/build.gradle   | 26 
 runners/flink/1.10/job-server/build.gradle | 31 --
 .../streaming/AbstractStreamOperatorCompat.java| 30 -
 .../beam/runners/flink/FlinkRunnerTestCompat.java  | 42 -
 .../beam/runners/flink/RemoteMiniClusterImpl.java  | 71 --
 .../runners/flink/SourceTransformationCompat.java  | 28 -
 runners/flink/1.11/build.gradle|  4 +-
 .../runners/flink/streaming/StreamSources.java |  0
 runners/flink/1.12/build.gradle|  4 +-
 runners/flink/1.13/build.gradle|  4 +-
 runners/flink/flink_runner.gradle  |  2 +-
 .../beam/runners/flink/FlinkPipelineRunner.java|  7 +--
 .../org/apache/beam/runners/flink/FlinkRunner.java |  7 +--
 .../flink/FlinkExecutionEnvironmentsTest.java  | 34 +++
 .../FlinkPipelineExecutionEnvironmentTest.java | 22 +++
 sdks/go/examples/stringsplit/stringsplit.go|  2 +-
 sdks/java/testing/nexmark/build.gradle |  2 +-
 sdks/java/testing/tpcds/build.gradle   |  2 +-
 sdks/python/apache_beam/transforms/sql_test.py |  4 +-
 settings.gradle.kts|  4 --
 .../site/content/en/contribute/release-guide.md|  2 +-
 .../content/en/documentation/dsls/sql/shell.md |  6 +-
 .../site/content/en/documentation/runners/flink.md |  6 +-
 .../en/documentation/sdks/java/testing/nexmark.md  |  6 +-
 41 files changed, 57 insertions(+), 355 deletions(-)
 delete mode 100644 runners/flink/1.10/build.gradle
 delete mode 100644 runners/flink/1.10/job-server-container/build.gradle
 delete mode 100644 runners/flink/1.10/job-server/build.gradle
 delete mode 100644 
runners/flink/1.10/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
 delete mode 100644 
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/FlinkRunnerTestCompat.java
 delete mode 100644 
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java
 delete mode 100644 
runners/flink/1.10/src/test/java/org/apache/beam/runners/flink/SourceTransformationCompat.java
 rename runners/flink/{1.10 => 
1.11}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java 
(100%)


[beam] branch master updated (b39ff90 -> ae82018)

2021-06-01 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b39ff90  Merge pull request #14899 from y1chi/prebuild_timeout
 add 10b6f34  [BEAM-12427] Ignore the AutoValue_* classes generated in 
"generated" folder from build process. Appears for sdks>java>io module.
 new ae82018  Merge pull request #14913: [BEAM-12427] Ignore the 
AutoValue_* classes generated in "generated" in sdks>java>io

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .gitignore | 1 +
 1 file changed, 1 insertion(+)


[beam] 01/01: Merge pull request #14913: [BEAM-12427] Ignore the AutoValue_* classes generated in "generated" in sdks>java>io

2021-06-01 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit ae820187ad5f8fc4c27504aa708068476e299b8f
Merge: b39ff90 10b6f34
Author: Ismaël Mejía 
AuthorDate: Tue Jun 1 21:35:29 2021 +0200

Merge pull request #14913: [BEAM-12427] Ignore the AutoValue_* classes 
generated in "generated" in sdks>java>io

 .gitignore | 1 +
 1 file changed, 1 insertion(+)


[beam] branch master updated (441dc8b -> 3876ddb)

2021-05-31 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 441dc8b  Merge pull request #14905: [BEAM-12277] Update Flink 1.13 
version to 1.13.1
 add 780e9ca  [BEAM-12424] Update Flink 1.12 to version 1.12.4
 add 3876ddb  Merge pull request #14907: [BEAM-12424] Update Flink 1.12 to 
version 1.12.4

No new revisions were added by this update.

Summary of changes:
 runners/flink/1.12/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (d8c5da8 -> 441dc8b)

2021-05-31 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d8c5da8  [BEAM-11608] Development environment set up automation 
(#14584)
 add 4031432  [BEAM-12277] Update Flink 1.13 version to 1.13.1
 add 441dc8b  Merge pull request #14905: [BEAM-12277] Update Flink 1.13 
version to 1.13.1

No new revisions were added by this update.

Summary of changes:
 runners/flink/1.13/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] 01/01: Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build target.

2021-05-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 00eb4207506720e7367098ad904bb37ba6de88a9
Merge: 2516cac 881da20
Author: Ismaël Mejía 
AuthorDate: Fri May 28 15:38:17 2021 +0200

Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build target.

 .test-infra/jenkins/CommonTestProperties.groovy|  2 +-
 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 runners/flink/1.13/build.gradle| 33 +++
 .../flink/1.13/job-server-container/build.gradle   | 26 
 runners/flink/1.13/job-server/build.gradle | 31 ++
 .../runners/flink/streaming/StreamSources.java | 69 ++
 .../beam/runners/flink/metrics/FileReporter.java   |  8 ++-
 .../python/apache_beam/options/pipeline_options.py |  2 +-
 .../runners/portability/flink_runner_test.py   | 49 +++
 settings.gradle.kts|  4 ++
 .../site/content/en/documentation/runners/flink.md | 24 +++-
 12 files changed, 220 insertions(+), 31 deletions(-)


[beam] branch master updated (2516cac -> 00eb420)

2021-05-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2516cac  Merge pull request #14897: [BEAM-12415] Update Spark 3 
version to 3.1.2
 add b342805  [BEAM-12277] Add flink 1.13 build target.
 add 8ad8965  [BEAM-12277] Make Flink 1.13 the default Flink version to run 
the tests
 add 8be1cb1  [BEAM-12277] Update expected metric name formatting.
 add 881da20  [BEAM-12277] fix mistake in source overrides
 new 00eb420  Merge pull request #14719: [BEAM-12277] Add Flink 1.13 build 
target.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .test-infra/jenkins/CommonTestProperties.groovy|  2 +-
 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 runners/flink/{1.12 => 1.13}/build.gradle  |  8 ++--
 .../job-server-container/build.gradle  |  0
 .../flink/{1.10 => 1.13}/job-server/build.gradle   |  2 +-
 .../runners/flink/streaming/StreamSources.java |  0
 .../beam/runners/flink/metrics/FileReporter.java   |  8 +++-
 .../python/apache_beam/options/pipeline_options.py |  2 +-
 .../runners/portability/flink_runner_test.py   | 49 +++---
 settings.gradle.kts|  4 ++
 .../site/content/en/documentation/runners/flink.md | 24 +--
 12 files changed, 66 insertions(+), 36 deletions(-)
 copy runners/flink/{1.12 => 1.13}/build.gradle (84%)
 copy runners/flink/{1.12 => 1.13}/job-server-container/build.gradle (100%)
 copy runners/flink/{1.10 => 1.13}/job-server/build.gradle (95%)
 copy runners/flink/{1.12 => 
1.13}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java 
(100%)


[beam] branch master updated (b03e429 -> 2516cac)

2021-05-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b03e429  Merge pull request #14854: [BEAM-12384] Set output 
typeDescriptor explictly in Read.Bounded transform
 add b6fca2b  [BEAM-12415] Update Spark 3 version to 3.1.2
 add 2516cac  Merge pull request #14897: [BEAM-12415] Update Spark 3 
version to 3.1.2

No new revisions were added by this update.

Summary of changes:
 runners/spark/3/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (31988c8 -> b03e429)

2021-05-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 31988c8  Merge pull request #14898 from boyuanzz/fix
 add 331c67c  [BEAM-12384] Refine generic types on Read.Bounded internals
 add 3b705c1  [BEAM-12384] Set output typeDescriptor explictly in 
Read.Bounded transform
 add 83bccf9  [BEAM-12384] Infer typeDescriptor from coder if 
typeDescriptor is not set explicitly.
 add b03e429  Merge pull request #14854: [BEAM-12384] Set output 
typeDescriptor explictly in Read.Bounded transform

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/beam/sdk/io/Read.java | 56 --
 .../org/apache/beam/sdk/values/PCollection.java|  8 +++-
 .../test/java/org/apache/beam/sdk/io/ReadTest.java | 43 +
 3 files changed, 80 insertions(+), 27 deletions(-)


[beam] branch master updated (1c63759 -> 2abbd37)

2021-05-26 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 1c63759  [BEAM-9547] Add additional unliftable grouped aggregations, 
add describe to DataFrame and Series (#14850)
 add 248dc26  [BEAM-12240] Add Java 17 as a valid environment
 add 2abbd37  Merge pull request #14880: [BEAM-12240] Add Java 17 as a 
valid environment

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/beam/runners/core/construction/Environments.java   | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


[beam] branch master updated (c18a06c -> df3b4d8)

2021-05-24 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c18a06c  Merge pull request #14867 from ihji/BEAM-12390
 add a4d535b  [BEAM-12241] Update vendored bytebuddy to version 1.11.0
 add df3b4d8  Merge pull request #14824: [BEAM-12241] Update vendored 
bytebuddy to version 1.11.0

No new revisions were added by this update.

Summary of changes:
 .../org/apache/beam/gradle/BeamModulePlugin.groovy |  2 +-
 sdks/java/core/build.gradle|  2 +-
 .../apache/beam/sdk/coders/RowCoderGenerator.java  | 40 ++--
 .../beam/sdk/schemas/utils/AutoValueUtils.java | 44 ++---
 .../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 26 
 .../apache/beam/sdk/schemas/utils/AvroUtils.java   | 16 ++---
 .../utils/ByteBuddyLocalVariableManager.java   | 10 +--
 .../beam/sdk/schemas/utils/ByteBuddyUtils.java | 72 +++---
 .../beam/sdk/schemas/utils/ConvertHelpers.java | 28 -
 .../beam/sdk/schemas/utils/JavaBeanUtils.java  | 34 +-
 .../apache/beam/sdk/schemas/utils/POJOUtils.java   | 46 +++---
 .../sdk/schemas/utils/SelectByteBuddyHelpers.java  | 62 +--
 .../reflect/ByteBuddyDoFnInvokerFactory.java   | 68 ++--
 .../reflect/ByteBuddyOnTimerInvokerFactory.java| 32 +-
 .../reflect/StableInvokerNamingStrategy.java   |  4 +-
 sdks/java/extensions/protobuf/build.gradle |  2 +-
 .../extensions/protobuf/ProtoByteBuddyUtils.java   | 62 +--
 17 files changed, 275 insertions(+), 275 deletions(-)


svn commit: r47844 - /dev/beam/2.29.0/

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 21:28:29 2021
New Revision: 47844

Log:
Move 2.29.0 to release repository

Removed:
dev/beam/2.29.0/



svn commit: r47843 - in /release/beam: ./ 2.29.0/ 2.29.0/python/

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 21:16:14 2021
New Revision: 47843

Log:
Move 2.29.0 to release repository

Added:
release/beam/2.29.0/
release/beam/2.29.0/apache-beam-2.29.0-source-release.zip   (with props)
release/beam/2.29.0/apache-beam-2.29.0-source-release.zip.asc
release/beam/2.29.0/apache-beam-2.29.0-source-release.zip.sha512
release/beam/2.29.0/python/
release/beam/2.29.0/python/apache-beam-2.29.0.zip   (with props)
release/beam/2.29.0/python/apache-beam-2.29.0.zip.asc
release/beam/2.29.0/python/apache-beam-2.29.0.zip.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl 
  (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-macosx_10_9_x86_64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl   
(with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_i686.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl  
 (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux1_x86_64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl 
  (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_i686.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl
   (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-manylinux2010_x86_64.whl.sha512
release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl   (with 
props)
release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.asc
release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win32.whl.sha512
release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl   
(with props)
release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp36-cp36m-win_amd64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl 
  (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-macosx_10_9_x86_64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl   
(with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_i686.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl  
 (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux1_x86_64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl 
  (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_i686.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl
   (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-manylinux2010_x86_64.whl.sha512
release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl   (with 
props)
release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.asc
release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win32.whl.sha512
release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl   
(with props)
release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp37-cp37m-win_amd64.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl  
 (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-macosx_10_9_x86_64.whl.sha512
release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl 
  (with props)

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.asc

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_i686.whl.sha512

release/beam/2.29.0/python/apache_beam-2.29.0-cp38-cp38-manylinux1_x86_64.whl

svn commit: r47842 - in /dev/beam: extensions/ vendor/

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 20:34:11 2021
New Revision: 47842

Log:
Remove empty directories for extensions and vendor

Removed:
dev/beam/extensions/
dev/beam/vendor/



svn commit: r47841 - in /release/beam/vendor/beam-vendor-bytebuddy-1_11_0: ./ 0.1/

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 20:32:25 2021
New Revision: 47841

Log:
Move Apache Beam vendored Byte Buddy 1.11.0 v0.1 to release branch

Added:
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/

release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
   (with props)

release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc

release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512

Added: 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
==
Binary file - no diff available.

Propchange: 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
--
svn:mime-type = application/octet-stream

Added: 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
==
--- 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
 (added)
+++ 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
 Thu May 20 20:32:25 2021
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEENBVjFynhWzMFGttnCp2vZxO4Y0kFAmCiiIUACgkQCp2vZxO4
+Y0kxwg//Rk4VH2BFbhQg//kc8KGjmjyT22exx1M55ttHmDaYmRlGR8YYBoLg6nfK
+uybzgQVHlAlMLqv3CuHyVX0YrA+iBc9AtvbwuIvHDhkwV5asyXbBhkUJ6nOzy6WY
+wUWOA7TTpFEOZUJiVDfOLYICo7QEuStJSVaEFome1JSFyK18cIy7uAKQfYIC1n4S
+Sl32RKYg4EXJACULO6sOnQq9D4sBeiqJ9IJcV/uGPwgN3ByJAk/GzsTnPsUrVA+B
+8tsa0USllAhMB6jTr3fGIOMDHaJJnIMr+qi/0Wmuc1Lq07tSsUtK8jeTVrtYy6zr
+FMQ6g2RKZq8ofQfPyPVLD7Glfb8COnJASgcGY6Iwlht9PRCmvUBaBuKN6Xz5pruS
+QHbf+EsiWi4dU9VX/aWI8w7csKLh2mOiQf/8ARIqPdQTmGTsfmQyKfZQzjPgq+lg
+Qw1j8lU/hMjCQmRU6RnDToJP70+THEK/PffiyycFsjSbkoGtu6CjROdjWVQQJALi
+jtqxf0KxH+jaYmQ3DfwToj4CdeVoY1IpoqXv+Jt+qboWE+xiUJYpsTzDq2Agy8+d
+kKTk/8dU/sxFNwwo2DgkryGw0sk+0PZfGAirBjVhn2CnAfsoaJvdWBqYnRCYLXYn
+FrV5PKzsrTMe8SZK183KJudKpASGXj+t2dxRTSF3MMxk3HJJIlE=
+=GL+y
+-END PGP SIGNATURE-

Added: 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
==
--- 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
 (added)
+++ 
release/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
 Thu May 20 20:32:25 2021
@@ -0,0 +1 @@
+e368451efee117d44d1286e87566e8a085395dad0c4c0051cd7e15b3cf87c56cfb8f63168efc5e1b44ccf408f90b1a3242915abdc780b41bb12630a454573697
  apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip




svn commit: r47840 - /dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 20:21:17 2021
New Revision: 47840

Log:
Move Apache Beam vendored Byte Buddy 1.11.0 v0.1 to release branch

Removed:
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/



svn commit: r47839 - in /release/beam: 2.25.0/ 2.26.0/ 2.27.0/ 2.28.0/ extensions/ vendor/beam-vendor-bytebuddy-1_10_8/ vendor/beam-vendor-bytebuddy-1_9_3/ vendor/beam-vendor-grpc-1_21_0/ vendor/beam-

2021-05-20 Thread iemejia
Author: iemejia
Date: Thu May 20 20:17:19 2021
New Revision: 47839

Log:
Remove releases already moved to archive.apache.org

Removed:
release/beam/2.25.0/
release/beam/2.26.0/
release/beam/2.27.0/
release/beam/2.28.0/
release/beam/extensions/
release/beam/vendor/beam-vendor-bytebuddy-1_10_8/
release/beam/vendor/beam-vendor-bytebuddy-1_9_3/
release/beam/vendor/beam-vendor-grpc-1_21_0/
release/beam/vendor/beam-vendor-grpc-1_26_0/
release/beam/vendor/beam-vendor-guava-26_0-jre/
release/beam/vendor/calcite/



[beam] branch master updated (8463a05 -> e0e3432)

2021-05-20 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8463a05  [BEAM-11227] Upgrade vendored gRPC to 1.36.0 with test fix 
(#14833)
 add 922f7f6  [BEAM-12342] Upgrade Spark 2 to version 2.4.8
 add e0e3432  Merge pull request #14823: [BEAM-12342] Upgrade Spark 2 to 
version 2.4.8

No new revisions were added by this update.

Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 runners/spark/2/build.gradle| 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


svn commit: r47768 - in /dev/beam/vendor/beam-vendor-bytebuddy-1_11_0: ./ 0.1/

2021-05-17 Thread iemejia
Author: iemejia
Date: Mon May 17 15:29:42 2021
New Revision: 47768

Log:
Add vendor bytebuddy 1.11.0

Added:
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/

dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
   (with props)

dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc

dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512

Added: 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
==
Binary file - no diff available.

Propchange: 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip
--
svn:mime-type = application/octet-stream

Added: 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
==
--- 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
 (added)
+++ 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.asc
 Mon May 17 15:29:42 2021
@@ -0,0 +1,16 @@
+-BEGIN PGP SIGNATURE-
+
+iQIzBAABCgAdFiEENBVjFynhWzMFGttnCp2vZxO4Y0kFAmCiiIUACgkQCp2vZxO4
+Y0kxwg//Rk4VH2BFbhQg//kc8KGjmjyT22exx1M55ttHmDaYmRlGR8YYBoLg6nfK
+uybzgQVHlAlMLqv3CuHyVX0YrA+iBc9AtvbwuIvHDhkwV5asyXbBhkUJ6nOzy6WY
+wUWOA7TTpFEOZUJiVDfOLYICo7QEuStJSVaEFome1JSFyK18cIy7uAKQfYIC1n4S
+Sl32RKYg4EXJACULO6sOnQq9D4sBeiqJ9IJcV/uGPwgN3ByJAk/GzsTnPsUrVA+B
+8tsa0USllAhMB6jTr3fGIOMDHaJJnIMr+qi/0Wmuc1Lq07tSsUtK8jeTVrtYy6zr
+FMQ6g2RKZq8ofQfPyPVLD7Glfb8COnJASgcGY6Iwlht9PRCmvUBaBuKN6Xz5pruS
+QHbf+EsiWi4dU9VX/aWI8w7csKLh2mOiQf/8ARIqPdQTmGTsfmQyKfZQzjPgq+lg
+Qw1j8lU/hMjCQmRU6RnDToJP70+THEK/PffiyycFsjSbkoGtu6CjROdjWVQQJALi
+jtqxf0KxH+jaYmQ3DfwToj4CdeVoY1IpoqXv+Jt+qboWE+xiUJYpsTzDq2Agy8+d
+kKTk/8dU/sxFNwwo2DgkryGw0sk+0PZfGAirBjVhn2CnAfsoaJvdWBqYnRCYLXYn
+FrV5PKzsrTMe8SZK183KJudKpASGXj+t2dxRTSF3MMxk3HJJIlE=
+=GL+y
+-END PGP SIGNATURE-

Added: 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
==
--- 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
 (added)
+++ 
dev/beam/vendor/beam-vendor-bytebuddy-1_11_0/0.1/apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip.sha512
 Mon May 17 15:29:42 2021
@@ -0,0 +1 @@
+e368451efee117d44d1286e87566e8a085395dad0c4c0051cd7e15b3cf87c56cfb8f63168efc5e1b44ccf408f90b1a3242915abdc780b41bb12630a454573697
  apache-beam-d93c591deb21237ddb656583d7ef7a4debba-source-release.zip




[beam] branch master updated: [BEAM-12343] GroupByKeyTest for changing WindowFn from GlobalWindow after GBK

2021-05-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 3fd92d0  [BEAM-12343] GroupByKeyTest for changing WindowFn from 
GlobalWindow after GBK
 new d93c591  Merge pull request #14667:[BEAM-12343] GroupByKeyTest for 
changing WindowFn from GlobalWindow after GBK
3fd92d0 is described below

commit 3fd92d0df24f671a9f5705860250acc3afd11e47
Author: Jan Lukavsky 
AuthorDate: Wed Apr 28 09:38:54 2021 +0200

[BEAM-12343] GroupByKeyTest for changing WindowFn from GlobalWindow after 
GBK
---
 .../apache/beam/sdk/transforms/GroupByKeyTest.java | 40 ++
 1 file changed, 40 insertions(+)

diff --git 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
index 88d40ac..fd849a5 100644
--- 
a/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
+++ 
b/sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java
@@ -70,8 +70,10 @@ import org.apache.beam.sdk.values.KV;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.TimestampedValue;
+import org.apache.beam.sdk.values.TypeDescriptors;
 import org.apache.beam.sdk.values.WindowingStrategy;
 import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableList;
+import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Streams;
 import org.checkerframework.checker.nullness.qual.Nullable;
 import org.hamcrest.Matcher;
 import org.joda.time.Duration;
@@ -585,6 +587,44 @@ public class GroupByKeyTest implements Serializable {
 }
 
 @Test
+@Category(ValidatesRunner.class)
+public void testRewindowWithTimestampCombiner() {
+  PCollection> input =
+  p.apply(
+  Create.timestamped(
+  TimestampedValue.of(KV.of("foo", 1), new Instant(1)),
+  TimestampedValue.of(KV.of("foo", 4), new Instant(4)),
+  TimestampedValue.of(KV.of("bar", 3), new Instant(3)),
+  TimestampedValue.of(KV.of("foo", 9), new Instant(9
+  .apply(
+  "GlobalWindows",
+  Window.>configure()
+  .withTimestampCombiner(TimestampCombiner.LATEST));
+
+  PCollection> result =
+  input
+  .apply(GroupByKey.create())
+  .apply(
+  MapElements.into(
+  TypeDescriptors.kvs(
+  TypeDescriptors.strings(), 
TypeDescriptors.integers()))
+  .via(kv -> KV.of(kv.getKey(), sum(kv.getValue()
+  .apply("FixedWindows", 
Window.into(FixedWindows.of(Duration.millis(1;
+
+  PAssert.that(result)
+  .inWindow(new IntervalWindow(new Instant(9), new Instant(10)))
+  .containsInAnyOrder(KV.of("foo", 14))
+  .inWindow(new IntervalWindow(new Instant(3), new Instant(4)))
+  .containsInAnyOrder(KV.of("bar", 3));
+
+  p.run();
+}
+
+private static int sum(Iterable parts) {
+  return Streams.stream(parts).mapToInt(e -> e).sum();
+}
+
+@Test
 @Category(NeedsRunner.class)
 public void testIdentityWindowFnPropagation() {
 


[beam] branch master updated (beb0d16 -> d9c113e)

2021-05-12 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from beb0d16  Merge pull request #14746 from ibzib/BEAM-12302
 add 7fc292b  [BEAM-12241] Upgrade vendored bytebuddy to version 1.11.0
 add d9c113e  Merge pull request #14766: [BEAM-12241] Upgrade vendored 
bytebuddy to version 1.11.0

No new revisions were added by this update.

Summary of changes:
 settings.gradle.kts| 2 +-
 vendor/{bytebuddy-1_10_8 => bytebuddy-1_11_0}/build.gradle.kts | 8 
 2 files changed, 5 insertions(+), 5 deletions(-)
 rename vendor/{bytebuddy-1_10_8 => bytebuddy-1_11_0}/build.gradle.kts (89%)


[beam] branch master updated (c3c4933 -> 9dbc7b7)

2021-05-12 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c3c4933  Merge pull request #14783 [BEAM-12326] Preserve resource 
hints across transform substitution.
 add a1c8f4b  [BEAM-10670] Use non-SDF based translation for Read by 
default on Spark Runner
 add 9dbc7b7  Merge pull request #14755: [BEAM-10670] Use non-SDF based 
translation for Read by default on Spark Runner

No new revisions were added by this update.

Summary of changes:
 .../structuredstreaming/SparkStructuredStreamingRunner.java|  8 ++--
 .../main/java/org/apache/beam/runners/spark/SparkRunner.java   |  8 ++--
 .../org/apache/beam/runners/spark/SparkRunnerDebugger.java | 10 --
 .../org/apache/beam/runners/spark/SparkRunnerDebuggerTest.java |  2 +-
 4 files changed, 9 insertions(+), 19 deletions(-)


[beam] 01/01: Merge pull request #14751: [BEAM-11055] Fix mistaken version of log4j in HCatalog

2021-05-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit d6d6f30723fbfb1a0472fe814b419232e06952a5
Merge: 5da41e6 d589b62
Author: Ismaël Mejía 
AuthorDate: Sat May 8 06:45:53 2021 +0200

Merge pull request #14751: [BEAM-11055] Fix mistaken version of log4j in 
HCatalog

 sdks/java/io/hcatalog/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (5da41e6 -> d6d6f30)

2021-05-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5da41e6  Merge pull request #14757 from 
iemejia/BEAM-12282-vendor-grpc-update-revert
 add d589b62  [BEAM-11055] Fix mistaken version of log4j in HCatalog
 new d6d6f30  Merge pull request #14751: [BEAM-11055] Fix mistaken version 
of log4j in HCatalog

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/hcatalog/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (7af241e -> 8787c13)

2021-05-06 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7af241e  Merge pull request #14739 from 
kennknowles/actually-2.29.0-was-later
 add 9fef5b4  Only run one pipeline in testWriteViaSink
 add 8787c13  Merge pull request #14740: [BEAM-7320] Only run one pipeline 
in TextIOWriteTest.testWriteViaSink

No new revisions were added by this update.

Summary of changes:
 .../test/java/org/apache/beam/sdk/io/TextIOWriteTest.java  | 14 --
 1 file changed, 14 deletions(-)


[beam] branch master updated (66caaa9 -> 7f0d11e)

2021-05-04 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 66caaa9  Merge pull request #14708: [BEAM-12271] Make max width for 
code snippets adaptive with text blocks
 add 0547c64  [BEAM-12280] Upgrade Flink runner to Flink version 1.12.3
 add 7f0d11e  Merge pull request #14721: [BEAM-12280] Upgrade Flink runner 
to Flink version 1.12.3

No new revisions were added by this update.

Summary of changes:
 runners/flink/1.12/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (b9e8aca -> 07e7613a)

2021-05-03 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from b9e8aca  Merge pull request #14709 from pabloem/revert-pr13617
 add 79ea6dd  [BEAM-12253] Change 
Read.UnboundedSourceAsSDFRestrictionTracker to use the cache.
 add 07e7613a Merge pull request #14704: [BEAM-12253] Change 
Read.UnboundedSourceAsSDFRestrictionTracker.getSplitBacklog to use the reader 
cache

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/beam/sdk/io/Read.java | 41 --
 1 file changed, 23 insertions(+), 18 deletions(-)


[beam] 03/03: Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and ArrayList object creations

2021-04-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7418c84de2b68b06e0318766ca6c69f4d2144390
Merge: 87e1164 be906f3
Author: Ismaël Mejía 
AuthorDate: Thu Apr 29 08:13:58 2021 +0200

Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce String and 
ArrayList object creations

 .../beam/runners/core/InMemoryTimerInternals.java  | 46 --
 .../main/java/org/apache/beam/sdk/values/Row.java  | 28 ++---
 .../java/org/apache/beam/sdk/values/RowUtils.java  | 13 ++
 3 files changed, 41 insertions(+), 46 deletions(-)


[beam] 02/03: [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils

2021-04-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit be906f38c5cac14c465332b8179022085ed76c13
Author: Daniel Kulp 
AuthorDate: Wed Apr 28 14:21:13 2021 -0400

[BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
---
 .../main/java/org/apache/beam/sdk/values/Row.java  | 28 ++
 .../java/org/apache/beam/sdk/values/RowUtils.java  | 13 ++
 2 files changed, 25 insertions(+), 16 deletions(-)

diff --git a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
index a19d9b3..9aa9277 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/Row.java
@@ -847,23 +847,19 @@ public abstract class Row implements Serializable {
 + " fields.");
   }
 
-  FieldOverrides fieldOverrides = new FieldOverrides(schema);
-  fieldOverrides.setOverrides(this.values);
-
-  Row row;
-  if (!fieldOverrides.isEmpty()) {
-row =
-(Row)
-new RowFieldMatcher()
-.match(
-new CapturingRowCases(schema, fieldOverrides),
-FieldType.row(schema),
-new RowPosition(FieldAccessDescriptor.create()),
-null);
-  } else {
-row = new RowWithStorage(schema, Collections.emptyList());
+  if (!values.isEmpty()) {
+FieldOverrides fieldOverrides = new FieldOverrides(schema, 
this.values);
+if (!fieldOverrides.isEmpty()) {
+  return (Row)
+  new RowFieldMatcher()
+  .match(
+  new CapturingRowCases(schema, fieldOverrides),
+  FieldType.row(schema),
+  new RowPosition(FieldAccessDescriptor.create()),
+  null);
+}
   }
-  return row;
+  return new RowWithStorage(schema, Collections.emptyList());
 }
   }
 
diff --git 
a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java 
b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
index d3b1c21..a5f6210 100644
--- a/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
+++ b/sdks/java/core/src/main/java/org/apache/beam/sdk/values/RowUtils.java
@@ -236,6 +236,11 @@ class RowUtils {
   this.rootSchema = rootSchema;
 }
 
+FieldOverrides(Schema rootSchema, List overrides) {
+  this.topNode = new FieldAccessNode(rootSchema, overrides);
+  this.rootSchema = rootSchema;
+}
+
 boolean isEmpty() {
   return topNode.isEmpty();
 }
@@ -270,6 +275,14 @@ class RowUtils {
 nestedAccess = Lists.newArrayList();
   }
 
+  FieldAccessNode(Schema schema, List overrides) {
+fieldOverrides = 
Lists.newArrayListWithExpectedSize(schema.getFieldCount());
+for (Object value : overrides) {
+  fieldOverrides.add(new FieldOverride(value));
+}
+nestedAccess = Lists.newArrayList();
+  }
+
   boolean isEmpty() {
 return fieldOverrides.isEmpty() && nestedAccess.isEmpty();
   }


[beam] branch master updated (87e1164 -> 7418c84)

2021-04-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 87e1164  Merge pull request #14676: Revert "[BEAM-3713] Moving 
integration tests from nose to pytest (#14481)"
 new b87f1f0  [BEAM-12247] Reduce memory/string creations in 
InMemoryTimerInternals
 new be906f3  [BEAM-12248] Reduce ArrayList allocation in Row/RowUtils
 new 7418c84  Merge pull request #14672: [BEAM-12248][BEAM-12247] Reduce 
String and ArrayList object creations

The 3 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/runners/core/InMemoryTimerInternals.java  | 46 --
 .../main/java/org/apache/beam/sdk/values/Row.java  | 28 ++---
 .../java/org/apache/beam/sdk/values/RowUtils.java  | 13 ++
 3 files changed, 41 insertions(+), 46 deletions(-)


[beam] 01/03: [BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals

2021-04-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b87f1f01197a78d60fb08a0ee7a4ef96bc4e08eb
Author: Daniel Kulp 
AuthorDate: Wed Apr 28 14:16:31 2021 -0400

[BEAM-12247] Reduce memory/string creations in InMemoryTimerInternals
---
 .../beam/runners/core/InMemoryTimerInternals.java  | 46 --
 1 file changed, 16 insertions(+), 30 deletions(-)

diff --git 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
index 8be9081..d0b3bed 100644
--- 
a/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
+++ 
b/runners/core-java/src/main/java/org/apache/beam/runners/core/InMemoryTimerInternals.java
@@ -63,6 +63,9 @@ public class InMemoryTimerInternals implements TimerInternals 
{
   /** Current synchronized processing time. */
   private Instant synchronizedProcessingTime = 
BoundedWindow.TIMESTAMP_MIN_VALUE;
 
+  /** Class.getSimpleName() cached to avoid allocations for tracing. */
+  private static final String SIMPLE_NAME = 
InMemoryTimerInternals.class.getSimpleName();
+
   @Override
   public @Nullable Instant currentOutputWatermarkTime() {
 return outputWatermarkTime;
@@ -125,17 +128,12 @@ public class InMemoryTimerInternals implements 
TimerInternals {
   @Deprecated
   @Override
   public void setTimer(TimerData timerData) {
-WindowTracing.trace("{}.setTimer: {}", getClass().getSimpleName(), 
timerData);
+WindowTracing.trace("{}.setTimer: {}", SIMPLE_NAME, timerData);
 
-@Nullable
-TimerData existing =
-existingTimers.get(
-timerData.getNamespace(), timerData.getTimerId() + '+' + 
timerData.getTimerFamilyId());
+@Nullable String colKey = timerData.getTimerId() + '+' + 
timerData.getTimerFamilyId();
+TimerData existing = existingTimers.get(timerData.getNamespace(), colKey);
 if (existing == null) {
-  existingTimers.put(
-  timerData.getNamespace(),
-  timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
-  timerData);
+  existingTimers.put(timerData.getNamespace(), colKey, timerData);
   timersForDomain(timerData.getDomain()).add(timerData);
 } else {
   checkArgument(
@@ -149,10 +147,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 NavigableSet timers = 
timersForDomain(timerData.getDomain());
 timers.remove(existing);
 timers.add(timerData);
-existingTimers.put(
-timerData.getNamespace(),
-timerData.getTimerId() + '+' + timerData.getTimerFamilyId(),
-timerData);
+existingTimers.put(timerData.getNamespace(), colKey, timerData);
   }
 }
   }
@@ -216,7 +211,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 newInputWatermark);
 WindowTracing.trace(
 "{}.advanceInputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
 inputWatermarkTime,
 newInputWatermark);
 inputWatermarkTime = newInputWatermark;
@@ -229,7 +224,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 if (newOutputWatermark.isAfter(inputWatermarkTime)) {
   WindowTracing.trace(
   "{}.advanceOutputWatermark: clipping output watermark from {} to {}",
-  getClass().getSimpleName(),
+  SIMPLE_NAME,
   newOutputWatermark,
   inputWatermarkTime);
   adjustedOutputWatermark = inputWatermarkTime;
@@ -244,7 +239,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 adjustedOutputWatermark);
 WindowTracing.trace(
 "{}.advanceOutputWatermark: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
 outputWatermarkTime,
 adjustedOutputWatermark);
 outputWatermarkTime = adjustedOutputWatermark;
@@ -259,10 +254,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 processingTime,
 newProcessingTime);
 WindowTracing.trace(
-"{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
-processingTime,
-newProcessingTime);
+"{}.advanceProcessingTime: from {} to {}", SIMPLE_NAME, 
processingTime, newProcessingTime);
 processingTime = newProcessingTime;
   }
 
@@ -277,7 +269,7 @@ public class InMemoryTimerInternals implements 
TimerInternals {
 newSynchronizedProcessingTime);
 WindowTracing.trace(
 "{}.advanceProcessingTime: from {} to {}",
-getClass().getSimpleName(),
+SIMPLE_NAME,
 synchronizedProcessingTime,
 newSynchronizedProcessingTime);
 

[beam] branch master updated (bc2fb2a -> b1a67ef)

2021-04-28 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bc2fb2a  Merge pull request #14645 from [BEAM-12188] Bump snakeyaml to 
1.28
 add 6ebadec  [BEAM-12243] TPC-DS: use SQL "substring()" instead of 
"substr()"
 add b1a67ef  Merge pull request #14670: [BEAM-12243] TPC-DS: use SQL 
"substring()" instead of "substr()"

No new revisions were added by this update.

Summary of changes:
 sdks/java/testing/tpcds/src/main/resources/queries/query15.sql  | 2 +-
 sdks/java/testing/tpcds/src/main/resources/queries/query19.sql  | 2 +-
 sdks/java/testing/tpcds/src/main/resources/queries/query23a.sql | 4 ++--
 sdks/java/testing/tpcds/src/main/resources/queries/query23b.sql | 4 ++--
 sdks/java/testing/tpcds/src/main/resources/queries/query45.sql  | 2 +-
 sdks/java/testing/tpcds/src/main/resources/queries/query62.sql  | 6 +++---
 sdks/java/testing/tpcds/src/main/resources/queries/query79.sql  | 4 ++--
 sdks/java/testing/tpcds/src/main/resources/queries/query8.sql   | 8 
 sdks/java/testing/tpcds/src/main/resources/queries/query85.sql  | 4 ++--
 sdks/java/testing/tpcds/src/main/resources/queries/query99.sql  | 6 +++---
 10 files changed, 21 insertions(+), 21 deletions(-)


[beam] branch master updated (4202fb8 -> ea107d0)

2021-04-27 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 4202fb8  Merge pull request #14661 - Fix worker_pool plumbing to use a 
mutable List.
 add 62b9c0f  [BEAM-9283] Disable caching ValidatesRunner tests tasks
 add ea107d0  Merge pull request #14640: [BEAM-9283] Disable caching 
ValidatesRunner tests tasks

No new revisions were added by this update.

Summary of changes:
 runners/flink/flink_runner.gradle | 2 ++
 runners/spark/spark_runner.gradle | 6 ++
 2 files changed, 8 insertions(+)


[beam] branch master updated (e13e548 -> 9505f4e)

2021-04-27 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e13e548  [BEAM-10029] Adds Grafana config for Spanner IO Perf Metrics 
(#14643)
 add ef654f6  Close ZipOutputStream
 add 9505f4e  Merge pull request #14631: [BEAM-12220] Close ZipOutputStream 
to avoid native memory leak

No new revisions were added by this update.

Summary of changes:
 sdks/java/core/src/main/java/org/apache/beam/sdk/util/ZipFiles.java | 1 +
 1 file changed, 1 insertion(+)


[beam] branch master updated (618b060 -> f61d100)

2021-04-26 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 618b060  Merge pull request #14626: [BEAM-12214] RedisIO.readAll() was 
removed since it is deprecated since Beam 2.13.0
 add 85a5560  [BEAM-12216] MqttIO.create() with clientId constructor 
removed because it was deprecated since Beam 2.13.0
 add f61d100  Merge pull request #14627: [BEAM-12216] MqttIO.create() with 
clientId constructor removed as it is deprecated since Beam 2.13.0

No new revisions were added by this update.

Summary of changes:
 CHANGES.md   |  2 ++
 .../main/java/org/apache/beam/sdk/io/mqtt/MqttIO.java| 16 
 2 files changed, 2 insertions(+), 16 deletions(-)


[beam] branch master updated (018f5e7 -> 9209c75)

2021-04-23 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 018f5e7  Merge pull request #14575 from [BEAM-12212] Adds  
--sdkContainerImage as new Java Dataflow PipelineOption
 add bf17110  [BEAM-12217] MongoDbIO: Read.withFilter() and 
Read.withProjection() are removed
 add 9209c75  Merge pull request #14628: [BEAM-12217] MongoDbIO: 
Read.withFilter() and Read.withProjection() are removed since they are 
deprecated since Beam 2.12.0

No new revisions were added by this update.

Summary of changes:
 CHANGES.md |  2 ++
 .../org/apache/beam/sdk/io/mongodb/MongoDbIO.java  | 37 --
 2 files changed, 2 insertions(+), 37 deletions(-)


[beam] branch master updated (f15294a -> 3246690)

2021-04-22 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f15294a  [BEAM-8691] Upgrading bigtable-client-core to 1.19.1
 add 62e0f38  [BEAM-12210] Use formatting string for checkArgument to avoid 
excess String appends
 add fc45ad4  Fix spotless
 add 3246690  Merge pull request #14620: [BEAM-12210] Use formatting string 
for checkArgument to avoid excess …

No new revisions were added by this update.

Summary of changes:
 .../src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)


[beam] branch master updated (2e9ee8c -> 93ecc1d)

2021-04-22 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2e9ee8c  [BEAM-2085] Fixups for Python resource hints. (#14605)
 add 8e0ee41  [BEAM-12207] Remove log messages about files to stage.
 add 4e3decb  [BEAM-12207] Enable nullness check on 
SparkPipelineRunner.java.
 add 93ecc1d  Merge pull request #14616: [BEAM-12207] Remove log messages 
about files to stage.

No new revisions were added by this update.

Summary of changes:
 .../org/apache/beam/runners/spark/SparkPipelineRunner.java | 10 +++---
 1 file changed, 3 insertions(+), 7 deletions(-)


[beam] branch master updated (667ec40 -> 985e2f0)

2021-04-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 667ec40  Merge pull request #14504: Refactor PR template to separate 
test types and label test variants
 add 3ea492d  [BEAM-12172] Bump gradle to 6.8.3
 add 985e2f0  Merge pull request #14543: [BEAM-12172] Bump gradle to 6.8.3

No new revisions were added by this update.

Summary of changes:
 gradle/wrapper/gradle-wrapper.properties | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (97af077 -> 8e66956)

2021-04-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 97af077  [BEAM-9547] DataFrame.corr cleanup (#14327)
 add f9d4805  [BEAM-12091] Make file staging uniform among runners
 add 8e66956  Merge pull request #14520: [BEAM-12091] Make file staging 
uniform among runners

No new revisions were added by this update.

Summary of changes:
 .../construction/resources/PipelineResources.java  | 37 +-
 .../resources/PipelineResourcesTest.java   | 78 +++---
 .../flink/FlinkPipelineExecutionEnvironment.java   | 18 +
 .../SparkStructuredStreamingRunner.java| 20 +-
 .../runners/spark/SparkCommonPipelineOptions.java  | 27 +---
 .../beam/runners/spark/SparkPipelineRunner.java|  8 ---
 .../org/apache/beam/runners/spark/SparkRunner.java | 18 +
 .../beam/runners/twister2/Twister2Runner.java  | 33 +
 .../beam/sdk/util/common/ReflectHelpers.java   |  2 +-
 9 files changed, 114 insertions(+), 127 deletions(-)


[beam] branch master updated (a86dc06 -> b10ce99)

2021-04-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a86dc06  Bump container pandas version to 1.2.4 (#14524)
 add 992c378  [BEAM-2888] Added packages.confluent.io maven repo
 add b10ce99  Merge pull request #14545: [BEAM-2888] Added 
"packages.confluent.io" maven repo

No new revisions were added by this update.

Summary of changes:
 .test-infra/validate-runner/build.gradle | 3 +++
 1 file changed, 3 insertions(+)


[beam] 03/04: [BEAM-11712] Fix static analysis warnings and typos on TPC-DS module

2021-04-13 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 8fe0c5cee1bab62680ebd92a51aed8f3da80e190
Author: Ismaël Mejía 
AuthorDate: Tue Apr 13 14:20:54 2021 +0200

[BEAM-11712] Fix static analysis warnings and typos on TPC-DS module
---
 .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java|   3 +-
 .../java/org/apache/beam/sdk/tpcds/CsvToRow.java   |   4 +-
 .../org/apache/beam/sdk/tpcds/QueryReader.java |   3 +-
 .../java/org/apache/beam/sdk/tpcds/RowToCsv.java   |   2 +-
 .../apache/beam/sdk/tpcds/SqlTransformRunner.java  |   2 +-
 .../beam/sdk/tpcds/TableSchemaJSONLoader.java  |   6 +-
 .../beam/sdk/tpcds/TpcdsParametersReader.java  |   4 +-
 .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java  |  11 +--
 .../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 102 ++---
 .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java  |   3 +-
 .../apache/beam/sdk/tpcds/TpcdsSchemasTest.java|   6 +-
 11 files changed, 68 insertions(+), 78 deletions(-)

diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 69e676f..304fdd2 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -211,8 +211,7 @@ public class BeamSqlEnvRunner {
 // Transform the result from PCollection into 
PCollection, and write it to the
 // location where results are stored.
 PCollection rowStrings =
-rows.apply(
-MapElements.into(TypeDescriptors.strings()).via((Row row) -> 
row.toString()));
+
rows.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString));
 rowStrings.apply(
 TextIO.write()
 .to(
diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
index d66b128..d6c8ed8 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/CsvToRow.java
@@ -33,8 +33,8 @@ import org.apache.commons.csv.CSVFormat;
  */
 public class CsvToRow extends PTransform, PCollection>
 implements Serializable {
-  private Schema schema;
-  private CSVFormat csvFormat;
+  private final Schema schema;
+  private final CSVFormat csvFormat;
 
   public CSVFormat getCsvFormat() {
 return csvFormat;
diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
index 7b00a37..c6f3253 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/QueryReader.java
@@ -35,7 +35,6 @@ public class QueryReader {
*/
   public static String readQuery(String queryFileName) throws Exception {
 String path = "queries/" + queryFileName + ".sql";
-String query = Resources.toString(Resources.getResource(path), 
Charsets.UTF_8);
-return query;
+return Resources.toString(Resources.getResource(path), Charsets.UTF_8);
   }
 }
diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
index 40a8cc5..a087948 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/RowToCsv.java
@@ -33,7 +33,7 @@ import org.apache.commons.csv.CSVFormat;
  */
 public class RowToCsv extends PTransform, PCollection>
 implements Serializable {
-  private CSVFormat csvFormat;
+  private final CSVFormat csvFormat;
 
   public RowToCsv(CSVFormat csvFormat) {
 this.csvFormat = csvFormat;
diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
index 4f56c1a..bea0261 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/SqlTransformRunner.java
@@ -189,7 +189,7 @@ public class SqlTransformRunner {
   try {
 tables
 .apply(SqlTransform.query(queryString))
-.apply(MapElements.into(TypeDescriptors.strings()).via((Row row) 
-> row.toString()))
+
.apply(MapElements.into(TypeDescriptors.strings()).via(Row::toString))
 .apply(
 TextIO.write()

[beam] 04/04: Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark runner

2021-04-13 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit b3ef2035abf9ca2dd94a11a1a6aa4440df28adb9
Merge: f805f1c 8fe0c5c
Author: Ismaël Mejía 
AuthorDate: Tue Apr 13 14:22:12 2021 +0200

Merge pull request #14373: [BEAM-11712] Run TPC-DS via BeamSQL and Spark 
runner

 sdks/java/testing/tpcds/README.md  |   68 +
 sdks/java/testing/tpcds/build.gradle   |  108 +-
 .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java|  327 +++--
 .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java  |   50 +-
 .../java/org/apache/beam/sdk/tpcds/CsvToRow.java   |   47 +-
 .../org/apache/beam/sdk/tpcds/QueryReader.java |   51 +-
 .../java/org/apache/beam/sdk/tpcds/RowToCsv.java   |   38 +-
 .../apache/beam/sdk/tpcds/SqlTransformRunner.java  |  314 +++--
 .../apache/beam/sdk/tpcds/SummaryGenerator.java|  219 ++--
 .../beam/sdk/tpcds/TableSchemaJSONLoader.java  |  162 +--
 .../org/apache/beam/sdk/tpcds/TpcdsOptions.java|   40 +-
 .../beam/sdk/tpcds/TpcdsOptionsRegistrar.java  |   10 +-
 .../beam/sdk/tpcds/TpcdsParametersReader.java  |  136 +-
 .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java   |   54 +-
 .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java  |  120 +-
 .../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++--
 ...pcdsOptionsRegistrar.java => package-info.java} |   16 +-
 .../org/apache/beam/sdk/tpcds/QueryReaderTest.java |  361 +++---
 .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java  |  261 ++--
 .../beam/sdk/tpcds/TpcdsParametersReaderTest.java  |  110 +-
 .../apache/beam/sdk/tpcds/TpcdsSchemasTest.java|  183 ++-
 21 files changed, 2158 insertions(+), 1853 deletions(-)


[beam] 02/04: [BEAM-11712] Add options for input/output paths, make it run via SparkRunner

2021-04-13 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit a407d79680d01c35760f3fe4e76cd4192e34edd1
Author: Alexey Romanenko 
AuthorDate: Tue Mar 30 18:04:22 2021 +0200

[BEAM-11712] Add options for input/output paths, make it run via SparkRunner
---
 sdks/java/testing/tpcds/README.md  | 68 ++
 sdks/java/testing/tpcds/build.gradle   | 11 +++-
 .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java| 14 ++---
 .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java  | 18 +-
 .../org/apache/beam/sdk/tpcds/QueryReader.java | 49 +---
 .../apache/beam/sdk/tpcds/SqlTransformRunner.java  |  9 +--
 .../beam/sdk/tpcds/TableSchemaJSONLoader.java  | 43 +-
 .../org/apache/beam/sdk/tpcds/TpcdsOptions.java| 14 +
 .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java  |  4 +-
 9 files changed, 123 insertions(+), 107 deletions(-)

diff --git a/sdks/java/testing/tpcds/README.md 
b/sdks/java/testing/tpcds/README.md
new file mode 100644
index 000..89f8073
--- /dev/null
+++ b/sdks/java/testing/tpcds/README.md
@@ -0,0 +1,68 @@
+
+
+# TPC-DS Benchmark
+
+## Google Dataflow Runner
+
+To execute TPC-DS benchmark for 1Gb dataset on Google Dataflow, run the 
following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+  --runner=DataflowRunner \
+  --queries=3,26,55 \
+  --tpcParallel=2 \
+  --dataDirectory=/path/to/tpcds_data/ \  
+  --project=apache-beam-testing \
+  --stagingLocation=gs://beamsql_tpcds_1/staging \
+  --tempLocation=gs://beamsql_tpcds_2/temp \
+  --dataDirectory=/path/to/tpcds_data/ \  
+  --region=us-west1 \
+  --maxNumWorkers=10"
+```
+
+To run a query using ZetaSQL planner (currently Query96 can be run using 
ZetaSQL), set the plannerName as below. If not specified, the default planner 
is Calcite.
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.args="--dataSize=1G \
+  --runner=DataflowRunner \
+  --queries=96 \
+  --tpcParallel=2 \
+  --dataDirectory=/path/to/tpcds_data/ \  
+  --plannerName=org.apache.beam.sdk.extensions.sql.zetasql.ZetaSQLQueryPlanner 
\
+  --project=apache-beam-testing \
+  --stagingLocation=gs://beamsql_tpcds_1/staging \
+  --tempLocation=gs://beamsql_tpcds_2/temp \
+  --region=us-west1 \
+  --maxNumWorkers=10"
+```
+
+## Spark Runner
+
+To execute TPC-DS benchmark with Query3 for 1Gb dataset on Apache Spark 2.x, 
run the following example command from the command line:
+
+```bash
+./gradlew :sdks:java:testing:tpcds:run -Ptpcds.runner=":runners:spark:2" 
-Ptpcds.args=" \
+  --runner=SparkRunner \
+  --queries=3 \
+  --tpcParallel=1 \
+  --dataDirectory=/path/to/tpcds_data/ \
+  --dataSize=1G \
+  --resultsDirectory=/path/to/tpcds_results/"
+```
diff --git a/sdks/java/testing/tpcds/build.gradle 
b/sdks/java/testing/tpcds/build.gradle
index 6237776..79fb1e8 100644
--- a/sdks/java/testing/tpcds/build.gradle
+++ b/sdks/java/testing/tpcds/build.gradle
@@ -33,7 +33,7 @@ def tpcdsArgsProperty = "tpcds.args"
 def tpcdsRunnerProperty = "tpcds.runner"
 def tpcdsRunnerDependency = project.findProperty(tpcdsRunnerProperty)
 ?: ":runners:direct-java"
-def shouldProvideSpark = ":runners:spark".equals(tpcdsRunnerDependency)
+def shouldProvideSpark = ":runners:spark:2".equals(tpcdsRunnerDependency)
 def isDataflowRunner = 
":runners:google-cloud-dataflow-java".equals(tpcdsRunnerDependency)
 def runnerConfiguration = ":runners:direct-java".equals(tpcdsRunnerDependency) 
? "shadow" : null
 
@@ -88,6 +88,15 @@ if (shouldProvideSpark) {
 }
 }
 
+// Execute the TPC-DS queries or suites via Gradle.
+//
+// Parameters:
+//   -Ptpcds.runner
+//   Specify a runner subproject, such as ":runners:spark:2" or 
":runners:flink:1.10"
+//   Defaults to ":runners:direct-java"
+//
+//   -Ptpcds.args
+//   Specify the command line for invoking 
org.apache.beam.sdk.tpcds.BeamTpcds
 task run(type: JavaExec) {
 def tpcdsArgsStr = project.findProperty(tpcdsArgsProperty) ?: ""
 def tpcdsArgsList = new ArrayList()
diff --git 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
index 43b97d2..69e676f 100644
--- 
a/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
+++ 
b/sdks/java/testing/tpcds/src/main/java/org/apache/beam/sdk/tpcds/BeamSqlEnvRunner.java
@@ -17,6 +17,8 @@
  */
 package org.apache.beam.sdk.tpcds;
 
+import static org.apache.beam.sdk.util.Preconditions.checkArgumentNotNull;
+
 import com.alibaba.fastjson.JSONObject;
 import java.util.ArrayList;
 import java.util.Arrays;
@@ -6

[beam] branch master updated (f805f1c -> b3ef203)

2021-04-13 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from f805f1c  Merge pull request #14499 from [BEAM-11408, BEAM-11772] Add 
explicit output typehints to ensure coder determinism for BQ with auto-sharding
 new 28eec3f  [BEAM-11712] Make up-to-date build file and codestyle
 new a407d79  [BEAM-11712] Add options for input/output paths, make it run 
via SparkRunner
 new 8fe0c5c  [BEAM-11712] Fix static analysis warnings and typos on TPC-DS 
module
 new b3ef203  Merge pull request #14373: [BEAM-11712] Run TPC-DS via 
BeamSQL and Spark runner

The 4 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/testing/tpcds/README.md  |   68 +
 sdks/java/testing/tpcds/build.gradle   |  108 +-
 .../apache/beam/sdk/tpcds/BeamSqlEnvRunner.java|  327 +++--
 .../java/org/apache/beam/sdk/tpcds/BeamTpcds.java  |   50 +-
 .../java/org/apache/beam/sdk/tpcds/CsvToRow.java   |   47 +-
 .../org/apache/beam/sdk/tpcds/QueryReader.java |   51 +-
 .../java/org/apache/beam/sdk/tpcds/RowToCsv.java   |   38 +-
 .../apache/beam/sdk/tpcds/SqlTransformRunner.java  |  314 +++--
 .../apache/beam/sdk/tpcds/SummaryGenerator.java|  219 ++--
 .../beam/sdk/tpcds/TableSchemaJSONLoader.java  |  162 +--
 .../org/apache/beam/sdk/tpcds/TpcdsOptions.java|   40 +-
 .../beam/sdk/tpcds/TpcdsOptionsRegistrar.java  |   10 +-
 .../beam/sdk/tpcds/TpcdsParametersReader.java  |  136 +-
 .../java/org/apache/beam/sdk/tpcds/TpcdsRun.java   |   54 +-
 .../org/apache/beam/sdk/tpcds/TpcdsRunResult.java  |  120 +-
 .../org/apache/beam/sdk/tpcds/TpcdsSchemas.java| 1336 ++--
 .../org/apache/beam/sdk/tpcds}/package-info.java   |4 +-
 .../org/apache/beam/sdk/tpcds/QueryReaderTest.java |  361 +++---
 .../beam/sdk/tpcds/TableSchemaJSONLoaderTest.java  |  261 ++--
 .../beam/sdk/tpcds/TpcdsParametersReaderTest.java  |  110 +-
 .../apache/beam/sdk/tpcds/TpcdsSchemasTest.java|  183 ++-
 21 files changed, 2159 insertions(+), 1840 deletions(-)
 create mode 100644 sdks/java/testing/tpcds/README.md
 copy sdks/java/testing/{load-tests/src/main/java/org/apache/beam/sdk/loadtests 
=> tpcds/src/main/java/org/apache/beam/sdk/tpcds}/package-info.java (92%)


[beam] branch master updated (253bf38 -> b908f59)

2021-04-12 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 253bf38  Merge pull request #14464 from ibzib/BEAM-12123
 add ca05c57  [BEAM-2303] Support SpecificData in AvroCoder
 add b908f59  Merge pull request #14410: [BEAM-2303] Support SpecificData 
in AvroCoder

No new revisions were added by this update.

Summary of changes:
 .../java/org/apache/beam/sdk/coders/AvroCoder.java | 23 
 .../org/apache/beam/sdk/coders/AvroCoderTest.java  | 43 --
 2 files changed, 55 insertions(+), 11 deletions(-)


[beam] branch master updated (62ada38 -> 752798e)

2021-04-12 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 62ada38  Merge pull request #14506: [BEAM-11903] Bump objenesis to 3.2
 add f725953  [BEAM-12151] Bump Apache Parquet to 1.12.0
 add 752798e  Merge pull request #14509: [BEAM-12151] Bump Apache Parquet 
to 1.12.0

No new revisions were added by this update.

Summary of changes:
 sdks/java/io/parquet/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (0416017 -> 62ada38)

2021-04-11 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 0416017  Merge pull request #14450: [BEAM-12040, BEAM-11934] Remove 
the option withRunnerDeterminedShardingUnboundedInternal; add a check for 
merging windows
 add 0b35b27  [BEAM-11903] Bump objenesis to 3.2
 add 62ada38  Merge pull request #14506: [BEAM-11903] Bump objenesis to 3.2

No new revisions were added by this update.

Summary of changes:
 sdks/java/extensions/kryo/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (c18e3cf -> e398a16)

2021-04-10 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c18e3cf  Merge pull request #14503 from [BEAM-12143] Fix PubsubReader 
to populate message id correctly
 add e398a16  Merge pull request #14472: [BEAM-12148] Align Spark runner 
jackson dependency version with Beam's

No new revisions were added by this update.

Summary of changes:
 .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy  | 2 ++
 runners/spark/spark_runner.gradle   | 6 +-
 2 files changed, 7 insertions(+), 1 deletion(-)


[beam] branch master updated (9601bde -> e6767c1)

2021-04-08 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9601bde  [BEAM-11227] Try reverting #14295: Moving from vendored gRPC 
1.26 to 1.36 (#14466)
 add 2cca8f1  [BEAM-12092] Bump jedis to 3.5.2
 add e6767c1  Merge pull request #14471: [BEAM-12092] Bump jedis to 3.5.2

No new revisions were added by this update.

Summary of changes:
 sdks/java/io/redis/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] 01/01: Merge pull request #14203: [BEAM-11948] Drop support for Flink 1.8 and 1.9

2021-04-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 572a99bab07e53e043887243e2b1e69120563be5
Merge: cb31b7b 05b3fd3
Author: Ismaël Mejía 
AuthorDate: Thu Apr 8 06:25:51 2021 +0200

Merge pull request #14203: [BEAM-11948] Drop support for Flink 1.8 and 1.9

 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 runners/flink/1.10/build.gradle|  4 +-
 .../beam/runners/flink/FlinkCapabilities.java  | 34 
 .../streaming/AbstractStreamOperatorCompat.java|  0
 .../beam/runners/flink/RemoteMiniClusterImpl.java  |  0
 .../runners/flink/SourceTransformationCompat.java  |  0
 runners/flink/1.11/build.gradle|  4 +-
 runners/flink/1.12/build.gradle|  4 +-
 runners/flink/1.8/build.gradle | 34 
 .../flink/1.8/job-server-container/build.gradle| 26 -
 runners/flink/1.8/job-server/build.gradle  | 31 ---
 .../beam/runners/flink/FlinkCapabilities.java  | 34 
 .../streaming/io/BeamStoppableFunction.java| 29 --
 .../beam/runners/flink/FlinkRunnerTestCompat.java  | 43 ---
 .../runners/flink/streaming/StreamSources.java | 50 -
 runners/flink/1.9/build.gradle | 33 
 .../flink/1.9/job-server-container/build.gradle| 26 -
 runners/flink/1.9/job-server/build.gradle  | 31 ---
 .../runners/flink/streaming/StreamSources.java | 62 --
 .../flink/FlinkBatchTransformTranslators.java  | 10 +---
 .../beam/runners/flink/FlinkPipelineRunner.java|  8 ---
 .../org/apache/beam/runners/flink/FlinkRunner.java |  7 ---
 .../translation/functions/FlinkDoFnFunction.java   | 12 +
 .../translation/types/CoderTypeSerializer.java |  0
 .../translation/types/EncodedValueSerializer.java  |  0
 .../streaming/io/BeamStoppableFunction.java|  0
 .../flink/batch/NonMergingGroupByKeyTest.java  |  5 --
 .../translation/types/CoderTypeSerializerTest.java |  0
 settings.gradle.kts|  8 ---
 .../site/content/en/documentation/runners/flink.md | 25 ++---
 31 files changed, 29 insertions(+), 494 deletions(-)

diff --cc 
runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
index 396000b,96b8781..b911567
--- 
a/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
+++ 
b/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPipelineRunner.java
@@@ -43,10 -43,9 +43,9 @@@ import org.apache.beam.sdk.metrics.Metr
  import org.apache.beam.sdk.metrics.MetricsOptions;
  import org.apache.beam.sdk.options.PipelineOptions;
  import org.apache.beam.sdk.options.PipelineOptionsFactory;
 -import org.apache.beam.vendor.grpc.v1p26p0.com.google.protobuf.Struct;
 +import org.apache.beam.vendor.grpc.v1p36p0.com.google.protobuf.Struct;
  import 
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.base.Preconditions;
  import org.apache.flink.api.common.JobExecutionResult;
- import org.apache.flink.runtime.util.EnvironmentInformation;
  import org.checkerframework.checker.nullness.qual.Nullable;
  import org.kohsuke.args4j.CmdLineException;
  import org.kohsuke.args4j.CmdLineParser;


[beam] branch master updated (cb31b7b -> 572a99b)

2021-04-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from cb31b7b  [BEAM-7372] cleanup codes for py2 compatibility from 
apache_beam/examples/snippets/*.py and apache_beam/examples/*.py (#1)
 add 05b3fd3  [BEAM-11948] Drop support for Flink 1.8 and 1.9
 new 572a99b  Merge pull request #14203: [BEAM-11948] Drop support for 
Flink 1.8 and 1.9

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 CHANGES.md |  1 +
 gradle.properties  |  2 +-
 runners/flink/1.10/build.gradle|  4 +-
 .../beam/runners/flink/FlinkCapabilities.java  | 34 
 .../streaming/AbstractStreamOperatorCompat.java|  0
 .../beam/runners/flink/RemoteMiniClusterImpl.java  |  0
 .../runners/flink/SourceTransformationCompat.java  |  0
 runners/flink/1.11/build.gradle|  4 +-
 runners/flink/1.12/build.gradle|  4 +-
 runners/flink/1.8/build.gradle | 34 
 .../flink/1.8/job-server-container/build.gradle| 26 -
 runners/flink/1.8/job-server/build.gradle  | 31 ---
 .../beam/runners/flink/FlinkCapabilities.java  | 34 
 .../streaming/io/BeamStoppableFunction.java| 29 --
 .../beam/runners/flink/FlinkRunnerTestCompat.java  | 43 ---
 .../runners/flink/streaming/StreamSources.java | 50 -
 runners/flink/1.9/build.gradle | 33 
 .../flink/1.9/job-server-container/build.gradle| 26 -
 runners/flink/1.9/job-server/build.gradle  | 31 ---
 .../runners/flink/streaming/StreamSources.java | 62 --
 .../flink/FlinkBatchTransformTranslators.java  | 10 +---
 .../beam/runners/flink/FlinkPipelineRunner.java|  8 ---
 .../org/apache/beam/runners/flink/FlinkRunner.java |  7 ---
 .../translation/functions/FlinkDoFnFunction.java   | 12 +
 .../translation/types/CoderTypeSerializer.java |  0
 .../translation/types/EncodedValueSerializer.java  |  0
 .../streaming/io/BeamStoppableFunction.java|  0
 .../flink/batch/NonMergingGroupByKeyTest.java  |  5 --
 .../translation/types/CoderTypeSerializerTest.java |  0
 settings.gradle.kts|  8 ---
 .../site/content/en/documentation/runners/flink.md | 25 ++---
 31 files changed, 29 insertions(+), 494 deletions(-)
 delete mode 100644 
runners/flink/1.10/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
 rename runners/flink/{1.8 => 
1.10}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
 (100%)
 rename runners/flink/{1.8 => 
1.10}/src/test/java/org/apache/beam/runners/flink/RemoteMiniClusterImpl.java 
(100%)
 rename runners/flink/{1.8 => 
1.10}/src/test/java/org/apache/beam/runners/flink/SourceTransformationCompat.java
 (100%)
 delete mode 100644 runners/flink/1.8/build.gradle
 delete mode 100644 runners/flink/1.8/job-server-container/build.gradle
 delete mode 100644 runners/flink/1.8/job-server/build.gradle
 delete mode 100644 
runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
 delete mode 100644 
runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
 delete mode 100644 
runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/FlinkRunnerTestCompat.java
 delete mode 100644 
runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
 delete mode 100644 runners/flink/1.9/build.gradle
 delete mode 100644 runners/flink/1.9/job-server-container/build.gradle
 delete mode 100644 runners/flink/1.9/job-server/build.gradle
 delete mode 100644 
runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
 rename runners/flink/{1.8 => 
}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
 (100%)
 rename runners/flink/{1.8 => 
}/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
 (100%)
 rename runners/flink/{1.9 => 
}/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
 (100%)
 rename runners/flink/{1.8 => 
}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
 (100%)


[beam] branch master updated (86ac487 -> 968abf4)

2021-04-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 86ac487  [BEAM-11961] InfluxDBIOIT failing with unauthorized error 
(#14215)
 add e3d2654  [BEAM-12066] Bump classgraph to 4.8.104
 add 968abf4  Merge pull request #14443: [BEAM-12066] Bump classgraph to 
4.8.104

No new revisions were added by this update.

Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (3216fcb -> b183b9a)

2021-04-07 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3216fcb  Merge pull request #14412 from [BEAM-8696] protobuf-java 
3.14.0 in line with libraries BOM 16.3.0
 add b8ff868  [BEAM-12095] Add unit tests for path_to_beam_jar(artifact_id).
 add b183b9a  Merge pull request #14448: [BEAM-12095] Add unit tests for 
path_to_beam_jar(artifact_id)

No new revisions were added by this update.

Summary of changes:
 .../apache_beam/utils/subprocess_server_test.py| 22 ++
 1 file changed, 22 insertions(+)


[beam] branch master updated (abbe14f -> 7134cfd)

2021-04-05 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from abbe14f  [BEAM-12083] Nexmark Query 13. (#14404)
 add 37db190  [BEAM-4106] Add FileStagingOptions and merge staging file 
options between runners
 add 7134cfd  Merge pull request #14423: [BEAM-4106] Add FileStagingOptions 
and merge staging file options between runners

No new revisions were added by this update.

Summary of changes:
 .../beam/runners/flink/FlinkPipelineOptions.java   | 18 ++---
 .../options/DataflowPipelineWorkerPoolOptions.java | 19 ++
 .../runners/spark/SparkCommonPipelineOptions.java  | 18 +++--
 .../runners/twister2/Twister2PipelineOptions.java  | 11 +++-
 .../beam/sdk/options/FileStagingOptions.java}  | 30 +++---
 .../beam/sdk/options/PortablePipelineOptions.java  | 18 +
 6 files changed, 26 insertions(+), 88 deletions(-)
 copy 
sdks/java/{extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlSeekableTable.java
 => core/src/main/java/org/apache/beam/sdk/options/FileStagingOptions.java} 
(58%)


[beam] branch master updated (e5b3333 -> 5f13eff)

2021-04-02 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e5b  Merge pull request #14414 [BEAM-10663] Re-enable 
CrossLanguageKafkaIOTest
 add 32af8d2  [BEAM-12088] Make file staging uniform among Spark Runners
 add 5f13eff  Merge pull request #14417: [BEAM-12088] Make file staging 
uniform among Spark Runners

No new revisions were added by this update.

Summary of changes:
 .../SparkStructuredStreamingRunner.java|  7 ++--
 .../translation/PipelineTranslator.java| 21 
 .../runners/spark/SparkCommonPipelineOptions.java  | 36 +
 .../beam/runners/spark/SparkPipelineOptions.java   | 37 --
 .../beam/runners/spark/SparkPipelineRunner.java|  2 +-
 .../org/apache/beam/runners/spark/SparkRunner.java |  6 ++--
 6 files changed, 44 insertions(+), 65 deletions(-)


[beam] branch master updated (2c619c8 -> 1e60f38)

2021-04-02 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2c619c8  [BEAM-7372] delete codes for compatibility of py2 from 
apache_beam/io except for io/gcp (#14352)
 add 9e0b378  [BEAM-11213] Display Beam Metrics in Spark History Server for 
Classic Runner too
 add 1e60f38  Merge pull request #14409: [BEAM-11213] Display Beam Metrics 
in Spark History Server for Classic Runner too

No new revisions were added by this update.

Summary of changes:
 .../beam/runners/spark/SparkPipelineOptions.java   | 12 
 .../beam/runners/spark/SparkPipelineRunner.java| 39 ++-
 .../org/apache/beam/runners/spark/SparkRunner.java | 59 +++-
 .../beam/runners/spark/util/SparkCommon.java   | 79 ++
 .../beam/runners/spark/util/SparkCompat.java   |  6 +-
 .../spark/metrics/SparkMetricsPusherTest.java  |  2 +-
 6 files changed, 131 insertions(+), 66 deletions(-)
 create mode 100644 
runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCommon.java


[beam] branch master updated (288d379 -> ec9e413)

2021-03-31 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 288d379  Merge pull request #14386 from iemejia/update-python-deps
 add 50031c6  Update website build dependencies
 add ec9e413  Merge pull request #14387: Update website build dependencies

No new revisions were added by this update.

Summary of changes:
 website/www/yarn.lock | 355 +++---
 1 file changed, 164 insertions(+), 191 deletions(-)


[beam] branch master updated (fd3075f -> 460e093)

2021-03-31 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from fd3075f  Merge pull request #13995 from [BEAM-11322] Apache Beam 
Example to tokenize sensitive data
 add 6eb3c24  Update dependencies
 add eb8ceeb  Update dependencies
 add 460e093  Merge pull request #14379: Update python dependencies

No new revisions were added by this update.

Summary of changes:
 .test-infra/jenkins/metrics_report/requirements.txt | 2 +-
 sdks/python/container/base_image_requirements.txt   | 4 ++--
 sdks/python/setup.py| 2 +-
 3 files changed, 4 insertions(+), 4 deletions(-)


[beam] branch master updated (a16bbf7 -> 7eff49f)

2021-03-30 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a16bbf7  Merge pull request #14339: Roll forward PJS with Pubsub fix
 add 4a1b0aa  [BEAM-7093] Add Spark 3 and Java 11 info into the CHANGES file
 add 7eff49f  Merge pull request #14378: [BEAM-7093] Add Spark 3 and Java 
11 info into the CHANGES file

No new revisions were added by this update.

Summary of changes:
 CHANGES.md | 2 ++
 1 file changed, 2 insertions(+)


[beam] branch master updated (475bbbc -> 776bf54)

2021-03-26 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 475bbbc  Merge pull request #14346: [BEAM-12054] CassandraIO: flush 
mutator in @FinishBundle
 add d1f4808  [BEAM-12050] Sickbay portable Flink tests of event time 
timers with TestStream
 add 776bf54  Merge pull request #14333: [BEAM-12050] Sickbay portable 
Flink tests of event time timers with TestStream

No new revisions were added by this update.

Summary of changes:
 runners/flink/job-server/flink_job_server.gradle | 4 
 1 file changed, 4 insertions(+)


[beam] branch master updated (a2da64f -> 75460dc)

2021-03-25 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from a2da64f  [BEAM-11935] Updates Dataflow SDK Harness map to set 
Environment ID (#14189)
 add 54c75db  [BEAM-11815] AWS SDK v1, DynamoDBIO should read all data with 
limits
 add ce214c0  [BEAM-11815] AWS SDK v2, DynamoDBIO reads all data with limits
 add 75460dc  Merge pull request #14314: [BEAM-11815] DynamoDBIO should 
read all data with limits

No new revisions were added by this update.

Summary of changes:
 .../beam/sdk/io/aws/dynamodb/DynamoDBIO.java   | 17 ++---
 .../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java   | 34 ++
 .../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java  | 23 
 .../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java  | 41 ++
 4 files changed, 104 insertions(+), 11 deletions(-)


[beam] branch master updated (8ac1eb5 -> 5bfdc3b)

2021-03-19 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 8ac1eb5  Merge pull request #14282: [BEAM-7093] Update some 
out-of-date Gradle Spark instructions.
 add b661b87  [BEAM-7078] Bump com.amazonaws:amazon-kinesis-client to 
version 1.14.2
 add 5bfdc3b  Merge pull request #14283: [BEAM-7078] Bump 
com.amazonaws:amazon-kinesis-client to version 1.14.2

No new revisions were added by this update.

Summary of changes:
 sdks/java/io/kinesis/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


[beam] branch master updated (d6b020a -> 8ac1eb5)

2021-03-19 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from d6b020a  Merge pull request #14279 from apache/tvalentyn-patch-1
 add 209fd5d  [BEAM-7093] Update some out-of-date Gradle Spark instructions.
 add 8ac1eb5  Merge pull request #14282: [BEAM-7093] Update some 
out-of-date Gradle Spark instructions.

No new revisions were added by this update.

Summary of changes:
 sdks/python/apache_beam/runners/portability/spark_runner.py | 2 +-
 .../python/apache_beam/runners/portability/spark_uber_jar_job_server.py | 2 +-
 website/www/site/content/en/get-started/quickstart-go.md| 2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)


[beam] 01/01: Merge pull request #14269: [BEAM-11992] Run CrossLanguage ValidatesRunner for Spark 3

2021-03-19 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 40eef355c8321bf6f376c68043db4c589bad78d9
Merge: 22a8b18 ba14ca2
Author: Ismaël Mejía 
AuthorDate: Fri Mar 19 18:42:30 2021 +0100

Merge pull request #14269: [BEAM-11992] Run CrossLanguage ValidatesRunner 
for Spark 3

 ...mmit_CrossLanguageValidatesRunner_Spark3.groovy | 49 ++
 README.md  |  2 +-
 release/src/main/scripts/mass_comment.py   |  1 +
 3 files changed, 51 insertions(+), 1 deletion(-)


[beam] branch master updated (22a8b18 -> 40eef35)

2021-03-19 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 22a8b18  Add an option to create Dataflow piplines from a snapshot for 
python sdk (#14278)
 add ba14ca2  [BEAM-11992] Run CrossLanguage ValidatesRunner for Spark 3
 new 40eef35  Merge pull request #14269: [BEAM-11992] Run CrossLanguage 
ValidatesRunner for Spark 3

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 ...ob_PostCommit_CrossLanguageValidatesRunner_Spark3.groovy} | 12 ++--
 README.md|  2 +-
 release/src/main/scripts/mass_comment.py |  1 +
 3 files changed, 8 insertions(+), 7 deletions(-)
 copy 
.test-infra/jenkins/{job_PostCommit_CrossLanguageValidatesRunner_Direct.groovy 
=> job_PostCommit_CrossLanguageValidatesRunner_Spark3.groovy} (84%)


[beam] branch master updated (e96f69a -> 5c9c8c0)

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from e96f69a  Revert "Increase the timeout  for ValidatesContainer suite" 
(#14219)
 add ff4366d  [BEAM-11023] Fix testGroupByKeyWithBadEqualsHashCode failing 
on Spark Structured Streaming runner
 add 96bf00a  [BEAM-11023] Fix testLargeKeys100MB on Spark Structured 
Streaming runner
 add 3ac902a  [BEAM-11023] Change access level GroupByKeyTest
 new 5c9c8c0  Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest 
testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark 
Structured Streaming runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 runners/spark/spark_runner.gradle | 2 ++
 .../src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java  | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)


[beam] 01/01: Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark Structured Streaming runner

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5c9c8c0d1ef0f8224fac77265873d5f6afc431d2
Merge: e96f69a 3ac902a
Author: Ismaël Mejía 
AuthorDate: Thu Mar 18 21:53:03 2021 +0100

Merge pull request #14275: [BEAM-11023] Fix GroupByKeyTest 
testLargeKeys100MB and testGroupByKeyWithBadEqualsHashCode failing on Spark 
Structured Streaming runner

 runners/spark/spark_runner.gradle | 2 ++
 .../src/test/java/org/apache/beam/sdk/transforms/GroupByKeyTest.java  | 4 ++--
 2 files changed, 4 insertions(+), 2 deletions(-)


[beam] branch master updated (3fc2ab1 -> aee24ad)

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 3fc2ab1  [BEAM-9547] Add NDFrame to doctests, implement a few more 
operations (#14236)
 add 9113c6f  [BEAM-9283] Add Java 11 Jpms compatibility tests for Spark 
runner
 new aee24ad  Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms 
compatibility tests for Spark runner

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 ...Java11.groovy => job_PostCommit_Java_Jpms_Spark_Java11.groovy} | 8 
 sdks/java/testing/jpms-tests/build.gradle | 8 
 2 files changed, 12 insertions(+), 4 deletions(-)
 copy .test-infra/jenkins/{job_PostCommit_Java_Jpms_Flink_Java11.groovy => 
job_PostCommit_Java_Jpms_Spark_Java11.groovy} (85%)


[beam] 01/01: Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms compatibility tests for Spark runner

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit aee24ad4aa78d5b1a5258a1003c08714808edcd5
Merge: 3fc2ab1 9113c6f
Author: Ismaël Mejía 
AuthorDate: Thu Mar 18 19:13:05 2021 +0100

Merge pull request #14270: [BEAM-9283] Add Java 11 Jpms compatibility tests 
for Spark runner

 .../job_PostCommit_Java_Jpms_Spark_Java11.groovy   | 49 ++
 sdks/java/testing/jpms-tests/build.gradle  |  8 
 2 files changed, 57 insertions(+)


[beam] branch master updated (540957e -> c4f62ca)

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 540957e  Merge pull request #14253: [BEAM-11992] Run CrossLanguage 
ValidatesRunner tests only for Spark 2
 add 6db22c6  [BEAM-8778] Bump software.amazon.awssdk version to 2.15.31
 new c4f62ca  Merge pull request #14264: [BEAM-8778] Bump 
software.amazon.awssdk version to 2.15.31

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy |  5 -
 sdks/java/io/{kinesis => amazon-web-services}/OWNERS   |  0
 sdks/java/io/{kinesis => amazon-web-services2}/OWNERS  |  0
 sdks/java/io/amazon-web-services2/build.gradle | 10 --
 4 files changed, 8 insertions(+), 7 deletions(-)
 copy sdks/java/io/{kinesis => amazon-web-services}/OWNERS (100%)
 copy sdks/java/io/{kinesis => amazon-web-services2}/OWNERS (100%)



[beam] 01/01: Merge pull request #14264: [BEAM-8778] Bump software.amazon.awssdk version to 2.15.31

2021-03-18 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit c4f62ca933b570ec300f501dc02f360054d5634a
Merge: 540957e 6db22c6
Author: Ismaël Mejía 
AuthorDate: Thu Mar 18 12:40:45 2021 +0100

Merge pull request #14264: [BEAM-8778] Bump software.amazon.awssdk version 
to 2.15.31

 .../main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy |  5 -
 sdks/java/io/amazon-web-services/OWNERS|  4 
 sdks/java/io/amazon-web-services2/OWNERS   |  4 
 sdks/java/io/amazon-web-services2/build.gradle | 10 --
 4 files changed, 16 insertions(+), 7 deletions(-)



[beam] 01/01: Merge pull request #14253: [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for Spark 2

2021-03-17 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 540957effe63ebe093dd1da82ae7f0d8f9aa482a
Merge: 9791ef9 abc7b20
Author: Ismaël Mejía 
AuthorDate: Thu Mar 18 07:12:57 2021 +0100

Merge pull request #14253: [BEAM-11992] Run CrossLanguage ValidatesRunner 
tests only for Spark 2

 .../jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy | 1 -
 1 file changed, 1 deletion(-)



[beam] branch master updated (9791ef9 -> 540957e)

2021-03-17 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 9791ef9  Merge pull request #14237 from [BEAM-11979] Ignore not 
serializable filter fields in python MongoDBI…
 add abc7b20  [BEAM-11992] Run CrossLanguage ValidatesRunner tests only for 
Spark 2
 new 540957e  Merge pull request #14253: [BEAM-11992] Run CrossLanguage 
ValidatesRunner tests only for Spark 2

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../jenkins/job_PostCommit_CrossLanguageValidatesRunner_Spark.groovy | 1 -
 1 file changed, 1 deletion(-)



[beam] branch master updated (25bad0e -> 8d6fa737)

2021-03-16 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 25bad0e  Merge pull request #14246: [BEAM-11726] Bump Clickhouse 
version to 0.2.6
 add 7af3d13  [BEAM-11764] Bump com.amazonaws version to 1.11.974
 add 8d6fa737 Merge pull request #14247: [BEAM-11764] Bump com.amazonaws 
version to 1.11.974

No new revisions were added by this update.

Summary of changes:
 buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)



[beam] branch master updated (7ff7ceb -> 25bad0e)

2021-03-16 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 7ff7ceb  Create 2.30.0 entry in CHANGES.md (#14250)
 add 3d4518f  [BEAM-11726] Bump Clickhouse version to "0.2.6"
 add 25bad0e  Merge pull request #14246: [BEAM-11726] Bump Clickhouse 
version to 0.2.6

No new revisions were added by this update.

Summary of changes:
 sdks/java/io/clickhouse/build.gradle | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)



[beam] 01/01: Merge pull request #14217: [BEAM-8221] Fix NPE while reading from non-existent Kafka topic

2021-03-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 08a9d54b60cd4743afeb61e697fa6016279b1ac6
Merge: 88d687b b2ce15e
Author: Ismaël Mejía 
AuthorDate: Mon Mar 15 11:39:55 2021 +0100

Merge pull request #14217: [BEAM-8221] Fix NPE while reading from 
non-existent Kafka topic

 .../beam/sdk/io/kafka/KafkaUnboundedSource.java|  7 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 26 ++
 2 files changed, 32 insertions(+), 1 deletion(-)



[beam] branch master updated (88d687b -> 08a9d54)

2021-03-15 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 88d687b  Merge pull request #14228: [BEAM-7092] Fix artifact names for 
Spark separated modules + upgrade to Spark 3.1.1
 add b2ce15e  [BEAM-8221] Fix NPE while reading from non-existent Kafka 
topic
 new 08a9d54  Merge pull request #14217: [BEAM-8221] Fix NPE while reading 
from non-existent Kafka topic

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../beam/sdk/io/kafka/KafkaUnboundedSource.java|  7 +-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 26 ++
 2 files changed, 32 insertions(+), 1 deletion(-)



[beam] branch master updated (153876f -> 88d687b)

2021-03-14 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 153876f  Merge pull request #14216: [BEAM-7093] Support Spark 3 in 
Spark runner
 add b388776  [BEAM-7092] Upgrade Spark 3 to version 3.1.1
 add f06ffcb  [BEAM-7092] Fix artifact name for Spark 2 runner to be the 
same as before
 add 1fdf315  [BEAM-7092] Make Spark 3 runner name consistent with other 
modules '-$VERSION'
 add 88d687b  Merge pull request #14228: [BEAM-7092] Fix artifact names for 
Spark separated modules + upgrade to Spark 3.1.1

No new revisions were added by this update.

Summary of changes:
 runners/spark/3/build.gradle | 4 ++--
 runners/spark/3/job-server/build.gradle  | 2 +-
 runners/spark/job-server/spark_job_server.gradle | 1 +
 runners/spark/spark_runner.gradle| 2 ++
 sdks/python/test-suites/portable/common.gradle   | 2 +-
 5 files changed, 7 insertions(+), 4 deletions(-)



[beam] branch master updated (c925ff8 -> 153876f)

2021-03-13 Thread iemejia
This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from c925ff8  Merge pull request #14154: [BEAM-11910] Increase the bag page 
limit for continuation pages
 add 9c0f608  [BEAM-9282] Move structured streaming runner into Spark 2 
specific module
 add 47fd176  [BEAM-9282] Separate modules for Spark 2/3
 add 17710e5  [BEAM-9282] Separate modules for Spark 2/3 job-server
 add e86862c  [BEAM-9282] Separate modules for Spark 2/3 job-server 
container
 add 44b7a87  [BEAM-7092] Run PostCommit tests for Spark 3 module too
 add cc29b5b  [BEAM-7092] Update tests invocation for Spark 2 module
 add b234bb4  [BEAM-9283] Add Spark 3 test jobs to the CI (Java 11)
 add 07e52fc  [BEAM-11654] Publish Spark 2 and 3 specific Job-Server 
containers
 add d9f3157  [BEAM-7092] Add paranamer 2.8 license to container (Spark 3 / 
Avro)
 add 153876f  Merge pull request #14216: [BEAM-7093] Support Spark 3 in 
Spark runner

No new revisions were added by this update.

Summary of changes:
 .test-infra/jenkins/CommonTestProperties.groovy|  4 +-
 ...ommit_CrossLanguageValidatesRunner_Spark.groovy |  3 +-
 .../job_PostCommit_Java_Nexmark_Spark.groovy   |  8 +-
 ...Java_PortableValidatesRunner_Spark_Batch.groovy |  3 +-
 ..._PortableValidatesRunner_Spark_Streaming.groovy |  3 +-
 ...ob_PostCommit_Java_ValidatesRunner_Spark.groovy |  3 +-
 ...ValidatesRunner_SparkStructuredStreaming.groovy |  2 +-
 ...ommit_Java_ValidatesRunner_Spark_Java11.groovy} | 13 ++-
 build.gradle.kts   |  5 +-
 .../org/apache/beam/gradle/BeamModulePlugin.groovy |  7 +-
 examples/java/build.gradle |  2 +-
 examples/kotlin/build.gradle   |  2 +-
 release/build.gradle.kts   |  2 +-
 release/src/main/scripts/run_rc_validation.sh  |  2 +-
 runners/core-construction-java/build.gradle|  3 +
 runners/{flink/1.8 => spark/2}/build.gradle| 13 +--
 .../{flink/1.9 => spark/2}/job-server/build.gradle |  4 +-
 .../2/job-server/container}/build.gradle   |  5 +-
 .../SparkStructuredStreamingPipelineOptions.java   |  0
 .../SparkStructuredStreamingPipelineResult.java|  0
 .../SparkStructuredStreamingRunner.java|  0
 .../SparkStructuredStreamingRunnerRegistrar.java   |  0
 .../aggregators/AggregatorsAccumulator.java|  0
 .../aggregators/NamedAggregators.java  |  0
 .../aggregators/NamedAggregatorsAccumulator.java   |  0
 .../aggregators/package-info.java  |  0
 .../structuredstreaming/examples/WordCount.java|  0
 .../metrics/AggregatorMetric.java  |  0
 .../metrics/AggregatorMetricSource.java|  0
 .../metrics/CompositeSource.java   |  0
 .../metrics/MetricsAccumulator.java|  0
 .../MetricsContainerStepMapAccumulator.java|  0
 .../metrics/SparkBeamMetric.java   |  0
 .../metrics/SparkBeamMetricSource.java |  0
 .../metrics/SparkMetricsContainerStepMap.java  |  0
 .../metrics/WithMetricsSupport.java|  0
 .../structuredstreaming/metrics/package-info.java  |  0
 .../metrics/sink/CodahaleCsvSink.java  |  0
 .../metrics/sink/CodahaleGraphiteSink.java |  0
 .../metrics/sink/package-info.java |  0
 .../spark/structuredstreaming/package-info.java|  0
 .../translation/PipelineTranslator.java|  0
 .../translation/SchemaHelpers.java |  0
 .../translation/SparkTransformOverrides.java   |  0
 .../translation/TransformTranslator.java   |  0
 .../translation/TranslationContext.java|  0
 .../translation/batch/AggregatorCombiner.java  |  0
 .../batch/CombinePerKeyTranslatorBatch.java|  0
 .../CreatePCollectionViewTranslatorBatch.java  |  0
 .../translation/batch/DatasetSourceBatch.java  |  0
 .../translation/batch/DoFnFunction.java|  0
 .../translation/batch/DoFnRunnerWithMetrics.java   |  0
 .../translation/batch/FlattenTranslatorBatch.java  |  0
 .../batch/GroupByKeyTranslatorBatch.java   |  0
 .../translation/batch/ImpulseTranslatorBatch.java  |  0
 .../translation/batch/ParDoTranslatorBatch.java|  0
 .../translation/batch/PipelineTranslatorBatch.java |  0
 .../translation/batch/ProcessContext.java  |  0
 .../batch/ReadSourceTranslatorBatch.java   |  0
 .../batch/ReshuffleTranslatorBatch.java|  0
 .../batch/WindowAssignTranslatorBatch.java |  0
 .../GroupAlsoByWindowViaOutputBufferFn.java|  0
 .../batch/functions/NoOpStepContext.java   |  0
 .../batch/functions/SparkSideInputReader.java  |  0
 .../translation/batch/functions/package-info.java  |  0
 .../translation/batch/package-info.java|  0
 .../translation/helpers/CoderHelper

  1   2   3   4   5   6   7   8   9   10   >