[beam] branch master updated (113461a -> 0dc2ad2)
This is an automated email from the ASF dual-hosted git repository.
mxm pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 113461a [BEAM-8160] Add FnApi execution mode instruction (#9628)
add ce27e30 [BEAM-7962] Drop support for Flink 1.5 and 1.6
add 661237c [BEAM-7962] Update version compatibility section on Flink
Runner page
add 0dc2ad2 Merge pull request #9632: [BEAM-7962] Drop support for Flink
1.5 and 1.6
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/CommonTestProperties.groovy| 2 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 8 ++---
...Java_PortableValidatesRunner_Flink_Batch.groovy | 2 +-
..._PortableValidatesRunner_Flink_Streaming.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 6 ++--
examples/java/build.gradle | 2 +-
examples/kotlin/build.gradle | 2 +-
project-mappings | 6 ++--
release/build.gradle | 2 +-
release/src/main/scripts/run_rc_validation.sh | 2 +-
runners/flink/1.5/build.gradle | 34 --
.../flink/1.5/job-server-container/build.gradle| 26 -
runners/flink/1.5/job-server/build.gradle | 31
runners/flink/1.6/build.gradle | 34 --
.../flink/1.6/job-server-container/build.gradle| 26 -
runners/flink/1.6/job-server/build.gradle | 31
runners/flink/1.7/build.gradle | 4 +--
.../translation/types/CoderTypeSerializer.java | 0
.../translation/types/EncodedValueSerializer.java | 0
.../FlinkBroadcastStateInternalsTest.java | 0
.../flink/streaming/FlinkStateInternalsTest.java | 0
.../translation/types/CoderTypeSerializerTest.java | 0
runners/flink/flink_runner.gradle | 11 ++-
sdks/go/test/build.gradle | 4 +--
sdks/java/testing/nexmark/build.gradle | 2 +-
.../runners/portability/flink_runner.py| 2 +-
sdks/python/test-suites/portable/py2/build.gradle | 14 -
sdks/python/test-suites/portable/py35/build.gradle | 2 +-
sdks/python/test-suites/portable/py36/build.gradle | 2 +-
sdks/python/test-suites/portable/py37/build.gradle | 2 +-
settings.gradle| 8 -
website/src/contribute/release-guide.md| 2 +-
website/src/documentation/dsls/sql/shell.md| 6 ++--
website/src/documentation/runners/flink.md | 29 +++---
website/src/documentation/sdks/nexmark.md | 6 ++--
36 files changed, 68 insertions(+), 244 deletions(-)
delete mode 100644 runners/flink/1.5/build.gradle
delete mode 100644 runners/flink/1.5/job-server-container/build.gradle
delete mode 100644 runners/flink/1.5/job-server/build.gradle
delete mode 100644 runners/flink/1.6/build.gradle
delete mode 100644 runners/flink/1.6/job-server-container/build.gradle
delete mode 100644 runners/flink/1.6/job-server/build.gradle
rename runners/flink/{1.5 =>
1.7}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
(100%)
rename runners/flink/{1.5 =>
1.7}/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
(100%)
rename runners/flink/{1.5 =>
1.7}/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
(100%)
rename runners/flink/{1.5 =>
1.7}/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
(100%)
rename runners/flink/{1.5 =>
1.7}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
(100%)
[beam] branch asf-site updated: Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
This is an automated email from the ASF dual-hosted git repository.
git-site-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/asf-site by this push:
new 345c0dd Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
345c0dd is described below
commit 345c0ddcb93eae5002b0e616b576c9a96c8a0ca4
Author: jenkins
AuthorDate: Tue Sep 24 09:29:33 2019 +
Publishing website 2019/09/24 09:29:33 at commit 0dc2ad2
---
.../contribute/release-guide/index.html| 2 +-
.../documentation/dsls/sql/shell/index.html| 6 ++---
.../documentation/runners/flink/index.html | 29 +++---
.../sdks/java/testing/nexmark/index.html | 6 ++---
4 files changed, 32 insertions(+), 11 deletions(-)
diff --git a/website/generated-content/contribute/release-guide/index.html
b/website/generated-content/contribute/release-guide/index.html
index 40403b8..8f4a0fa 100644
--- a/website/generated-content/contribute/release-guide/index.html
+++ b/website/generated-content/contribute/release-guide/index.html
@@ -1601,7 +1601,7 @@ Thanks everyone!
Flink Local Runner
-./gradlew
:runners:flink:1.5:runQuickstartJavaFlinkLocal \
+./gradlew
:runners:flink:1.8:runQuickstartJavaFlinkLocal \
-Prepourl=https://repository.apache.org/content/repositories/orgapachebeam-${KEY}
\
-Pver=${RELEASE_VERSION}
diff --git a/website/generated-content/documentation/dsls/sql/shell/index.html
b/website/generated-content/documentation/dsls/sql/shell/index.html
index ec1fb7a..bed0830 100644
--- a/website/generated-content/documentation/dsls/sql/shell/index.html
+++ b/website/generated-content/documentation/dsls/sql/shell/index.html
@@ -301,7 +301,7 @@ limitations under the License.
To use Beam SQL shell, you must first clone the https://github.com/apache/beam;>Beam SDK repository. Then, from the
root of the repository clone, execute the following commands to run the
shell:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
./sdks/java/extensions/sql/shell/build/install/shell/bin/shell
@@ -393,7 +393,7 @@ No rows affected (0.042 seconds)
Make sure the SQL shell includes the desired runner. Add the
corresponding project id to the -Pbeam.sql.shell.bundled parameter of the
Gradle invocation (https://github.com/apache/beam/blob/master/sdks/java/extensions/sql/shell/build.gradle;>source
code, https://github.com/apache/beam/blob/master/settings.gradle;>project
ids). For example, use the following command to include Flink runner and
KafkaIO:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' installDist
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' installDist
@@ -422,7 +422,7 @@ No rows affected (0.042 seconds)
You can also build your own standalone package for SQL shell using distZip or distTar tasks. For example:
-./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.5,:sdks:java:io:kafka' distZip
+./gradlew -p
sdks/java/extensions/sql/shell
-Pbeam.sql.shell.bundled=':runners:flink:1.8,:sdks:java:io:kafka' distZip
ls ./sdks/java/extensions/sql/shell/build/distributions/
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.tar
beam-sdks-java-extensions-sql-shell-2.6.0-SNAPSHOT.zip
diff --git a/website/generated-content/documentation/runners/flink/index.html
b/website/generated-content/documentation/runners/flink/index.html
index b3ad611..921907c 100644
--- a/website/generated-content/documentation/runners/flink/index.html
+++ b/website/generated-content/documentation/runners/flink/index.html
@@ -319,12 +319,33 @@ period.
Artifact Id
- =2.13.0
+ 2.17.0
1.8.x
beam-runners-flink-1.8
- =2.10.0
+ 1.7.x
+ beam-runners-flink-1.7
+
+
+ 2.13.0 - 2.16.0
+ 1.8.x
+ beam-runners-flink-1.8
+
+
+ 1.7.x
+ beam-runners-flink-1.7
+
+
+ 1.6.x
+ beam-runners-flink-1.6
+
+
+ 1.5.x
+ beam-runners-flink_2.11
+
+
+ 2.10.0 - 2.16.0
1.7.x
beam-runners-flink-1.7
@@ -464,7 +485,7 @@ download it on the Downloads page. In the
available. To run a pipeline on an embedded Flink cluster:
-1. Start the JobService endpoint: ./gradlew
:runners:flink:1.5:job-server:runShadow
+1. Start the JobService endpoint: ./gradlew
:runners:flink:1.8:job-server:runShadow
@@ -497,7 +518,7 @@ To run on a separate https://ci.apache.org/projects/flink/flink-docs-re
1. Start a Flink cluster which exposes the Rest
interface on localhost:8081 by default.
-2. Start JobService with Flink Rest endpoint:
./gradlew
:runners:flink:1.5:job-server:runShadow
[beam] branch master updated (0dc2ad2 -> fce6380)
This is an automated email from the ASF dual-hosted git repository. mxm pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 0dc2ad2 Merge pull request #9632: [BEAM-7962] Drop support for Flink 1.5 and 1.6 add c4f4109 [BEAM-8157] Introduce encode_nested method on Python SDK Coder add 61b8fa0 [BEAM-8157] Ensure key encoding for state requests is consistent across SDKs add fce6380 Merge pull request #9484: [BEAM-8157] Ensure key encoding for state requests is consistent across SDKs No new revisions were added by this update. Summary of changes: .../FlinkStreamingPortablePipelineTranslator.java | 6 - .../streaming/ExecutableStageDoFnOperator.java | 8 ++- .../wrappers/streaming/FlinkKeyUtils.java | 11 ++--- .../streaming/state/FlinkStateInternals.java | 12 ++ .../wrappers/streaming/DoFnOperatorTest.java | 12 +- .../wrappers/streaming/FlinkKeyUtilsTest.java | 26 +++--- .../beam/fn/harness/state/FnApiStateAccessor.java | 6 - sdks/python/apache_beam/coders/coders.py | 8 +++ .../apache_beam/runners/worker/bundle_processor.py | 6 +++-- 9 files changed, 53 insertions(+), 42 deletions(-)
[beam] branch master updated (6e3c352 -> d61c227)
This is an automated email from the ASF dual-hosted git repository. lgajowy pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6e3c352 Merge pull request #9449: [BEAM-7660] Create Python ParDo load test job on Flink add b063a9e [BEAM-8256] Set fixed number of workers for file-based IOITs add d61c227 Merge pull request #9596: [BEAM-8256] Set fixed number of workers for Java IOITs No new revisions were added by this update. Summary of changes: .../job_PerformanceTests_FileBasedIO_IT.groovy | 108 + .../job_PerformanceTests_HadoopFormat.groovy | 26 ++--- .../jenkins/job_PerformanceTests_JDBC.groovy | 26 ++--- .../job_PerformanceTests_MongoDBIO_IT.groovy | 20 ++-- 4 files changed, 107 insertions(+), 73 deletions(-)
[beam] branch master updated (fce6380 -> 6e3c352)
This is an automated email from the ASF dual-hosted git repository.
lgajowy pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from fce6380 Merge pull request #9484: [BEAM-8157] Ensure key encoding for
state requests is consistent across SDKs
add cb3772c [BEAM-7660] Create Python ParDo load test job on Flink
add 8bb6982 [BEAM-7660] Parameter names changes
add 6e3c352 Merge pull request #9449: [BEAM-7660] Create Python ParDo
load test job on Flink
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/LoadTestsBuilder.groovy| 2 +-
.../jenkins/job_LoadTests_CoGBK_Java.groovy| 40 +++
.../job_LoadTests_Combine_Flink_Python.groovy | 26 ++---
.../jenkins/job_LoadTests_Combine_Java.groovy | 31 +++---
.../jenkins/job_LoadTests_Combine_Python.groovy| 32 +++---
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 60 +-
.test-infra/jenkins/job_LoadTests_GBK_Java.groovy | 67 +--
.../jenkins/job_LoadTests_GBK_Python.groovy| 50 -
.../job_LoadTests_GBK_Python_reiterate.groovy | 20 ++--
.../jenkins/job_LoadTests_Java_Smoke.groovy| 32 +++---
...ovy => job_LoadTests_ParDo_Flink_Python.groovy} | 124 -
.../jenkins/job_LoadTests_ParDo_Java.groovy| 40 +++
.../jenkins/job_LoadTests_ParDo_Python.groovy | 34 +++---
.../jenkins/job_LoadTests_Python_Smoke.groovy | 18 ++-
.../job_LoadTests_coGBK_Flink_Python.groovy| 32 +++---
.../jenkins/job_LoadTests_coGBK_Python.groovy | 38 +++
.../job_PerformanceTests_BigQueryIO_Java.groovy| 22 ++--
.../job_PerformanceTests_BigQueryIO_Python.groovy | 20 ++--
.../jenkins/job_PerformanceTests_Python.groovy | 12 +-
19 files changed, 313 insertions(+), 387 deletions(-)
copy .test-infra/jenkins/{job_LoadTests_coGBK_Flink_Python.groovy =>
job_LoadTests_ParDo_Flink_Python.groovy} (52%)
[beam] branch master updated (d61c227 -> f4a1358)
This is an automated email from the ASF dual-hosted git repository.
pabloem pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from d61c227 Merge pull request #9596: [BEAM-8256] Set fixed number of
workers for Java IOITs
add f4a1358 [BEAM-8131] Provide Kubernetes setup for Prometheus (#9482)
No new revisions were added by this update.
Summary of changes:
.test-infra/metrics/README.md | 31 +++--
.../apply_configmaps.sh} | 10 +-
.test-infra/metrics/beamprometheus-deploy.yaml | 125 +
.test-infra/metrics/docker-compose.yml | 28 -
.../alertmanager/config/alertmanager.yml} | 35 +++---
.../prometheus/config/prometheus.yml} | 24 +++-
.../prometheus/config/rules.yml} | 29 ++---
7 files changed, 240 insertions(+), 42 deletions(-)
copy .test-infra/{kubernetes/kafka-cluster/setup-cluster.sh =>
metrics/apply_configmaps.sh} (67%)
create mode 100644 .test-infra/metrics/beamprometheus-deploy.yaml
copy .test-infra/metrics/{grafana/provisioning/dashboards/all.yaml =>
prometheus/alertmanager/config/alertmanager.yml} (61%)
copy .test-infra/metrics/{sync/jenkins/Dockerfile =>
prometheus/prometheus/config/prometheus.yml} (67%)
copy .test-infra/metrics/{grafana/provisioning/dashboards/all.yaml =>
prometheus/prometheus/config/rules.yml} (58%)
[beam] branch master updated (ac45af9 -> 40f2562)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from ac45af9 [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories add 15e4bd1 [BEAM-8302] Fix PostCommit_XVR_Flink add 40f2562 Merge pull request #9644 [BEAM-8302] Fix PostCommit_XVR_Flink No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/transforms/external_test.py | 3 +++ 1 file changed, 3 insertions(+)
[beam] branch release-2.16.0 updated (f4171eb -> 5da269f)
This is an automated email from the ASF dual-hosted git repository. markliu pushed a change to branch release-2.16.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from f4171eb Merge pull request #9623: [BEAM-8224] Fix bug in _fn_takes_side_inputs (#9563) add 8b6c471 [BEAM-8286] replace dead intersphinx link for google-cloud-python add 5da269f Merge pull request #9654: [BEAM-8286] replace dead intersphinx link for google-cloud-python No new revisions were added by this update. Summary of changes: sdks/python/scripts/generate_pydoc.sh | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-)
[beam] branch master updated: More guarding against missing grpc.
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 32f03f9 More guarding against missing grpc. new 6eb8605 Merge pull request #9645 from robertwb/no-grpc 32f03f9 is described below commit 32f03f969aa82da2ebecd9edb5ca4a630f32aa47 Author: Robert Bradshaw AuthorDate: Mon Sep 23 17:15:19 2019 -0700 More guarding against missing grpc. --- sdks/python/apache_beam/transforms/external.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sdks/python/apache_beam/transforms/external.py b/sdks/python/apache_beam/transforms/external.py index 0e241b7..75fe766 100644 --- a/sdks/python/apache_beam/transforms/external.py +++ b/sdks/python/apache_beam/transforms/external.py @@ -39,13 +39,13 @@ from apache_beam.typehints.native_type_compatibility import convert_to_beam_type from apache_beam.typehints.trivial_inference import instance_to_type from apache_beam.typehints.typehints import Union from apache_beam.typehints.typehints import UnionConstraint -from apache_beam.utils import subprocess_server # Protect against environments where grpc is not available. # pylint: disable=wrong-import-order, wrong-import-position, ungrouped-imports try: import grpc from apache_beam.portability.api import beam_expansion_api_pb2_grpc + from apache_beam.utils import subprocess_server except ImportError: grpc = None # pylint: enable=wrong-import-order, wrong-import-position, ungrouped-imports
[beam] branch master updated (6eb8605 -> 031b378)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 6eb8605 Merge pull request #9645 from robertwb/no-grpc add 899f45c [BEAM-5820] Switch Beam SQL to vendored Calcite add 0d9972f not include shadow direct runner dependency in nexmark add 031b378 [BEAM-5820] Use vendored calcite No new revisions were added by this update. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 1 + .../src/main/resources/beam/checkstyle.xml | 8 ++ sdks/java/extensions/sql/build.gradle | 119 + sdks/java/extensions/sql/datacatalog/build.gradle | 3 +- .../sql/example/BeamSqlDataCatalogExample.java | 2 +- .../datacatalog/DataCatalogTableProvider.java | 4 +- .../sql/meta/provider/datacatalog/SchemaUtils.java | 4 +- .../sql/meta/provider/datacatalog/TableUtils.java | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 2 +- sdks/java/extensions/sql/shell/build.gradle| 4 +- .../extensions/sql/src/main/codegen/config.fmpp| 10 +- .../beam/sdk/extensions/sql/SqlTransform.java | 4 +- .../extensions/sql/TableNameExtractionUtils.java | 14 +-- .../sdk/extensions/sql/impl/BeamCalciteSchema.java | 17 +-- .../sql/impl/BeamCalciteSchemaFactory.java | 20 ++-- .../sdk/extensions/sql/impl/BeamCalciteTable.java | 30 +++--- .../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 14 +-- .../sql/impl/BeamSqlPipelineOptionsRegistrar.java | 2 +- .../extensions/sql/impl/BeamTableStatistics.java | 14 +-- .../sql/impl/CalciteConnectionWrapper.java | 16 +-- .../extensions/sql/impl/CalciteFactoryWrapper.java | 24 ++--- .../extensions/sql/impl/CalciteQueryPlanner.java | 72 ++--- .../sdk/extensions/sql/impl/JdbcConnection.java| 8 +- .../beam/sdk/extensions/sql/impl/JdbcDriver.java | 30 +++--- .../beam/sdk/extensions/sql/impl/JdbcFactory.java | 36 --- .../beam/sdk/extensions/sql/impl/QueryPlanner.java | 2 +- .../extensions/sql/impl/ScalarFunctionImpl.java| 55 +- .../extensions/sql/impl/TableResolutionUtils.java | 6 +- .../beam/sdk/extensions/sql/impl/UdafImpl.java | 12 +-- .../beam/sdk/extensions/sql/impl/UdfImpl.java | 12 ++- .../sql/impl/UdfImplReflectiveFunctionBase.java| 19 ++-- .../sql/impl/parser/SqlCheckConstraint.java| 18 ++-- .../sql/impl/parser/SqlColumnDeclaration.java | 20 ++-- .../sql/impl/parser/SqlCreateExternalTable.java| 30 +++--- .../extensions/sql/impl/parser/SqlDdlNodes.java| 20 ++-- .../extensions/sql/impl/parser/SqlDropObject.java | 24 ++--- .../extensions/sql/impl/parser/SqlDropTable.java | 10 +- .../sql/impl/parser/SqlSetOptionBeam.java | 20 ++-- .../extensions/sql/impl/planner/BeamCostModel.java | 11 +- .../sql/impl/planner/BeamJavaTypeFactory.java | 12 +-- .../sql/impl/planner/BeamRelDataTypeSystem.java| 4 +- .../extensions/sql/impl/planner/BeamRuleSets.java | 56 +- .../sql/impl/planner/NodeStatsMetadata.java| 12 +-- .../sql/impl/planner/RelMdNodeStats.java | 12 +-- .../sql/impl/rel/BeamAggregationRel.java | 20 ++-- .../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 68 ++-- .../extensions/sql/impl/rel/BeamCoGBKJoinRel.java | 14 +-- .../sql/impl/rel/BeamEnumerableConverter.java | 40 +++ .../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java | 22 ++-- .../extensions/sql/impl/rel/BeamIOSourceRel.java | 14 +-- .../extensions/sql/impl/rel/BeamIntersectRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 34 +++--- .../sql/impl/rel/BeamLogicalConvention.java| 12 +-- .../sdk/extensions/sql/impl/rel/BeamMinusRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamRelNode.java | 12 ++- .../sql/impl/rel/BeamSetOperatorRelBase.java | 2 +- .../sql/impl/rel/BeamSideInputJoinRel.java | 14 +-- .../sql/impl/rel/BeamSideInputLookupJoinRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamSortRel.java | 30 +++--- .../extensions/sql/impl/rel/BeamSqlRelUtils.java | 6 +- .../extensions/sql/impl/rel/BeamUncollectRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamUnionRel.java | 14 +-- .../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 24 ++--- .../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 20 ++-- .../sdk/extensions/sql/impl/rel/package-info.java | 5 +- .../sql/impl/rule/BeamAggregationRule.java | 24 ++--- .../sql/impl/rule/BeamBasicAggregationRule.java| 14 +-- .../sdk/extensions/sql/impl/rule/BeamCalcRule.java | 14 +-- .../sql/impl/rule/BeamCoGBKJoinRule.java | 12 +-- .../sql/impl/rule/BeamEnumerableConverterRule.java | 8 +-
[beam] branch master updated (031b378 -> ac45af9)
This is an automated email from the ASF dual-hosted git repository.
lcwik pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 031b378 [BEAM-5820] Use vendored calcite
add 77bfad4 Remove the 'experimental' word from Jet Runner
modules/directories
add ac45af9 [BEAM-7305] Remove the 'experimental' word from Jet Runner
modules/directories
No new revisions were added by this update.
Summary of changes:
runners/{jet-experimental => jet}/build.gradle | 0
.../src/main/java/org/apache/beam/runners/jet/DAGBuilder.java | 0
.../org/apache/beam/runners/jet/FailedRunningPipelineResults.java | 0
.../src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java | 0
.../main/java/org/apache/beam/runners/jet/JetPipelineOptions.java | 0
.../main/java/org/apache/beam/runners/jet/JetPipelineResult.java| 0
.../src/main/java/org/apache/beam/runners/jet/JetRunner.java| 0
.../main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java | 0
.../java/org/apache/beam/runners/jet/JetTransformTranslator.java| 0
.../java/org/apache/beam/runners/jet/JetTransformTranslators.java | 0
.../java/org/apache/beam/runners/jet/JetTranslationContext.java | 0
.../src/main/java/org/apache/beam/runners/jet/Utils.java| 0
.../java/org/apache/beam/runners/jet/metrics/AbstractMetric.java| 0
.../main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java | 0
.../java/org/apache/beam/runners/jet/metrics/DistributionImpl.java | 0
.../main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java| 0
.../java/org/apache/beam/runners/jet/metrics/JetMetricResults.java | 0
.../org/apache/beam/runners/jet/metrics/JetMetricsContainer.java| 0
.../main/java/org/apache/beam/runners/jet/metrics/package-info.java | 0
.../src/main/java/org/apache/beam/runners/jet/package-info.java | 0
.../java/org/apache/beam/runners/jet/processors/AbstractParDoP.java | 0
.../java/org/apache/beam/runners/jet/processors/AssignWindowP.java | 0
.../java/org/apache/beam/runners/jet/processors/BoundedSourceP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/FlattenP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/ImpulseP.java | 0
.../main/java/org/apache/beam/runners/jet/processors/ParDoP.java| 0
.../java/org/apache/beam/runners/jet/processors/StatefulParDoP.java | 0
.../org/apache/beam/runners/jet/processors/UnboundedSourceP.java| 0
.../src/main/java/org/apache/beam/runners/jet/processors/ViewP.java | 0
.../java/org/apache/beam/runners/jet/processors/WindowGroupP.java | 0
.../java/org/apache/beam/runners/jet/processors/package-info.java | 0
.../java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java| 0
.../src/test/java/org/apache/beam/runners/jet/TestJetRunner.java| 0
.../src/test/java/org/apache/beam/runners/jet/TestStreamP.java | 0
.../examples/src/main/resources/archetype-resources/pom.xml | 6 +++---
settings.gradle | 2 +-
36 files changed, 4 insertions(+), 4 deletions(-)
rename runners/{jet-experimental => jet}/build.gradle (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetRunner.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/Utils.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java (100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
(100%)
rename runners/{jet-experimental =>
jet}/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java (100%)
rename runners/{jet-experimental
[beam] branch master updated: [BEAM-8293] prescriptive log message for artifact retrieval failure
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
The following commit(s) were added to refs/heads/master by this push:
new 340c320 [BEAM-8293] prescriptive log message for artifact retrieval
failure
new f07bd7a Merge pull request #9646 from ibzib/retrieve-fail
340c320 is described below
commit 340c3202b1e5824b959f5f9f626e4c7c7842a3cb
Author: Kyle Weaver
AuthorDate: Mon Sep 23 17:45:16 2019 -0700
[BEAM-8293] prescriptive log message for artifact retrieval failure
---
.../artifact/BeamFileSystemArtifactRetrievalService.java| 6 +-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
index ff7e9ba..06f26f1 100644
---
a/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
+++
b/runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/BeamFileSystemArtifactRetrievalService.java
@@ -89,7 +89,11 @@ public class BeamFileSystemArtifactRetrievalService
responseObserver.onNext(response);
responseObserver.onCompleted();
} catch (Exception e) {
- LOG.info("GetManifest for {} failed", token, e);
+ LOG.warn(
+ "GetManifest for {} failed. Make sure the artifact staging directory
(configurable "
+ + "via --artifacts-dir argument to the job server) is accessible
to workers.",
+ token,
+ e);
responseObserver.onError(e);
}
}
[beam] branch release-2.16.0 updated (5da269f -> 8884112)
This is an automated email from the ASF dual-hosted git repository. markliu pushed a change to branch release-2.16.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 5da269f Merge pull request #9654: [BEAM-8286] replace dead intersphinx link for google-cloud-python add 69f6ffe [BEAM-8299] Upgrade Jackson to version 2.9.10 add 8884112 Merge pull request #9637: [release-2.16.0][BEAM-8299] Upgrade Jackson to version 2.9.10 No new revisions were added by this update. Summary of changes: .../src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy| 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-)
[beam] branch release-2.16.0 updated (8884112 -> dc072d6)
This is an automated email from the ASF dual-hosted git repository. markliu pushed a change to branch release-2.16.0 in repository https://gitbox.apache.org/repos/asf/beam.git. from 8884112 Merge pull request #9637: [release-2.16.0][BEAM-8299] Upgrade Jackson to version 2.9.10 add eb40ca8 Avoid comparison with param.default in type signature analysis. add 869a05b [BEAM-8301] Fix incomparable defaults. add dc072d6 Merge pull request #9657: [BEAM-8301] Cherry-pick default argument comparison fixes No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/pipeline_test.py| 22 +++ sdks/python/apache_beam/runners/common.py | 28 - sdks/python/apache_beam/transforms/core.py | 2 +- sdks/python/apache_beam/typehints/decorators.py | 2 +- 4 files changed, 38 insertions(+), 16 deletions(-)
[beam] branch master updated (f07bd7a -> 2acbfbd)
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from f07bd7a Merge pull request #9646 from ibzib/retrieve-fail add b32e332 [BEAM-8021] Stop publishing unneeded artifacts add 5ea2878 [BEAM-8021] Add Automatic-Module-Name headers to beam's artifacts. new 2acbfbd Merge branch 'lgajowy-BEAM-8021' The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../org/apache/beam/gradle/BeamModulePlugin.groovy | 33 ++ examples/java/build.gradle | 2 +- examples/kotlin/build.gradle | 2 +- model/fn-execution/build.gradle| 5 +++- model/job-management/build.gradle | 10 --- model/pipeline/build.gradle| 5 +++- runners/apex/build.gradle | 2 +- runners/core-construction-java/build.gradle| 2 +- runners/core-java/build.gradle | 2 +- runners/direct-java/build.gradle | 24 runners/extensions-java/metrics/build.gradle | 2 +- runners/flink/flink_runner.gradle | 3 +- runners/flink/job-server/flink_job_server.gradle | 1 + runners/gearpump/build.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../worker/windmill/build.gradle | 1 + runners/java-fn-execution/build.gradle | 2 +- runners/jet/build.gradle | 2 +- runners/local-java/build.gradle| 1 + runners/reference/java/build.gradle| 2 +- runners/samza/build.gradle | 2 +- runners/samza/job-server/build.gradle | 1 + runners/spark/build.gradle | 2 +- runners/spark/job-server/build.gradle | 1 + sdks/java/build-tools/build.gradle | 2 +- sdks/java/core/build.gradle| 23 --- sdks/java/extensions/euphoria/build.gradle | 2 +- .../google-cloud-platform-core/build.gradle| 2 +- sdks/java/extensions/jackson/build.gradle | 1 + sdks/java/extensions/join-library/build.gradle | 2 +- sdks/java/extensions/kryo/build.gradle | 1 + sdks/java/extensions/protobuf/build.gradle | 2 +- sdks/java/extensions/sketching/build.gradle| 2 +- sdks/java/extensions/sorter/build.gradle | 2 +- sdks/java/extensions/sql/build.gradle | 1 + sdks/java/extensions/sql/datacatalog/build.gradle | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 1 + sdks/java/extensions/zetasketch/build.gradle | 2 +- sdks/java/fn-execution/build.gradle| 2 +- sdks/java/harness/build.gradle | 1 + sdks/java/io/amazon-web-services/build.gradle | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 2 +- sdks/java/io/amqp/build.gradle | 2 +- sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 +- sdks/java/io/cassandra/build.gradle| 2 +- sdks/java/io/clickhouse/build.gradle | 1 + sdks/java/io/common/build.gradle | 2 +- .../elasticsearch-tests-2/build.gradle | 1 + .../elasticsearch-tests-5/build.gradle | 1 + .../elasticsearch-tests-6/build.gradle | 1 + .../elasticsearch-tests-common/build.gradle| 1 + sdks/java/io/elasticsearch/build.gradle| 2 +- sdks/java/io/file-based-io-tests/build.gradle | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 + sdks/java/io/hadoop-common/build.gradle| 2 +- sdks/java/io/hadoop-file-system/build.gradle | 2 +- sdks/java/io/hadoop-format/build.gradle| 2 +- sdks/java/io/hbase/build.gradle| 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- sdks/java/io/jdbc/build.gradle | 2 +- sdks/java/io/jms/build.gradle | 2 +- sdks/java/io/kafka/build.gradle| 2 +- sdks/java/io/kinesis/build.gradle | 2 +- sdks/java/io/kudu/build.gradle | 2 +- sdks/java/io/mongodb/build.gradle | 2 +- sdks/java/io/mqtt/build.gradle | 2 +- sdks/java/io/parquet/build.gradle | 2 +- sdks/java/io/rabbitmq/build.gradle | 2 +- sdks/java/io/redis/build.gradle| 2 +- sdks/java/io/solr/build.gradle | 2 +-
[beam] 01/01: Merge branch 'lgajowy-BEAM-8021'
This is an automated email from the ASF dual-hosted git repository. lcwik pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git commit 2acbfbd23a23ee6eed77cbe74690f66f985c9815 Merge: f07bd7a 5ea2878 Author: Luke Cwik AuthorDate: Tue Sep 24 14:02:44 2019 -0700 Merge branch 'lgajowy-BEAM-8021' .../org/apache/beam/gradle/BeamModulePlugin.groovy | 33 ++ examples/java/build.gradle | 2 +- examples/kotlin/build.gradle | 2 +- model/fn-execution/build.gradle| 5 +++- model/job-management/build.gradle | 10 --- model/pipeline/build.gradle| 5 +++- runners/apex/build.gradle | 2 +- runners/core-construction-java/build.gradle| 2 +- runners/core-java/build.gradle | 2 +- runners/direct-java/build.gradle | 24 runners/extensions-java/metrics/build.gradle | 2 +- runners/flink/flink_runner.gradle | 3 +- runners/flink/job-server/flink_job_server.gradle | 1 + runners/gearpump/build.gradle | 2 +- runners/google-cloud-dataflow-java/build.gradle| 2 +- .../worker/windmill/build.gradle | 1 + runners/java-fn-execution/build.gradle | 2 +- runners/jet/build.gradle | 2 +- runners/local-java/build.gradle| 1 + runners/reference/java/build.gradle| 2 +- runners/samza/build.gradle | 2 +- runners/samza/job-server/build.gradle | 1 + runners/spark/build.gradle | 2 +- runners/spark/job-server/build.gradle | 1 + sdks/java/build-tools/build.gradle | 2 +- sdks/java/core/build.gradle| 23 --- sdks/java/extensions/euphoria/build.gradle | 2 +- .../google-cloud-platform-core/build.gradle| 2 +- sdks/java/extensions/jackson/build.gradle | 1 + sdks/java/extensions/join-library/build.gradle | 2 +- sdks/java/extensions/kryo/build.gradle | 1 + sdks/java/extensions/protobuf/build.gradle | 2 +- sdks/java/extensions/sketching/build.gradle| 2 +- sdks/java/extensions/sorter/build.gradle | 2 +- sdks/java/extensions/sql/build.gradle | 1 + sdks/java/extensions/sql/datacatalog/build.gradle | 2 +- sdks/java/extensions/sql/hcatalog/build.gradle | 2 +- sdks/java/extensions/sql/jdbc/build.gradle | 1 + sdks/java/extensions/zetasketch/build.gradle | 2 +- sdks/java/fn-execution/build.gradle| 2 +- sdks/java/harness/build.gradle | 1 + sdks/java/io/amazon-web-services/build.gradle | 2 +- sdks/java/io/amazon-web-services2/build.gradle | 2 +- sdks/java/io/amqp/build.gradle | 2 +- sdks/java/io/bigquery-io-perf-tests/build.gradle | 2 +- sdks/java/io/cassandra/build.gradle| 2 +- sdks/java/io/clickhouse/build.gradle | 1 + sdks/java/io/common/build.gradle | 2 +- .../elasticsearch-tests-2/build.gradle | 1 + .../elasticsearch-tests-5/build.gradle | 1 + .../elasticsearch-tests-6/build.gradle | 1 + .../elasticsearch-tests-common/build.gradle| 1 + sdks/java/io/elasticsearch/build.gradle| 2 +- sdks/java/io/file-based-io-tests/build.gradle | 2 +- sdks/java/io/google-cloud-platform/build.gradle| 1 + sdks/java/io/hadoop-common/build.gradle| 2 +- sdks/java/io/hadoop-file-system/build.gradle | 2 +- sdks/java/io/hadoop-format/build.gradle| 2 +- sdks/java/io/hbase/build.gradle| 2 +- sdks/java/io/hcatalog/build.gradle | 2 +- sdks/java/io/jdbc/build.gradle | 2 +- sdks/java/io/jms/build.gradle | 2 +- sdks/java/io/kafka/build.gradle| 2 +- sdks/java/io/kinesis/build.gradle | 2 +- sdks/java/io/kudu/build.gradle | 2 +- sdks/java/io/mongodb/build.gradle | 2 +- sdks/java/io/mqtt/build.gradle | 2 +- sdks/java/io/parquet/build.gradle | 2 +- sdks/java/io/rabbitmq/build.gradle | 2 +- sdks/java/io/redis/build.gradle| 2 +- sdks/java/io/solr/build.gradle | 2 +- sdks/java/io/synthetic/build.gradle| 2 +- sdks/java/io/tika/build.gradle | 2 +- sdks/java/io/xml/build.gradle | 2 +- sdks/java/javadoc/build.gradle | 2 +- sdks/java/maven-archetypes/examples/build.gradle | 2 +- sdks/java/maven-archetypes/starter/build.gradle| 2
[beam] branch master updated: [BEAM-8311] Fix py mongodbio display data
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a commit to branch master in repository https://gitbox.apache.org/repos/asf/beam.git The following commit(s) were added to refs/heads/master by this push: new 656cc76 [BEAM-8311] Fix py mongodbio display data new 4ea54ca Merge pull request #9656 from y1chi/BEAM-8311 656cc76 is described below commit 656cc76a217457c2b66d4f37d2681359d1fb662e Author: Yichi Zhang AuthorDate: Tue Sep 24 14:17:06 2019 -0700 [BEAM-8311] Fix py mongodbio display data --- sdks/python/apache_beam/io/mongodbio.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/sdks/python/apache_beam/io/mongodbio.py b/sdks/python/apache_beam/io/mongodbio.py index 39fec8d..6004ca1 100644 --- a/sdks/python/apache_beam/io/mongodbio.py +++ b/sdks/python/apache_beam/io/mongodbio.py @@ -203,8 +203,8 @@ class _BoundedMongoSource(iobase.BoundedSource): res['uri'] = self.uri res['database'] = self.db res['collection'] = self.coll -res['filter'] = self.filter -res['project'] = self.projection +res['filter'] = json.dumps(self.filter) +res['projection'] = str(self.projection) res['mongo_client_spec'] = json.dumps(self.spec) return res
[beam] branch master updated (2acbfbd -> 80bfcc6)
This is an automated email from the ASF dual-hosted git repository. robertwb pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 2acbfbd Merge branch 'lgajowy-BEAM-8021' add 7bee065 [BEAM-8301] Fix incomparable defaults. add 80bfcc6 Merge pull request #9641 [BEAM-8301] Fix incomparable defaults. No new revisions were added by this update. Summary of changes: sdks/python/apache_beam/pipeline_test.py | 22 ++ sdks/python/apache_beam/runners/common.py | 28 ++-- 2 files changed, 36 insertions(+), 14 deletions(-)
[beam] branch pabloem-patch-1 created (now 7644806)
This is an automated email from the ASF dual-hosted git repository. pabloem pushed a change to branch pabloem-patch-1 in repository https://gitbox.apache.org/repos/asf/beam.git. at 7644806 Fixing issue in AvroCoder documentation. No new revisions were added by this update.
[beam] branch master updated (4ea54ca -> a40d643)
This is an automated email from the ASF dual-hosted git repository. kenn pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/beam.git. from 4ea54ca Merge pull request #9656 from y1chi/BEAM-8311 add a40d643 [BEAM-3372] Remove duplicated zone in Dataflow worker options (#9578) No new revisions were added by this update. Summary of changes: .../options/DataflowPipelineWorkerPoolOptions.java | 17 ++--- .../beam/sdk/extensions/gcp/options/GcpOptions.java | 1 + 2 files changed, 3 insertions(+), 15 deletions(-)
