Build failed in Jenkins: beam_PerformanceTests_Python #900

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam3 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 67234a15b56ddeed1d4dd90d276bad7e5003822c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67234a15b56ddeed1d4dd90d276bad7e5003822c
Commit message: "Merge pull request #4603: [BEAM-3620] Remove older Kafka 
versions from build time support"
 > git rev-list 67234a15b56ddeed1d4dd90d276bad7e5003822c # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3471126445077613477.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins821368487411391225.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8865254936675380375.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2056938800619628828.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/43/41/033a273f9a25cb63050a390ee8397acbc7eae2159195d85f06f17e7be45a/setuptools-38.5.1-py2.py3-none-any.whl#md5=908b8b5e50bf429e520b2b5fa1b350e5
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6350526133859850870.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5952185686393851707.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement alread

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #144

[Apache Jenkins Server]

See 

svn commit: r24925 - in /dev/beam/2.3.0: apache-beam-2.3.0-python.zip.asc apache-beam-2.3.0-source-release.zip apache-beam-2.3.0-source-release.zip.asc apache-beam-2.3.0-source-release.zip.md5 apache-

[jbonofre]

Author: jbonofre
Date: Sun Feb 11 05:24:22 2018
New Revision: 24925

Log:
Update 2.3.0 release

Modified:
dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5
dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.sha1

Modified: dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc
==
--- dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-python.zip.asc Sun Feb 11 05:24:22 2018
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJae+A7AAoJEL/y7kLIKC52X58QAJkn6JwZD7je+NfPI0feP7Jj
-Cg+BIZ0il6STZfJKO2JfFGyMzr2Djq2iXjJcvXaNMTfltMdVCIhe9HnkGhit70fI
-mErQlC1I4qJcTmOTtAgda0KpfXSK6lf2CFfguThcAjWj+mMlhnXb9BMptlxrOGQl
-J0vfffbBeeXcnSmQaetnBOZJogmcOsBDCgmqi5gB8j6GTtBFdU+7g0JhTDvGSGyI
-AvyYxoQiP9hhvV/e2lUee95iY4rJEKymJBfQHwWJBjr5zreqjaXd1hCv2/4cyYGd
-hm2x/JQAO1XsMk4OM6Qi11QTyKWG7x6GSqrTnm8YMeCDvqBMp05Bhm2WWVq2UvgM
-gqonrqZncsBu4/mGzYr2aTMGBByWDuN9SUFwtZs59dvKtwIFX3fld7/ebHyCc8CO
-3RQkaCUM0QKoq6IFArZBvZHFdUP1zBgFXtwGrsxQUY639ljaExnudf6X2fSI/mfD
-PcksRlZu4EQ6v12krLW+L/SFsISIu6nHWLQwTcHlr4OZGyshFvneFT3EBxm46s4A
-o6xivYDEM2X7+wBrxtYKWymbez7NZSgLz6Pc3B/6860B+RwHJjWmVNfHCr8XSBOp
-rnVIMI3clQ0eCz377VHBOVBXhU0rrCCmjg3XfRVJ5L6QMzgK8PUgus7nMMAhoaw0
-rq009gMzCyTxQSihp6Dg
-=il66
+iQIcBAABAgAGBQJaf9M6AAoJEL/y7kLIKC52+qYP/iIiBWeg77BnSDVjqUCUyVrM
+i3gyPpxdCG0AB9LxUoS3EzXYYxlnOaVcutY1pyHm+67gQvO54ouFcAV1I75p9DS8
+/WuTBgC62mz5Xt1pK2tsI8zGxvS6p+w2DWQCzBpT/rdOzv76brE7KT7PgazvRh16
+xdngOlbSVbhJm8EviajxZG5LfL/smNSS814OouOYzvkna6nv45s7kRQIoCfyWFre
+Ium2o3Wy3lqm77Sfq7aTsmCSbzJrAmIaXkIpuhTsLzRwXt5SYaHtof+IbD1g7CMj
+DmbBCVNPtNWSEIaHvPKuZRQ8KDSYidNMKyGc34gZA8PvZeA7ZWwWuDbQPpvyYJG0
+xLxlGWH5cIZuh1NcMVw0FpSTtGJbHwE8p8FFjMf7RnfTpgouiY6+kejD1U/8S6sD
+OetPNUxRU19lZJhYRsk7u+s8dU4U/SMEIVHtpyDojpRCFyc4rgGOzTpDS8HL4Za+
+LOWwPms7lCJffi5nrLl4HPhFLy8ProBnDp444OstPY7h9Hrb7k4Btp9Ycy8dTeZA
+ERkpN6CZORZqTBPtIGelcQ4UDlI4/L4YYglzmM6L9Ho6idf565I6oDq+Oj0XAYhN
+rFjqOU0vHkT5gZU4IdPicKvNlX/YACz55tTRPjbOaukoq51KfNbVzZQ9AZBJ8xv1
+Y0GOm/1kvWH3x5o1V9kO
+=MzXl
 -END PGP SIGNATURE-

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip
==
Binary files - no diff available.

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc
==
--- dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.asc Sun Feb 11 05:24:22 
2018
@@ -1,17 +1,17 @@
 -BEGIN PGP SIGNATURE-
 Version: GnuPG v1
 
-iQIcBAABAgAGBQJae1SfAAoJEL/y7kLIKC52+20QAJkebTvBEu+IUA7oo2h4KakS
-rdtofRT0E/lNB2i09jya8qPuYnLIbKcZZAl2AlFxQ6ngzanXdi1QBaMkpYEsYP0L
-AP25F6QnnuilhOBIv5S80bxa47QQsd498UMvBnwiWXIWw/ZaBfgpMLNL71+OUOGc
-rqMaXyCbvwr45hoIAy3r1OHCDwYm58iAcuVUEyNTaAHlxvYXPXTsOS7kAGCTQANg
-W1v04jXQcenxZNrb+D4oFNs6nHCT1yxPBgU1mEk7u/RRRdw/IFf9xG0AADytzupc
-v7F/Ssa4QjpUoRIvPrmYWF4PAF6b+AOAfqah/kJV8fxrqtlsbw8Qum74ekAmCcIE
-/kIhct7ezGiEDrGw6qWFLzfFH/CD4dii7v+Ut0prBSJAXcCmcLqrg0zCujRbCWuz
-LPdC5lc7KLgNN7vw7yH45jQ26zCe0YJvDYUNJmWM1UAAiwieqMIwSPilMhaF+Pd/
-ykROjtFrjgAiwBbJ7MOSgLut2HperpeDqKpLCcAI0pngKIQSSEywEMxbWbSltXIz
-Q81PRffBA7SOkmIuVT1qMQjR4Ie8saJxflJc8JPwJlXUr0Tc7afA+3Zov9+VMERo
-XGsH0QOLYQoQwT1109xI8PzRgdCdRTpOe4FOgRtdxAZ6+zhEKkS046PGNNR6E/Fv
-ymytpT6yvDzQ9mzQ5kuL
-=/d2I
+iQIcBAABAgAGBQJaf1L8AAoJEL/y7kLIKC52QykP/0S5FZVT+D4Meb3xSMRQ6KKx
+vPC7+D1RpmHVRnRd6sDr5H47iE2+OsIt6THazpaxVI/9DDtQJtVfsl0Bl7cUek4R
+BhufusaqpEC32nuQsLKuZEPbA8zQNruNkkWdTYX/cTnZmkcNj4WivOJugj7g1Lro
+bTFqGvfdqTAYUq+kfHA3nGMuAH43MsVINHxU7PFNf9zukOXB7b4tSuzrEd9JyCK2
+QisP+8d6/K+di6M3WSYFLxjtAJW3n69LjtxEFb2h2lsmp2vsaRe9grVzcZivkkI3
+nYRdB1xkqXu0JH7+XA+UmkUO0VSsECSrj58FWbNw6eEoXgR/skeZP0dU9IKKoQEE
+cNwtuiQsQNsh6NanVky32NfwNt+oFNxXNDPYeYl3IQYgQjvIJC5R/rOmk4N39Ivj
+KlAQEAlPh7htzSmlZIOExOqsIeQr3P2Bjc1fwoa+UCOJ/WuADk+5OdujAWTnPjo/
+g9hOWUTJD7MmB06sZiRXj1Paa0BRj10T62UUGwL2nGeuzjRC8OEW6B2NTuvQ/oVL
+vugF7Bcwf4Ucg3XxrsOXdeieSrFdV/uSHTfoem3VssIdVG1g/lgK267qtHKAazBJ
+lT4cCVPbf0BXkUxV6indMSDEMEhkMQgGh9zT8OEPZp4ZlBy45F1KK0UJIqqviFWj
+jWJETGiQkVTDqDeTGavp
+=3Zmm
 -END PGP SIGNATURE-

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5
==
--- dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5 (original)
+++ dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.md5 Sun Feb 11 05:24:22 
2018
@@ -1 +1 @@
-5288ab03de17ff82ed0d80ffd41fae9b  apache-beam-2.3.0-source-release.zip
+4bfbaad87e8c44d80a2d43e6a7059cf4  apache-beam-2.3.0-source-release.zip

Modified: dev/beam/2.3.0/apache-beam-2.3.0-source-release.zip.sha1
==
--- dev/beam/2.3.0/apac

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #22

[Apache Jenkins Server]

See 


--
[...truncated 98.84 KB...]
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_client.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/dataflow_v1b3_messages.py
 -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying 
apache_beam/runners/dataflow/internal/clients/dataflow/message_matchers_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/internal/clients/dataflow
copying apache_beam/runners/dataflow/native_io/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/iobase_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.4.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universa

[jira] [Updated] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2445:
-
Priority: Major  (was: Minor)

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin reassigned BEAM-2445:


Assignee: Xu Mingmin

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Assignee: Xu Mingmin
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-2445) DSL SQL to use service locator pattern to automatically register UDFs

[Xu Mingmin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xu Mingmin updated BEAM-2445:
-
Fix Version/s: 2.4.0

> DSL SQL to use service locator pattern to automatically register UDFs
> -
>
> Key: BEAM-2445
> URL: https://issues.apache.org/jira/browse/BEAM-2445
> Project: Beam
>  Issue Type: Improvement
>  Components: dsl-sql
>Reporter: Luke Cwik
>Priority: Major
> Fix For: 2.4.0
>
>
> Use a service locator pattern to find UDFs that can be registered. The 
> service loader can be used to register UDFs for standard functions via DSL 
> SQL, additional UDFs registered by third party libraries, and end user 
> created UDFs.
> Example ServiceLoader usage within Apache Beam to find coder providers:
> https://github.com/apache/beam/blob/7126fdc6ee5671e99a2dede3f25ba616aa0e8fa4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L147



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #143

[Apache Jenkins Server]

See 


Changes:

[rangadi] Remove older Kafka versions from build time support.

[rangadi] Move kafka-clients dependency to provided scope.

--
[...truncated 13.42 KB...]
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: 
proto-google-cloud-pubsub-v1[grpc]<0.16dev,>=0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.4.0.dev0)
Installing collected packages: hdfs, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_TextIOIT] $ /bin/bash -xe 
/tmp/jenkins3539405974818769352.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.textioit_pkb_results 
--temp_dir= 
--official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT 
'--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/143/]'
 '--beam_extra_mvn_properties=[filesystem=gcs]'
2018-02-11 00:00:28,771 9a1711d9 MainThread INFO Verbose logging to: 

2018-02-11 00:00:28,771 9a1711d9 MainThread INFO PerfKitBenchmarker 
version: v1.12.0-260-ge324e61
2018-02-11 00:00:28,772 9a1711d9 MainThread INFO Flag values:
--beam_extra_mvn_properties=[filesystem=gcs]
--beam_it_class=org.apache.beam.sdk.io.text.TextIOIT
--beam_it_timeout=1200
--beam_sdk=java
--temp_dir=
--maven_binary=/home/jenkins/tools/maven/latest/bin/mvn
--beam_it_options=[--project=apache-beam-testing,--tempRoot=gs://temp-storage-for-perf-tests,--numberOfRecords=100,--filenamePrefix=gs://temp-storage-for-perf-tests/beam_PerformanceTests_TextIOIT/143/]
--beam_prebuilt
--project=apache-beam-testing
--bigquery_table=beam_performance.textioit_pkb_results
--official
--beam_it_module=sdks/java/io/file-based-io-tests
--dpb_log_level=INFO
--beam_it_profile=io-it
--benchmarks=beam_integration_benchmark
2018-02-11 00:00:29,037 9a1711d9 MainThread WARNING  The key "flags" was not in 
the default config, but was in user overrides. This may indicate a typo.
2018-02-11 00:00:29,037 9a1711d9 MainThread INFO Initializing the edw 
service decoder
2018-02-11 00:00:29,149 9a1711d9 MainThr

Build failed in Jenkins: beam_PerformanceTests_Python #899

[Apache Jenkins Server]

See 


Changes:

[rangadi] Remove older Kafka versions from build time support.

[rangadi] Move kafka-clients dependency to provided scope.

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 67234a15b56ddeed1d4dd90d276bad7e5003822c (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67234a15b56ddeed1d4dd90d276bad7e5003822c
Commit message: "Merge pull request #4603: [BEAM-3620] Remove older Kafka 
versions from build time support"
 > git rev-list 67ba5531c71d273dbbfa356f244bb2bb670b2bdf # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7671503057843117792.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5702978633318382297.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8768207372680132830.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3188304659742349448.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2208171215857969546.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6058804148053490386.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may caus

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #21

[Apache Jenkins Server]

$TAG \
--staging_location=$GCS_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 399.200s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180210-11
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:65198ef094bca057410f435fa2734181b9e91a60c7341b9aadbc1cf9046ab3fd
Deleted: sha256:b42973a596aa29b3cd5fb31b3dc12df54c590025406424276506245e03a63c8a
Deleted: sha256:97f393e4d959e9b91a784df78be2c4b554c3db03b3810ee4b91a15d013f52800
Deleted: sha256:98ca1ab37bdb5668d8f149233f0c3745b6fc557c699b88e04e8fee328cd7c965
Deleted: sha256:b15fdf9a85051c16e6730cb6c3db6e668856630ded0e01430077f93b0eb2f802
Deleted: sha256:d9a78e114e4a565117a237cda1f31420f3d975e3255f4b984c9ef2c5374d3867
Deleted: sha256:5f21586489c8c135389ecfe0fadc25c6e123454309d01b8916b356a446746cae
Deleted: sha256:30d14c9b093f3af493c50a6c450bb824fb545b7526be7ec513f0d886b6d523e7
Deleted: sha256:183c22853e575ddbb03f3174a727ec28ccc03943950d71684edd96db5768f973
gcloud container images delete $CONTAINER:$TAG --quiet
Usage: gcloud container [optional flags] 
  group may be   clusters | node-pools | operations
  command may be get-server-config

Deploy and manage clusters of machines for running containers.

flags:
  Run `gcloud container --help`
  for the full list of available flags for this command.

global flags:
  Run `gcloud -h` for a description of flags available to all commands.

command groups:
  clusters   Deploy and teardown Google Container Engine clusters.
  node-pools Create and delete operations for Google Container
 Engine node pools.
  operations Get and list operations for Google Container Engine
 clusters.

commands:
  get-server-config  Get Container Engine server config.


For more detailed information on this command and its flags, run:
  gcloud container --help

ERROR: (gcloud.container) Invalid choice: 'images'.

Valid choices are [clusters, get-server-config, node-pools, operations].
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user xuming...@users.noreply.github.com
Not sending mail to unregistered user jiang...@gmail.com
Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user j...@nanthrax.net
Not sending mail to unregistered user g...@telligent-data.com
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user dariusz.aniszew...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user ccla...@bluewin.ch
Not sending mail to unregistered user fjetum...@gmail.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user aromanenko@gmail.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user mott...@gmail.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user z...@giggles.nyc.corp.google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user joey.bar...@gmail.com

[jira] [Resolved] (BEAM-3620) Deprecate older kafka clients and make kafka clients a provided dependency

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía resolved BEAM-3620.

Resolution: Fixed

> Deprecate older kafka clients and make kafka clients a provided dependency
> --
>
> Key: BEAM-3620
> URL: https://issues.apache.org/jira/browse/BEAM-3620
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Kafka versions 0.10.1 and above have better support for timestamps. We intend 
> to deprecate older versions as a first step towards dropping the support 
> entirely.
> The older versions (0.9.x and 0.10.0) at run time still work, but KafkaIO 
> itself does not support these at build time. It lets us remove some of the 
> reflection based code used to support older versions at build time. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3620) Deprecate older kafka clients and make kafka clients a provided dependency

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-3620:
---
Summary: Deprecate older kafka clients and make kafka clients a provided 
dependency  (was: Deprecate older Kafka clients)

> Deprecate older kafka clients and make kafka clients a provided dependency
> --
>
> Key: BEAM-3620
> URL: https://issues.apache.org/jira/browse/BEAM-3620
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Affects Versions: 2.3.0
>Reporter: Raghu Angadi
>Assignee: Raghu Angadi
>Priority: Major
> Fix For: 2.4.0
>
>  Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> Kafka versions 0.10.1 and above have better support for timestamps. We intend 
> to deprecate older versions as a first step towards dropping the support 
> entirely.
> The older versions (0.9.x and 0.10.0) at run time still work, but KafkaIO 
> itself does not support these at build time. It lets us remove some of the 
> reflection based code used to support older versions at build time. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (67ba553 -> 67234a1)

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 67ba553  Merge pull request #4653 from jbonofre/BEAM-3668
 add 05d4f3a  Remove older Kafka versions from build time support.
 add 8faf5f3  Move kafka-clients dependency to provided scope.
 new 67234a1  Merge pull request #4603: [BEAM-3620] Remove older Kafka 
versions from build time support

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/kafka/pom.xml |  1 +
 .../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java | 65 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |  9 ++-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 60 
 4 files changed, 57 insertions(+), 78 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[beam] 01/01: Merge pull request #4603: [BEAM-3620] Remove older Kafka versions from build time support

[iemejia]

This is an automated email from the ASF dual-hosted git repository.

iemejia pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 67234a15b56ddeed1d4dd90d276bad7e5003822c
Merge: 67ba553 8faf5f3
Author: Ismaël Mejía 
AuthorDate: Sat Feb 10 22:50:29 2018 +0100

Merge pull request #4603: [BEAM-3620] Remove older Kafka versions from 
build time support

[BEAM-3620] Remove older Kafka versions from build time support.

 sdks/java/io/kafka/pom.xml |  1 +
 .../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java | 65 +-
 .../java/org/apache/beam/sdk/io/kafka/KafkaIO.java |  9 ++-
 .../org/apache/beam/sdk/io/kafka/KafkaIOTest.java  | 60 
 4 files changed, 57 insertions(+), 78 deletions(-)


-- 
To stop receiving notification emails like this one, please contact
ieme...@apache.org.

[jira] [Created] (BEAM-3685) It should be an error to run a Pipeline without ever specifying options

[Thomas Groh (JIRA)]

Thomas Groh created BEAM-3685:
-

 Summary: It should be an error to run a Pipeline without ever 
specifying options
 Key: BEAM-3685
 URL: https://issues.apache.org/jira/browse/BEAM-3685
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-core
Reporter: Thomas Groh
Assignee: Kenneth Knowles


Doing so lets users run some pipelines without specifying any configuration, 
which is dangerous.

 

At minimum, it should log a very obvious warning.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.3.0 updated: [maven-release-plugin] rollback changes from release preparation of v2.3.0-RC3

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new 4abe27c  [maven-release-plugin] rollback changes from release 
preparation of v2.3.0-RC3
4abe27c is described below

commit 4abe27cb390de3ab97c52c15f7a083f82f808fdc
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Feb 10 21:50:16 2018 +0100

[maven-release-plugin] rollback changes from release preparation of 
v2.3.0-RC3
---
 examples/java/pom.xml   | 2 +-
 examples/pom.xml| 2 +-
 model/fn-execution/pom.xml  | 2 +-
 model/job-management/pom.xml| 2 +-
 model/pipeline/pom.xml  | 2 +-
 model/pom.xml   | 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/gcp/gcemd/pom.xml   | 2 +-
 runners/gcp/gcsproxy/pom.xml| 2 +-
 runners/gcp/pom.xml | 2 +-
 runners/gearpump/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/java-fn-execution/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml | 2 +-
 runners/local-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/reference/java/pom.xml  | 2 +-
 runners/reference/job-server/pom.xml| 2 +-
 runners/reference/pom.xml   | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/go/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/container/pom.xml | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sketching/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/extensions/sql/pom.xml| 2 +-
 sdks/java/fn-execution/pom.xml  | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amazon-web-services/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml| 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/file-based-io-tests/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 

[beam] branch release-2.3.0 updated: [maven-release-plugin] prepare release v2.3.0-RC3

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new 081de89  [maven-release-plugin] prepare release v2.3.0-RC3
081de89 is described below

commit 081de891e05a6c84db0f3fd03e3ad95f362d5c4c
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Feb 10 21:49:44 2018 +0100

[maven-release-plugin] prepare release v2.3.0-RC3
---
 examples/java/pom.xml   | 2 +-
 examples/pom.xml| 2 +-
 model/fn-execution/pom.xml  | 2 +-
 model/job-management/pom.xml| 2 +-
 model/pipeline/pom.xml  | 2 +-
 model/pom.xml   | 2 +-
 pom.xml | 4 ++--
 runners/apex/pom.xml| 2 +-
 runners/core-construction-java/pom.xml  | 2 +-
 runners/core-java/pom.xml   | 2 +-
 runners/direct-java/pom.xml | 2 +-
 runners/flink/pom.xml   | 2 +-
 runners/gcp/gcemd/pom.xml   | 2 +-
 runners/gcp/gcsproxy/pom.xml| 2 +-
 runners/gcp/pom.xml | 2 +-
 runners/gearpump/pom.xml| 2 +-
 runners/google-cloud-dataflow-java/pom.xml  | 2 +-
 runners/java-fn-execution/pom.xml   | 2 +-
 runners/local-artifact-service-java/pom.xml | 2 +-
 runners/local-java/pom.xml  | 2 +-
 runners/pom.xml | 2 +-
 runners/reference/java/pom.xml  | 2 +-
 runners/reference/job-server/pom.xml| 2 +-
 runners/reference/pom.xml   | 2 +-
 runners/spark/pom.xml   | 2 +-
 sdks/go/pom.xml | 2 +-
 sdks/java/build-tools/pom.xml   | 2 +-
 sdks/java/container/pom.xml | 2 +-
 sdks/java/core/pom.xml  | 2 +-
 sdks/java/extensions/google-cloud-platform-core/pom.xml | 2 +-
 sdks/java/extensions/jackson/pom.xml| 2 +-
 sdks/java/extensions/join-library/pom.xml   | 2 +-
 sdks/java/extensions/pom.xml| 2 +-
 sdks/java/extensions/protobuf/pom.xml   | 2 +-
 sdks/java/extensions/sketching/pom.xml  | 2 +-
 sdks/java/extensions/sorter/pom.xml | 2 +-
 sdks/java/extensions/sql/pom.xml| 2 +-
 sdks/java/fn-execution/pom.xml  | 2 +-
 sdks/java/harness/pom.xml   | 2 +-
 sdks/java/io/amazon-web-services/pom.xml| 2 +-
 sdks/java/io/amqp/pom.xml   | 2 +-
 sdks/java/io/cassandra/pom.xml  | 2 +-
 sdks/java/io/common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-2/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-5/pom.xml  | 2 +-
 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/pom.xml | 2 +-
 sdks/java/io/elasticsearch-tests/pom.xml| 2 +-
 sdks/java/io/elasticsearch/pom.xml  | 2 +-
 sdks/java/io/file-based-io-tests/pom.xml| 2 +-
 sdks/java/io/google-cloud-platform/pom.xml  | 2 +-
 sdks/java/io/hadoop-common/pom.xml  | 2 +-
 sdks/java/io/hadoop-file-system/pom.xml | 2 +-
 sdks/java/io/hadoop-input-format/pom.xml| 2 +-
 sdks/java/io/hbase/pom.xml  | 2 +-
 sdks/java/io/hcatalog/pom.xml   | 2 +-
 sdks/java/io/jdbc/pom.xml   | 2 +-
 sdks/java/io/jms/pom.xml| 2 +-
 sdks/java/io/kafka/pom.xml  | 2 +-
 sdks/java/io/kinesis/pom.xml

[beam] annotated tag v2.3.0-RC3 created (now d1c62ec)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to annotated tag v2.3.0-RC3
in repository https://gitbox.apache.org/repos/asf/beam.git.


  at d1c62ec  (tag)
 tagging 081de891e05a6c84db0f3fd03e3ad95f362d5c4c (commit)
 replaces v2.3.0-RC2
  by Jean-Baptiste Onofré
  on Sat Feb 10 21:50:06 2018 +0100

- Log -
[maven-release-plugin] copy for tag v2.3.0-RC3
---

No new revisions were added by this update.

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

Build failed in Jenkins: beam_PerformanceTests_Python #898

[Apache Jenkins Server]

See 


Changes:

[tgroh] Add a Primitive Impulse PTransform

[tgroh] Add a single-stage fusion implementation

[jb] [BEAM-3668] Quick workaround fix for netty conflict waiting better fix

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 67ba5531c71d273dbbfa356f244bb2bb670b2bdf (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 67ba5531c71d273dbbfa356f244bb2bb670b2bdf
Commit message: "Merge pull request #4653 from jbonofre/BEAM-3668"
 > git rev-list 5056e06b7bbb6a70e00797e13de99a7492237be0 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5837845295811352826.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins357812789014159322.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4962743173791763017.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins6280206364873797403.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins309365571231561307.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5221231240337594637.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cau

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #20

[Apache Jenkins Server]

/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 445.711s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180210-174822
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:295bf498480317abe53869b062e279f82066fd9231c082e91514c9be108b0534
Deleted: sha256:33bbc6a04dfde8543534371dc2447412de6f43155f5e33ac4ec777e0b8200a4f
Deleted: sha256:85f2d833c13ea1e58d93dd06df63556f4805077c24a6b01cdcd440b7c9102858
Deleted: sha256:8503c00c85a7d125ccb15f8c2d2aa7020fb6dc134f37f5373f758e4db18de043
Deleted: sha256:de5da3f54473d4e21af7bc7ddcb6bca8d5192903c93617c8c4df5aa0fab18260
Deleted: sha256:c271eb02eb9adb6e737a046e850a5719c1ee06e4c68b586db35b26deacc7d750
Deleted: sha256:5e5ce1803f0fd9a3b262ab65ec48aa853914dbc2805c09f205b641b83f93022c
Deleted: sha256:842fcad8189ab558bbc7ebad7e788f6db3ac0d4126be6df433ead49c30cc4bfc
Deleted: sha256:c8dfa7adbcca05ec0bf382f3a0240f7235e422b9ed90807c6496a59a49bdcd31
gcloud container images delete $CONTAINER:$TAG --quiet
Usage: gcloud container [optional flags] 
  group may be   clusters | node-pools | operations
  command may be get-server-config

Deploy and manage clusters of machines for running containers.

flags:
  Run `gcloud container --help`
  for the full list of available flags for this command.

global flags:
  Run `gcloud -h` for a description of flags available to all commands.

command groups:
  clusters   Deploy and teardown Google Container Engine clusters.
  node-pools Create and delete operations for Google Container
 Engine node pools.
  operations Get and list operations for Google Container Engine
 clusters.

commands:
  get-server-config  Get Container Engine server config.


For more detailed information on this command and its flags, run:
  gcloud container --help

ERROR: (gcloud.container) Invalid choice: 'images'.

Valid choices are [clusters, get-server-config, node-pools, operations].
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user xuming...@users.noreply.github.com
Not sending mail to unregistered user jiang...@gmail.com
Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user j...@nanthrax.net
Not sending mail to unregistered user g...@telligent-data.com
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user dariusz.aniszew...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user ccla...@bluewin.ch
Not sending mail to unregistered user fjetum...@gmail.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user aromanenko@gmail.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user mott...@gmail.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user z...@giggles.nyc.corp.google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user joey.bar...@gmail.com

[jira] [Resolved] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-3668.

Resolution: Fixed

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: full_log.txt
>
>  Time Spent: 2h 20m
>  Remaining Estimate: 0h
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContex

[beam] branch release-2.3.0 updated: [BEAM-3668] Quick workaround fix for netty conflict waiting better fix by BEAM-3519

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new 2fdb200  [BEAM-3668] Quick workaround fix for netty conflict waiting 
better fix by BEAM-3519
2fdb200 is described below

commit 2fdb200cbedae64ff88c4fa624926b595b8aab90
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Feb 10 08:39:21 2018 +0100

[BEAM-3668] Quick workaround fix for netty conflict waiting better fix by 
BEAM-3519
---
 sdks/java/io/hadoop-input-format/pom.xml  |  3 +++
 sdks/java/io/jdbc/pom.xml |  3 +++
 .../src/main/resources/archetype-resources/pom.xml| 19 +++
 3 files changed, 25 insertions(+)

diff --git a/sdks/java/io/hadoop-input-format/pom.xml 
b/sdks/java/io/hadoop-input-format/pom.xml
index 3bf3064..0fbd13a 100644
--- a/sdks/java/io/hadoop-input-format/pom.xml
+++ b/sdks/java/io/hadoop-input-format/pom.xml
@@ -75,6 +75,9 @@
 
 
   spark-runner
+  
+4.0.43.Final
+  
   
 
   org.apache.beam
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index 3af599e..17eb9f5 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -40,6 +40,9 @@
 
 
   spark-runner
+  
+4.0.43.Final
+  
   
 
   org.apache.beam
diff --git 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
index 9da3499..410e24b 100644
--- 
a/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
+++ 
b/sdks/java/maven-archetypes/examples/src/main/resources/archetype-resources/pom.xml
@@ -243,6 +243,9 @@
   spark-runner
   
+  
+4.0.43.Final
+  
   
 
   org.apache.beam
@@ -274,6 +277,22 @@
   ${jackson.version}
   runtime
 
+
+
+  org.apache.beam
+  beam-sdks-java-io-google-cloud-platform
+  ${beam.version}
+  
+
+  io.grpc
+  grpc-netty
+
+
+  io.netty
+  netty-handler
+
+  
+
   
 
   

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3519:
---
Fix Version/s: (was: 2.3.0)
   2.4.0

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Blocker
> Fix For: 2.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-3519) GCP IO exposes netty on its API surface, causing conflicts with runners

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-3519:
---
Priority: Critical  (was: Blocker)

> GCP IO exposes netty on its API surface, causing conflicts with runners
> ---
>
> Key: BEAM-3519
> URL: https://issues.apache.org/jira/browse/BEAM-3519
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Chamikara Jayalath
>Priority: Critical
> Fix For: 2.4.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Google Cloud Platform IOs module leaks netty this causes conflicts in 
> particular with execution systems that use conflicting versions of such 
> modules. 
>  For the case there is a dependency conflict with the Spark Runner version of 
> netty, see: BEAM-3492



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] 01/01: Merge pull request #4653 from jbonofre/BEAM-3668

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 67ba5531c71d273dbbfa356f244bb2bb670b2bdf
Merge: bd1f87d 12396d8
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Feb 10 18:47:49 2018 +0100

Merge pull request #4653 from jbonofre/BEAM-3668

[BEAM-3668] Quick workaround fix for netty conflict waiting better fix by 
BEAM-3519

 sdks/java/io/hadoop-input-format/pom.xml  |  3 +++
 sdks/java/io/jdbc/pom.xml |  3 +++
 .../src/main/resources/archetype-resources/pom.xml| 19 +++
 3 files changed, 25 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] branch master updated (bd1f87d -> 67ba553)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from bd1f87d  Merge pull request #4575: Add a single-stage fusion 
implementation
 add 12396d8  [BEAM-3668] Quick workaround fix for netty conflict waiting 
better fix by BEAM-3519
 new 67ba553  Merge pull request #4653 from jbonofre/BEAM-3668

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/hadoop-input-format/pom.xml  |  3 +++
 sdks/java/io/jdbc/pom.xml |  3 +++
 .../src/main/resources/archetype-resources/pom.xml| 19 +++
 3 files changed, 25 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #19

[Apache Jenkins Server]

_LOCATION/staging-validatesrunner-test \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 403.761s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180210-152742
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:643804e7007ab4595042c9cfc82a570043585d118f32089c848898a608e71b96
Deleted: sha256:f38c469be2f8470fbf0a5e9faf4b15b3e1727a29405a7151e15522f3d643cf45
Deleted: sha256:38c09e99ee57631180437b8b632cadcf8a7cf9da2471d75e0f35e11da5c3f5d0
Deleted: sha256:a185c81ae04c671fd2116d757c869af5796f427e565f4a23be654fc8a2fd2c6f
Deleted: sha256:0fd385696f7daa4a8a926e20a40ed41b3c990807537e0ee44805fe3eb561f343
Deleted: sha256:224adcf4c512aebc75322d6848a1e0f9cd6292c242b17e087ea6c8d37b5d6513
Deleted: sha256:5c3b64e7b37ec02dd585b6262bf04753d9df7bc3dbcd3ec8a8c527c702ec19d9
Deleted: sha256:2c794ca25ac2dcd40adfb1ab3b48471aff58292d0ed33e64004e1253f48f74f4
Deleted: sha256:0b095261f3bce9a5c6a0abcffd76a3875a7cce25c9962d3f53e468591de69ddb
gcloud container images delete $CONTAINER:$TAG --quiet
Usage: gcloud container [optional flags] 
  group may be   clusters | node-pools | operations
  command may be get-server-config

Deploy and manage clusters of machines for running containers.

flags:
  Run `gcloud container --help`
  for the full list of available flags for this command.

global flags:
  Run `gcloud -h` for a description of flags available to all commands.

command groups:
  clusters   Deploy and teardown Google Container Engine clusters.
  node-pools Create and delete operations for Google Container
 Engine node pools.
  operations Get and list operations for Google Container Engine
 clusters.

commands:
  get-server-config  Get Container Engine server config.


For more detailed information on this command and its flags, run:
  gcloud container --help

ERROR: (gcloud.container) Invalid choice: 'images'.

Valid choices are [clusters, get-server-config, node-pools, operations].
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user xuming...@users.noreply.github.com
Not sending mail to unregistered user jiang...@gmail.com
Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user j...@nanthrax.net
Not sending mail to unregistered user g...@telligent-data.com
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user dariusz.aniszew...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user ccla...@bluewin.ch
Not sending mail to unregistered user fjetum...@gmail.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user aromanenko@gmail.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user mott...@gmail.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user z...@giggles.nyc.corp.google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user joey.bar...@gmail.com

[beam] branch master updated (5056e06 -> bd1f87d)

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 5056e06  Merge pull request #4500 from jbonofre/BEAM-793
 add 9c6b5ff  Add a Primitive Impulse PTransform
 add 96da623  Add a single-stage fusion implementation
 new bd1f87d  Merge pull request #4575: Add a single-stage fusion 
implementation

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../resources/org/apache/beam/model/common_urns.md |   1 +
 ...KeyTranslation.java => ImpulseTranslation.java} |  28 +-
 .../core/construction/PTransformTranslation.java   |   1 +
 .../core/construction/graph/ExecutableStage.java   |  84 +++
 .../graph/GreedilyFusedExecutableStage.java| 195 +
 .../graph/GreedyPCollectionFusers.java | 257 +++
 .../graph/GreedilyFusedExecutableStageTest.java| 826 +
 .../org/apache/beam/sdk/transforms/Impulse.java|  59 ++
 8 files changed, 1437 insertions(+), 14 deletions(-)
 copy 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/{GroupByKeyTranslation.java
 => ImpulseTranslation.java} (72%)
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/ExecutableStage.java
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedilyFusedExecutableStage.java
 create mode 100644 
runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/GreedyPCollectionFusers.java
 create mode 100644 
runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/GreedilyFusedExecutableStageTest.java
 create mode 100644 
sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Impulse.java

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

[beam] 01/01: Merge pull request #4575: Add a single-stage fusion implementation

[tgroh]

This is an automated email from the ASF dual-hosted git repository.

tgroh pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit bd1f87db3f72d6969b470b5532b31004a4c8a3f7
Merge: 5056e06 96da623
Author: Thomas Groh 
AuthorDate: Sat Feb 10 07:08:08 2018 -0800

Merge pull request #4575: Add a single-stage fusion implementation

[BEAM-3565] Add a single-stage fusion implementation

 .../resources/org/apache/beam/model/common_urns.md |   1 +
 .../core/construction/ImpulseTranslation.java  |  65 ++
 .../core/construction/PTransformTranslation.java   |   1 +
 .../core/construction/graph/ExecutableStage.java   |  84 +++
 .../graph/GreedilyFusedExecutableStage.java| 195 +
 .../graph/GreedyPCollectionFusers.java | 257 +++
 .../graph/GreedilyFusedExecutableStageTest.java| 826 +
 .../org/apache/beam/sdk/transforms/Impulse.java|  59 ++
 8 files changed, 1488 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
tg...@apache.org.

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #5930

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #897

[Apache Jenkins Server]

See 


Changes:

[jb] [BEAM-793] Add backoff support in JdbcIO Write

--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam7 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 5056e06b7bbb6a70e00797e13de99a7492237be0 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 5056e06b7bbb6a70e00797e13de99a7492237be0
Commit message: "Merge pull request #4500 from jbonofre/BEAM-793"
 > git rev-list 959748d3f5851b2c55b76ba55168ef9fe3381518 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7880557541166470474.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1743051203744816413.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8762727440896774168.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3411991460026477517.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5942578897369914037.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1681661042754828352.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more informat

Build failed in Jenkins: beam_PostCommit_Python_ValidatesContainer_Dataflow #18

[Apache Jenkins Server]

st \
--temp_location=$GCS_LOCATION/temp-validatesrunner-test \
--output=$GCS_LOCATION/output \
--sdk_location=$SDK_LOCATION \
--num_workers=1"
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/container/local/lib/python2.7/site-packages/setuptools/dist.py>:355:
 UserWarning: Normalizing '2.4.0.dev' to '2.4.0.dev0'
  normalized_version,
running nosetests
running egg_info
writing requirements to apache_beam.egg-info/requires.txt
writing apache_beam.egg-info/PKG-INFO
writing top-level names to apache_beam.egg-info/top_level.txt
writing dependency_links to apache_beam.egg-info/dependency_links.txt
writing entry points to apache_beam.egg-info/entry_points.txt
reading manifest file 'apache_beam.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching 'README.md'
warning: no files found matching 'NOTICE'
warning: no files found matching 'LICENSE'
writing manifest file 'apache_beam.egg-info/SOURCES.txt'
<https://builds.apache.org/job/beam_PostCommit_Python_ValidatesContainer_Dataflow/ws/src/sdks/python/apache_beam/io/gcp/gcsio.py>:166:
 DeprecationWarning: object() takes no parameters
  super(GcsIO, cls).__new__(cls, storage_client))
test_wordcount_fnapi_it (apache_beam.examples.wordcount_it_test.WordCountIT) 
... ok

--
Ran 1 test in 393.992s

OK

# Delete the container locally and remotely
docker rmi $CONTAINER:$TAG
Untagged: us.gcr.io/apache-beam-testing/jenkins/python:20180210-092812
Untagged: 
us.gcr.io/apache-beam-testing/jenkins/python@sha256:0fee89c0b766fe49a87cf24d8b6b0ec23af45a02098cf326d1c9d3480d89fec6
Deleted: sha256:85419c3478002d7df3c382e6934cb812781e7729294019fe92e5904dcac831be
Deleted: sha256:a3efad53b34c5f42940624ad734b851cc6cf03dbb93732441b00164faa2d129e
Deleted: sha256:4783c48b042e04340f98806bb83d776fdc808b49bdfdf97755d8fa339723a495
Deleted: sha256:c4d3decfeba93387e8cc45482bde39cd862dee4b135f707e79afb3669773156f
Deleted: sha256:e03d13ec404dd72a814f05fda41a6e70914b43d723b44c88a7d68419835fae23
Deleted: sha256:cc9eff9e767086022086eff0ab21974b6f92ba7d74b8667258331c9153f9146c
Deleted: sha256:9dc7d459880adae43267ef3597bad7a8c8baba12e6f982d7111fcc40360682d1
Deleted: sha256:07d40e1594c58df9e03b29ab7a10ed9b91676a4afd60565a8754b99adec7d6a5
gcloud container images delete $CONTAINER:$TAG --quiet
Usage: gcloud container [optional flags] 
  group may be   clusters | node-pools | operations
  command may be get-server-config

Deploy and manage clusters of machines for running containers.

flags:
  Run `gcloud container --help`
  for the full list of available flags for this command.

global flags:
  Run `gcloud -h` for a description of flags available to all commands.

command groups:
  clusters   Deploy and teardown Google Container Engine clusters.
  node-pools Create and delete operations for Google Container
 Engine node pools.
  operations Get and list operations for Google Container Engine
 clusters.

commands:
  get-server-config  Get Container Engine server config.


For more detailed information on this command and its flags, run:
  gcloud container --help

ERROR: (gcloud.container) Invalid choice: 'images'.

Valid choices are [clusters, get-server-config, node-pools, operations].
Build step 'Execute shell' marked build as failure
Not sending mail to unregistered user daniel.o.program...@gmail.com
Not sending mail to unregistered user xuming...@users.noreply.github.com
Not sending mail to unregistered user jiang...@gmail.com
Not sending mail to unregistered user pawel.pk.kaczmarc...@gmail.com
Not sending mail to unregistered user sid...@google.com
Not sending mail to unregistered user hero...@google.com
Not sending mail to unregistered user kirpic...@google.com
Not sending mail to unregistered user aljoscha.kret...@gmail.com
Not sending mail to unregistered user apill...@google.com
Not sending mail to unregistered user j...@nanthrax.net
Not sending mail to unregistered user g...@telligent-data.com
Not sending mail to unregistered user eh...@google.com
Not sending mail to unregistered user dariusz.aniszew...@polidea.com
Not sending mail to unregistered user ekirpic...@gmail.com
Not sending mail to unregistered user ccla...@bluewin.ch
Not sending mail to unregistered user fjetum...@gmail.com
Not sending mail to unregistered user mari...@mariagh.svl.corp.google.com
Not sending mail to unregistered user aromanenko@gmail.com
Not sending mail to unregistered user mair...@google.com
Not sending mail to unregistered user mott...@gmail.com
Not sending mail to unregistered user c...@google.com
Not sending mail to unregistered user w...@google.com
Not sending mail to unregistered user z...@giggles.nyc.corp.google.com
Not sending mail to unregistered user git...@alasdairhodge.co.uk
Not sending mail to unregistered user ke...@google.com
Not sending mail to unregistered user k...@google.com
Not sending mail to unregistered user joey.bar...@gmail.com

Jenkins build became unstable: beam_PostCommit_Java_MavenInstall #5929

[Apache Jenkins Server]

See 

[jira] [Assigned] (BEAM-3668) Apache Spark Java Quickstart fails 2.3.0 RC2

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3668:
--

Assignee: Jean-Baptiste Onofré  (was: Ismaël Mejía)

> Apache Spark Java Quickstart fails 2.3.0 RC2
> 
>
> Key: BEAM-3668
> URL: https://issues.apache.org/jira/browse/BEAM-3668
> Project: Beam
>  Issue Type: Bug
>  Components: examples-java, runner-spark
>Affects Versions: 2.3.0
>Reporter: Luke Cwik
>Assignee: Jean-Baptiste Onofré
>Priority: Blocker
> Fix For: 2.3.0
>
> Attachments: full_log.txt
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Running the quickstart command fails:
> {code:java}
> mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount \
>  -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" 
> -Pspark-runner
> {code}
> with:
> {code:java}
> SEVERE: Error sending result 
> StreamResponse{streamId=/jars/json4s-ast_2.11-3.2.11.jar, byteCount=82421, 
> body=FileSegmentManagedBuffer{file=/usr/local/google/home/lcwik/.m2/repository/org/json4s/json4s-ast_2.11/3.2.11/json4s-ast_2.11-3.2.11.jar,
>  offset=0, length=82421}} to /100.100.214.107:48220; closing connection
> java.lang.AbstractMethodError
> at io.netty.util.ReferenceCountUtil.touch(ReferenceCountUtil.java:73)
> at 
> io.netty.channel.DefaultChannelPipeline.touch(DefaultChannelPipeline.java:107)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:811)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at 
> io.netty.handler.codec.MessageToMessageEncoder.write(MessageToMessageEncoder.java:111)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite(AbstractChannelHandlerContext.java:731)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:817)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:724)
> at io.netty.handler.timeout.IdleStateHandler.write(IdleStateHandler.java:305)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWrite0(AbstractChannelHandlerContext.java:739)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeWriteAndFlush(AbstractChannelHandlerContext.java:802)
> at 
> io.netty.channel.AbstractChannelHandlerContext.write(AbstractChannelHandlerContext.java:815)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:795)
> at 
> io.netty.channel.AbstractChannelHandlerContext.writeAndFlush(AbstractChannelHandlerContext.java:832)
> at 
> io.netty.channel.DefaultChannelPipeline.writeAndFlush(DefaultChannelPipeline.java:1032)
> at io.netty.channel.AbstractChannel.writeAndFlush(AbstractChannel.java:296)
> at 
> org.apache.spark.network.server.TransportRequestHandler.respond(TransportRequestHandler.java:192)
> at 
> org.apache.spark.network.server.TransportRequestHandler.processStreamRequest(TransportRequestHandler.java:148)
> at 
> org.apache.spark.network.server.TransportRequestHandler.handle(TransportRequestHandler.java:109)
> at 
> org.apache.spark.network.server.TransportChannelHandler.channelRead(TransportChannelHandler.java:118)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:349)
> at 
> io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:341)
> at 
> org.apache.spark.network.util.TransportFrameDecoder.channelRead(TransportFrameDecoder.java:85)
> at 
> io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
> at 
> io.netty

[jira] [Resolved] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré resolved BEAM-793.
---
Resolution: Fixed

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch release-2.3.0 updated: [BEAM-793] Add backoff support in JdbcIO Write

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch release-2.3.0
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/release-2.3.0 by this push:
 new e91a03f  [BEAM-793] Add backoff support in JdbcIO Write
e91a03f is described below

commit e91a03f096fe0dda7fd444a3389cbe1630b11a39
Author: Jean-Baptiste Onofré 
AuthorDate: Fri Jan 26 13:12:12 2018 +0100

[BEAM-793] Add backoff support in JdbcIO Write
---
 sdks/java/io/jdbc/build.gradle |   1 +
 sdks/java/io/jdbc/pom.xml  |  21 +++-
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   | 138 -
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 118 +++---
 4 files changed, 227 insertions(+), 51 deletions(-)

diff --git a/sdks/java/io/jdbc/build.gradle b/sdks/java/io/jdbc/build.gradle
index ab488c9..d610a4e 100644
--- a/sdks/java/io/jdbc/build.gradle
+++ b/sdks/java/io/jdbc/build.gradle
@@ -35,6 +35,7 @@ dependencies {
   shadow project(path: ":sdks:java:core", configuration: "shadow")
   shadow library.java.findbugs_jsr305
   shadow "org.apache.commons:commons-dbcp2:2.1.1"
+  testCompile project(path: ":sdks:java:core", configuration: "shadowTest")
   testCompile project(path: ":runners:direct-java", configuration: "shadow")
   testCompile project(path: ":sdks:java:io:common", configuration: "shadow")
   testCompile project(":sdks:java:io:common").sourceSets.test.output
diff --git a/sdks/java/io/jdbc/pom.xml b/sdks/java/io/jdbc/pom.xml
index 6264e9e..3af599e 100644
--- a/sdks/java/io/jdbc/pom.xml
+++ b/sdks/java/io/jdbc/pom.xml
@@ -264,6 +264,11 @@
 
 
 
+  org.slf4j
+  slf4j-api
+
+
+
   com.google.code.findbugs
   jsr305
 
@@ -274,6 +279,11 @@
   2.1.1
 
 
+
+  joda-time
+  joda-time
+
+
 
 
   com.google.auto.value
@@ -317,11 +327,6 @@
 
 
   org.slf4j
-  slf4j-api
-  test
-
-
-  org.slf4j
   slf4j-jdk14
   test
 
@@ -332,6 +337,12 @@
 
 
   org.apache.beam
+  beam-sdks-java-core
+  test
+  tests
+
+
+  org.apache.beam
   beam-sdks-java-io-common
   test
   tests
diff --git 
a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java 
b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
index 8b47aa9..1a6b54b 100644
--- a/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
+++ b/sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcIO.java
@@ -20,11 +20,15 @@ package org.apache.beam.sdk.io.jdbc;
 import static com.google.common.base.Preconditions.checkArgument;
 
 import com.google.auto.value.AutoValue;
+import java.io.IOException;
 import java.io.Serializable;
 import java.sql.Connection;
 import java.sql.PreparedStatement;
 import java.sql.ResultSet;
 import java.sql.SQLException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Predicate;
 import javax.annotation.Nullable;
 import javax.sql.DataSource;
 import org.apache.beam.sdk.annotations.Experimental;
@@ -39,11 +43,18 @@ import org.apache.beam.sdk.transforms.Reshuffle;
 import org.apache.beam.sdk.transforms.SerializableFunctions;
 import org.apache.beam.sdk.transforms.View;
 import org.apache.beam.sdk.transforms.display.DisplayData;
+import org.apache.beam.sdk.util.BackOff;
+import org.apache.beam.sdk.util.BackOffUtils;
+import org.apache.beam.sdk.util.FluentBackoff;
+import org.apache.beam.sdk.util.Sleeper;
 import org.apache.beam.sdk.values.PBegin;
 import org.apache.beam.sdk.values.PCollection;
 import org.apache.beam.sdk.values.PCollectionView;
 import org.apache.beam.sdk.values.PDone;
 import org.apache.commons.dbcp2.BasicDataSource;
+import org.joda.time.Duration;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
 
 /**
  * IO to read and write data on JDBC.
@@ -134,6 +145,9 @@ import org.apache.commons.dbcp2.BasicDataSource;
  */
 @Experimental(Experimental.Kind.SOURCE_SINK)
 public class JdbcIO {
+
+  private static final Logger LOG = LoggerFactory.getLogger(JdbcIO.class);
+
   /**
* Read data from a JDBC datasource.
*
@@ -164,9 +178,22 @@ public class JdbcIO {
   public static  Write write() {
 return new AutoValue_JdbcIO_Write.Builder()
 .setBatchSize(DEFAULT_BATCH_SIZE)
+.setRetryStrategy(new DefaultRetryStrategy())
 .build();
   }
 
+  /**
+   * This is the default {@link Predicate} we use to detect DeadLock.
+   * It basically test if the {@link SQLException#getSQLState()} equals 40001.
+   * 40001 is the SQL State used by most of database to identify deadlock.
+   */
+  public static class DefaultRetryStrategy implements RetryStrategy {
+@Override
+public boolean apply(SQLException e) {
+  return (e.getSQLState().equals("40001"));
+}
+  }
+
   private JdbcIO() {}
 
 

[jira] [Updated] (BEAM-793) JdbcIO can create a deadlock when parallelism is greater than 1

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré updated BEAM-793:
--
Fix Version/s: (was: 2.4.0)
   2.3.0

> JdbcIO can create a deadlock when parallelism is greater than 1
> ---
>
> Key: BEAM-793
> URL: https://issues.apache.org/jira/browse/BEAM-793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-extensions
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
> Fix For: 2.3.0
>
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> With the following JdbcIO configuration, if the parallelism is greater than 
> 1, we can have a {{Deadlock found when trying to get lock; try restarting 
> transaction}}.
> {code}
> MysqlDataSource dbCfg = new MysqlDataSource();
> dbCfg.setDatabaseName("db");
> dbCfg.setUser("user");
> dbCfg.setPassword("pass");
> dbCfg.setServerName("localhost");
> dbCfg.setPortNumber(3306);
> p.apply(Create.of(data))
> .apply(JdbcIO. Long>>write()
> 
> .withDataSourceConfiguration(JdbcIO.DataSourceConfiguration.create(dbCfg))
> .withStatement("INSERT INTO 
> smth(loc,event_type,hash,begin_date,end_date) VALUES(?, ?, ?, ?, ?) ON 
> DUPLICATE KEY UPDATE event_type=VALUES(event_type),end_date=VALUES(end_date)")
> .withPreparedStatementSetter(new 
> JdbcIO.PreparedStatementSetter Long>>() {
> public void setParameters(Tuple5 Integer, ByteString, Long, Long> element, PreparedStatement statement)
> throws Exception {
> statement.setInt(1, element.f0);
> statement.setInt(2, element.f1);
> statement.setBytes(3, 
> element.f2.toByteArray());
> statement.setLong(4, element.f3);
> statement.setLong(5, element.f4);
> }
> }));
> {code}
> This can happen due to the {{autocommit}}. I'm going to investigate.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (959748d -> 5056e06)

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 959748d  Merge pull request #3886: [BEAM-419] Fixing SE_BAD_FIELD 
FindBug in CombineFnUtil
 add e8ac922  [BEAM-793] Add backoff support in JdbcIO Write
 new 5056e06  Merge pull request #4500 from jbonofre/BEAM-793

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/java/io/jdbc/build.gradle |   1 +
 sdks/java/io/jdbc/pom.xml  |  21 +++-
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   | 138 -
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 118 +++---
 4 files changed, 227 insertions(+), 51 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.

[beam] 01/01: Merge pull request #4500 from jbonofre/BEAM-793

[jbonofre]

This is an automated email from the ASF dual-hosted git repository.

jbonofre pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 5056e06b7bbb6a70e00797e13de99a7492237be0
Merge: 959748d e8ac922
Author: Jean-Baptiste Onofré 
AuthorDate: Sat Feb 10 09:12:34 2018 +0100

Merge pull request #4500 from jbonofre/BEAM-793

[BEAM-793] Add backoff support in JdbcIO Write

 sdks/java/io/jdbc/build.gradle |   1 +
 sdks/java/io/jdbc/pom.xml  |  21 +++-
 .../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java   | 138 -
 .../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java| 118 +++---
 4 files changed, 227 insertions(+), 51 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
jbono...@apache.org.