Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5125

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5206

[Apache Jenkins Server]

See 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4386

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Spark #1450

[Apache Jenkins Server]

See 


Changes:

[ehudm] Update Dataflow Beam container version.

[robertwb] Remove obsolete MapTaskRunner.

[ccy] Revert #4666 "Use beam.io.WriteToBigQuery()"

[github] [BEAM-3806] Fix direct-runner hang (#4829)

--
[...truncated 90.52 KB...]
'apache-beam-testing:bqjob_r5e68bdef17b2db08_01620e9104a0_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 06:20:09,938 d3ceba86 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 06:20:28,734 d3ceba86 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 06:20:31,521 d3ceba86 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.76s,  CPU:0.38s,  MaxMemory:25328kb 
STDOUT: Upload complete.
Waiting on bqjob_r7aee8018af6d3fa5_01620e915981_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r7aee8018af6d3fa5_01620e915981_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r7aee8018af6d3fa5_01620e915981_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 06:20:31,522 d3ceba86 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 06:20:52,536 d3ceba86 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 06:20:55,048 d3ceba86 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.47s,  CPU:0.38s,  MaxMemory:25412kb 
STDOUT: Upload complete.
Waiting on bqjob_r6b7254370c4553c0_01620e91b625_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r6b7254370c4553c0_01620e91b625_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r6b7254370c4553c0_01620e91b625_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 06:20:55,049 d3ceba86 MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 06:21:10,922 d3ceba86 MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 06:21:15,485 d3ceba86 MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:04.55s,  CPU:0.42s,  MaxMemory:26128kb 
STDOUT: Upload complete.
Waiting on bqjob_r3f27326d3bf9c16e_01620e91ffda_1 ... (0s) Current status: 
RUNNING   

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Apex #3665

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-3820) SolrIO: Allow changing batchSize for writes

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394058#comment-16394058
 ] 

Jean-Baptiste Onofré commented on BEAM-3820:


It makes sense, I already did the change on some IOs (like {{JdbcIO}}).

> SolrIO: Allow changing batchSize for writes
> ---
>
> Key: BEAM-3820
> URL: https://issues.apache.org/jira/browse/BEAM-3820
> Project: Beam
>  Issue Type: Improvement
>  Components: io-java-solr
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Tim Robertson
>Assignee: Ismaël Mejía
>Priority: Trivial
>
> The SolrIO hard codes the batchSize for writes at 1000.  It would be a good 
> addition to allow the user to set the batchSize explicitly (similar to the 
> ElasticsearchIO)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Python #1006

[Apache Jenkins Server]

See 


Changes:

[ehudm] Update Dataflow Beam container version.

[robertwb] Remove obsolete MapTaskRunner.

[ccy] Revert #4666 "Use beam.io.WriteToBigQuery()"

[github] [BEAM-3806] Fix direct-runner hang (#4829)

--
[...truncated 1.11 KB...]
 > git rev-list --no-walk 66d6876a164498623c6419e07ed1ca4bde759434 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1642781944875400830.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2301095571331710073.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins1056548211769785322.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2113875930816926051.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins7189489620304260314.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3949047843813085076.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in 
/home/jenkins/.local/lib/python2.7/site-packages (from absl-py->-r 

Jenkins build is back to normal : beam_PerformanceTests_TextIOIT #251

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (BEAM-3823) Separate RAT (and probably Checkstyle, etc) into their own Jenkins job

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394056#comment-16394056
 ] 

Jean-Baptiste Onofré edited comment on BEAM-3823 at 3/10/18 6:06 AM:
-

Why not moving RAT in a dedicated profile and adding this profile on Jenkins 
(for the jobs we want) ?


was (Author: jbonofre):
Why not move RAT in a dedicated profile and adding this profile on Jenkins (for 
the jobs we want) ?

> Separate RAT (and probably Checkstyle, etc) into their own Jenkins job
> --
>
> Key: BEAM-3823
> URL: https://issues.apache.org/jira/browse/BEAM-3823
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Currently, RAT is added to both Java and Go presubmit (or is about to be).
> So these tests will go red, making it a little bit harder to see at a glance 
> the status of a PR. Since there are no dependencies, these can be another 
> status that will return quickly but also not obscure other signal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3823) Separate RAT (and probably Checkstyle, etc) into their own Jenkins job

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jean-Baptiste Onofré reassigned BEAM-3823:
--

Assignee: Jean-Baptiste Onofré

> Separate RAT (and probably Checkstyle, etc) into their own Jenkins job
> --
>
> Key: BEAM-3823
> URL: https://issues.apache.org/jira/browse/BEAM-3823
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Currently, RAT is added to both Java and Go presubmit (or is about to be).
> So these tests will go red, making it a little bit harder to see at a glance 
> the status of a PR. Since there are no dependencies, these can be another 
> status that will return quickly but also not obscure other signal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3823) Separate RAT (and probably Checkstyle, etc) into their own Jenkins job

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16394056#comment-16394056
 ] 

Jean-Baptiste Onofré commented on BEAM-3823:


Why not move RAT in a dedicated profile and adding this profile on Jenkins (for 
the jobs we want) ?

> Separate RAT (and probably Checkstyle, etc) into their own Jenkins job
> --
>
> Key: BEAM-3823
> URL: https://issues.apache.org/jira/browse/BEAM-3823
> Project: Beam
>  Issue Type: Bug
>  Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Jean-Baptiste Onofré
>Priority: Minor
>
> Currently, RAT is added to both Java and Go presubmit (or is about to be).
> So these tests will go red, making it a little bit harder to see at a glance 
> the status of a PR. Since there are no dependencies, these can be another 
> status that will return quickly but also not obscure other signal.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #236

[Apache Jenkins Server]

See 


Changes:

[ehudm] Update Dataflow Beam container version.

[robertwb] Remove obsolete MapTaskRunner.

[ccy] Revert #4666 "Use beam.io.WriteToBigQuery()"

[github] [BEAM-3806] Fix direct-runner hang (#4829)

--
[...truncated 11.74 KB...]
Collecting pyvcf<0.7.0,>=0.6.8 (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/usr/local/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.5.0.dev0)
  Using cached typing-3.6.4-py2-none-any.whl
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
./.env/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting google-apitools<=0.5.20,>=0.5.18 (from apache-beam==2.5.0.dev0)
  Using cached google_apitools-0.5.20-py2-none-any.whl
Collecting proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 (from 
apache-beam==2.5.0.dev0)
Collecting googledatastore==7.0.1 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-pubsub==0.26.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_pubsub-0.26.0-py2.py3-none-any.whl
Collecting proto-google-cloud-pubsub-v1==0.15.4 (from apache-beam==2.5.0.dev0)
Collecting google-cloud-bigquery==0.25.0 (from apache-beam==2.5.0.dev0)
  Using cached google_cloud_bigquery-0.25.0-py2.py3-none-any.whl
Collecting pyhamcrest<2.0,>=1.9 (from apache-beam==2.5.0.dev0)
  Using cached PyHamcrest-1.9.0-py2.py3-none-any.whl
Requirement already satisfied: enum34>=1.0.4 in 
/usr/local/lib/python2.7/dist-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/usr/local/lib/python2.7/dist-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
  Using cached pyasn1_modules-0.2.1-py2.py3-none-any.whl
Requirement already satisfied: pyasn1>=0.1.7 in 
/usr/local/lib/python2.7/dist-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.5.0.dev0)
Collecting fasteners>=0.14 (from 
google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached fasteners-0.14.1-py2.py3-none-any.whl
Collecting googleapis-common-protos<2.0dev,>=1.5.2 (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.5.0.dev0)
Collecting google-cloud-core<0.26dev,>=0.25.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_cloud_core-0.25.0-py2.py3-none-any.whl
Collecting gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Collecting monotonic>=0.1 (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
  Using cached monotonic-1.4-py2.py3-none-any.whl
Collecting google-auth<2.0.0dev,>=0.4.0 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth-1.4.1-py2.py3-none-any.whl
Collecting google-auth-httplib2 (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_auth_httplib2-0.0.3-py2.py3-none-any.whl
Collecting google-gax<0.16dev,>=0.15.7 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
  Using cached google_gax-0.15.16-py2.py3-none-any.whl
Collecting grpc-google-iam-v1<0.12dev,>=0.11.1 (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Collecting cachetools>=2.0.0 (from 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #234

[Apache Jenkins Server]

See 


Changes:

[ehudm] Update Dataflow Beam container version.

[robertwb] Remove obsolete MapTaskRunner.

[ccy] Revert #4666 "Use beam.io.WriteToBigQuery()"

[github] [BEAM-3806] Fix direct-runner hang (#4829)

--
[...truncated 10.12 KB...]
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/usr/lib/python2.7/dist-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
:318:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/security.html#snimissingwarning.
  SNIMissingWarning
:122:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/security.html#insecureplatformwarning.
  InsecurePlatformWarning
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #5

[Apache Jenkins Server]

See 


Changes:

[ehudm] Update Dataflow Beam container version.

[robertwb] Remove obsolete MapTaskRunner.

[ccy] Revert #4666 "Use beam.io.WriteToBigQuery()"

[github] [BEAM-3806] Fix direct-runner hang (#4829)

--
[...truncated 7.69 KB...]
Successfully installed numpy-1.13.3
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins8664735570801088295.sh
+ .env/bin/pip install -e 'src/sdks/python/[gcp,test]'
Obtaining 
file://
Requirement already satisfied: avro<2.0.0,>=1.8.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: crcmod<2.0,>=1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: dill==0.2.6 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: grpcio<2,>=1.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Collecting hdfs<3.0.0,>=2.1.0 (from apache-beam==2.5.0.dev0)
Requirement already satisfied: httplib2<0.10,>=0.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: mock<3.0.0,>=1.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: oauth2client<5,>=2.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: protobuf<4,>=3.5.0.post1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyyaml<4.0.0,>=3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyvcf<0.7.0,>=0.6.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: six<1.12,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: typing<3.7.0,>=3.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: futures<4.0.0,>=3.1.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-apitools<=0.5.20,>=0.5.18 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0 
in /home/jenkins/.local/lib/python2.7/site-packages (from 
apache-beam==2.5.0.dev0)
Requirement already satisfied: googledatastore==7.0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-pubsub==0.26.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: proto-google-cloud-pubsub-v1==0.15.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-bigquery==0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: pyhamcrest<2.0,>=1.9 in 
/home/jenkins/.local/lib/python2.7/site-packages (from apache-beam==2.5.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
grpcio<2,>=1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: requests>=2.7.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: docopt in /usr/local/lib/python2.7/dist-packages 
(from hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: funcsigs>=1; python_version < "3.3" in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pbr>=0.11 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
mock<3.0.0,>=1.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pyasn1>=0.1.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: pyasn1-modules>=0.0.5 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: rsa>=3.1.4 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
oauth2client<5,>=2.0.1->apache-beam==2.5.0.dev0)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from protobuf<4,>=3.5.0.post1->apache-beam==2.5.0.dev0)
Requirement already satisfied: fasteners>=0.14 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5124

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3803) Dataflow runner should handle metrics per the spec

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3803?focusedWorklogId=79165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79165
 ]

ASF GitHub Bot logged work on BEAM-3803:


Author: ASF GitHub Bot
Created on: 10/Mar/18 04:41
Start Date: 10/Mar/18 04:41
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4841: BEAM-3803: 
Dataflow runner implements metrics contract
URL: https://github.com/apache/beam/pull/4841#issuecomment-372002295
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79165)
Time Spent: 4.5h  (was: 4h 20m)

> Dataflow runner should handle metrics per the spec
> --
>
> Key: BEAM-3803
> URL: https://issues.apache.org/jira/browse/BEAM-3803
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Labels: nexmark
>  Time Spent: 4.5h
>  Remaining Estimate: 0h
>
> The dataflow runner only supports committed metrics for batch jobs and 
> attempted metrics for streaming jobs. It should always support attempted 
> metrics and throw an UnsupportedOperationException when the metrics are 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Apex #3664

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5205

[Apache Jenkins Server]

See 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4403

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?focusedWorklogId=79161=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79161
 ]

ASF GitHub Bot logged work on BEAM-3824:


Author: ASF GitHub Bot
Created on: 10/Mar/18 03:17
Start Date: 10/Mar/18 03:17
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #4846: [BEAM-3824] Revert 
#4666 "Use beam.io.WriteToBigQuery()"
URL: https://github.com/apache/beam/pull/4846
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/apache_beam/examples/complete/game/game_stats.py 
b/sdks/python/apache_beam/examples/complete/game/game_stats.py
index 32f6f15e3a8..1f13ed180f6 100644
--- a/sdks/python/apache_beam/examples/complete/game/game_stats.py
+++ b/sdks/python/apache_beam/examples/complete/game/game_stats.py
@@ -163,6 +163,43 @@ def process(self, team_score, 
window=beam.DoFn.WindowParam):
 }
 
 
+class WriteToBigQuery(beam.PTransform):
+  """Generate, format, and write BigQuery table row information."""
+  def __init__(self, table_name, dataset, schema):
+"""Initializes the transform.
+Args:
+  table_name: Name of the BigQuery table to use.
+  dataset: Name of the dataset to use.
+  schema: Dictionary in the format {'column_name': 'bigquery_type'}
+"""
+super(WriteToBigQuery, self).__init__()
+self.table_name = table_name
+self.dataset = dataset
+self.schema = schema
+
+  def get_schema(self):
+"""Build the output table schema."""
+return ', '.join(
+'%s:%s' % (col, self.schema[col]) for col in self.schema)
+
+  def get_table(self, pipeline):
+"""Utility to construct an output table reference."""
+project = pipeline.options.view_as(GoogleCloudOptions).project
+return '%s:%s.%s' % (project, self.dataset, self.table_name)
+
+  def expand(self, pcoll):
+table = self.get_table(pcoll.pipeline)
+return (
+pcoll
+| 'ConvertToRow' >> beam.Map(
+lambda elem: {col: elem[col] for col in self.schema})
+| beam.io.Write(beam.io.BigQuerySink(
+table,
+schema=self.get_schema(),
+create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
+write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)))
+
+
 # [START abuse_detect]
 class CalculateSpammyUsers(beam.PTransform):
   """Filter out all but those users with a high clickrate, which we will
@@ -243,8 +280,7 @@ def run(argv=None):
   options = PipelineOptions(pipeline_args)
 
   # We also require the --project option to access --dataset
-  project = options.view_as(GoogleCloudOptions).project
-  if project is None:
+  if options.view_as(GoogleCloudOptions).project is None:
 parser.print_usage()
 print(sys.argv[0] + ': error: argument --project is required')
 sys.exit(1)
@@ -260,8 +296,6 @@ def run(argv=None):
   # Enforce that this pipeline is always run in streaming mode
   options.view_as(StandardOptions).streaming = True
 
-  table_spec_prefix = '{}:{}.{}'.format(project, args.dataset, args.table_name)
-
   with beam.Pipeline(options=options) as p:
 # Read events from Pub/Sub using custom timestamps
 raw_events = (
@@ -298,13 +332,6 @@ def run(argv=None):
 # updates for late data. Uses the side input derived above --the set of
 # suspected robots-- to filter out scores from those users from the sum.
 # Write the results to BigQuery.
-team_table_spec = table_spec_prefix + '_teams'
-team_table_schema = (
-'team:STRING, '
-'total_score:INTEGER, '
-'window_start:STRING, '
-'processing_time: STRING')
-
 (raw_events  # pylint: disable=expression-not-assigned
  | 'WindowIntoFixedWindows' >> beam.WindowInto(
  beam.window.FixedWindows(fixed_window_duration))
@@ -317,20 +344,19 @@ def run(argv=None):
  | 'ExtractAndSumScore' >> ExtractAndSumScore('team')
  # [END filter_and_calc]
  | 'TeamScoresDict' >> beam.ParDo(TeamScoresDict())
- | 'WriteTeamScoreSums' >> beam.io.WriteToBigQuery(
- team_table_spec,
- schema=team_table_schema,
- create_disposition=beam.io.BigQueryDisposition.CREATE_IF_NEEDED,
- write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND))
+ | 'WriteTeamScoreSums' >> WriteToBigQuery(
+ args.table_name + '_teams', args.dataset, {
+ 'team': 'STRING',
+ 'total_score': 'INTEGER',
+ 'window_start': 'STRING',
+ 'processing_time': 'STRING',
+ }))
 
 # [START session_calc]
 # Detect user sessions-- that is, a burst of activity separated by a gap
 # from further 

[beam] branch master updated (2dcff0e -> fc772c0)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2dcff0e  Merge pull request #4845 from robertwb/runner-cleanup
 add 75e286a  Revert #4666 "Use beam.io.WriteToBigQuery()"
 new fc772c0  Merge pull request #4846 from charlesccychen/revert-pr-4666

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../examples/complete/game/game_stats.py   | 73 ---
 .../examples/complete/game/hourly_team_score.py| 57 +++
 .../examples/complete/game/leader_board.py | 81 +++---
 3 files changed, 148 insertions(+), 63 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4846 from charlesccychen/revert-pr-4666

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fc772c08f62960b6b944b07bc38df20a1054d3f9
Merge: 2dcff0e 75e286a
Author: Ahmet Altay 
AuthorDate: Fri Mar 9 19:17:12 2018 -0800

Merge pull request #4846 from charlesccychen/revert-pr-4666

[BEAM-3824] Revert #4666 "Use beam.io.WriteToBigQuery()"

 .../examples/complete/game/game_stats.py   | 73 ---
 .../examples/complete/game/hourly_team_score.py| 57 +++
 .../examples/complete/game/leader_board.py | 81 +++---
 3 files changed, 148 insertions(+), 63 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

Jenkins build is unstable: beam_PostCommit_Java_ValidatesRunner_Spark #4384

[Apache Jenkins Server]

See 

[beam] branch master updated (2180974 -> 2dcff0e)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 2180974  [BEAM-3806] Fix direct-runner hang (#4829)
 add 582fcea  Remove obsolete MapTaskRunner.
 new 2dcff0e  Merge pull request #4845 from robertwb/runner-cleanup

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/portability/fn_api_runner_test.py  | 187 +++-
 .../runners/portability/maptask_executor_runner.py | 472 -
 .../portability/maptask_executor_runner_test.py| 223 --
 3 files changed, 175 insertions(+), 707 deletions(-)
 delete mode 100644 
sdks/python/apache_beam/runners/portability/maptask_executor_runner.py
 delete mode 100644 
sdks/python/apache_beam/runners/portability/maptask_executor_runner_test.py

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4845 from robertwb/runner-cleanup

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 2dcff0e7f47fe8bfb37346c78ebb9e19020f532a
Merge: 2180974 582fcea
Author: Robert Bradshaw 
AuthorDate: Fri Mar 9 17:47:04 2018 -0800

Merge pull request #4845 from robertwb/runner-cleanup

Remove obsolete MapTaskRunner.

 .../runners/portability/fn_api_runner_test.py  | 187 +++-
 .../runners/portability/maptask_executor_runner.py | 472 -
 .../portability/maptask_executor_runner_test.py| 223 --
 3 files changed, 175 insertions(+), 707 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] branch release-2.4.0 updated (237130c -> fadab7f)

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a change to branch release-2.4.0
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 237130c  Update dataflow container version.
 add 65d40df  Revert #4666 "Use beam.io.WriteToBigQuery()"
 new fadab7f  Merge pull request #4847 from charlesccychen/revert-pr-4666-cp

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../examples/complete/game/game_stats.py   | 73 ---
 .../examples/complete/game/hourly_team_score.py| 57 +++
 .../examples/complete/game/leader_board.py | 81 +++---
 3 files changed, 148 insertions(+), 63 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

[beam] 01/01: Merge pull request #4847 from charlesccychen/revert-pr-4666-cp

[robertwb]

This is an automated email from the ASF dual-hosted git repository.

robertwb pushed a commit to branch release-2.4.0
in repository https://gitbox.apache.org/repos/asf/beam.git

commit fadab7f42cd9ffdae1a7a53917dd8c390c9bd220
Merge: 237130c 65d40df
Author: Robert Bradshaw 
AuthorDate: Fri Mar 9 17:34:29 2018 -0800

Merge pull request #4847 from charlesccychen/revert-pr-4666-cp

CP #4846 (Revert #4666 "Use beam.io.WriteToBigQuery()")

 .../examples/complete/game/game_stats.py   | 73 ---
 .../examples/complete/game/hourly_team_score.py| 57 +++
 .../examples/complete/game/leader_board.py | 81 +++---
 3 files changed, 148 insertions(+), 63 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
rober...@apache.org.

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Flink #5204

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-3806) DirectRunner hangs if multiple timers set in the same bundle

[Ben Chambers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3806?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ben Chambers resolved BEAM-3806.

   Resolution: Fixed
Fix Version/s: 2.5.0

Fixed by PR https://github.com/apache/beam/pull/4829

> DirectRunner hangs if multiple timers set in the same bundle
> 
>
> Key: BEAM-3806
> URL: https://issues.apache.org/jira/browse/BEAM-3806
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Ben Chambers
>Assignee: Thomas Groh
>Priority: Major
> Fix For: 2.5.0
>
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> See the repro below:
> {code:java}
> package com.simbly.beam.cassandra;
> import org.apache.beam.sdk.coders.KvCoder;
> import org.apache.beam.sdk.coders.StringUtf8Coder;
> import org.apache.beam.sdk.state.TimeDomain;
> import org.apache.beam.sdk.state.Timer;
> import org.apache.beam.sdk.state.TimerSpec;
> import org.apache.beam.sdk.state.TimerSpecs;
> import org.apache.beam.sdk.testing.PAssert;
> import org.apache.beam.sdk.testing.TestPipeline;
> import org.apache.beam.sdk.testing.TestStream;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.joda.time.Duration;
> import org.junit.Rule;
> import org.junit.Test;
> public class DirectRunnerTest {
>   @Rule
>   public TestPipeline pipeline = TestPipeline.create();
>   @Test
>   public void badTimerBehavior() {
> TestStream> stream = TestStream
> .create(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
> .addElements(KV.of("key1", "v1"))
> .advanceWatermarkToInfinity();
> PCollection result = pipeline
> .apply(stream)
> .apply(ParDo.of(new TestDoFn()));
> PAssert.that(result).containsInAnyOrder("It works");
> pipeline.run().waitUntilFinish();
>   }
>   private static class TestDoFn extends DoFn, String> {
> @TimerId("timer")
> private final TimerSpec timer = TimerSpecs.timer(TimeDomain.EVENT_TIME);
> @ProcessElement
> public void process(ProcessContext c,
> @TimerId("timer") Timer timer) {
>   timer.offset(Duration.standardMinutes(10)).setRelative();
>   timer.offset(Duration.standardMinutes(30)).setRelative();
> }
> @OnTimer("timer")
> public void onTimer(OnTimerContext c, @TimerId("timer") Timer timer) {
>   c.output("It works");
> }
>   }
> }
> {code}
> From inspection, this seems to be caused by the logic in 
> [WatermarkManager|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java#L313],
>  which does the following if there are multiple timers for akey:
>  # Adds the first timer to the `pendingTimers`, `keyTimers`, and 
> `existingTimersForKey`.
>  # Removes the first timer from `keyTimers`
>  # Adds the second timer to `keyTimers` and `existingTimersForKey`.
> This leads to inconsistencies since pendingTimers has only the first timer, 
> keyTimers only the second, and existingTimers has both. This becomes more 
> problematic since one of these lists is used for *firing* (and thus releasing 
> holds) and the other is used for holds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3806) DirectRunner hangs if multiple timers set in the same bundle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3806?focusedWorklogId=79153=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79153
 ]

ASF GitHub Bot logged work on BEAM-3806:


Author: ASF GitHub Bot
Created on: 10/Mar/18 01:07
Start Date: 10/Mar/18 01:07
Worklog Time Spent: 10m 
  Work Description: bjchambers closed pull request #4829: [BEAM-3806] Fix 
direct-runner hang
URL: https://github.com/apache/beam/pull/4829
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index b01f166b625..747a6671941 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -26,7 +26,6 @@
 import com.google.common.collect.ComparisonChain;
 import com.google.common.collect.HashBasedTable;
 import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Ordering;
 import com.google.common.collect.SortedMultiset;
@@ -39,6 +38,7 @@
 import java.util.EnumMap;
 import java.util.HashMap;
 import java.util.HashSet;
+import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.NavigableSet;
@@ -143,7 +143,7 @@
* timestamp which indicates we have received all of the data and there will 
be no more on-time or
* late data. This value is represented by {@link 
WatermarkManager#THE_END_OF_TIME}.
*/
-  private interface Watermark {
+  @VisibleForTesting interface Watermark {
 /**
  * Returns the current value of this watermark.
  */
@@ -211,13 +211,13 @@ public static WatermarkUpdate fromTimestamps(Instant 
oldTime, Instant currentTim
*
* See {@link #refresh()} for more information.
*/
-  private static class AppliedPTransformInputWatermark implements Watermark {
+  @VisibleForTesting static class AppliedPTransformInputWatermark implements 
Watermark {
 private final Collection inputWatermarks;
 private final SortedMultiset pendingElements;
 
 // This tracks only the quantity of timers at each timestamp, for quickly 
getting the cross-key
 // minimum
-private final SortedMultiset pendingTimers;
+private final SortedMultiset pendingTimers;
 
 // Entries in this table represent the authoritative timestamp for which
 // a per-key-and-StateNamespace timer is set.
@@ -290,15 +290,15 @@ private synchronized void 
removePending(CommittedBundle completed) {
   pendingElements.remove(completed);
 }
 
-private synchronized Instant getEarliestTimerTimestamp() {
+@VisibleForTesting synchronized Instant getEarliestTimerTimestamp() {
   if (pendingTimers.isEmpty()) {
 return BoundedWindow.TIMESTAMP_MAX_VALUE;
   } else {
-return pendingTimers.firstEntry().getElement();
+return pendingTimers.firstEntry().getElement().getTimestamp();
   }
 }
 
-private synchronized void updateTimers(TimerUpdate update) {
+@VisibleForTesting synchronized void updateTimers(TimerUpdate update) {
   NavigableSet keyTimers =
   objectTimers.computeIfAbsent(update.key, k -> new TreeSet<>());
   Table existingTimersForKey =
@@ -311,10 +311,12 @@ private synchronized void updateTimers(TimerUpdate 
update) {
   existingTimersForKey.get(timer.getNamespace(), 
timer.getTimerId());
 
   if (existingTimer == null) {
-pendingTimers.add(timer.getTimestamp());
+pendingTimers.add(timer);
 keyTimers.add(timer);
   } else if (!existingTimer.equals(timer)) {
+pendingTimers.remove(existingTimer);
 keyTimers.remove(existingTimer);
+pendingTimers.add(timer);
 keyTimers.add(timer);
   } // else the timer is already set identically, so noop
 
@@ -329,7 +331,7 @@ private synchronized void updateTimers(TimerUpdate update) {
   existingTimersForKey.get(timer.getNamespace(), 
timer.getTimerId());
 
   if (existingTimer != null) {
-pendingTimers.remove(existingTimer.getTimestamp());
+pendingTimers.remove(existingTimer);
 keyTimers.remove(existingTimer);
 existingTimersForKey.remove(existingTimer.getNamespace(), 
existingTimer.getTimerId());
   }
@@ -338,12 +340,14 @@ private synchronized void 

[beam] branch master updated: [BEAM-3806] Fix direct-runner hang (#4829)

[bchambers]

This is an automated email from the ASF dual-hosted git repository.

bchambers pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git


The following commit(s) were added to refs/heads/master by this push:
 new 2180974  [BEAM-3806] Fix direct-runner hang (#4829)
2180974 is described below

commit 218097454f69288aa68e457ce0f9ea2e40dbb86b
Author: Ben Chambers 
AuthorDate: Fri Mar 9 17:07:51 2018 -0800

[BEAM-3806] Fix direct-runner hang (#4829)

Fix handling of timers in WatermarkManager

1. TimerUpdates should use an ordered collection so that the most recent
   timestamp set by user code is the one taken throughout the system.
2. Application of TimerUpdates needs to keep all three data structures
   in sync.
3. Add unit tests that the data structures are consistent.

Added ParDoTest hangs when run with the DirectRunner prior to the other 
changes.
---
 .../beam/runners/direct/WatermarkManager.java  | 38 ---
 .../beam/runners/direct/WatermarkManagerTest.java  | 54 ++
 .../org/apache/beam/sdk/transforms/ParDoTest.java  | 33 +
 3 files changed, 108 insertions(+), 17 deletions(-)

diff --git 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
index b01f166..747a667 100644
--- 
a/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
+++ 
b/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java
@@ -26,7 +26,6 @@ import com.google.common.base.MoreObjects;
 import com.google.common.collect.ComparisonChain;
 import com.google.common.collect.HashBasedTable;
 import com.google.common.collect.ImmutableList;
-import com.google.common.collect.ImmutableSet;
 import com.google.common.collect.Iterables;
 import com.google.common.collect.Ordering;
 import com.google.common.collect.SortedMultiset;
@@ -39,6 +38,7 @@ import java.util.Collections;
 import java.util.EnumMap;
 import java.util.HashMap;
 import java.util.HashSet;
+import java.util.LinkedHashSet;
 import java.util.List;
 import java.util.Map;
 import java.util.NavigableSet;
@@ -143,7 +143,7 @@ class WatermarkManager {
* timestamp which indicates we have received all of the data and there will 
be no more on-time or
* late data. This value is represented by {@link 
WatermarkManager#THE_END_OF_TIME}.
*/
-  private interface Watermark {
+  @VisibleForTesting interface Watermark {
 /**
  * Returns the current value of this watermark.
  */
@@ -211,13 +211,13 @@ class WatermarkManager {
*
* See {@link #refresh()} for more information.
*/
-  private static class AppliedPTransformInputWatermark implements Watermark {
+  @VisibleForTesting static class AppliedPTransformInputWatermark implements 
Watermark {
 private final Collection inputWatermarks;
 private final SortedMultiset pendingElements;
 
 // This tracks only the quantity of timers at each timestamp, for quickly 
getting the cross-key
 // minimum
-private final SortedMultiset pendingTimers;
+private final SortedMultiset pendingTimers;
 
 // Entries in this table represent the authoritative timestamp for which
 // a per-key-and-StateNamespace timer is set.
@@ -290,15 +290,15 @@ class WatermarkManager {
   pendingElements.remove(completed);
 }
 
-private synchronized Instant getEarliestTimerTimestamp() {
+@VisibleForTesting synchronized Instant getEarliestTimerTimestamp() {
   if (pendingTimers.isEmpty()) {
 return BoundedWindow.TIMESTAMP_MAX_VALUE;
   } else {
-return pendingTimers.firstEntry().getElement();
+return pendingTimers.firstEntry().getElement().getTimestamp();
   }
 }
 
-private synchronized void updateTimers(TimerUpdate update) {
+@VisibleForTesting synchronized void updateTimers(TimerUpdate update) {
   NavigableSet keyTimers =
   objectTimers.computeIfAbsent(update.key, k -> new TreeSet<>());
   Table existingTimersForKey =
@@ -311,10 +311,12 @@ class WatermarkManager {
   existingTimersForKey.get(timer.getNamespace(), 
timer.getTimerId());
 
   if (existingTimer == null) {
-pendingTimers.add(timer.getTimestamp());
+pendingTimers.add(timer);
 keyTimers.add(timer);
   } else if (!existingTimer.equals(timer)) {
+pendingTimers.remove(existingTimer);
 keyTimers.remove(existingTimer);
+pendingTimers.add(timer);
 keyTimers.add(timer);
   } // else the timer is already set identically, so noop
 
@@ -329,7 +331,7 @@ class WatermarkManager {
   existingTimersForKey.get(timer.getNamespace(), 
timer.getTimerId());
 
   if 

[jira] [Created] (BEAM-3825) Gradle presubmit not running all tests

[Ben Chambers (JIRA)]

Ben Chambers created BEAM-3825:
--

 Summary: Gradle presubmit not running all tests
 Key: BEAM-3825
 URL: https://issues.apache.org/jira/browse/BEAM-3825
 Project: Beam
  Issue Type: Bug
  Components: testing
Reporter: Ben Chambers
Assignee: Jason Kuster


For PR https://github.com/apache/beam/pull/4829

The following test failed (flakily) and succeeded on retry in Maven:

[https://builds.apache.org/job/beam_PreCommit_Java_MavenInstall/org.apache.beam$beam-runners-direct-java/18505/testReport/org.apache.beam.sdk.io/FileIOTest/testMatchWatchForNewFiles/]

 

While investigating the original failure, I was unable to find it in the logs 
of the Gradle presubmit, making we worry that it may not be covering all tsets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3818) Add support for the streaming side inputs in the Python DirectRunner

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3818?focusedWorklogId=79147=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79147
 ]

ASF GitHub Bot logged work on BEAM-3818:


Author: ASF GitHub Bot
Created on: 10/Mar/18 00:39
Start Date: 10/Mar/18 00:39
Worklog Time Spent: 10m 
  Work Description: mariapython commented on issue #4838: [BEAM-3818] Add 
support for streaming side inputs in the DirectRunner
URL: https://github.com/apache/beam/pull/4838#issuecomment-371984800
 
 
   R: @charlesccychen 
   cc: @aaltay, @robertwb 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79147)
Time Spent: 10m
Remaining Estimate: 0h

> Add support for the streaming side inputs in the Python DirectRunner
> 
>
> Key: BEAM-3818
> URL: https://issues.apache.org/jira/browse/BEAM-3818
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-py-core
>Reporter: María GH
>Assignee: María GH
>Priority: Minor
> Fix For: 3.0.0
>
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> The streaming DirectRunner should support streaming side input semantics.  
> Currently, side inputs are only available for globally-windowed side input 
> PCollections.
> Also, empty side inputs cause a pipeline stall.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam] branch master updated (66d6876 -> 7b7a104)

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.


from 66d6876  Merge pull request #4824: [BEAM-3802] Move metrics caching up 
a level
 add e99c5ba  Update Dataflow Beam container version.
 new 7b7a104  Merge pull request #4839 from udim/pubsub-fix

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 sdks/python/apache_beam/runners/dataflow/internal/dependency.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[beam] 01/01: Merge pull request #4839 from udim/pubsub-fix

[altay]

This is an automated email from the ASF dual-hosted git repository.

altay pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git

commit 7b7a1043222c9bb137c5233eab8d92aeb3da7147
Merge: 66d6876 e99c5ba
Author: Ahmet Altay 
AuthorDate: Fri Mar 9 16:36:08 2018 -0800

Merge pull request #4839 from udim/pubsub-fix

Update Dataflow Beam container version.

 sdks/python/apache_beam/runners/dataflow/internal/dependency.py | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

-- 
To stop receiving notification emails like this one, please contact
al...@apache.org.

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393863#comment-16393863
 ] 

Chamikara Jayalath commented on BEAM-3824:
--

Thanks Charles. Above PR is being reverted. Assigning the JIRA to David to 
update and resubmit the PR (not a release blocker).

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 

[jira] [Updated] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Valentyn Tymofieiev (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Valentyn Tymofieiev updated BEAM-3824:
--
Affects Version/s: 2.4.0

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Affects Versions: 2.4.0
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 537, in _reraise_augmented
>  

[jira] [Assigned] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath reassigned BEAM-3824:


Assignee: David Cavazos  (was: Chamikara Jayalath)

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: David Cavazos
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 537, in _reraise_augmented

[jira] [Work logged] (BEAM-3803) Dataflow runner should handle metrics per the spec

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3803?focusedWorklogId=79141=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79141
 ]

ASF GitHub Bot logged work on BEAM-3803:


Author: ASF GitHub Bot
Created on: 10/Mar/18 00:26
Start Date: 10/Mar/18 00:26
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4841: BEAM-3803: Dataflow 
runner implements metrics contract
URL: https://github.com/apache/beam/pull/4841#issuecomment-371983285
 
 
   run java gradle precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79141)
Time Spent: 4h 20m  (was: 4h 10m)

> Dataflow runner should handle metrics per the spec
> --
>
> Key: BEAM-3803
> URL: https://issues.apache.org/jira/browse/BEAM-3803
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Labels: nexmark
>  Time Spent: 4h 20m
>  Remaining Estimate: 0h
>
> The dataflow runner only supports committed metrics for batch jobs and 
> attempted metrics for streaming jobs. It should always support attempted 
> metrics and throw an UnsupportedOperationException when the metrics are 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3803) Dataflow runner should handle metrics per the spec

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3803?focusedWorklogId=79139=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79139
 ]

ASF GitHub Bot logged work on BEAM-3803:


Author: ASF GitHub Bot
Created on: 10/Mar/18 00:25
Start Date: 10/Mar/18 00:25
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4841: BEAM-3803: Dataflow 
runner implements metrics contract
URL: https://github.com/apache/beam/pull/4841#issuecomment-371983155
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79139)
Time Spent: 4h 10m  (was: 4h)

> Dataflow runner should handle metrics per the spec
> --
>
> Key: BEAM-3803
> URL: https://issues.apache.org/jira/browse/BEAM-3803
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Labels: nexmark
>  Time Spent: 4h 10m
>  Remaining Estimate: 0h
>
> The dataflow runner only supports committed metrics for batch jobs and 
> attempted metrics for streaming jobs. It should always support attempted 
> metrics and throw an UnsupportedOperationException when the metrics are 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Spark #1449

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3457] Drop Go Maven PreCommit

[herohde] [BEAM-3804] Build Go SDK container with gradle

[apilloud] [BEAM-3802] Remove broken cachedMetricResults

[cclauss] [BEAM-1251] Change unicode --> six.text_type for Python 3 - again

[herohde] Fix package name in Go container gradle build

--
[...truncated 90.70 KB...]
'apache-beam-testing:bqjob_r7094a7c6c4a35b47_01620d463dbf_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 00:18:52,107 084a3afa MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 00:19:18,442 084a3afa MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 00:19:21,256 084a3afa MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.79s,  CPU:0.34s,  MaxMemory:25280kb 
STDOUT: Upload complete.
Waiting on bqjob_r4b6e7ca7812e2a68_01620d46b034_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r4b6e7ca7812e2a68_01620d46b034_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r4b6e7ca7812e2a68_01620d46b034_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 00:19:21,256 084a3afa MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 00:19:37,118 084a3afa MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 00:19:39,890 084a3afa MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.75s,  CPU:0.33s,  MaxMemory:25312kb 
STDOUT: Upload complete.
Waiting on bqjob_r74dc6cabac489723_01620d46f932_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r74dc6cabac489723_01620d46f932_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r74dc6cabac489723_01620d46f932_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-03-10 00:19:39,891 084a3afa MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-03-10 00:19:59,026 084a3afa MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-03-10 00:20:01,938 084a3afa MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1,  WallTime:0:02.89s,  CPU:0.36s,  MaxMemory:25392kb 
STDOUT: Upload complete.
Waiting on 

[jira] [Commented] (BEAM-3817) Incompatible input encoding running Tornadoes example on dataflow

[Henning Rohde (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393856#comment-16393856
 ] 

Henning Rohde commented on BEAM-3817:
-

I took a look and the job uses the wrong container image: 
"gcr.io//beam/gcemd", not the "gcr.io//beam/go" image. The 
stackdriver logs show that is fails to start up, because it expects different 
arguments.

> Incompatible input encoding running Tornadoes example on dataflow
> -
>
> Key: BEAM-3817
> URL: https://issues.apache.org/jira/browse/BEAM-3817
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Braden Bassingthwaite
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Trying to run:
> go run tornadoes.go --output=:bbass.tornadoes --project  
> --runner dataflow --staging_location=gs://bbass/tornadoes 
> --worker_harness_container_image=gcr.io//beam/go
> Found here:
> [https://github.com/apache/beam/blob/master/sdks/go/examples/cookbook/tornadoes/tornadoes.go]
> I can run it locally but I get the error on Dataflow:
> (8fa522c2bb03a769): Workflow failed. Causes: (8fa522c2bb03ab04): Incompatible 
> input encoding. 
>  
> I built the worker_harness_container_image using:
> mvn clean install -DskipTests -Pbuild-containers 
> -Ddocker-repository-root=gcr.io//beam
>  
> Thanks!
>  
> Very excited to start using the golang beam sdk! great work!
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Charles Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393855#comment-16393855
 ] 

Charles Chen commented on BEAM-3824:


I found that the root cause is [https://github.com/apache/beam/pull/4666].  The 
pipeline passes with this change reversed.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

Jenkins build is back to normal : beam_PerformanceTests_TFRecordIOIT #232

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PerformanceTests_Python #1005

[Apache Jenkins Server]

See 


Changes:

[herohde] [BEAM-3457] Drop Go Maven PreCommit

[herohde] [BEAM-3804] Build Go SDK container with gradle

[apilloud] [BEAM-3802] Remove broken cachedMetricResults

[cclauss] [BEAM-1251] Change unicode --> six.text_type for Python 3 - again

[herohde] Fix package name in Go container gradle build

--
[...truncated 1.33 KB...]
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins498390656689407407.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3562087228917105324.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins5288758864350628877.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4303336847699464537.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/ad/dc/fcced9ec3f2561c0cbe8eb6527eef7cf4f4919a2b3a07891a36e846635af/setuptools-38.5.2-py2.py3-none-any.whl#md5=abd3307cdce6fb543b5a4d0e3e98bdb6
Downloading/unpacking pip from 
https://pypi.python.org/packages/b6/ac/7015eb97dc749283ffdec1c3a88ddb8ae03b8fad0f0e611408f196358da3/pip-9.0.1-py2.py3-none-any.whl#md5=297dbd16ef53bcef0447d245815f5144
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2111574020502214142.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins4692268690038569384.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt 

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393847#comment-16393847
 ] 

Chamikara Jayalath commented on BEAM-3824:
--

I confirmed that this works for 2.3 but doesn't work for HEAD with 
BundleBasedDirectRunner so seems to be a regression unrelated to the new runner.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #4400

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-1589) Add OnWindowExpiration method to Stateful DoFn

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1589?focusedWorklogId=79137=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79137
 ]

ASF GitHub Bot logged work on BEAM-1589:


Author: ASF GitHub Bot
Created on: 10/Mar/18 00:06
Start Date: 10/Mar/18 00:06
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4482: [BEAM-1589]  
Added @onWindowExpiration annotation.
URL: https://github.com/apache/beam/pull/4482#issuecomment-371980672
 
 
   Ah, sorry I waited long enough that this needs a rebase to resolve the 
conflicts.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79137)
Time Spent: 10m
Remaining Estimate: 0h

> Add OnWindowExpiration method to Stateful DoFn
> --
>
> Key: BEAM-1589
> URL: https://issues.apache.org/jira/browse/BEAM-1589
> Project: Beam
>  Issue Type: New Feature
>  Components: runner-core, sdk-java-core
>Reporter: Jingsong Lee
>Assignee: Batkhuyag Batsaikhan
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> See BEAM-1517
> This allows the user to do some work before the state's garbage collection.
> It seems kind of annoying, but on the other hand forgetting to set a final 
> timer to flush state is probably data loss most of the time.
> FlinkRunner does this work very simply, but other runners, such as 
> DirectRunner, need to traverse all the states to do this, and maybe it's a 
> little hard.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3793) SDK harness should log bad gRPC responses

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3793.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> SDK harness should log bad gRPC responses
> -
>
> Key: BEAM-3793
> URL: https://issues.apache.org/jira/browse/BEAM-3793
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ben Sidhom
>Assignee: Henning Rohde
>Priority: Minor
>  Labels: portability
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> As of now, if a null ProvisionInfo is passed to the boot code, it panics when 
> trying to convert to JSON. It should at the very least log a useful error 
> message if a bad ProvisionInfo is received.
>  
> Sample stack trace:
> panic: reflect: call of reflect.Value.Field on zero Value
> goroutine 1 [running]:
> reflect.Value.Field(0x0, 0x0, 0x0, 0x0, 0x0, 0x1, 0x81f2e0)
>  /Users/sidhom/.gradle/go/binary/1.9/go/src/reflect/value.go:759 +0x12e
> github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb.(*Marshaler).marshalObject(0xc42015cb10,
>  0xc42010, 0xb7a7e0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc4201bbbd0, 0x410e58)
>  
> /Users/sidhom/Downloads/git/beam/sdks/java/container/.gogradle/project_gopath/src/github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb/jsonpb.go:207
>  +0x1040
> github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb.(*Marshaler).Marshal(0xc42015cb10,
>  0xb70ae0, 0xc4201ac770, 0xb7a7e0, 0x0, 0x8a53e0, 0x4c3601)
>  
> /Users/sidhom/Downloads/git/beam/sdks/java/container/.gogradle/project_gopath/src/github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb/jsonpb.go:122
>  +0xc1
> github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb.(*Marshaler).MarshalToString(0xc42015cb10,
>  0xb7a7e0, 0x0, 0x1, 0x15, 0x0, 0x0)
>  
> /Users/sidhom/Downloads/git/beam/sdks/java/container/.gogradle/project_gopath/src/github.com/apache/beam/sdks/java/boot/vendor/github.com/golang/protobuf/jsonpb/jsonpb.go:128
>  +0x6e
> github.com/apache/beam/sdks/java/boot/vendor/github.com/apache/beam/sdks/go/pkg/beam/provision.ProtoToJSON(0x0,
>  0x12, 0xc4201bbdc0, 0x1, 0x1)
>  
> /Users/sidhom/Downloads/git/beam/sdks/java/container/.gogradle/project_gopath/src/github.com/apache/beam/sdks/java/boot/vendor/github.com/apache/beam/sdks/go/pkg/beam/provision/provison.go:79
>  +0x51
> main.main()
>  
> /Users/sidhom/Downloads/git/beam/sdks/java/container/.gogradle/project_gopath/src/github.com/apache/beam/sdks/java/boot/boot.go:80
>  +0x309



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Charles Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393843#comment-16393843
 ] 

Charles Chen commented on BEAM-3824:


It looks like this is not an issue with the new DirectRunner implementation–the 
same error occurs when we use the old BundleBasedDirectRunner.  It is possible 
there is some other difference between 2.3.0 and now (maybe a dependency 
issue?).

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #4

[Apache Jenkins Server]

See 


Changes:

[markliu] [BEAM-3750] Remove TestPipeline.covnertToArgs from integration tests

[apilloud] [Nexmark][SQL] Convert SQL Rows to Java models

[apilloud] [Nexmark][SQL] Output java pojos

[apilloud] [Nexmark][SQL] Add model tests for sql

[herohde] [BEAM-3457] Drop Go Maven PreCommit

[markliu] Fix integration tests that use NumberedSharedFiles

[herohde] [BEAM-3804] Build Go SDK container with gradle

[herohde] [BEAM-3793] Validate provision response and add beamctl support

[cclauss] [BEAM-1251] Fix basestring, file(), reduce(), and xrange() for Python 
3

[apilloud] [Nexmark] Use queue for Query 6 model

[apilloud] [Nexmark] Ensure enough data to produce output

[altay] Fix topic URIs.

[apilloud] [BEAM-3802] Remove broken cachedMetricResults

[cclauss] [BEAM-1251] Change unicode --> six.text_type for Python 3 - again

[herohde] Fix package name in Go container gradle build

[apilloud] [Nexmark] Sickbay query 6

[chamikara] [BEAM-3734] Performance tests for XmlIO  (#4747)

--
[...truncated 14.43 KB...]
Requirement already satisfied: googleapis-common-protos<2.0dev,>=1.5.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
proto-google-cloud-datastore-v1<=0.90.4,>=0.90.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: google-cloud-core<0.26dev,>=0.25.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: idna<2.7,>=2.5 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: chardet<3.1.0,>=3.0.2 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: certifi>=2017.4.17 in 
/usr/local/lib/python2.7/dist-packages (from 
requests>=2.7.0->hdfs<3.0.0,>=2.1.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: monotonic>=0.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
fasteners>=0.14->google-apitools<=0.5.20,>=0.5.18->apache-beam==2.5.0.dev0)
Requirement already satisfied: google-auth<2.0.0dev,>=0.4.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: google-auth-httplib2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: google-gax<0.16dev,>=0.15.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: grpc-google-iam-v1<0.12dev,>=0.11.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: cachetools>=2.0.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-auth<2.0.0dev,>=0.4.0->google-cloud-core<0.26dev,>=0.25.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: future<0.17dev,>=0.16.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Requirement already satisfied: ply==3.8 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
google-gax<0.16dev,>=0.15.7->gapic-google-cloud-pubsub-v1<0.16dev,>=0.15.0->google-cloud-pubsub==0.26.0->apache-beam==2.5.0.dev0)
Installing collected packages: hdfs, apache-beam
  Running setup.py develop for apache-beam
Successfully installed apache-beam hdfs-2.1.0
[beam_PerformanceTests_XmlIOIT] $ /bin/bash -xe 
/tmp/jenkins6570920591093042887.sh
+ .env/bin/python PerfKitBenchmarker/pkb.py --project=apache-beam-testing 
--dpb_log_level=INFO --maven_binary=/home/jenkins/tools/maven/latest/bin/mvn 
--bigquery_table=beam_performance.xmlioit_pkb_results 
--temp_dir= 
--official=true --benchmarks=beam_integration_benchmark --beam_it_timeout=1200 
--beam_it_profile=io-it --beam_prebuilt=true --beam_sdk=java 
--beam_it_module=sdks/java/io/file-based-io-tests 
--beam_it_class=org.apache.beam.sdk.io.xml.XmlIOIT 

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Charles Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393808#comment-16393808
 ] 

Charles Chen commented on BEAM-3824:


Thanks, I will send you a PR to make sure we use the old DirectRunner 
implementation in this case.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393802#comment-16393802
 ] 

Valentyn Tymofieiev commented on BEAM-3824:
---

I don't think we tested this example on FnAPI runner before, so sounds like 
this error may be a result of recent changes in Direct runner implementation.

[~charleschen]/[~robertwb] what's the suggested course of action here?

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393797#comment-16393797
 ] 

Chamikara Jayalath commented on BEAM-3824:
--

cc: [~charleschen] in case this is related to recent updates to DirectRunner.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Chamikara Jayalath (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393795#comment-16393795
 ] 

Chamikara Jayalath commented on BEAM-3824:
--

Did this work with Fn API runner before ? I've only tried out BQ with Python 
batch runner.

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4399

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-3802] Remove broken cachedMetricResults

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 

[jira] [Work logged] (BEAM-3803) Dataflow runner should handle metrics per the spec

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3803?focusedWorklogId=79128=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79128
 ]

ASF GitHub Bot logged work on BEAM-3803:


Author: ASF GitHub Bot
Created on: 09/Mar/18 23:09
Start Date: 09/Mar/18 23:09
Worklog Time Spent: 10m 
  Work Description: apilloud commented on issue #4841: BEAM-3803: Dataflow 
runner implements metrics contract
URL: https://github.com/apache/beam/pull/4841#issuecomment-371971449
 
 
   run java precommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79128)
Time Spent: 4h  (was: 3h 50m)

> Dataflow runner should handle metrics per the spec
> --
>
> Key: BEAM-3803
> URL: https://issues.apache.org/jira/browse/BEAM-3803
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Andrew Pilloud
>Assignee: Andrew Pilloud
>Priority: Major
>  Labels: nexmark
>  Time Spent: 4h
>  Remaining Estimate: 0h
>
> The dataflow runner only supports committed metrics for batch jobs and 
> attempted metrics for streaming jobs. It should always support attempted 
> metrics and throw an UnsupportedOperationException when the metrics are 
> missing.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[beam-site] 02/02: This closes #397

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 3629fd4170c1b6c5cd22ce31c43da1f04b6646dc
Merge: afb4164 315acb1
Author: Mergebot 
AuthorDate: Fri Mar 9 15:08:16 2018 -0800

This closes #397

 src/documentation/dsls/sql.md | 337 --
 1 file changed, 226 insertions(+), 111 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] branch mergebot updated (941d495 -> 3629fd4)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 941d495  This closes #396
 discard c06a5a1  Update site to mention the 2 Python lint envs
 add 770fe8f  Update site to mention the 2 Python lint envs
 add afb4164  This closes #396
 new 315acb1  Update SQL doc to match new APIs
 new 3629fd4  This closes #397

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (941d495)
\
 N -- N -- N   refs/heads/mergebot (3629fd4)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 src/documentation/dsls/sql.md | 337 --
 1 file changed, 226 insertions(+), 111 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
mergebot-r...@apache.org.

[beam-site] 01/02: Update SQL doc to match new APIs

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 315acb1b25fd573d126160409284dd7ec46750cd
Author: akedin 
AuthorDate: Fri Mar 2 11:16:18 2018 -0800

Update SQL doc to match new APIs

BeamSql.querySimple() and queryMulti() were combined into query().
BeamRecord was renamed to Row. Factory methods and builders were added to 
it.
---
 src/documentation/dsls/sql.md | 337 --
 1 file changed, 226 insertions(+), 111 deletions(-)

diff --git a/src/documentation/dsls/sql.md b/src/documentation/dsls/sql.md
index 2f6fafc..a6289dc 100644
--- a/src/documentation/dsls/sql.md
+++ b/src/documentation/dsls/sql.md
@@ -7,104 +7,145 @@ permalink: /documentation/dsls/sql/
 
 # Beam SQL
 
-* TOC
-{:toc}
-
 This page describes the implementation of Beam SQL, and how to simplify a Beam 
pipeline with DSL APIs.
 
 ## 1. Overview {#overview}
 
-SQL is a well-adopted standard to process data with concise syntax. With DSL 
APIs (currently available only in Java), now `PCollection`s can be queried with 
standard SQL statements, like a regular table. The DSL APIs leverage [Apache 
Calcite](http://calcite.apache.org/) to parse and optimize SQL queries, then 
translate into a composite Beam `PTransform`. In this way, both SQL and normal 
Beam `PTransform`s can be mixed in the same pipeline.
+SQL is a well-adopted standard to process data with concise syntax. With DSL 
APIs (currently available only in Java), now `PCollections` can be queried with 
standard SQL statements, like a regular table. The DSL APIs leverage [Apache 
Calcite](http://calcite.apache.org/) to parse and optimize SQL queries, then 
translate into a composite Beam `PTransform`. In this way, both SQL and normal 
Beam `PTransforms` can be mixed in the same pipeline.
 
 There are two main pieces to the SQL DSL API:
 
-* [BeamRecord]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest }}/index.html?org/apache/beam/sdk/values/BeamRecord.html): 
a new data type used to define composite records (i.e., rows) that consist of 
multiple, named columns of primitive data types. All SQL DSL queries must be 
made against collections of type `PCollection`. Note that 
`BeamRecord` itself is not SQL-specific, however, and may also be used in 
pipelines that do not utilize SQL.
-* [BeamSql]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/BeamSql.html): the interface 
for creating `PTransforms` from SQL queries.
+* [BeamSql]({{ site.baseurl }}/documentation/sdks/javadoc/{{ 
site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/BeamSql.html): the interface 
for creating `PTransforms` from SQL queries;
+* [Row]({{ site.baseurl }}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/values/Row.html) contains named columns with 
corresponding data types. Beam SQL queries can be made only against collections 
of type `PCollection`;
 
 We'll look at each of these below.
 
 ## 2. Usage of DSL APIs {#usage}
 
-### BeamRecord
+### Row
 
-Before applying a SQL query to a `PCollection`, the data in the collection 
must be in `BeamRecord` format. A `BeamRecord` represents a single, immutable 
row in a Beam SQL `PCollection`. The names and types of the fields/columns in 
the record are defined by its associated [BeamRecordType]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/values/BeamRecordType.html); for SQL queries, 
you should use the [BeamRecordSqlType]({{ site.baseurl [...]
+Before applying a SQL query to a `PCollection`, the data in the collection 
must be in `Row` format. A `Row` represents a single, immutable record in a 
Beam SQL `PCollection`. The names and types of the fields/columns in the row 
are defined by its associated [RowType]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/values/RowType.html).
+For SQL queries, you should use the [RowSqlType.builder()]({{ site.baseurl 
}}/documentation/sdks/javadoc/{{ site.release_latest 
}}/index.html?org/apache/beam/sdk/extensions/sql/RowSqlType.html) to create 
`RowTypes`, it allows creating schemas with all supported SQL types (see [Data 
Types](#data-types) for more details on supported primitive data types).
 
 
-A `PCollection` can be created explicitly or implicitly:
+A `PCollection` can be obtained multiple ways, for example:
 
-Explicitly:
-  * **From in-memory data** (typically for unit testing). In this case, the 
record type and coder must be specified explicitly:
-```
+  * **From in-memory data** (typically for unit testing).
+
+**Note:** you have to explicitly specify the `Row` coder. In this example 
we're doing it by calling `Create.of(..).withCoder()`:
+
+

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79123=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79123
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:48
Start Date: 09/Mar/18 22:48
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-371967470
 
 
   +R: @alanmyrvold 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79123)
Time Spent: 62h 20m  (was: 62h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 62h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79124=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79124
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:48
Start Date: 09/Mar/18 22:48
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: [BEAM-3339] Mobile 
gaming automation for Java nightly snapshot on core runners
URL: https://github.com/apache/beam/pull/4788#issuecomment-371967506
 
 
   +R: @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79124)
Time Spent: 62.5h  (was: 62h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 62.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3806) DirectRunner hangs if multiple timers set in the same bundle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3806?focusedWorklogId=79122=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79122
 ]

ASF GitHub Bot logged work on BEAM-3806:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:47
Start Date: 09/Mar/18 22:47
Worklog Time Spent: 10m 
  Work Description: bjchambers commented on issue #4829: [BEAM-3806] Fix 
direct-runner hang
URL: https://github.com/apache/beam/pull/4829#issuecomment-371967233
 
 
   Run Java PreCommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79122)
Time Spent: 20m  (was: 10m)

> DirectRunner hangs if multiple timers set in the same bundle
> 
>
> Key: BEAM-3806
> URL: https://issues.apache.org/jira/browse/BEAM-3806
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Ben Chambers
>Assignee: Thomas Groh
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> See the repro below:
> {code:java}
> package com.simbly.beam.cassandra;
> import org.apache.beam.sdk.coders.KvCoder;
> import org.apache.beam.sdk.coders.StringUtf8Coder;
> import org.apache.beam.sdk.state.TimeDomain;
> import org.apache.beam.sdk.state.Timer;
> import org.apache.beam.sdk.state.TimerSpec;
> import org.apache.beam.sdk.state.TimerSpecs;
> import org.apache.beam.sdk.testing.PAssert;
> import org.apache.beam.sdk.testing.TestPipeline;
> import org.apache.beam.sdk.testing.TestStream;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.joda.time.Duration;
> import org.junit.Rule;
> import org.junit.Test;
> public class DirectRunnerTest {
>   @Rule
>   public TestPipeline pipeline = TestPipeline.create();
>   @Test
>   public void badTimerBehavior() {
> TestStream> stream = TestStream
> .create(KvCoder.of(StringUtf8Coder.of(), StringUtf8Coder.of()))
> .addElements(KV.of("key1", "v1"))
> .advanceWatermarkToInfinity();
> PCollection result = pipeline
> .apply(stream)
> .apply(ParDo.of(new TestDoFn()));
> PAssert.that(result).containsInAnyOrder("It works");
> pipeline.run().waitUntilFinish();
>   }
>   private static class TestDoFn extends DoFn, String> {
> @TimerId("timer")
> private final TimerSpec timer = TimerSpecs.timer(TimeDomain.EVENT_TIME);
> @ProcessElement
> public void process(ProcessContext c,
> @TimerId("timer") Timer timer) {
>   timer.offset(Duration.standardMinutes(10)).setRelative();
>   timer.offset(Duration.standardMinutes(30)).setRelative();
> }
> @OnTimer("timer")
> public void onTimer(OnTimerContext c, @TimerId("timer") Timer timer) {
>   c.output("It works");
> }
>   }
> }
> {code}
> From inspection, this seems to be caused by the logic in 
> [WatermarkManager|https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/WatermarkManager.java#L313],
>  which does the following if there are multiple timers for akey:
>  # Adds the first timer to the `pendingTimers`, `keyTimers`, and 
> `existingTimersForKey`.
>  # Removes the first timer from `keyTimers`
>  # Adds the second timer to `keyTimers` and `existingTimersForKey`.
> This leads to inconsistencies since pendingTimers has only the first timer, 
> keyTimers only the second, and existingTimers has both. This becomes more 
> problematic since one of these lists is used for *firing* (and thus releasing 
> holds) and the other is used for holds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Dataflow #5121

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1082

[Apache Jenkins Server]

See 


Changes:

[apilloud] [BEAM-3802] Remove broken cachedMetricResults

--
[...truncated 121.24 KB...]
pipeline.run()
  File 
"
 line 102, in run
result = super(TestPipeline, self).run()
  File 
"
 line 369, in run
self.to_runner_api(), self.runner, self._options).run(False)
  File 
"
 line 382, in run
return self.runner.run_pipeline(self)
  File 
"
 line 285, in run_pipeline
return_context=True)
  File 
"
 line 580, in to_runner_api
root_transform_id = context.transforms.get_id(self._root_transform())
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 810, in to_runner_api
for part in self.parts],
  File 
"
 line 60, in get_id
self._id_to_proto[id] = obj.to_runner_api(self._pipeline_context)
  File 
"
 line 808, in to_runner_api
spec=transform_to_runner_api(self.transform, context),
  File 
"
 line 805, in transform_to_runner_api
return transform.to_runner_api(context)
  File 
"
 line 542, in to_runner_api
urn, typed_param = self.to_runner_api_parameter(context)
  File 
"
 line 559, in to_runner_api_parameter
wrappers_pb2.BytesValue(value=pickler.dumps(self)))
  File 
"
 line 193, in dumps
s = dill.dumps(o)
  File 
"
 line 259, in dumps
dump(obj, file, protocol, byref, fmode, recurse)#, strictio)
  File 
"
 line 252, in dump
pik.dump(obj)
  File "/usr/lib/python2.7/pickle.py", line 224, in dump
self.save(obj)
  File "/usr/lib/python2.7/pickle.py", line 331, in save
self.save_reduce(obj=obj, *rv)
  File "/usr/lib/python2.7/pickle.py", line 396, in save_reduce
save(cls)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 94, in wrapper
obj=obj)
  File "/usr/lib/python2.7/pickle.py", line 401, in save_reduce
save(args)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File "/usr/lib/python2.7/pickle.py", line 562, in save_tuple
save(element)
  File "/usr/lib/python2.7/pickle.py", line 286, in save
f(self, obj) # Call unbound method with explicit self
  File 
"
 line 165, in new_save_module_dict
return old_save_module_dict(pickler, obj)
  File 
"
 line 841, in save_module_dict
StockPickler.save_dict(pickler, obj)
  File "/usr/lib/python2.7/pickle.py", line 649, in save_dict

[jira] [Commented] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Valentyn Tymofieiev (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16393681#comment-16393681
 ] 

Valentyn Tymofieiev commented on BEAM-3824:
---

Also do we understand why the error did not surface earlier?

> BQ sink fails on Direct Runner for 2.4.0 RC2
> 
>
> Key: BEAM-3824
> URL: https://issues.apache.org/jira/browse/BEAM-3824
> Project: Beam
>  Issue Type: Bug
>  Components: runner-direct
>Reporter: Valentyn Tymofieiev
>Assignee: Chamikara Jayalath
>Priority: Blocker
>
> python -m apache_beam.examples.complete.game.hourly_team_score 
> --project=$PROJECT --dataset=beam_release_2_4_0 
> --input=gs://$BUCKET/mobile/first_5000_gaming_data.csv
> The pipeline fails with:
> INFO:root:finish  output_tags=['out'], 
> receivers=[ConsumerSet[WriteTeamScoreSums/WriteToBigQuery.out0, 
> coder=WindowedValueCoder[FastPrimitivesCoder], len(consumers)=0]]> 
> Traceback (most recent call last):
>  File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
>  "__main__", fname, loader, pkg_name) 
>  File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
>  exec code in run_globals 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 276, in <
> module> 
>  run() 
>  File 
> "/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
>  line 270, in r
> un 
>  write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 389, in __exit__
>  self.run().wait_until_finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 369, in run
>  self.to_runner_api(), self.runner, self._options).run(False) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
>  line 382, in run
>  return self.runner.run_pipeline(self) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
>  line 129, in run_pip
> eline 
>  return runner.run_pipeline(pipeline)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 215, in ru
> n_pipeline 
>  return self.run_via_runner_api(pipeline.to_runner_api())
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 218, in ru
> n_via_runner_api 
>  return self.run_stages(*self.create_stages(pipeline_proto))
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 837, in ru
> n_stages 
>  pcoll_buffers, safe_coders).process_bundle.metrics
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 938, in ru
> n_stage 
>  self._progress_frequency).process_bundle(data_input, data_output)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1110, in p
> rocess_bundle 
>  result_future = self._controller.control_handler.push(process_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
>  line 1003, in p
> ush 
>  response = self.worker.do_instruction(request)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 185, in do_instruc
> tion 
>  request.instruction_id) 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
>  line 202, in process_bu
> ndle 
>  processor.process_bundle(instruction_id)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
>  line 298, in proc
> ess_bundle 
>  op.finish() 
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
>  line 389, in finish
>  self.dofn_runner.finish()
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 517, in finish
>  self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
>  File 
> "/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
>  line 508, in _invoke_bundle_method
>  self._reraise_augmented(exn) 
>  File 
> 

Build failed in Jenkins: beam_PostCommit_Python_Verify #4398

[Apache Jenkins Server]

See 


Changes:

[cclauss] [BEAM-1251] Change unicode --> six.text_type for Python 3 - again

--
[...truncated 1.02 MB...]
copying apache_beam/runners/dataflow/native_io/streaming_create.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/dataflow/native_io
copying apache_beam/runners/direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/bundle_factory.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/clock.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/consumer_tracking_pipeline_visitor_test.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_metrics_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/evaluation_context.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/executor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/helper_transforms.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/sdf_direct_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/transform_evaluator.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/util.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/direct/watermark_manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/direct
copying apache_beam/runners/experimental/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental
copying apache_beam/runners/experimental/python_rpc_direct/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying 
apache_beam/runners/experimental/python_rpc_direct/python_rpc_direct_runner.py 
-> apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/experimental/python_rpc_direct/server.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/experimental/python_rpc_direct
copying apache_beam/runners/job/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/manager.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/job/utils.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/job
copying apache_beam/runners/portability/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/fn_api_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/maptask_executor_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_main.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/portability/universal_local_runner_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/portability
copying apache_beam/runners/test/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/test
copying apache_beam/runners/worker/__init__.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/bundle_processor.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/data_plane_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/log_handler_test.py -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.pxd -> 
apache-beam-2.5.0.dev0/apache_beam/runners/worker
copying apache_beam/runners/worker/logger.py -> 

Jenkins build is back to normal : beam_PostRelease_NightlySnapshot #121

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-3824) BQ sink fails on Direct Runner for 2.4.0 RC2

[Valentyn Tymofieiev (JIRA)]

Valentyn Tymofieiev created BEAM-3824:
-

 Summary: BQ sink fails on Direct Runner for 2.4.0 RC2
 Key: BEAM-3824
 URL: https://issues.apache.org/jira/browse/BEAM-3824
 Project: Beam
  Issue Type: Bug
  Components: runner-direct
Reporter: Valentyn Tymofieiev
Assignee: Chamikara Jayalath


python -m apache_beam.examples.complete.game.hourly_team_score 
--project=$PROJECT --dataset=beam_release_2_4_0 
--input=gs://$BUCKET/mobile/first_5000_gaming_data.csv

The pipeline fails with:

INFO:root:finish  
Traceback (most recent call last):
 File "/usr/lib/python2.7/runpy.py", line 174, in _run_module_as_main
 "__main__", fname, loader, pkg_name) 
 File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
 exec code in run_globals 
 File 
"/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
 line 276, in <
module> 
 run() 
 File 
"/tmp/release_testing/r2.4.0_env/lib/python2.7/site-packages/apache_beam/examples/complete/game/hourly_team_score.py",
 line 270, in r
un 
 write_disposition=beam.io.BigQueryDisposition.WRITE_APPEND)) 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
 line 389, in __exit__
 self.run().wait_until_finish() 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
 line 369, in run
 self.to_runner_api(), self.runner, self._options).run(False) 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/pipeline.py",
 line 382, in run
 return self.runner.run_pipeline(self) 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
 line 129, in run_pip
eline 
 return runner.run_pipeline(pipeline)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 215, in ru
n_pipeline 
 return self.run_via_runner_api(pipeline.to_runner_api())
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 218, in ru
n_via_runner_api 
 return self.run_stages(*self.create_stages(pipeline_proto))
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 837, in ru
n_stages 
 pcoll_buffers, safe_coders).process_bundle.metrics
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 938, in ru
n_stage 
 self._progress_frequency).process_bundle(data_input, data_output)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 1110, in p
rocess_bundle 
 result_future = self._controller.control_handler.push(process_bundle)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
 line 1003, in p
ush 
 response = self.worker.do_instruction(request)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 185, in do_instruc
tion 
 request.instruction_id) 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
 line 202, in process_bu
ndle 
 processor.process_bundle(instruction_id)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
 line 298, in proc
ess_bundle 
 op.finish() 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/worker/operations.py",
 line 389, in finish
 self.dofn_runner.finish()
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
 line 517, in finish
 self._invoke_bundle_method(self.do_fn_invoker.invoke_finish_bundle)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
 line 508, in _invoke_bundle_method
 self._reraise_augmented(exn) 
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/apache_beam/runners/common.py",
 line 537, in _reraise_augmented
 six.raise_from(new_exn, original_traceback)
 File 
"/tmp/release_testing/r2.4.0_env/local/lib/python2.7/site-packages/six.py", 
line 737, in raise_from
 raise value 
RuntimeError: Could not successfully insert rows to BigQuery table 
[google.com:clouddfe:beam_release_2_4_0.leader_board]. Errors: []
 index: 0>, ]
 index: 1>, ]
 index: 2>, ]
 index: 3>, ]
...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79106=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79106
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173577413
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79103=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79103
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173575019
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79098=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79098
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574620
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79104=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79104
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173575323
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79101=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79101
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173575420
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79108=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79108
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173576028
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79099=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79099
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574737
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79105=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79105
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173576181
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79096=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79096
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574242
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79111=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79111
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173575549
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79100=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79100
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574801
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79110=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79110
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173576223
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79102=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79102
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574856
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,748 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create()).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+@Nullable abstract String queue();
+@Nullable abstract String exchange();
+@Nullable abstract String exchangeType();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79107=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79107
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574874
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,748 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+
+import javax.annotation.Nullable;
+
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.ByteArrayCoder;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create()).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+@Nullable abstract String queue();
+@Nullable abstract String exchange();
+@Nullable abstract String exchangeType();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int 

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79109=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79109
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173577348
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-1240) Create RabbitMqIO

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-1240?focusedWorklogId=79097=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79097
 ]

ASF GitHub Bot logged work on BEAM-1240:


Author: ASF GitHub Bot
Created on: 09/Mar/18 22:02
Start Date: 09/Mar/18 22:02
Worklog Time Spent: 10m 
  Work Description: jkff commented on a change in pull request #1729: 
[BEAM-1240] Create RabbitMqIO
URL: https://github.com/apache/beam/pull/1729#discussion_r173574461
 
 

 ##
 File path: 
sdks/java/io/rabbitmq/src/main/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqIO.java
 ##
 @@ -0,0 +1,711 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.rabbitmq;
+
+import static com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import com.rabbitmq.client.AMQP;
+import com.rabbitmq.client.Channel;
+import com.rabbitmq.client.Connection;
+import com.rabbitmq.client.ConnectionFactory;
+import com.rabbitmq.client.QueueingConsumer;
+import java.io.IOException;
+import java.io.Serializable;
+import java.net.URISyntaxException;
+import java.security.KeyManagementException;
+import java.security.NoSuchAlgorithmException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.NoSuchElementException;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.annotations.Experimental;
+import org.apache.beam.sdk.coders.Coder;
+import org.apache.beam.sdk.coders.SerializableCoder;
+import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.transforms.DoFn;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.ParDo;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PDone;
+import org.joda.time.Duration;
+import org.joda.time.Instant;
+
+/**
+ * A IO to publish or consume messages with a RabbitMQ broker.
+ *
+ * Consuming messages from RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Read} returns an unbounded {@link PCollection} 
containing RabbitMQ
+ * messages body (as {@code byte[]}).
+ *
+ * To configure a RabbitMQ source, you have to provide a RabbitMQ {@code 
URI} to connect
+ * to a RabbitMQ broker. The following example illustrates various options for 
configuring the
+ * source:
+ *
+ * {@code
+ *
+ *  pipeline.apply(
+ *
RabbitMqIO.read().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE")
+ *
+ * }
+ *
+ * Publishing messages to RabbitMQ server
+ *
+ * {@link RabbitMqIO} {@link Write} can send {@code byte[]} to a RabbitMQ 
server queue.
+ *
+ * As for the {@link Read}, the {@link Write} is configured with a RabbitMQ
+ * {@link ConnectionConfig}.
+ *
+ * For instance:
+ *
+ * {@code
+ *
+ *  pipeline
+ *.apply(...) // provide PCollection
+ *
.apply(RabbitMqIO.write().withUri("amqp://user:password@localhost:5672").withQueue("QUEUE"));
+ *
+ * }
+ */
+@Experimental(Experimental.Kind.SOURCE_SINK)
+public class RabbitMqIO {
+
+  public static Read read() {
+return new AutoValue_RabbitMqIO_Read.Builder()
+.setConnectionConfig(ConnectionConfig.create()).setQueueDeclare(false)
+
.setMaxReadTime(null).setMaxNumRecords(Long.MAX_VALUE).setUseCorrelationId(false).build();
+  }
+
+  public static Write write() {
+return new AutoValue_RabbitMqIO_Write.Builder()
+.setConnectionConfig(ConnectionConfig.create())
+.setExchangeDeclare(false).build();
+  }
+
+  private RabbitMqIO() {
+  }
+
+  /**
+   * Describe a connection configuration to a RabbitMQ server.
+   */
+  @AutoValue
+  public abstract static class ConnectionConfig implements Serializable {
+
+@Nullable abstract String uri();
+
+abstract int networkRecoveryInterval();
+abstract boolean automaticRecovery();
+abstract boolean topologyRecovery();
+
+abstract int connectionTimeout();
+abstract int requestedChannelMax();
+abstract int requestedFrameMax();
+abstract int requestedHeartbeat();
+
+abstract Builder builder();
+
+

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79088=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79088
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370687641
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79088)
Time Spent: 61h 40m  (was: 61.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79089
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370881722
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79089)
Time Spent: 61h 50m  (was: 61h 40m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61h 50m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79085=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79085
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370665517
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79085)
Time Spent: 61h 10m  (was: 61h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79091=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79091
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370910010
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79091)
Time Spent: 62h 10m  (was: 62h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 62h 10m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79090=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79090
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370898387
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79090)
Time Spent: 62h  (was: 61h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 62h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79082=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79082
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371906539
 
 
   Run Seed Job


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79082)
Time Spent: 60h 40m  (was: 60.5h)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 60h 40m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79087
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370673256
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79087)
Time Spent: 61.5h  (was: 61h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79086=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79086
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370672090
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79086)
Time Spent: 61h 20m  (was: 61h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79081=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79081
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371643954
 
 
   Run Python ReleaseCandidate


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79081)
Time Spent: 60.5h  (was: 60h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 60.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79080=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79080
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371643689
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79080)
Time Spent: 60h 20m  (was: 60h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 60h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79084=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79084
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370662991
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79084)
Time Spent: 61h  (was: 60h 50m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 61h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79083=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79083
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:47
Start Date: 09/Mar/18 21:47
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371908599
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79083)
Time Spent: 60h 50m  (was: 60h 40m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 60h 50m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79075=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79075
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:46
Start Date: 09/Mar/18 21:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371369984
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79075)
Time Spent: 59.5h  (was: 59h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 59.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79051=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79051
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:46
Start Date: 09/Mar/18 21:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-370650233
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79051)
Time Spent: 55.5h  (was: 55h 20m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 55.5h
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3339) Create post-release testing of the nightly snapshots

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3339?focusedWorklogId=79062=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-79062
 ]

ASF GitHub Bot logged work on BEAM-3339:


Author: ASF GitHub Bot
Created on: 09/Mar/18 21:46
Start Date: 09/Mar/18 21:46
Worklog Time Spent: 10m 
  Work Description: yifanzou commented on issue #4788: BEAM-3339 Mobile 
gaming automation for Java nightly snapshot -- testing
URL: https://github.com/apache/beam/pull/4788#issuecomment-371021438
 
 
   Run Dataflow PostRelease


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 79062)
Time Spent: 57h 20m  (was: 57h 10m)

> Create post-release testing of the nightly snapshots
> 
>
> Key: BEAM-3339
> URL: https://issues.apache.org/jira/browse/BEAM-3339
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Alan Myrvold
>Assignee: Jason Kuster
>Priority: Major
>  Time Spent: 57h 20m
>  Remaining Estimate: 0h
>
> The nightly java snapshots in 
> https://repository.apache.org/content/groups/snapshots/org/apache/beam should 
> be verified by following the 
> https://beam.apache.org/get-started/quickstart-java/ instructions, to verify 
> that the release is usable.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)