Build failed in Jenkins: beam_PostCommit_Python_Verify #3215

[Apache Jenkins Server]

See 


--
[...truncated 44.47 KB...]
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting six<1.11,>=1.9 (from apache-beam==2.2.0.dev0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.6-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-agYiAW-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpWKtny3pip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-agYiAW-build/setup.py", line 203, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-agYiAW-build/setup.py", line 143, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, six, funcsigs, 
pbr, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: six 1.11.0
Uninstalling six-1.11.0:
  Successfully uninstalled six-1.11.0
  Found existing installation: protobuf 3.4.0
Uninstalling protobuf-3.4.0:
  Successfully uninstalled protobuf-3.4.0
  Running setup.py install for apache-beam: started
Running setup.py install for apache-beam: finished with status 'error'
Complete output from 

[jira] [Commented] (BEAM-2724) MSEC counters should support Structured Names in Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180118#comment-16180118
 ] 

ASF GitHub Bot commented on BEAM-2724:
--

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/3786


> MSEC counters should support Structured Names in Dataflow
> -
>
> Key: BEAM-2724
> URL: https://issues.apache.org/jira/browse/BEAM-2724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2724) MSEC counters should support Structured Names in Dataflow

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16180119#comment-16180119
 ] 

ASF GitHub Bot commented on BEAM-2724:
--

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/3786

[BEAM-2724] Preparing statesampler to work with structured names




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam ssampler-structured

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3786


commit 6897c0998bf14440f20ec6263d0e4efd2c81924a
Author: Pablo 
Date:   2017-08-29T22:23:20Z

Preparing statesampler to work with structured names

commit 9e6f3c00ac9ac908020cb9d9ad81ef1807c38ad7
Author: Pablo 
Date:   2017-09-21T15:57:00Z

Support iotarget for ssampler

commit b3d629fb27b7150b79f1bfcea973f5ff0c0a1b78
Author: Pablo 
Date:   2017-09-21T17:53:57Z

Improving documentation

commit b6e79645b5d6d2d356755f0d64f5244f9b3fa9e1
Author: Pablo 
Date:   2017-09-25T17:59:33Z

Fix missing arg

commit f10b3747bb7c869dab00ffc28d50b4cab4e238aa
Author: Pablo 
Date:   2017-09-25T22:47:21Z

Addressing comments

commit 932f5a1e632d7666f2e671095c603b69a0c823ec
Author: Pablo 
Date:   2017-09-25T23:13:54Z

Fix nit




> MSEC counters should support Structured Names in Dataflow
> -
>
> Key: BEAM-2724
> URL: https://issues.apache.org/jira/browse/BEAM-2724
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Reporter: Pablo Estrada
>Assignee: Pablo Estrada
>




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3786: [BEAM-2724] Preparing statesampler to work with str...

[pabloem]

Github user pabloem closed the pull request at:

https://github.com/apache/beam/pull/3786


---

[GitHub] beam pull request #3786: [BEAM-2724] Preparing statesampler to work with str...

[pabloem]

GitHub user pabloem reopened a pull request:

https://github.com/apache/beam/pull/3786

[BEAM-2724] Preparing statesampler to work with structured names




You can merge this pull request into a Git repository by running:

$ git pull https://github.com/pabloem/incubator-beam ssampler-structured

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3786.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3786


commit 6897c0998bf14440f20ec6263d0e4efd2c81924a
Author: Pablo 
Date:   2017-08-29T22:23:20Z

Preparing statesampler to work with structured names

commit 9e6f3c00ac9ac908020cb9d9ad81ef1807c38ad7
Author: Pablo 
Date:   2017-09-21T15:57:00Z

Support iotarget for ssampler

commit b3d629fb27b7150b79f1bfcea973f5ff0c0a1b78
Author: Pablo 
Date:   2017-09-21T17:53:57Z

Improving documentation

commit b6e79645b5d6d2d356755f0d64f5244f9b3fa9e1
Author: Pablo 
Date:   2017-09-25T17:59:33Z

Fix missing arg

commit f10b3747bb7c869dab00ffc28d50b4cab4e238aa
Author: Pablo 
Date:   2017-09-25T22:47:21Z

Addressing comments

commit 932f5a1e632d7666f2e671095c603b69a0c823ec
Author: Pablo 
Date:   2017-09-25T23:13:54Z

Fix nit




---

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #4046

[Apache Jenkins Server]

See 

[beam-site] 01/01: Prepare repository for deployment.

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit 07615eaabc0c84ad2f2ac6818d5e00622b997c9d
Author: Mergebot 
AuthorDate: Tue Sep 26 01:24:09 2017 +

Prepare repository for deployment.
---
 .../runners/capability-matrix/index.html   | 30 +++---
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/content/documentation/runners/capability-matrix/index.html 
b/content/documentation/runners/capability-matrix/index.html
index 225fdaf..8939869 100644
--- a/content/documentation/runners/capability-matrix/index.html
+++ b/content/documentation/runners/capability-matrix/index.html
@@ -1168,7 +1168,7 @@
 
 
 
-
+
 
 
   
@@ -1248,7 +1248,7 @@
 
 
 
-~
+
 
 
   
@@ -1314,7 +1314,7 @@
 
 
 
-
+
 
 
   
@@ -1354,7 +1354,7 @@
 
 
 
-
+
 
 
   
@@ -2206,7 +2206,7 @@
 
 
 
-No: It is a 
batch-only runner, and intermediate trigger firings are effectively 
meaningless.
+No: batch-only 
runner
 
 
   
@@ -2246,7 +2246,7 @@
 
 
 
-No: Currently 
watermark progress jumps from the beginning of time to the end of time once the 
input has been fully consumed, thus no additional triggering granularity is 
available.
+No
 
 
   
@@ -2286,7 +2286,7 @@
 
 
 
-No: From the 
perspective of triggers, processing time currently jumps from the beginning of 
time to the end of time once the input has been fully consumed, thus no 
additional triggering granularity is available.
+No
 
 
   
@@ -2326,7 +2326,7 @@
 
 
 
-No: Elements are 
processed in the largest bundles possible, so count-based triggers are 
effectively meaningless.
+No
 
 
   
@@ -2366,7 +2366,7 @@
 
 
 
-No: pending model 
support
+No
 
 
   
@@ -2406,7 +2406,7 @@
 
 
 
-Yes: fully 
supported
+No
 
 
   
@@ -2446,7 +2446,7 @@
 
 
 
-No: No data is ever 
late.
+No
 
 
   
@@ -2486,7 +2486,7 @@
 
 
 
-Partially: not 
implemented
+No
 
 
   
@@ -2552,7 +2552,7 @@
 
 
 
-Yes: fully 
supported
+No: batch-only 
runner
 
 
   
@@ -2592,7 +2592,7 @@
 
 
 
-Yes: fully 
supported
+No
 
 
   
@@ -2632,7 +2632,7 @@
 
 
 
-No: pending model 
support
+No
 
 
   

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch asf-site updated (ed155be -> 07615ea)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch asf-site
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


from ed155be  Prepare repository for deployment.
 add dc0da0b  Update Mapreduce capability matrix when/how entries
 add 8c112de  This closes #324
 new 07615ea  Prepare repository for deployment.

The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../runners/capability-matrix/index.html   | 30 +++---
 src/_data/capability-matrix.yml| 30 +++---
 2 files changed, 30 insertions(+), 30 deletions(-)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 01/02: Update Mapreduce capability matrix when/how entries

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit dc0da0b3ef6d666a8bea5014a9a24f7dafdcd975
Author: melissa 
AuthorDate: Thu Sep 21 09:41:58 2017 -0700

Update Mapreduce capability matrix when/how entries
---
 src/_data/capability-matrix.yml | 30 +++---
 1 file changed, 15 insertions(+), 15 deletions(-)

diff --git a/src/_data/capability-matrix.yml b/src/_data/capability-matrix.yml
index 1c1171d..b0ea35a 100644
--- a/src/_data/capability-matrix.yml
+++ b/src/_data/capability-matrix.yml
@@ -577,7 +577,7 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: It is a batch-only runner, and intermediate trigger firings 
are effectively meaningless.
+l2: batch-only runner
 l3: ''
 
   - name: Event-time triggers
@@ -608,7 +608,7 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: Currently watermark progress jumps from the beginning of time 
to the end of time once the input has been fully consumed, thus no additional 
triggering granularity is available.
+l2: ''
 l3: ''
 
   - name: Processing-time triggers
@@ -639,7 +639,7 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: From the perspective of triggers, processing time currently 
jumps from the beginning of time to the end of time once the input has been 
fully consumed, thus no additional triggering granularity is available.
+l2: ''
 l3: ''
 
   - name: Count triggers
@@ -670,7 +670,7 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: Elements are processed in the largest bundles possible, so 
count-based triggers are effectively meaningless.
+l2: ''
 l3: ''
 
   - name: '[Meta]data driven triggers'
@@ -702,7 +702,7 @@ categories:
 l3:
   - class: mapreduce
 l1: 'No'
-l2: pending model support
+l2: ''
 l3:
 
   - name: Composite triggers
@@ -732,8 +732,8 @@ categories:
 l2: ''
 l3: ''
   - class: mapreduce
-l1: 'Yes'
-l2: fully supported
+l1: 'No'
+l2: ''
 l3: ''
 
   - name: Allowed lateness
@@ -764,7 +764,7 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: No data is ever late.
+l2: ''
 l3: ''
 
   - name: Timers
@@ -794,8 +794,8 @@ categories:
 l2: not implemented
 l3: ''
   - class: mapreduce
-l1: 'Partially'
-l2: not implemented
+l1: 'No'
+l2: ''
 l3: ''
 
   - description: How do refinements relate?
@@ -833,8 +833,8 @@ categories:
 l2: fully supported
 l3: ''
   - class: mapreduce
-l1: 'Yes'
-l2: fully supported
+l1: 'No'
+l2: batch-only runner
 l3: ''
 
   - name: Accumulating
@@ -864,8 +864,8 @@ categories:
 l2: ''
 l3: ''
   - class: mapreduce
-l1: 'Yes'
-l2: fully supported
+l1: 'No'
+l2: ''
 l3: ''
 
   - name: 'Accumulating  Retracting'
@@ -897,5 +897,5 @@ categories:
 l3: ''
   - class: mapreduce
 l1: 'No'
-l2: pending model support
+l2: ''
 l3: ''

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

[beam-site] branch mergebot updated (cd4fe42 -> 8c112de)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard cd4fe42  This closes #325
 discard c43098e  Manually add missing generated files
 new dc0da0b  Update Mapreduce capability matrix when/how entries
 new 8c112de  This closes #324

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (cd4fe42)
\
 N -- N -- N   refs/heads/mergebot (8c112de)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 content/contribute/docker-images/index.html | 383 
 content/images/logo_gearpump.png| Bin 4691 -> 0 bytes
 src/_data/capability-matrix.yml |  30 +--
 3 files changed, 15 insertions(+), 398 deletions(-)
 delete mode 100644 content/contribute/docker-images/index.html
 delete mode 100644 content/images/logo_gearpump.png

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4879

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3926

[Apache Jenkins Server]

See 

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #4045

[Apache Jenkins Server]

See 

[GitHub] beam pull request #3898: [BEAM-2899] Add a Bare-bones ReferenceRunner Job Se...

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3898

[BEAM-2899] Add a Bare-bones ReferenceRunner Job Service

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
This will eventually accept Job API calls and execute a Pipeline using
the ReferenceRunner backend.

This change exists primarily to create the appropriate module and POM.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam ulr_execution

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3898.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3898


commit 7cee5dc586c2851cd673baf05173ce0c480a9b41
Author: Thomas Groh 
Date:   2017-08-04T17:58:32Z

Add a Bare-bones ReferenceRunner Job Service

This will eventually accept Job API calls and execute a Pipeline using
the ReferenceRunner backend.

This change exists primarily to create the appropriate module and POM.




---

[jira] [Commented] (BEAM-2899) Universal Local Runner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179967#comment-16179967
 ] 

ASF GitHub Bot commented on BEAM-2899:
--

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3898

[BEAM-2899] Add a Bare-bones ReferenceRunner Job Service

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
This will eventually accept Job API calls and execute a Pipeline using
the ReferenceRunner backend.

This change exists primarily to create the appropriate module and POM.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam ulr_execution

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3898.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3898


commit 7cee5dc586c2851cd673baf05173ce0c480a9b41
Author: Thomas Groh 
Date:   2017-08-04T17:58:32Z

Add a Bare-bones ReferenceRunner Job Service

This will eventually accept Job API calls and execute a Pipeline using
the ReferenceRunner backend.

This change exists primarily to create the appropriate module and POM.




> Universal Local Runner
> --
>
> Key: BEAM-2899
> URL: https://issues.apache.org/jira/browse/BEAM-2899
> Project: Beam
>  Issue Type: Improvement
>  Components: runner-core
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>  Labels: portability
>
> To make the portability effort tractable, we should implement a Universal 
> Local Runner (ULR) in Java that runs in a single server process plus docker 
> containers for the SDK harness containers. It would serve multiple purposes:
>   (1) A reference implementation for other runners. Ideally, any new feature 
> should be implemented in the ULR first.
>   (2) A fully-featured test runner for SDKs who participate in the 
> portability framework. It thus complements the direct runners.
>   (3) A test runner for user code that depends on or customizes the runtime 
> environment. For example, a DoFn that shells out has a dependency that may be 
> satisfied on the user's desktop (and thus works fine on the direct runner), 
> but perhaps not by the container harness image. The ULR allows for an easy 
> way to find out.
> The Java direct runner presumably has lots of pieces that can be reused.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4878

[Apache Jenkins Server]

See 


Changes:

[tgroh] Update the Dataflow v1b3 API version

--
[...truncated 1.19 MB...]
2017-09-25T23:22:19.383 [INFO] Downloading: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
2017-09-25T23:22:19.441 [INFO] Downloaded: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar (12 KB at 90.1 
KB/sec)
2017-09-25T23:22:19.441 [INFO] Downloading: 
http://conjars.org/repo/org/apache/storm/storm-core/1.0.1/storm-core-1.0.1.jar
2017-09-25T23:22:19.496 [INFO] Downloaded: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
 (48 KB at 263.4 KB/sec)
2017-09-25T23:22:19.496 [INFO] Downloading: 
http://conjars.org/repo/org/clojure/clojure/1.7.0/clojure-1.7.0.jar
2017-09-25T23:22:19.497 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
2017-09-25T23:22:19.674 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
 (43 KB at 119.3 KB/sec)
2017-09-25T23:22:19.723 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
 (246 KB at 605.6 KB/sec)
2017-09-25T23:22:19.772 [INFO] Downloaded: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
 (230 KB at 506.1 KB/sec)
2017-09-25T23:22:20.022 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar 
(680 KB at 965.1 KB/sec)
2017-09-25T23:22:20.026 [INFO] Downloading: 
http://clojars.org/repo/org/scalatest/scalatest_2.10/2.2.6/scalatest_2.10-2.2.6.jar
2017-09-25T23:22:20.034 [INFO] Downloading: 
http://clojars.org/repo/org/apache/storm/storm-core/1.0.1/storm-core-1.0.1.jar
2017-09-25T23:22:20.035 [INFO] Downloading: 
http://clojars.org/repo/org/clojure/clojure/1.7.0/clojure-1.7.0.jar
2017-09-25T23:22:20.107 [INFO] Downloading: 
https://clojars.org/repo/org/clojure/clojure/1.7.0/clojure-1.7.0.jar
[JENKINS] Archiving disabled
2017-09-25T23:22:20.948 [INFO]  
   
2017-09-25T23:22:20.948 [INFO] 

2017-09-25T23:22:20.949 [INFO] Skipping Apache Beam :: Parent
2017-09-25T23:22:20.949 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-25T23:22:20.949 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-09-25T23:23:15.033 [INFO] 

2017-09-25T23:23:15.033 [INFO] Reactor Summary:
2017-09-25T23:23:15.033 [INFO] 
2017-09-25T23:23:15.034 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 45.387 s]
2017-09-25T23:23:15.034 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SUCCESS [ 14.936 s]
2017-09-25T23:23:15.034 [INFO] Apache Beam :: SDKs 
 SUCCESS [  7.409 s]
2017-09-25T23:23:15.034 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3925

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2975) Results of ReadableState.read() should be snapshots of the underlying state

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179931#comment-16179931
 ] 

Luke Cwik commented on BEAM-2975:
-

Also, what are the semantics of readLater(), should it return a view of the 
state when readLater was called or when read() is called.

{code:java}
@StateId("tag") BagState state;
BagState laterInts = state.readLater();
state.add(17);
Iterable ints = laterInts.read();
// what should be inside ints?
{code}

How do you expect this to work in the Fn API world since all appends done by 
the SDK harness will be doing a blind append so it can't know how many values 
are within state without reading it first so code like:
{code:java}
@StateId("tag") BagState state;
Iterable ints = state.read();
for (int i  = 0; i < big number; ++i) {
  state.add(17);
}
// SDK harness writes the appends out to the runner harness.
// Only way to know what ints stored is by having the SDK harness read it all 
before writing out so it can limit the length of the ints iterable.
{code}


> Results of ReadableState.read() should be snapshots of the underlying state
> ---
>
> Key: BEAM-2975
> URL: https://issues.apache.org/jira/browse/BEAM-2975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Mills
>Assignee: Daniel Mills
>Priority: Minor
>
> Future modification of state should not be reflected in previous calls to 
> read().  For example:
> @StateId("tag") BagState state;
> Iterable ints = state.read();
> state.add(17);
> // ints should still be empty here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[2/2] beam git commit: Change checksum back to md5

[tgroh]

Change checksum back to md5

The harness boot code depends on the type of checksum being an md5


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/166fadb6
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/166fadb6
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/166fadb6

Branch: refs/heads/master
Commit: 166fadb6687d03aa556775feb2fce49e807e78ec
Parents: 446f311
Author: Thomas Groh 
Authored: Mon Sep 25 09:58:20 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 16:00:11 2017 -0700

--
 .../src/main/proto/beam_artifact_api.proto   | 15 ---
 1 file changed, 4 insertions(+), 11 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/166fadb6/sdks/common/runner-api/src/main/proto/beam_artifact_api.proto
--
diff --git a/sdks/common/runner-api/src/main/proto/beam_artifact_api.proto 
b/sdks/common/runner-api/src/main/proto/beam_artifact_api.proto
index f713fa7..1ca535b 100644
--- a/sdks/common/runner-api/src/main/proto/beam_artifact_api.proto
+++ b/sdks/common/runner-api/src/main/proto/beam_artifact_api.proto
@@ -60,18 +60,11 @@ message ArtifactMetadata {
   string name = 1;
 
   // (Optional) The Unix-like permissions of the artifact
-  int32 permissions = 2;
+  uint32 permissions = 2;
 
-  // (Optional) The checksum of the artifact.
-  Checksum checksum = 3;
-}
-
-message Checksum {
-  // (Required) the algorithm used to generate this checksum
-  string algorithm = 1;
-
-  // (Required) the value of this checksum
-  bytes value = 2;
+  // (Optional) The md5 checksum of the artifact. Used, among other things, by 
harness boot code to
+  // validate the integrity of the artifact.
+  string md5 = 3;
 }
 
 // A collection of artifacts.

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4877

[Apache Jenkins Server]

See 


Changes:

[kirpichov] [BEAM-2467] Kinesis source watermark based on

--
[...truncated 1.19 MB...]
2017-09-25T23:05:49.349 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/storm/storm-core/1.0.1/storm-core-1.0.1.jar
 (19650 KB at 2738.3 KB/sec)
2017-09-25T23:05:49.680 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/github/rholder/snowball-stemmer/1.3.0.581.1/snowball-stemmer-1.3.0.581.1.jar
 (91 KB at 12.1 KB/sec)
2017-09-25T23:05:49.729 [INFO] Downloading: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
2017-09-25T23:05:49.729 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
2017-09-25T23:05:49.733 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar
2017-09-25T23:05:49.734 [INFO] Downloading: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
2017-09-25T23:05:49.733 [INFO] Downloading: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar
2017-09-25T23:05:49.847 [INFO] Downloaded: 
http://conjars.org/repo/riffle/riffle/0.1-dev/riffle-0.1-dev.jar (12 KB at 93.9 
KB/sec)
2017-09-25T23:05:49.847 [INFO] Downloading: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
2017-09-25T23:05:49.903 [INFO] Downloaded: 
http://conjars.org/repo/org/pentaho/pentaho-aggdesigner-algorithm/5.1.5-jhyde/pentaho-aggdesigner-algorithm-5.1.5-jhyde.jar
 (48 KB at 271.0 KB/sec)
2017-09-25T23:05:50.017 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-local/2.6.3/cascading-local-2.6.3.jar
 (43 KB at 149.1 KB/sec)
2017-09-25T23:05:50.096 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-hadoop/2.6.3/cascading-hadoop-2.6.3.jar
 (246 KB at 668.2 KB/sec)
2017-09-25T23:05:50.131 [INFO] Downloaded: 
http://conjars.org/repo/thirdparty/jgrapht-jdk1.6/0.8.1/jgrapht-jdk1.6-0.8.1.jar
 (230 KB at 577.5 KB/sec)
2017-09-25T23:05:50.459 [INFO] Downloaded: 
http://conjars.org/repo/cascading/cascading-core/2.6.3/cascading-core-2.6.3.jar 
(680 KB at 930.8 KB/sec)
[JENKINS] Archiving disabled
2017-09-25T23:05:51.225 [INFO]  
   
2017-09-25T23:05:51.225 [INFO] 

2017-09-25T23:05:51.225 [INFO] Skipping Apache Beam :: Parent
2017-09-25T23:05:51.225 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-25T23:05:51.225 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled2017-09-25T23:06:28.496 [INFO] 

2017-09-25T23:06:28.496 [INFO] Reactor Summary:
2017-09-25T23:06:28.496 [INFO] 
2017-09-25T23:06:28.496 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 27.627 s]
2017-09-25T23:06:28.496 [INFO] Apache Beam :: SDKs :: Java 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3924

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2975) Results of ReadableState.read() should be snapshots of the underlying state

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179920#comment-16179920
 ] 

Luke Cwik commented on BEAM-2975:
-

Is this really what users expect?

I would expect that read() is a live view and not a snapshot view.

Also, it seems as though no changes were made to 
https://github.com/apache/beam/blob/master/sdks/java/harness/src/main/java/org/apache/beam/fn/harness/FnApiDoFnRunner.java
 to reflect this change in the expectation.

> Results of ReadableState.read() should be snapshots of the underlying state
> ---
>
> Key: BEAM-2975
> URL: https://issues.apache.org/jira/browse/BEAM-2975
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-core
>Reporter: Daniel Mills
>Assignee: Daniel Mills
>Priority: Minor
>
> Future modification of state should not be reflected in previous calls to 
> read().  For example:
> @StateId("tag") BagState state;
> Iterable ints = state.read();
> state.add(17);
> // ints should still be empty here.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3895: Change checksum back to md5

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3895


---

[1/2] beam git commit: This closes #3895

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 446f31174 -> 352f106f9


This closes #3895


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/352f106f
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/352f106f
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/352f106f

Branch: refs/heads/master
Commit: 352f106f992a2ff4c08274685657242f04f967ef
Parents: 446f311 166fadb
Author: Thomas Groh 
Authored: Mon Sep 25 16:00:11 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 16:00:11 2017 -0700

--
 .../src/main/proto/beam_artifact_api.proto   | 15 ---
 1 file changed, 4 insertions(+), 11 deletions(-)
--

[jira] [Created] (BEAM-2988) PubSub subscriptions automatically created in Python DirectRunner not reliably cleaned up

[Charles Chen (JIRA)]

Charles Chen created BEAM-2988:
--

 Summary: PubSub subscriptions automatically created in Python 
DirectRunner not reliably cleaned up
 Key: BEAM-2988
 URL: https://issues.apache.org/jira/browse/BEAM-2988
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Ahmet Altay
Priority: Minor


In the Python DirectRunner, we attempt to garbage-collect automatically-created 
PubSub subscriptions here 
(https://github.com/apache/beam/blob/446f311746b74bdfb420ba71d76097d683fde9a1/sdks/python/apache_beam/runners/direct/transform_evaluator.py#L370).
  However, this mechanism may not succeed, since it is called on the __del__ 
method when other references may have already been garbage-collected.  We 
should further investigate the mechanism here and attempt to make it more 
reliable.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3923

[Apache Jenkins Server]

See 

[2/2] beam git commit: This closes #3896

[tgroh]

This closes #3896


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/446f3117
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/446f3117
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/446f3117

Branch: refs/heads/master
Commit: 446f311746b74bdfb420ba71d76097d683fde9a1
Parents: 3bf65dc 95ef860
Author: Thomas Groh 
Authored: Mon Sep 25 15:39:38 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 15:39:38 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--

[GitHub] beam pull request #3896: Update Beam's root pom.xml to use a newer API clien...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3896


---

[1/2] beam git commit: Update the Dataflow v1b3 API version

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master 3bf65dc7a -> 446f31174


Update the Dataflow v1b3 API version


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/95ef860b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/95ef860b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/95ef860b

Branch: refs/heads/master
Commit: 95ef860be12844a11e3bc986752d925157fca2e0
Parents: 3bf65dc
Author: Sunil Pedapudi 
Authored: Mon Sep 25 11:15:54 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 15:39:27 2017 -0700

--
 pom.xml | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/95ef860b/pom.xml
--
diff --git a/pom.xml b/pom.xml
index 236645c..3ccd8d8 100644
--- a/pom.xml
+++ b/pom.xml
@@ -112,7 +112,7 @@
 v1-rev6-1.22.0
 0.1.18
 v2-rev8-1.22.0
-v1b3-rev198-1.22.0
+v1b3-rev213-1.22.0
 0.5.160222
 1.4.0
 1.3.0

[jira] [Closed] (BEAM-2467) KinesisIO watermark based on approximateArrivalTimestamp

[Eugene Kirpichov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eugene Kirpichov closed BEAM-2467.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> KinesisIO watermark based on approximateArrivalTimestamp
> 
>
> Key: BEAM-2467
> URL: https://issues.apache.org/jira/browse/BEAM-2467
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Paweł Kaczmarczyk
>Assignee: Paweł Kaczmarczyk
> Fix For: 2.2.0
>
>
> In Kinesis we can start reading the stream at some point in the past during 
> the retention period (up to 7 days). With current approach for setting 
> record's timestamp and watermark (both are always set to current time, i.e. 
> Instant.now()), we can't observe the actual position in the stream.
> So the idea is to change this behaviour and set the record timestamp based on 
> the 
> [ApproximateArrivalTimestamp|http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html#Streams-Type-Record-ApproximateArrivalTimestamp].
>  Watermark will be set accordingly to the last read record's timestamp. 
> ApproximateArrivalTimestamp is still some approximation and may result in 
> having records with out-of-order timestamp's which in turn may result in some 
> events marked as late. This however should not be a frequent issue and even 
> if it happens it should be a matter of milliseconds or seconds so can be 
> handled even with a tiny allowedLateness setting



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2467) KinesisIO watermark based on approximateArrivalTimestamp

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179888#comment-16179888
 ] 

ASF GitHub Bot commented on BEAM-2467:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3851


> KinesisIO watermark based on approximateArrivalTimestamp
> 
>
> Key: BEAM-2467
> URL: https://issues.apache.org/jira/browse/BEAM-2467
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-java-extensions
>Reporter: Paweł Kaczmarczyk
>Assignee: Paweł Kaczmarczyk
>
> In Kinesis we can start reading the stream at some point in the past during 
> the retention period (up to 7 days). With current approach for setting 
> record's timestamp and watermark (both are always set to current time, i.e. 
> Instant.now()), we can't observe the actual position in the stream.
> So the idea is to change this behaviour and set the record timestamp based on 
> the 
> [ApproximateArrivalTimestamp|http://docs.aws.amazon.com/kinesis/latest/APIReference/API_Record.html#Streams-Type-Record-ApproximateArrivalTimestamp].
>  Watermark will be set accordingly to the last read record's timestamp. 
> ApproximateArrivalTimestamp is still some approximation and may result in 
> having records with out-of-order timestamp's which in turn may result in some 
> events marked as late. This however should not be a frequent issue and even 
> if it happens it should be a matter of milliseconds or seconds so can be 
> handled even with a tiny allowedLateness setting



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3851: [BEAM-2467] Kinesis source watermark based on appro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3851


---

[1/2] beam git commit: [BEAM-2467] Kinesis source watermark based on approximateArrivalTimestamp

[jkff]

Repository: beam
Updated Branches:
  refs/heads/master 1bd17d1b9 -> 3bf65dc7a


[BEAM-2467] Kinesis source watermark based on approximateArrivalTimestamp


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/21a62ed4
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/21a62ed4
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/21a62ed4

Branch: refs/heads/master
Commit: 21a62ed419bbb364b28007779af1f8d87d17dfb6
Parents: 1bd17d1
Author: Pawel Kaczmarczyk 
Authored: Thu Sep 14 12:52:16 2017 +0200
Committer: Eugene Kirpichov 
Committed: Mon Sep 25 15:19:35 2017 -0700

--
 .../sdk/io/kinesis/GetKinesisRecordsResult.java |  8 ++-
 .../beam/sdk/io/kinesis/KinesisReader.java  | 64 +
 .../sdk/io/kinesis/ShardRecordsIterator.java| 22 --
 .../sdk/io/kinesis/SimplifiedKinesisClient.java |  1 +
 .../beam/sdk/io/kinesis/AmazonKinesisMock.java  |  7 +-
 .../beam/sdk/io/kinesis/KinesisReaderTest.java  | 76 
 6 files changed, 155 insertions(+), 23 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/21a62ed4/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/GetKinesisRecordsResult.java
--
diff --git 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/GetKinesisRecordsResult.java
 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/GetKinesisRecordsResult.java
index f605f55..bbbffed 100644
--- 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/GetKinesisRecordsResult.java
+++ 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/GetKinesisRecordsResult.java
@@ -32,9 +32,10 @@ class GetKinesisRecordsResult {
 
   private final List records;
   private final String nextShardIterator;
+  private final long millisBehindLatest;
 
   public GetKinesisRecordsResult(List records, String 
nextShardIterator,
-  final String streamName, final String shardId) {
+  long millisBehindLatest, final String streamName, final String shardId) {
 this.records = transform(records, new Function() {
 
   @Nullable
@@ -45,6 +46,7 @@ class GetKinesisRecordsResult {
   }
 });
 this.nextShardIterator = nextShardIterator;
+this.millisBehindLatest = millisBehindLatest;
   }
 
   public List getRecords() {
@@ -54,4 +56,8 @@ class GetKinesisRecordsResult {
   public String getNextShardIterator() {
 return nextShardIterator;
   }
+
+  public long getMillisBehindLatest() {
+return millisBehindLatest;
+  }
 }

http://git-wip-us.apache.org/repos/asf/beam/blob/21a62ed4/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
--
diff --git 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
index e2fa474..1abcd98 100644
--- 
a/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
+++ 
b/sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisReader.java
@@ -23,7 +23,12 @@ import static com.google.common.collect.Lists.newArrayList;
 import java.io.IOException;
 import java.util.List;
 import java.util.NoSuchElementException;
+
 import org.apache.beam.sdk.io.UnboundedSource;
+import org.apache.beam.sdk.transforms.Min;
+import org.apache.beam.sdk.transforms.windowing.BoundedWindow;
+import org.apache.beam.sdk.util.MovingFunction;
+import org.joda.time.Duration;
 import org.joda.time.Instant;
 import org.slf4j.Logger;
 import org.slf4j.LoggerFactory;
@@ -35,12 +40,34 @@ import org.slf4j.LoggerFactory;
 class KinesisReader extends UnboundedSource.UnboundedReader {
 
   private static final Logger LOG = 
LoggerFactory.getLogger(KinesisReader.class);
+  /**
+   * Period of samples to determine watermark.
+   */
+  private static final Duration SAMPLE_PERIOD = Duration.standardMinutes(1);
+
+  /**
+   * Period of updates to determine watermark.
+   */
+  private static final Duration SAMPLE_UPDATE = Duration.standardSeconds(5);
+
+  /**
+   * Minimum number of unread messages required before considering updating 
watermark.
+   */
+  static final int MIN_WATERMARK_MESSAGES = 10;
+
+  /**
+   * Minimum number of SAMPLE_UPDATE periods over which unread messages should 
be spread
+   * before considering updating watermark.
+   */
+  private static final int MIN_WATERMARK_SPREAD = 2;
 
   private final SimplifiedKinesisClient kinesis;
   private final UnboundedSource source;
   private final CheckpointGenerator initialCheckpointGenerator;
   private RoundRobin 

[2/2] beam git commit: This closes #3851: [BEAM-2467] Kinesis source watermark based on approximateArrivalTimestamp

[jkff]

This closes #3851: [BEAM-2467] Kinesis source watermark based on 
approximateArrivalTimestamp


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/3bf65dc7
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/3bf65dc7
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/3bf65dc7

Branch: refs/heads/master
Commit: 3bf65dc7a89934e23c5675e2656d8f9c0950c685
Parents: 1bd17d1 21a62ed
Author: Eugene Kirpichov 
Authored: Mon Sep 25 15:20:08 2017 -0700
Committer: Eugene Kirpichov 
Committed: Mon Sep 25 15:20:08 2017 -0700

--
 .../sdk/io/kinesis/GetKinesisRecordsResult.java |  8 ++-
 .../beam/sdk/io/kinesis/KinesisReader.java  | 64 +
 .../sdk/io/kinesis/ShardRecordsIterator.java| 22 --
 .../sdk/io/kinesis/SimplifiedKinesisClient.java |  1 +
 .../beam/sdk/io/kinesis/AmazonKinesisMock.java  |  7 +-
 .../beam/sdk/io/kinesis/KinesisReaderTest.java  | 76 
 6 files changed, 155 insertions(+), 23 deletions(-)
--

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3922

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2982) PubSubIO.readMessages().fromSubscription(...) doesn't work with ValueProvider

[Thomas Groh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Thomas Groh resolved BEAM-2982.
---
   Resolution: Fixed
Fix Version/s: 2.2.0

> PubSubIO.readMessages().fromSubscription(...) doesn't work with ValueProvider
> -
>
> Key: BEAM-2982
> URL: https://issues.apache.org/jira/browse/BEAM-2982
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ben Chambers
>Assignee: Thomas Groh
> Fix For: 2.2.0
>
>
> Originally reported on Stack Overflow:
> https://stackoverflow.com/questions/46360584/apache-beam-template-runtime-context-error
> ---
> In the `PubsubUnboundedSource#expand` method we create the PubsubSource:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L1399
> Creating the PubsubSource calls `getSubscription` which attempts to get the 
> value out of a value provider.
> To support templatization, the PubsubSource needs to take the ValueProvider, 
> and only get the subscription out at pipeline execution time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2982) PubSubIO.readMessages().fromSubscription(...) doesn't work with ValueProvider

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179737#comment-16179737
 ] 

ASF GitHub Bot commented on BEAM-2982:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3888


> PubSubIO.readMessages().fromSubscription(...) doesn't work with ValueProvider
> -
>
> Key: BEAM-2982
> URL: https://issues.apache.org/jira/browse/BEAM-2982
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Reporter: Ben Chambers
>Assignee: Thomas Groh
>
> Originally reported on Stack Overflow:
> https://stackoverflow.com/questions/46360584/apache-beam-template-runtime-context-error
> ---
> In the `PubsubUnboundedSource#expand` method we create the PubsubSource:
> https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java#L1399
> Creating the PubsubSource calls `getSubscription` which attempts to get the 
> value out of a value provider.
> To support templatization, the PubsubSource needs to take the ValueProvider, 
> and only get the subscription out at pipeline execution time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3888: [BEAM-2982] Use the SubscriptionProvider in PubsubU...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3888


---

[1/2] beam git commit: This closes #3888

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master a2ff162c9 -> 1bd17d1b9


This closes #3888


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/1bd17d1b
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/1bd17d1b
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/1bd17d1b

Branch: refs/heads/master
Commit: 1bd17d1b95a6b27331626fa9bdbaa723969b710d
Parents: a2ff162 2e27526
Author: Thomas Groh 
Authored: Mon Sep 25 13:38:00 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 13:38:00 2017 -0700

--
 .../io/gcp/pubsub/PubsubUnboundedSource.java| 18 ---
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java| 55 
 2 files changed, 67 insertions(+), 6 deletions(-)
--

[2/2] beam git commit: Use the SubscriptionProvider in PubsubUnboundedSource

[tgroh]

Use the SubscriptionProvider in PubsubUnboundedSource

During expansion, a ValueProvider may not be accessible. This ensures
that if the subscription is based on a value provider, it will only be
evaluated when that ValueProvider is bound, rather than at construction
time.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/2e275264
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/2e275264
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/2e275264

Branch: refs/heads/master
Commit: 2e275264b21db45787833502e5e42907b05e28b8
Parents: a2ff162
Author: Thomas Groh 
Authored: Fri Sep 22 11:47:19 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 13:38:00 2017 -0700

--
 .../io/gcp/pubsub/PubsubUnboundedSource.java| 18 ---
 .../beam/sdk/io/gcp/pubsub/PubsubIOTest.java| 55 
 2 files changed, 67 insertions(+), 6 deletions(-)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/2e275264/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
--
diff --git 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
index bf3a121..2271786 100644
--- 
a/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
+++ 
b/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubUnboundedSource.java
@@ -61,6 +61,7 @@ import org.apache.beam.sdk.metrics.Counter;
 import org.apache.beam.sdk.metrics.SourceMetrics;
 import org.apache.beam.sdk.options.PipelineOptions;
 import org.apache.beam.sdk.options.ValueProvider;
+import org.apache.beam.sdk.options.ValueProvider.StaticValueProvider;
 import org.apache.beam.sdk.transforms.Combine;
 import org.apache.beam.sdk.transforms.DoFn;
 import org.apache.beam.sdk.transforms.PTransform;
@@ -1097,13 +1098,14 @@ public class PubsubUnboundedSource extends 
PTransform(desiredNumSplits);
   PubsubSource splitSource = this;
   if (subscriptionPath == null) {
-splitSource = new PubsubSource(outer, 
outer.createRandomSubscription(options));
+splitSource =
+new PubsubSource(
+outer, 
StaticValueProvider.of(outer.createRandomSubscription(options)));
   }
   for (int i = 0; i < desiredNumSplits * SCALE_OUT; i++) {
 // Since the source is immutable and Pubsub automatically shards we 
simply
@@ -1129,8 +1133,8 @@ public class PubsubUnboundedSource extends 
PTransform

[beam-site] 01/02: Manually add missing generated files

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit c43098e8e0a0d49fe99f70e2ad26ab97210ae28b
Author: melissa 
AuthorDate: Mon Sep 25 10:51:39 2017 -0700

Manually add missing generated files
---
 content/contribute/docker-images/index.html | 383 
 content/images/logo_gearpump.png| Bin 0 -> 4691 bytes
 2 files changed, 383 insertions(+)

diff --git a/content/contribute/docker-images/index.html 
b/content/contribute/docker-images/index.html
new file mode 100644
index 000..cd160aa
--- /dev/null
+++ b/content/contribute/docker-images/index.html
@@ -0,0 +1,383 @@
+
+
+  
+  
+  
+  
+  Beam Docker Images
+  
+  https://fonts.googleapis.com/css?family=Roboto:100,300,400; 
rel="stylesheet">
+  
+  https://ajax.googleapis.com/ajax/libs/jquery/2.2.0/jquery.min.js";>
+  
+  
+  https://beam.apache.org/contribute/docker-images/; data-proofer-ignore>
+  
+  https://beam.apache.org/feed.xml;>
+  
+
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
+(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new 
Date();a=s.createElement(o),
+
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
+
})(window,document,'script','//www.google-analytics.com/analytics.js','ga');
+ga('create', 'UA-73650088-1', 'auto');
+ga('send', 'pageview');
+  
+
+
+  
+
+
+  
+
+  
+
+  Toggle navigation
+  
+  
+  
+
+
+
+  
+
+  Get Started 
+  
+Beam Overview
+Quickstart - 
Java
+Quickstart - 
Python
+
+Example Walkthroughs
+WordCount
+Mobile 
Gaming
+
+Resources
+Downloads
+Support
+  
+
+
+  Documentation 
+  
+Using the Documentation
+
+Beam Concepts
+Programming 
Guide
+Additional 
Resources
+
+Pipeline Fundamentals
+Design Your 
Pipeline
+Create Your 
Pipeline
+Test 
Your Pipeline
+Pipeline I/O
+
+SDKs
+Java SDK
+Java SDK API Reference 
+
+Python SDK
+Python SDK API Reference 
+
+
+Runners
+Capability 
Matrix
+Direct Runner
+Apache Apex 
Runner
+Apache Flink 
Runner
+Apache Gearpump 
Runner
+Apache Spark 
Runner
+Cloud Dataflow 
Runner
+
+
+DSLs
+SQL
+  
+
+
+  Contribute 
+  
+Get Started Contributing
+
+Guides
+Contribution 
Guide
+Testing Guide
+Release Guide
+PTransform Style 
Guide
+Runner Authoring 
Guide
+
+Technical References
+Design 
Principles
+Ongoing 
Projects
+Source 
Repository
+Docker Images
+
+Promotion
+Presentation 
Materials
+Logos and Design
+
+Maturity Model
+Team
+  
+
+
+Blog
+  
+  
+
+  https://www.apache.org/foundation/press/kit/feather_small.png; alt="Apache 
Logo" style="height:20px;">
+  
+http://www.apache.org/;>ASF Homepage
+http://www.apache.org/licenses/;>License
+http://www.apache.org/security/;>Security
+http://www.apache.org/foundation/thanks.html;>Thanks
+http://www.apache.org/foundation/sponsorship.html;>Sponsorship
+https://www.apache.org/foundation/policies/conduct;>Code of 
Conduct
+  
+
+  
+
+
+
+
+  Docker Images
+
+Docker images allow to create a reproducible environment to build and test
+Beam. You can use the docker images by using the provided https://github.com/apache/beam/tree/master/sdks/java/build-tools/src/main/resources/docker;>Docker
 scripts.
+
+In this directory you will find scripts to build and run docker images for
+different purposes:
+
+
+  
+file: Create a Docker container from a Beam source 
code .zip file
+in a given environment. It is useful to test a specific version of Beam,
+for example to validate a release vote.
+  
+  
+git: Same as file but the Beam source code comes 
from the git repository,
+you can choose a given branch/tag/pull-request. Useful to test in a specific
+environment.
+  
+  
+release: It builds an end-user distribution of 
the latest version of Beam
+and its dependencies. 

[beam-site] branch mergebot updated (91aa43c -> cd4fe42)

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a change to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git.


 discard 91aa43c  This closes #324
 discard eeba9b3  Update Mapreduce capability matrix when/how entries
 new c43098e  Manually add missing generated files
 new cd4fe42  This closes #325

This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version.  This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:

 * -- * -- B -- O -- O -- O   (91aa43c)
\
 N -- N -- N   refs/heads/mergebot (cd4fe42)

You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.

Any revisions marked "omit" are not gone; other references still
refer to them.  Any revisions marked "discard" are gone forever.

The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails.  The revisions
listed as "add" were already present in the repository and have only
been added to this reference.


Summary of changes:
 .../docker-images}/index.html  | 227 -
 {src => content}/images/logo_gearpump.png  | Bin
 src/_data/capability-matrix.yml|  30 +--
 3 files changed, 144 insertions(+), 113 deletions(-)
 copy content/{get-started/quickstart-py => 
contribute/docker-images}/index.html (62%)
 copy {src => content}/images/logo_gearpump.png (100%)

-- 
To stop receiving notification emails like this one, please contact
['"commits@beam.apache.org" '].

[beam-site] 02/02: This closes #325

[mergebot-role]

This is an automated email from the ASF dual-hosted git repository.

mergebot-role pushed a commit to branch mergebot
in repository https://gitbox.apache.org/repos/asf/beam-site.git

commit cd4fe4255ffc39e784714438950990f34e46d48c
Merge: ed155be c43098e
Author: Mergebot 
AuthorDate: Mon Sep 25 19:56:39 2017 +

This closes #325

 content/contribute/docker-images/index.html | 383 
 content/images/logo_gearpump.png| Bin 0 -> 4691 bytes
 2 files changed, 383 insertions(+)

-- 
To stop receiving notification emails like this one, please contact
"commits@beam.apache.org" .

Jenkins build is back to normal : beam_PostCommit_Java_MavenInstall #4874

[Apache Jenkins Server]

See 

[jira] [Created] (BEAM-2987) Close of InputStream should be placed in finally block

[Ted Yu (JIRA)]

Ted Yu created BEAM-2987:


 Summary: Close of InputStream should be placed in finally block
 Key: BEAM-2987
 URL: https://issues.apache.org/jira/browse/BEAM-2987
 Project: Beam
  Issue Type: Bug
  Components: sdk-java-extensions
Reporter: Ted Yu
Assignee: Reuven Lax
Priority: Minor


In TikaSource#TikaReader#start():
{code}
try {
  parser.parse(is, tikaHandler, tikaMetadata, context);
  is.close();
{code}
The close() should be in finally block.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3897: Option to disable validation of BigtableIO.write ta...

[steveniemitz]

GitHub user steveniemitz opened a pull request:

https://github.com/apache/beam/pull/3897

Option to disable validation of BigtableIO.write target table

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tc-dc/beam bigtable-disable-validation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3897.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3897


commit 8674113c55db14b026037a3471803b3c259e5ac7
Author: steve 
Date:   2017-09-25T18:21:54Z

Option to disable validation of BigtableIO.write target table




---

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3209

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3921

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2985) BigQuery IO write transform is broken for DirectRunner

[Chamikara Jayalath (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chamikara Jayalath resolved BEAM-2985.
--
   Resolution: Fixed
Fix Version/s: 2.2.0

> BigQuery IO write transform is broken for DirectRunner
> --
>
> Key: BEAM-2985
> URL: https://issues.apache.org/jira/browse/BEAM-2985
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
> Fix For: 2.2.0
>
>
> I get following error when trying to run BigQuery tornadoes using 
> DirectRunner.
> DataflowRunner seems to be working fine.
> python -m apache_beam.examples.cookbook.bigquery_tornadoes --output 
> . --project 
>  Request missing required parameter projectId
>  Traceback for above exception (most recent call last):
>   File "apache_beam/utils/retry.py", line 175, in wrapper
> return fun(*args, **kwargs)
>   File "apache_beam/io/gcp/bigquery.py", line 828, in _get_table
> response = self.client.tables.Get(request)
>   File "apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", 
> line 608, in Get
> config, request, global_params=global_params)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 695, in _RunMethod
> download)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 676, in PrepareHttpRequest
> method_config, request, relative_path=url_builder.relative_path)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 580, in __ConstructRelativePath
> relative_path=relative_path)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/util.py",
>  line 124, in ExpandRelativePath
> 'Request missing required parameter %s' % param)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2985) BigQuery IO write transform is broken for DirectRunner

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179477#comment-16179477
 ] 

ASF GitHub Bot commented on BEAM-2985:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3892


> BigQuery IO write transform is broken for DirectRunner
> --
>
> Key: BEAM-2985
> URL: https://issues.apache.org/jira/browse/BEAM-2985
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Chamikara Jayalath
>Assignee: Chamikara Jayalath
>
> I get following error when trying to run BigQuery tornadoes using 
> DirectRunner.
> DataflowRunner seems to be working fine.
> python -m apache_beam.examples.cookbook.bigquery_tornadoes --output 
> . --project 
>  Request missing required parameter projectId
>  Traceback for above exception (most recent call last):
>   File "apache_beam/utils/retry.py", line 175, in wrapper
> return fun(*args, **kwargs)
>   File "apache_beam/io/gcp/bigquery.py", line 828, in _get_table
> response = self.client.tables.Get(request)
>   File "apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py", 
> line 608, in Get
> config, request, global_params=global_params)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 695, in _RunMethod
> download)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 676, in PrepareHttpRequest
> method_config, request, relative_path=url_builder.relative_path)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/base_api.py",
>  line 580, in __ConstructRelativePath
> relative_path=relative_path)
>   File 
> "/Users/chamikara/testing/beam_bq_09_22_2017/env1/lib/python2.7/site-packages/apitools/base/py/util.py",
>  line 124, in ExpandRelativePath
> 'Request missing required parameter %s' % param)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3892: [BEAM-2985] Updates WriteToBigQuery PTransform to g...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3892


---

[2/2] beam git commit: This closes #3892

[chamikara]

This closes #3892


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/a2ff162c
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/a2ff162c
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/a2ff162c

Branch: refs/heads/master
Commit: a2ff162c93e1cb812412b2b71f3d13a961c1d739
Parents: 06809b9 ae81003
Author: chamik...@google.com 
Authored: Mon Sep 25 11:10:44 2017 -0700
Committer: chamik...@google.com 
Committed: Mon Sep 25 11:10:44 2017 -0700

--
 sdks/python/apache_beam/io/gcp/bigquery.py | 3 +++
 1 file changed, 3 insertions(+)
--

[1/2] beam git commit: Updates WriteToBigQuery PTransform to get project id from GoogleCloudOptions when using DirectRunner.

[chamikara]

Repository: beam
Updated Branches:
  refs/heads/master 06809b982 -> a2ff162c9


Updates WriteToBigQuery PTransform to get project id from GoogleCloudOptions 
when using DirectRunner.

WriteToBigQuery PTransform behaves differently for DirectRunner and 
DataflowRunner when it comes to determining the project that the output table 
belongs to. If a project is not specified, DataflowRunner defauls to 
GoogleCloudOptions.project while DirectRunner does not. This PR fixes this 
inconsistency by defaulting to GoogleCloudOptions.project for DirectRunner as 
well.


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ae810034
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ae810034
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ae810034

Branch: refs/heads/master
Commit: ae8100348f43e50d27a500d4196fa08137ad81b8
Parents: 06809b9
Author: chamik...@google.com 
Authored: Fri Sep 22 17:59:50 2017 -0700
Committer: chamik...@google.com 
Committed: Mon Sep 25 11:10:22 2017 -0700

--
 sdks/python/apache_beam/io/gcp/bigquery.py | 3 +++
 1 file changed, 3 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ae810034/sdks/python/apache_beam/io/gcp/bigquery.py
--
diff --git a/sdks/python/apache_beam/io/gcp/bigquery.py 
b/sdks/python/apache_beam/io/gcp/bigquery.py
index 40611ac..ee79ae5 100644
--- a/sdks/python/apache_beam/io/gcp/bigquery.py
+++ b/sdks/python/apache_beam/io/gcp/bigquery.py
@@ -1412,6 +1412,9 @@ bigquery_v2_messages.TableSchema):
   raise TypeError('Unexpected schema argument: %s.' % schema)
 
   def expand(self, pcoll):
+if self.table_reference.projectId is None:
+  self.table_reference.projectId = pcoll.pipeline.options.view_as(
+  GoogleCloudOptions).project
 bigquery_write_fn = BigQueryWriteFn(
 table_id=self.table_reference.tableId,
 dataset_id=self.table_reference.datasetId,

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3919

[Apache Jenkins Server]

See 

[jira] [Resolved] (BEAM-2876) Add provision api proto

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-2876.
-
   Resolution: Fixed
Fix Version/s: 2.3.0

> Add provision api proto
> ---
>
> Key: BEAM-2876
> URL: https://issues.apache.org/jira/browse/BEAM-2876
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>  Labels: portability
> Fix For: 2.3.0
>
>
> As per discussion in https://s.apache.org/beam-fn-api-container-contract, we 
> need to define the provision API to allow boot code access to pipeline 
> options, in particular.
> It is proposed as a separate API instead of merging it with control or 
> artifact:
>(1) Not merging with control avoids having the boot code talk to control, 
> only to disconnect and have the SDK harness connect. The runner can't then 
> use the lifetime of the connection to be the lifetime of the SDK harness.
>(2) Not merging with artifact allows for simple, reusable artifact proxies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2876) Add provision api proto

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179410#comment-16179410
 ] 

ASF GitHub Bot commented on BEAM-2876:
--

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3885


> Add provision api proto
> ---
>
> Key: BEAM-2876
> URL: https://issues.apache.org/jira/browse/BEAM-2876
> Project: Beam
>  Issue Type: Sub-task
>  Components: beam-model
>Reporter: Henning Rohde
>Assignee: Henning Rohde
>  Labels: portability
>
> As per discussion in https://s.apache.org/beam-fn-api-container-contract, we 
> need to define the provision API to allow boot code access to pipeline 
> options, in particular.
> It is proposed as a separate API instead of merging it with control or 
> artifact:
>(1) Not merging with control avoids having the boot code talk to control, 
> only to disconnect and have the SDK harness connect. The runner can't then 
> use the lifetime of the connection to be the lifetime of the SDK harness.
>(2) Not merging with artifact allows for simple, reusable artifact proxies.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3885: [BEAM-2876] Add preliminary provision API

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/beam/pull/3885


---

[1/2] beam git commit: This closes #3885

[tgroh]

Repository: beam
Updated Branches:
  refs/heads/master aa2604a39 -> 06809b982


This closes #3885


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/06809b98
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/06809b98
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/06809b98

Branch: refs/heads/master
Commit: 06809b98200b75a8b51582558bdd7a9ad555ebaa
Parents: aa2604a ef1e719
Author: Thomas Groh 
Authored: Mon Sep 25 10:35:29 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 10:35:29 2017 -0700

--
 sdks/common/fn-api/pom.xml  |  5 ++
 .../src/main/proto/beam_provision_api.proto | 54 
 2 files changed, 59 insertions(+)
--

[2/2] beam git commit: [BEAM-2876] Add preliminary provision API

[tgroh]

[BEAM-2876] Add preliminary provision API


Project: http://git-wip-us.apache.org/repos/asf/beam/repo
Commit: http://git-wip-us.apache.org/repos/asf/beam/commit/ef1e7191
Tree: http://git-wip-us.apache.org/repos/asf/beam/tree/ef1e7191
Diff: http://git-wip-us.apache.org/repos/asf/beam/diff/ef1e7191

Branch: refs/heads/master
Commit: ef1e71917278c7998634734111fe9043d005b4a7
Parents: aa2604a
Author: Henning Rohde 
Authored: Fri Sep 22 10:41:28 2017 -0700
Committer: Thomas Groh 
Committed: Mon Sep 25 10:35:29 2017 -0700

--
 sdks/common/fn-api/pom.xml  |  5 ++
 .../src/main/proto/beam_provision_api.proto | 54 
 2 files changed, 59 insertions(+)
--


http://git-wip-us.apache.org/repos/asf/beam/blob/ef1e7191/sdks/common/fn-api/pom.xml
--
diff --git a/sdks/common/fn-api/pom.xml b/sdks/common/fn-api/pom.xml
index 6810667..d7e7584 100644
--- a/sdks/common/fn-api/pom.xml
+++ b/sdks/common/fn-api/pom.xml
@@ -92,6 +92,11 @@
 
 
 
+  com.google.guava
+  guava
+
+
+
   io.grpc
   grpc-core
 

http://git-wip-us.apache.org/repos/asf/beam/blob/ef1e7191/sdks/common/fn-api/src/main/proto/beam_provision_api.proto
--
diff --git a/sdks/common/fn-api/src/main/proto/beam_provision_api.proto 
b/sdks/common/fn-api/src/main/proto/beam_provision_api.proto
new file mode 100644
index 000..fb4f252
--- /dev/null
+++ b/sdks/common/fn-api/src/main/proto/beam_provision_api.proto
@@ -0,0 +1,54 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+/*
+ * Protocol Buffers describing the Provision API, for communicating with a 
runner
+ * for job and environment provisioning information over GRPC.
+ */
+
+syntax = "proto3";
+
+package org.apache.beam.fn.v1;
+
+option java_package = "org.apache.beam.fn.v1";
+option java_outer_classname = "ProvisionApi";
+
+import "google/protobuf/struct.proto";
+
+// A service to provide runtime provisioning information to the SDK harness
+// worker instances -- such as pipeline options, resource constaints and
+// other job metadata -- needed by an SDK harness instance to initialize.
+service ProvisionService {
+// Get provision information for the SDK harness worker instance.
+rpc GetProvisionInfo(GetProvisionInfoRequest) returns 
(GetProvisionInfoResponse);
+}
+
+// A request to get the provision info of a SDK harness worker instance.
+message GetProvisionInfoRequest { }
+
+// A response containing the provision info of a SDK harness worker instance.
+message GetProvisionInfoResponse {
+// (required) The job ID.
+string job_id = 1;
+// (required) The job name.
+string job_name = 2;
+
+// (required) Pipeline options. For non-template jobs, the options are
+// identical to what is passed to job submission.
+google.protobuf.Struct pipeline_options = 3;
+}

[GitHub] beam pull request #3895: Change checksum back to md5

[tgroh]

GitHub user tgroh opened a pull request:

https://github.com/apache/beam/pull/3895

Change checksum back to md5

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [ ] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [ ] Each commit in the pull request should have a meaningful subject 
line and body.
 - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [ ] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [ ] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [ ] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---
The harness boot code depends on the type of checksum being an md5



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tgroh/beam md5

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3895.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3895






---

[jira] [Commented] (BEAM-2956) DataflowRunner incorrectly reports the user agent for the Dataflow distribution

[Luke Cwik (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179243#comment-16179243
 ] 

Luke Cwik commented on BEAM-2956:
-

The remaining change is within the Dataflow distribution so this can be closed.

> DataflowRunner incorrectly reports the user agent for the Dataflow 
> distribution
> ---
>
> Key: BEAM-2956
> URL: https://issues.apache.org/jira/browse/BEAM-2956
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.0.0, 2.1.0
>Reporter: Luke Cwik
>Assignee: Luke Cwik
> Fix For: 2.2.0
>
>
> The DataflowRunner when distributed with the Dataflow SDK distribution may 
> incorrectly submit a user agent and properties from the Apache Beam 
> distribution.
> This occurs when the Apache Beam jars appear on the classpath before the 
> Dataflow SDK distribution. The fix is to not have two files at the same path 
> but to use two different paths, where the lack of the second path means that 
> we are using the Apache Beam distribution and its existence implies we are 
> using the Dataflow distribution.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2986) Support reading avro GenericRecords with BigQueryIO

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179176#comment-16179176
 ] 

ASF GitHub Bot commented on BEAM-2986:
--

GitHub user steveniemitz opened a pull request:

https://github.com/apache/beam/pull/3894

[BEAM-2986] Support for reading avro GenericRecords from BigQuery

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is an initial pass at support for reading `GenericRecord`s from 
BigQuery (rather than JSON TableRows).

A couple points of discussion:
- `SerializableFunction>` is a disgusting generic type, should this be refactored to be simply 
`ParseFnFactory` or similar?
- I removed `BigQueryJsonReader` and all references to it.  A non-split 
`createReader` now follows the same code-path as split readers.  I'm not sure 
if there's a better way to do this than introducing an aggregating reader like 
I did with `MergedBoundedReader`?  I'm not sure under which conditions each 
method is called, but this seemed like the best compromise.
- I followed the AvroIO convention of `readRecords().withParseFn(...)`, 
however it seems like it would make a better API to make `readRecords` take the 
`parseFn` directly, so it's impossible to even attempt to build a reader 
without a parseFn (that would then fail at `expand()` time.  Is there a reason 
for the design of AvroIO like that?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tc-dc/beam bigquery-avro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3894


commit 89796f9a862f1fe1a5a04124f04a8132382af9b2
Author: steve 
Date:   2017-09-22T15:07:07Z

Support for using raw avro records from BigQuery




> Support reading avro GenericRecords with BigQueryIO
> ---
>
> Key: BEAM-2986
> URL: https://issues.apache.org/jira/browse/BEAM-2986
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Steve Niemitz
>Assignee: Chamikara Jayalath
>
> It would be nice to expose the avro records from BigQuery exports directly, 
> rather than converting/serializing them to TableRow objects.
> The interface could be similar to AvroIO.parseGenericRecords, allowing a user 
> to pass in a custom parseFn to the reader.
> There's a thread on the dev@ mailing this w/ discussion:
> https://lists.apache.org/thread.html/a045726a0c8d7412f3556ee3aac881920afb040180f42112b393c68e@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[jira] [Commented] (BEAM-2986) Support reading avro GenericRecords with BigQueryIO

[Eugene Kirpichov (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16179175#comment-16179175
 ] 

Eugene Kirpichov commented on BEAM-2986:


To do this change in a compatible way, you may want to follow an approach 
similar to what 
https://github.com/apache/beam/commit/77ba7a35cdae0b036791cce0682beefeb3fd809b 
did for TextIO.Write.

> Support reading avro GenericRecords with BigQueryIO
> ---
>
> Key: BEAM-2986
> URL: https://issues.apache.org/jira/browse/BEAM-2986
> Project: Beam
>  Issue Type: New Feature
>  Components: sdk-java-gcp
>Reporter: Steve Niemitz
>Assignee: Chamikara Jayalath
>
> It would be nice to expose the avro records from BigQuery exports directly, 
> rather than converting/serializing them to TableRow objects.
> The interface could be similar to AvroIO.parseGenericRecords, allowing a user 
> to pass in a custom parseFn to the reader.
> There's a thread on the dev@ mailing this w/ discussion:
> https://lists.apache.org/thread.html/a045726a0c8d7412f3556ee3aac881920afb040180f42112b393c68e@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

[GitHub] beam pull request #3894: [BEAM-2986] Support for reading avro GenericRecords...

[steveniemitz]

GitHub user steveniemitz opened a pull request:

https://github.com/apache/beam/pull/3894

[BEAM-2986] Support for reading avro GenericRecords from BigQuery

Follow this checklist to help us incorporate your contribution quickly and 
easily:

 - [x] Make sure there is a [JIRA 
issue](https://issues.apache.org/jira/projects/BEAM/issues/) filed for the 
change (usually before you start working on it).  Trivial changes like typos do 
not require a JIRA issue.  Your pull request should address just this issue, 
without pulling in other changes.
 - [x] Each commit in the pull request should have a meaningful subject 
line and body.
 - [x] Format the pull request title like `[BEAM-XXX] Fixes bug in 
ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA 
issue.
 - [x] Write a pull request description that is detailed enough to 
understand what the pull request does, how, and why.
 - [x] Run `mvn clean verify` to make sure basic checks pass. A more 
thorough check will be performed on your pull request automatically.
 - [x] If this contribution is large, please file an Apache [Individual 
Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).

---

This is an initial pass at support for reading `GenericRecord`s from 
BigQuery (rather than JSON TableRows).

A couple points of discussion:
- `SerializableFunction>` is a disgusting generic type, should this be refactored to be simply 
`ParseFnFactory` or similar?
- I removed `BigQueryJsonReader` and all references to it.  A non-split 
`createReader` now follows the same code-path as split readers.  I'm not sure 
if there's a better way to do this than introducing an aggregating reader like 
I did with `MergedBoundedReader`?  I'm not sure under which conditions each 
method is called, but this seemed like the best compromise.
- I followed the AvroIO convention of `readRecords().withParseFn(...)`, 
however it seems like it would make a better API to make `readRecords` take the 
`parseFn` directly, so it's impossible to even attempt to build a reader 
without a parseFn (that would then fail at `expand()` time.  Is there a reason 
for the design of AvroIO like that?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tc-dc/beam bigquery-avro

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/beam/pull/3894.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3894


commit 89796f9a862f1fe1a5a04124f04a8132382af9b2
Author: steve 
Date:   2017-09-22T15:07:07Z

Support for using raw avro records from BigQuery




---

[jira] [Created] (BEAM-2986) Support reading avro GenericRecords with BigQueryIO

[Steve Niemitz (JIRA)]

Steve Niemitz created BEAM-2986:
---

 Summary: Support reading avro GenericRecords with BigQueryIO
 Key: BEAM-2986
 URL: https://issues.apache.org/jira/browse/BEAM-2986
 Project: Beam
  Issue Type: New Feature
  Components: sdk-java-gcp
Reporter: Steve Niemitz
Assignee: Chamikara Jayalath


It would be nice to expose the avro records from BigQuery exports directly, 
rather than converting/serializing them to TableRow objects.

The interface could be similar to AvroIO.parseGenericRecords, allowing a user 
to pass in a custom parseFn to the reader.

There's a thread on the dev@ mailing this w/ discussion:
https://lists.apache.org/thread.html/a045726a0c8d7412f3556ee3aac881920afb040180f42112b393c68e@%3Cdev.beam.apache.org%3E



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Build failed in Jenkins: beam_PostCommit_Python_Verify #3208

[Apache Jenkins Server]

See 


--
[...truncated 44.47 KB...]
Collecting mock<3.0.0,>=1.0.1 (from apache-beam==2.2.0.dev0)
  Using cached mock-2.0.0-py2.py3-none-any.whl
Collecting oauth2client<4.0.0,>=2.0.1 (from apache-beam==2.2.0.dev0)
Collecting protobuf<=3.3.0,>=3.2.0 (from apache-beam==2.2.0.dev0)
  Using cached protobuf-3.3.0-cp27-cp27mu-manylinux1_x86_64.whl
Collecting pyyaml<4.0.0,>=3.12 (from apache-beam==2.2.0.dev0)
Collecting six<1.11,>=1.9 (from apache-beam==2.2.0.dev0)
  Using cached six-1.10.0-py2.py3-none-any.whl
Collecting typing<3.7.0,>=3.6.0 (from apache-beam==2.2.0.dev0)
  Using cached typing-3.6.2-py2-none-any.whl
Requirement already satisfied: futures>=2.2.0 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Requirement already satisfied: enum34>=1.0.4 in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
grpcio<2.0,>=1.0->apache-beam==2.2.0.dev0)
Collecting funcsigs>=1; python_version < "3.3" (from 
mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached funcsigs-1.0.2-py2.py3-none-any.whl
Collecting pbr>=0.11 (from mock<3.0.0,>=1.0.1->apache-beam==2.2.0.dev0)
  Using cached pbr-3.1.1-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached rsa-3.4.2-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1_modules-0.1.4-py2.py3-none-any.whl
Collecting pyasn1>=0.1.7 (from 
oauth2client<4.0.0,>=2.0.1->apache-beam==2.2.0.dev0)
  Using cached pyasn1-0.3.6-py2.py3-none-any.whl
Requirement already satisfied: setuptools in 
./target/.tox/py27cython/lib/python2.7/site-packages (from 
protobuf<=3.3.0,>=3.2.0->apache-beam==2.2.0.dev0)
Building wheels for collected packages: apache-beam
  Running setup.py bdist_wheel for apache-beam: started
  Running setup.py bdist_wheel for apache-beam: finished with status 'error'
  Complete output from command 

 -u -c "import setuptools, 
tokenize;__file__='/tmp/pip-kiLcME-build/setup.py';f=getattr(tokenize, 'open', 
open)(__file__);code=f.read().replace('\r\n', 
'\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d 
/tmp/tmpCCf9g2pip-wheel- --python-tag cp27:
  
:351:
 UserWarning: Normalizing '2.2.0.dev' to '2.2.0.dev0'
normalized_version,
  running bdist_wheel
  running build
  running build_py
  Traceback (most recent call last):
File "", line 1, in 
File "/tmp/pip-kiLcME-build/setup.py", line 203, in 
  'test': generate_protos_first(test),
File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
  dist.run_commands()
File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
  self.run_command(cmd)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File 
"
 line 204, in run
  self.run_command('build')
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/usr/lib/python2.7/distutils/command/build.py", line 128, in run
  self.run_command(cmd_name)
File "/usr/lib/python2.7/distutils/cmd.py", line 326, in run_command
  self.distribution.run_command(command)
File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
  cmd_obj.run()
File "/tmp/pip-kiLcME-build/setup.py", line 143, in run
  gen_protos.generate_proto_files()
File "gen_protos.py", line 65, in generate_proto_files
  'Not in apache git tree; unable to find proto definitions.')
  RuntimeError: Not in apache git tree; unable to find proto definitions.
  
  
  Failed building wheel for apache-beam
  Running setup.py clean for apache-beam
Failed to build apache-beam
Installing collected packages: avro, crcmod, dill, httplib2, six, funcsigs, 
pbr, mock, pyasn1, rsa, pyasn1-modules, oauth2client, protobuf, pyyaml, typing, 
apache-beam
  Found existing installation: six 1.11.0
Uninstalling six-1.11.0:
  Successfully uninstalled six-1.11.0
  Found existing installation: protobuf 3.4.0
Uninstalling protobuf-3.4.0:
  Successfully uninstalled protobuf-3.4.0
  Running setup.py install for apache-beam: started
Running setup.py install for apache-beam: finished with status 'error'
Complete output from 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Spark #3148

[Apache Jenkins Server]

See 

[jira] [Commented] (BEAM-2840) BigQueryIO write is slow/fail with a bounded source

[Vincent Spiewak (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2840?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16178928#comment-16178928
 ] 

Vincent Spiewak commented on BEAM-2840:
---

I tried to reproduce the issue with a public dataset without success.
We have around 700+ dates tables (2 years +) of datas. Around 200k-300k events 
per day

> BigQueryIO write is slow/fail with a bounded source
> ---
>
> Key: BEAM-2840
> URL: https://issues.apache.org/jira/browse/BEAM-2840
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-java-gcp
>Affects Versions: 2.0.0
> Environment: Gougle Cloud Platform
>Reporter: Vincent Spiewak
>Assignee: Reuven Lax
> Attachments: PrepareWrite.BatchLoads.png
>
>
> BigQueryIO Writer is slow / fail if the input source is bounded.
> EDIT: Input BQ: 294 GB, 741,896,827 events
> If the input source is bounded (GCS / BQ select / ...), BigQueryIO Writer use 
> the 
> "[Method.FILE_LOADS|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1168];
>  instead of streaming inserts.
> Large amounts of input datas result in a  java.lang.OutOfMemoryError / Java 
> heap space (500 millions rows).
> !PrepareWrite.BatchLoads.png|thumbnail!
> We cannot use "Method.STREAMING_INSERTS" or control the batchs sizes since
> [withMaxFilesPerBundle|https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryIO.java#L1131]
>  is private :(
> Someone reported a similar problem with GCS -> BQ on Stackoverflow: 
> [Why is writing to BigQuery from a Dataflow/Beam pipeline 
> slow?|https://stackoverflow.com/questions/45889992/why-is-writing-to-bigquery-from-a-dataflow-beam-pipeline-slow#comment78954153_45889992]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3918

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_MavenInstall #4872

[Apache Jenkins Server]

See 


--
[...truncated 420.58 KB...]
2017-09-25T12:05:08.661 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/avro/avro-compiler/1.8.2/avro-compiler-1.8.2.jar
 (81 KB at 1169.2 KB/sec)
2017-09-25T12:05:08.662 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar
2017-09-25T12:05:08.674 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-plugin-api/2.0.10/maven-plugin-api-2.0.10.jar
 (13 KB at 151.7 KB/sec)
2017-09-25T12:05:08.674 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/joda-time/joda-time/2.7/joda-time-2.7.jar
2017-09-25T12:05:08.695 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact-manager/2.0.10/maven-artifact-manager-2.0.10.jar
 (57 KB at 545.9 KB/sec)
2017-09-25T12:05:08.695 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar
2017-09-25T12:05:08.723 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/commons/commons-compress/1.8.1/commons-compress-1.8.1.jar
 (357 KB at 2725.1 KB/sec)
2017-09-25T12:05:08.724 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.7/slf4j-simple-1.7.7.jar
2017-09-25T12:05:08.724 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-api/1.7.7/slf4j-api-1.7.7.jar
 (29 KB at 216.4 KB/sec)
2017-09-25T12:05:08.736 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/xerial/snappy/snappy-java/1.1.1.3/snappy-java-1.1.1.3.jar
 (576 KB at 3994.8 KB/sec)
2017-09-25T12:05:08.750 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/slf4j/slf4j-simple/1.7.7/slf4j-simple-1.7.7.jar
 (11 KB at 66.0 KB/sec)
2017-09-25T12:05:08.785 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/joda-time/joda-time/2.7/joda-time-2.7.jar 
(576 KB at 2981.7 KB/sec)
[WARNING] Failed to getClass for org.apache.avro.mojo.SchemaMojo
[JENKINS] Archiving disabled
2017-09-25T12:05:09.841 [INFO]  
   
2017-09-25T12:05:09.841 [INFO] 

2017-09-25T12:05:09.841 [INFO] Skipping Apache Beam :: Parent
2017-09-25T12:05:09.841 [INFO] This project has been banned from the build due 
to previous failures.
2017-09-25T12:05:09.841 [INFO] 

[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
[JENKINS] Archiving disabled
2017-09-25T12:05:34.502 [INFO] 

2017-09-25T12:05:34.502 [INFO] Reactor Summary:
2017-09-25T12:05:34.502 [INFO] 
2017-09-25T12:05:34.502 [INFO] Apache Beam :: Parent 
.. SUCCESS [ 27.463 s]
2017-09-25T12:05:34.502 [INFO] Apache Beam :: SDKs :: Java :: Build Tools 
. SUCCESS [ 15.453 s]
2017-09-25T12:05:34.502 [INFO] Apache Beam :: SDKs 
 SUCCESS [  6.068 s]
2017-09-25T12:05:34.502 [INFO] Apache Beam :: SDKs :: Common 

Jenkins build is back to normal : beam_PostCommit_Python_Verify #3207

[Apache Jenkins Server]

See 

Jenkins build is still unstable: beam_PostCommit_Java_ValidatesRunner_Flink #3917

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Spark #3147

[Apache Jenkins Server]

See 


--
[...truncated 218.58 KB...]
2017-09-25T06:03:10.750 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.17/maven-checkstyle-plugin-2.17.pom
2017-09-25T06:03:10.777 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.17/maven-checkstyle-plugin-2.17.pom
 (14 KB at 482.8 KB/sec)
2017-09-25T06:03:10.780 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.17/maven-checkstyle-plugin-2.17.jar
2017-09-25T06:03:10.816 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-checkstyle-plugin/2.17/maven-checkstyle-plugin-2.17.jar
 (107 KB at 2945.2 KB/sec)
2017-09-25T06:03:10.820 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.20/maven-surefire-plugin-2.20.pom
2017-09-25T06:03:10.846 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.20/maven-surefire-plugin-2.20.pom
 (7 KB at 243.3 KB/sec)
2017-09-25T06:03:10.847 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.20/surefire-2.20.pom
2017-09-25T06:03:10.875 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/surefire/surefire/2.20/surefire-2.20.pom
 (21 KB at 725.5 KB/sec)
2017-09-25T06:03:10.878 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.20/maven-surefire-plugin-2.20.jar
2017-09-25T06:03:10.908 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/plugins/maven-surefire-plugin/2.20/maven-surefire-plugin-2.20.jar
 (52 KB at 1732.6 KB/sec)
2017-09-25T06:03:10.912 [INFO] 
2017-09-25T06:03:10.912 [INFO] --- maven-clean-plugin:3.0.0:clean 
(default-clean) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:10.914 [INFO] Deleting 

 (includes = [**/*.pyc, **/*.egg-info/, **/sdks/python/LICENSE, 
**/sdks/python/NOTICE, **/sdks/python/README.md], excludes = [])
2017-09-25T06:03:11.021 [INFO] 
2017-09-25T06:03:11.021 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:11.128 [INFO] 
2017-09-25T06:03:11.128 [INFO] --- maven-enforcer-plugin:3.0.0-M1:enforce 
(enforce-banned-dependencies) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:11.236 [INFO] 
2017-09-25T06:03:11.236 [INFO] --- maven-remote-resources-plugin:1.5:process 
(process-resource-bundles) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:11.353 [INFO] 
2017-09-25T06:03:11.353 [INFO] --- maven-resources-plugin:3.0.2:resources 
(default-resources) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:11.354 [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
2017-09-25T06:03:11.355 [INFO] Copying 11 resources
2017-09-25T06:03:11.383 [INFO] Copying 3 resources
2017-09-25T06:03:11.491 [INFO] 
2017-09-25T06:03:11.491 [INFO] --- maven-compiler-plugin:3.6.2:compile 
(default-compile) @ beam-sdks-java-build-tools ---
2017-09-25T06:03:11.495 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.pom
2017-09-25T06:03:11.521 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-incremental/1.1/maven-shared-incremental-1.1.pom
 (5 KB at 178.1 KB/sec)
2017-09-25T06:03:11.524 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.1/maven-shared-utils-0.1.pom
2017-09-25T06:03:11.550 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/apache/maven/shared/maven-shared-utils/0.1/maven-shared-utils-0.1.pom
 (4 KB at 146.4 KB/sec)
2017-09-25T06:03:11.552 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/6.0_ALPHA/asm-6.0_ALPHA.pom
2017-09-25T06:03:11.578 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm/6.0_ALPHA/asm-6.0_ALPHA.pom
 (2 KB at 72.8 KB/sec)
2017-09-25T06:03:11.579 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/6.0_ALPHA/asm-parent-6.0_ALPHA.pom
2017-09-25T06:03:11.604 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/org/ow2/asm/asm-parent/6.0_ALPHA/asm-parent-6.0_ALPHA.pom
 (6 KB at 214.8 KB/sec)
2017-09-25T06:03:11.606 [INFO] Downloading: 
https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0-M6/qdox-2.0-M6.pom
2017-09-25T06:03:11.633 [INFO] Downloaded: 
https://repo.maven.apache.org/maven2/com/thoughtworks/qdox/qdox/2.0-M6/qdox-2.0-M6.pom
 (16 KB at 585.5 KB/sec)
2017-09-25T06:03:11.635 [INFO] Downloading: