[jira] [Updated] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[Charles Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Chen updated BEAM-4024:
---
Description: 
Recently, changes were made to support streaming side inputs in the Beam Python 
DirectRunner.  However, these changes cause problems when the 
BundleBasedDirectRunner is explicitly used.  When Beam Python tests are run 
with the BundleBasedDirectRunner (i.e. when the FnApiRunner is disabled), the 
following errors occur:

==
FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
--
Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
 line 159, in test_empty_write
 p | beam.Create([]) | beam.io.Write(sink) # pylint: 
disable=expression-not-assigned
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", line 
409, in __exit__
 self.run().wait_until_finish()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
 line 104, in run
 state = result.wait_until_finish()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 421, in wait_until_finish
 self._executor.await_completion()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 360, in await_completion
 self._executor.await_completion()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 406, in await_completion
 six.reraise(t, v, tb)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values
 assert not view.has_result
AssertionError: 
 >> begin captured logging << 
root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
root: INFO: Running pipeline with DirectRunner.
root: DEBUG: Creating 1 empty shard(s).
root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
batches: 1, num_threads: 1
root: INFO: Renamed 1 shards in 0.11 seconds.
root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values
 assert not view.has_result
AssertionError

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in 

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88286=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88286
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 06/Apr/18 00:10
Start Date: 06/Apr/18 00:10
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5032: [BEAM-4021] Work around 
tox bug.
URL: https://github.com/apache/beam/pull/5032#issuecomment-379112617
 
 
   I don't know why the maven build fails to run `pip --version` while the 
gradle one succeeds (same machine).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88286)
Time Spent: 1.5h  (was: 1h 20m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88257=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88257
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:44
Start Date: 05/Apr/18 22:44
Worklog Time Spent: 10m 
  Work Description: shoyer commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379098591
 
 
   OK, squashed and rebased. Jenkins is still failing due to 
https://issues.apache.org/jira/browse/BEAM-4021


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88257)
Time Spent: 6h  (was: 5h 50m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[Charles Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Charles Chen updated BEAM-4024:
---
Description: 
Recently, changes were made to support streaming side inputs in the Beam Python 
DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
[https://github.com/apache/beam/pull/4949]).  However, these changes cause 
problems when the BundleBasedDirectRunner is explicitly used.  When Beam Python 
tests are run with the BundleBasedDirectRunner (i.e. when the FnApiRunner is 
disabled), the following errors occur:

==
 FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
 --
 Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
 line 159, in test_empty_write
 p | beam.Create([]) | beam.io.Write(sink) # pylint: 
disable=expression-not-assigned
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", line 
409, in __exit__
 self.run().wait_until_finish()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
 line 104, in run
 state = result.wait_until_finish()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 421, in wait_until_finish
 self._executor.await_completion()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 360, in await_completion
 self._executor.await_completion()
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 406, in await_completion
 six.reraise(t, v, tb)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values
 assert not view.has_result
 AssertionError: 
  >> begin captured logging << 
 root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.
 root: INFO: Running pipeline with DirectRunner.
 root: DEBUG: Creating 1 empty shard(s).
 root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
batches: 1, num_threads: 1
 root: INFO: Renamed 1 shards in 0.11 seconds.
 root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values
 assert not view.has_result
 AssertionError

root: ERROR: Exception at bundle 
, 
due to an exception.
 Traceback (most recent call last):
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call
 side_input_values)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call
 self._completion_callback.handle_result(self, self._input_bundle, result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result
 input_committed_bundle, self._timer_firings, transform_result)
 File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result
 committed_bundle.get_elements_iterable(make_copy=True))
 File 

[jira] [Commented] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427772#comment-16427772
 ] 

Ahmet Altay commented on BEAM-4025:
---

cc: [~melap] 

We can update the documentation to explicitly mention that python 3 is not 
supported and link to BEAM-1251 if they are interested in following the status.

> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> config was fixed after 2.2.0 - 
> https://github.com/apache/beam/commit/807b4bcefd877440c12f7fc3b401446e21899904.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_TFRecordIOIT #338

[Apache Jenkins Server]

See 


--
[...truncated 148.87 KB...]
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT #356

[Apache Jenkins Server]

See 


--
[...truncated 146.19 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT #340

[Apache Jenkins Server]

See 


--
[...truncated 146.97 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the 

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT #342

[Apache Jenkins Server]

See 


--
[...truncated 146.48 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded 

[jira] [Commented] (BEAM-4022) beam_PreCommit_Python_MavenInstall failing with cython error

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427574#comment-16427574
 ] 

Udi Meiri commented on BEAM-4022:
-

This is duplicate of https://issues.apache.org/jira/browse/BEAM-4021

> beam_PreCommit_Python_MavenInstall failing with cython error
> 
>
> Key: BEAM-4022
> URL: https://issues.apache.org/jira/browse/BEAM-4022
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> beam_PreCommit_Python_MavenInstall is failing with (link to failing job 
> [https://builds.apache.org/view/A-D/view/ActiveMQ/job/beam_PreCommit_Python_MavenInstall/4425/console]
>  )
>  
> OK (skipped=44)
> py27-gcp runtests: commands[5] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/run_tox_cleanup.sh
> py27-cython2 create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/.tox/py27-cython2
> py27-cython2 installdeps: cython==0.26.1
> ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/bin/tox",
>  line 11, in 
> sys.exit(run_main())
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main
> main(args)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main
> retcode = Session(config).runcommand()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand
> return self.subcommand_test()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test
> if self.setupenv(venv):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv
> status = venv.update(action=action)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update
> self.hook.tox_testenv_install_deps(action=action, venv=self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__
> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec
> return self._inner_hookexec(hook, methods, kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in 
> firstresult=hook.spec_opts.get('firstresult'),
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall
> return outcome.get_result()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result
> _reraise(*ex)  # noqa
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 180, in _multicall
> res = hook_impl.function(*args)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 452, in tox_testenv_install_deps
> venv._install(deps, action=action)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 331, in _install
> 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88247=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88247
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:05
Start Date: 05/Apr/18 22:05
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179614042
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Hooks allow runners to tailor execution of the worker to allow for 
customization
+// of features used by the harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a Fn API instruction.
+type RequestHook func(context.Context, *fnpb.InstructionRequest) error
+
+// RunRequestHooks runs the hooks that handle a FnAPI request.
+func RunRequestHooks(ctx context.Context, req *fnpb.InstructionRequest) {
+   // The request hooks should not modify the request.
+   // TODO(wcn): pass the request by value to enforce? That's a perf hit.
+   // I'd rather trust users to do the right thing.
+   for n, h := range activeHooks {
+   if h.Req != nil {
+   if err := h.Req(ctx, req); err != nil {
+   log.Infof(ctx, "request hook %s failed: %v", n, 
err)
+   }
+   }
+   }
+}
+
+// ResponseHook is called when sending a Fn API instruction response.
+type ResponseHook func(context.Context, *fnpb.InstructionRequest, 
*fnpb.InstructionResponse) error
+
+// RunResponseHooks runs the hooks that handle a FnAPI response.
+func RunResponseHooks(ctx context.Context, req *fnpb.InstructionRequest, resp 
*fnpb.InstructionResponse) {
+   for n, h := range activeHooks {
+   if h.Resp != nil {

[jira] [Assigned] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka reassigned BEAM-4023:
--

Assignee: Ankur Goenka  (was: Henning Rohde)

> Log warning for missing worker id in FnApiControlClientPoolService
> --
>
> Key: BEAM-4023
> URL: https://issues.apache.org/jira/browse/BEAM-4023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Blocker
>
> When running the maven build in a machine without the valid auth credentials 
> the module breaks like this:
> {code:bash}
> {{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
> beam-runners-gcp-gcsproxy ---}}
> {{[INFO] Prepared command line : bin/go build -buildmode=default -o 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
> github.com/apache/beam/cmd/gcsproxy}}
> {{[ERROR] }}
> {{[ERROR] -Exec.Err-}}
> {{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
> {{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
> undefined: option.WithoutAuthentication}}
> {{[ERROR] }}
> {{}}
> {{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact 
> proxy FAILURE [  1.038 s]}}
> {{}}
> {{[INFO] BUILD FAILURE}}
> {{}}
> {{[ERROR] Failed to execute goal 
> com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
> beam-runners-gcp-gcsproxy: Can't find generated target file : 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
> {{[ERROR] }}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-4023:
---
Priority: Major  (was: Blocker)

> Log warning for missing worker id in FnApiControlClientPoolService
> --
>
> Key: BEAM-4023
> URL: https://issues.apache.org/jira/browse/BEAM-4023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> We should 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

[Ankur Goenka (JIRA)]

Ankur Goenka created BEAM-4023:
--

 Summary: Log warning for missing worker id in 
FnApiControlClientPoolService
 Key: BEAM-4023
 URL: https://issues.apache.org/jira/browse/BEAM-4023
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Ankur Goenka
Assignee: Henning Rohde


When running the maven build in a machine without the valid auth credentials 
the module breaks like this:

{code:bash}

{{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
beam-runners-gcp-gcsproxy ---}}
{{[INFO] Prepared command line : bin/go build -buildmode=default -o 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
github.com/apache/beam/cmd/gcsproxy}}
{{[ERROR] }}
{{[ERROR] -Exec.Err-}}
{{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
{{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication}}
{{[ERROR] }}
{{}}

{{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy 
FAILURE [  1.038 s]}}
{{}}
{{[INFO] BUILD FAILURE}}
{{}}
{{[ERROR] Failed to execute goal 
com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
beam-runners-gcp-gcsproxy: Can't find generated target file : 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
{{[ERROR] }}

{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-4023:
---
Description: We should   (was: When running the maven build in a machine 
without the valid auth credentials the module breaks like this:

{code:bash}

{{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
beam-runners-gcp-gcsproxy ---}}
{{[INFO] Prepared command line : bin/go build -buildmode=default -o 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
github.com/apache/beam/cmd/gcsproxy}}
{{[ERROR] }}
{{[ERROR] -Exec.Err-}}
{{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
{{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication}}
{{[ERROR] }}
{{}}

{{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy 
FAILURE [  1.038 s]}}
{{}}
{{[INFO] BUILD FAILURE}}
{{}}
{{[ERROR] Failed to execute goal 
com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
beam-runners-gcp-gcsproxy: Can't find generated target file : 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
{{[ERROR] }}

{code})

> Log warning for missing worker id in FnApiControlClientPoolService
> --
>
> Key: BEAM-4023
> URL: https://issues.apache.org/jira/browse/BEAM-4023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Blocker
>
> We should 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88260=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88260
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:47
Start Date: 05/Apr/18 22:47
Worklog Time Spent: 10m 
  Work Description: shoyer commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379098591
 
 
   OK, squashed and rebased. Jenkins is still failing due to 
https://issues.apache.org/jira/browse/BEAM-4021 but the Python test suite 
appears to be passing (it also passes locally for me)


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88260)
Time Spent: 6h 10m  (was: 6h)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 6h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Taylor Edmiston (JIRA)]

Taylor Edmiston created BEAM-4025:
-

 Summary: Non-working Python 3 versions are installable on PyPI
 Key: BEAM-4025
 URL: https://issues.apache.org/jira/browse/BEAM-4025
 Project: Beam
  Issue Type: Bug
  Components: sdk-py-core
Affects Versions: 2.2.0, 2.1.0, 2.0.0, 0.6.0
Reporter: Taylor Edmiston
Assignee: Ahmet Altay


Beam does not currently support Python 3, however, it's possible to install 
_some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
PyPI configuration was fixed at some point after 2.2.0.)

This affects the following 5 versions:
{code:java}
$ pip3 install apache-beam==-1
Collecting apache-beam==-1
 Could not find a version that satisfies the requirement apache-beam==-1 (from 
versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
No matching distribution found for apache-beam==-1{code}
This is a confusing experience for users (customers) to discover that it's 
incompatible only on import after installing the package (and not reading the 
docs) via the error message:
{code:java}
RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
releaselevel='final', serial=0)].{code}
Since it's not compatible anyway, it's confusing that the non-working package 
versions are installable.

I don't think there's an easy way to remove just the Python 3 build from the 
wheels or in the PyPI GUI.  As far as I can tell, this would require rebuilding 
+ re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88269=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88269
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:10
Start Date: 05/Apr/18 23:10
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5035: [BEAM-4024] Fix 
AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379103223
 
 
   Thank you LGTM.
   
   Should we run at least some batch unit tests with BundleBasedDirectRunner to 
prevent regressions?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88269)
Time Spent: 40m  (was: 0.5h)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #109

[Apache Jenkins Server]

See 


--
[...truncated 26.89 KB...]
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.

Build failed in Jenkins: beam_PerformanceTests_AvroIOIT_HDFS #14

[Apache Jenkins Server]

See 


--
[...truncated 161.05 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 from the shaded 
jar.
[INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.9.13 from the shaded jar.

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88226=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88226
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 20:30
Start Date: 05/Apr/18 20:30
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379066778
 
 
   The change itself LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88226)
Time Spent: 5h 20m  (was: 5h 10m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (BEAM-3521) Review and update the references of maven to gradle in the website

[Scott Wegner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Scott Wegner reassigned BEAM-3521:
--

Assignee: Scott Wegner

> Review and update the references of maven to gradle in the website
> --
>
> Key: BEAM-3521
> URL: https://issues.apache.org/jira/browse/BEAM-3521
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, website
>Reporter: Ismaël Mejía
>Assignee: Scott Wegner
>Priority: Major
>
> I suppose that the only maven reference that will stay probably should be the 
> one on the maven-archetype but I suppose this will be decided later on.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4023) Log warning for missing worker id in FnApiControlClientPoolService

[Ankur Goenka (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ankur Goenka updated BEAM-4023:
---
Description: We should log warning for missing worker id when connecting 
the GRPC channel.  (was: We should )

> Log warning for missing worker id in FnApiControlClientPoolService
> --
>
> Key: BEAM-4023
> URL: https://issues.apache.org/jira/browse/BEAM-4023
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
>
> We should log warning for missing worker id when connecting the GRPC channel.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3982) Transform libraries not registering their types

[Henning Rohde (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henning Rohde resolved BEAM-3982.
-
   Resolution: Fixed
Fix Version/s: 2.5.0

> Transform libraries not registering their types
> ---
>
> Key: BEAM-3982
> URL: https://issues.apache.org/jira/browse/BEAM-3982
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Bill Neubauer
>Assignee: Henning Rohde
>Priority: Minor
> Fix For: 2.5.0
>
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> The transform libraries in the SDK have structs-as-DoFns, but the types are 
> not registered. While this works in a direct runner, trying to run examples 
> using those libraries fails because the DoFns aren't serializable.
> It's not clear how to make the libraries intrinsically self-policing. At 
> least to ensure the examples work, these failures can be exposed easily using 
> the flags "--runner=dataflow --dry_run" to force the serialization error.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88262=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88262
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:51
Start Date: 05/Apr/18 22:51
Worklog Time Spent: 10m 
  Work Description: charlesccychen opened a new pull request #5035: 
[BEAM-4024] Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035
 
 
   This change fixes an AttributeError in the BundleBasedDirectRunner.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88262)
Time Spent: 10m
Remaining Estimate: 0h

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> 

[jira] [Commented] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Ahmet Altay (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427726#comment-16427726
 ] 

Ahmet Altay commented on BEAM-4025:
---

I agree this is confusing. Lack of python 3 support is already documented, if 
it would help we could add more documentation. However, I do not think it is 
worth the effort to patch and re-release old sdks.

> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> PyPI configuration was fixed at some point after 2.2.0.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88278=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88278
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:54
Start Date: 05/Apr/18 23:54
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179630905
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -33,6 +33,16 @@ extras = test
 whitelist_externals =
   find
   time
+deps =
+  grpcio-tools==1.3.5
+  cython: cython==0.26.1
+
+# These 2 magic command overrides are required for Jenkins builds.
 
 Review comment:
   This is bizarre and a good find. I wonder if it is any better with the new 
tox 3 (released on 4/2).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88278)
Time Spent: 0.5h  (was: 20m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in 

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88280=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88280
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:54
Start Date: 05/Apr/18 23:54
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179629335
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -37,7 +38,10 @@ task setupVirtualenv {
   args '-c', ". ${envdir}/bin/activate && pip install --upgrade tox"
 }
   }
-  outputs.files("${envdir}/bin/tox")
+  // Gradle will delete outputs whenever it thinks they are stale. Putting a
 
 Review comment:
   Is this a warning to reader? Or is there more work that can be done?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88280)
Time Spent: 40m  (was: 0.5h)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 40m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88279=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88279
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:54
Start Date: 05/Apr/18 23:54
Worklog Time Spent: 10m 
  Work Description: aaltay commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179630812
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -33,6 +33,16 @@ extras = test
 whitelist_externals =
   find
   time
+deps =
+  grpcio-tools==1.3.5
+  cython: cython==0.26.1
+
+# These 2 magic command overrides are required for Jenkins builds.
+# Otherwise we get "OSError: [Errno 2] No such file or directory" errors.
+# Source:
+# https://github.com/tox-dev/tox/issues/123#issuecomment-284714629
+install_command = {envbindir}/python {envbindir}/pip install {opts} {packages}
 
 Review comment:
   Do you need the `{envbindir}/python` part? `pip` might already be an 
executable.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88279)
Time Spent: 0.5h  (was: 20m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ 

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88285=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88285
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 06/Apr/18 00:09
Start Date: 06/Apr/18 00:09
Worklog Time Spent: 10m 
  Work Description: aaltay commented on issue #5032: [BEAM-4021] Work 
around tox bug.
URL: https://github.com/apache/beam/pull/5032#issuecomment-379112441
 
 
   Thanks for answering my questions. It looks good other than the test failure.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88285)
Time Spent: 1h 20m  (was: 1h 10m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 
> 

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88224=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88224
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 20:28
Start Date: 05/Apr/18 20:28
Worklog Time Spent: 10m 
  Work Description: robertwb commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379066276
 
 
   This looks unrelated to your changes. 
   
   Jenkins: retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88224)
Time Spent: 5h 10m  (was: 5h)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 5h 10m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88244
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 21:44
Start Date: 05/Apr/18 21:44
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #4959: [BEAM-3956] 
Preserve stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379086332
 
 
   Please squash the commits, thank you.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88244)
Time Spent: 5h 50m  (was: 5h 40m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 5h 50m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88264=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88264
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:52
Start Date: 05/Apr/18 22:52
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5035: [BEAM-4024] 
Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379099988
 
 
   R: @aaltay 
   CC: @mariapython 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88264)
Time Spent: 0.5h  (was: 20m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  

[jira] [Work logged] (BEAM-3437) Support schema in PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3437?focusedWorklogId=88216=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88216
 ]

ASF GitHub Bot logged work on BEAM-3437:


Author: ASF GitHub Bot
Created on: 05/Apr/18 19:47
Start Date: 05/Apr/18 19:47
Worklog Time Spent: 10m 
  Work Description: reuvenlax commented on issue #4964: [BEAM-3437] 
Introduce Schema class, and use it in BeamSQL
URL: https://github.com/apache/beam/pull/4964#issuecomment-379055305
 
 
   FYI that is exactly the point of this work. However the work to make
   schemas work on non-Row objects is still in the branch I have with JB. We
   decided it made sense to merge just this part of it so that we don't create
   huge conflicts with BeamSQL work.
   
   On Thu, Apr 5, 2018 at 12:43 PM Kenn Knowles 
   wrote:
   
   > OK - if it doesn't exist in the portable model and is just a feature of
   > the Java SDK then that is a solid avenue for exploration. For SQL we
   > actually want perhaps a lot more than this, as we'd like embedded SQL
   > transforms to be able to input and output non-Row types generically based
   > on schema.
   >
   > —
   > You are receiving this because you authored the thread.
   > Reply to this email directly, view it on GitHub
   > , or mute
   > the thread
   > 

   > .
   >
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88216)
Time Spent: 8h 20m  (was: 8h 10m)

> Support schema in PCollections
> --
>
> Key: BEAM-3437
> URL: https://issues.apache.org/jira/browse/BEAM-3437
> Project: Beam
>  Issue Type: Wish
>  Components: beam-model
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 8h 20m
>  Remaining Estimate: 0h
>
> As discussed with some people in the team, it would be great to add schema 
> support in {{PCollections}}. It will allow us:
> 1. To expect some data type in {{PTransforms}}
> 2. Improve some runners with additional features (I'm thinking about Spark 
> runner with data frames for instance).
> A technical draft document has been created: 
> https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=BhykQIs=5a203b46=comment_email_document
> I also started a PoC on a branch, I will update this Jira with a "discussion" 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3437) Support schema in PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3437?focusedWorklogId=88218=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88218
 ]

ASF GitHub Bot logged work on BEAM-3437:


Author: ASF GitHub Bot
Created on: 05/Apr/18 19:55
Start Date: 05/Apr/18 19:55
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4964: [BEAM-3437] 
Introduce Schema class, and use it in BeamSQL
URL: https://github.com/apache/beam/pull/4964#issuecomment-379057477
 
 
   Yea when green LGTM for merge from me. Incidentally it might be a good idea 
to move work from said branch to master just to get finer grained review. This 
PR probably had somewhere between 2 and 5 PRs worth of orthogonal changes in it 
(like switching things to joda time, moves, renames, and also the new 
functionality).


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88218)
Time Spent: 8h 40m  (was: 8.5h)

> Support schema in PCollections
> --
>
> Key: BEAM-3437
> URL: https://issues.apache.org/jira/browse/BEAM-3437
> Project: Beam
>  Issue Type: Wish
>  Components: beam-model
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 8h 40m
>  Remaining Estimate: 0h
>
> As discussed with some people in the team, it would be great to add schema 
> support in {{PCollections}}. It will allow us:
> 1. To expect some data type in {{PTransforms}}
> 2. Improve some runners with additional features (I'm thinking about Spark 
> runner with data frames for instance).
> A technical draft document has been created: 
> https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=BhykQIs=5a203b46=comment_email_document
> I also started a PoC on a branch, I will update this Jira with a "discussion" 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3516) SpannerWriteGroupFn does not respect mutation limits

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3516?focusedWorklogId=88222=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88222
 ]

ASF GitHub Bot logged work on BEAM-3516:


Author: ASF GitHub Bot
Created on: 05/Apr/18 20:16
Start Date: 05/Apr/18 20:16
Worklog Time Spent: 10m 
  Work Description: mairbek commented on issue #4860: [BEAM-3516] Spanner 
BatchFn does not respect mutation limits
URL: https://github.com/apache/beam/pull/4860#issuecomment-379062843
 
 
   lgtm, @chamikaramj can you take a look?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88222)
Time Spent: 1h 20m  (was: 1h 10m)

> SpannerWriteGroupFn does not respect mutation limits
> 
>
> Key: BEAM-3516
> URL: https://issues.apache.org/jira/browse/BEAM-3516
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Ryan Gordon
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1h 20m
>  Remaining Estimate: 0h
>
> When using SpannerIO.write(), if it happens to be a large batch or a table 
> with indexes its very possible it can hit the Spanner Mutations Limitation 
> and fail with the following error:
> {quote}Jan 02, 2018 2:42:59 PM 
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> SEVERE: 2018-01-02T22:42:57.873Z: (3e7c871d215e890b): 
> com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: 
> io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The transaction contains 
> too many mutations. Insert and update operations count with the multiplicity 
> of the number of columns they affect. For example, inserting values into one 
> key column and four non-key columns count as five mutations total for the 
> insert. Delete and delete range operations count as one mutation regardless 
> of the number of columns affected. The total mutation count includes any 
> changes to indexes that the transaction generates. Please reduce the number 
> of writes, or use fewer indexes. (Maximum number: 2)
> links {
>  description: "Cloud Spanner limits documentation."
>  url: "https://cloud.google.com/spanner/docs/limits;
> }
> at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
>  at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
>  at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
>  at 
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:404)
>  at 
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:376)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:729)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:726)
>  at com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:200)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:725)
>  at 
> com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:248)
>  at 
> com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.flushBatch(SpannerWriteGroupFn.java:108)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:79)
> {quote}
>  
> As a workaround we can override the "withBatchSizeBytes" to something much 
> smaller:
> {quote}mutations.apply("Write", SpannerIO
>    .write()
>    // Artificially reduce the max batch size b/c the batcher currently doesn't
>    // take into account the 2 mutation multiplicity limit
>    .withBatchSizeBytes(1024) // 1KB
>    .withProjectId("#PROJECTID#")
>    .withInstanceId("#INSTANCE#")
>    .withDatabaseId("#DATABASE#")
>  );
> {quote}
> While this is not as efficient, it at least allows it to work consistently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3516) SpannerWriteGroupFn does not respect mutation limits

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3516?focusedWorklogId=88223=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88223
 ]

ASF GitHub Bot logged work on BEAM-3516:


Author: ASF GitHub Bot
Created on: 05/Apr/18 20:16
Start Date: 05/Apr/18 20:16
Worklog Time Spent: 10m 
  Work Description: mairbek commented on issue #4860: [BEAM-3516] Spanner 
BatchFn does not respect mutation limits
URL: https://github.com/apache/beam/pull/4860#issuecomment-379063009
 
 
   retest this please


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88223)
Time Spent: 1.5h  (was: 1h 20m)

> SpannerWriteGroupFn does not respect mutation limits
> 
>
> Key: BEAM-3516
> URL: https://issues.apache.org/jira/browse/BEAM-3516
> Project: Beam
>  Issue Type: Bug
>  Components: runner-dataflow
>Affects Versions: 2.2.0
>Reporter: Ryan Gordon
>Assignee: Chamikara Jayalath
>Priority: Major
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> When using SpannerIO.write(), if it happens to be a large batch or a table 
> with indexes its very possible it can hit the Spanner Mutations Limitation 
> and fail with the following error:
> {quote}Jan 02, 2018 2:42:59 PM 
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> SEVERE: 2018-01-02T22:42:57.873Z: (3e7c871d215e890b): 
> com.google.cloud.spanner.SpannerException: INVALID_ARGUMENT: 
> io.grpc.StatusRuntimeException: INVALID_ARGUMENT: The transaction contains 
> too many mutations. Insert and update operations count with the multiplicity 
> of the number of columns they affect. For example, inserting values into one 
> key column and four non-key columns count as five mutations total for the 
> insert. Delete and delete range operations count as one mutation regardless 
> of the number of columns affected. The total mutation count includes any 
> changes to indexes that the transaction generates. Please reduce the number 
> of writes, or use fewer indexes. (Maximum number: 2)
> links {
>  description: "Cloud Spanner limits documentation."
>  url: "https://cloud.google.com/spanner/docs/limits;
> }
> at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerExceptionPreformatted(SpannerExceptionFactory.java:119)
>  at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:43)
>  at 
> com.google.cloud.spanner.SpannerExceptionFactory.newSpannerException(SpannerExceptionFactory.java:80)
>  at 
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.get(GrpcSpannerRpc.java:404)
>  at 
> com.google.cloud.spanner.spi.v1.GrpcSpannerRpc.commit(GrpcSpannerRpc.java:376)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:729)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl$2.call(SpannerImpl.java:726)
>  at com.google.cloud.spanner.SpannerImpl.runWithRetries(SpannerImpl.java:200)
>  at 
> com.google.cloud.spanner.SpannerImpl$SessionImpl.writeAtLeastOnce(SpannerImpl.java:725)
>  at 
> com.google.cloud.spanner.SessionPool$PooledSession.writeAtLeastOnce(SessionPool.java:248)
>  at 
> com.google.cloud.spanner.DatabaseClientImpl.writeAtLeastOnce(DatabaseClientImpl.java:37)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.flushBatch(SpannerWriteGroupFn.java:108)
>  at 
> org.apache.beam.sdk.io.gcp.spanner.SpannerWriteGroupFn.processElement(SpannerWriteGroupFn.java:79)
> {quote}
>  
> As a workaround we can override the "withBatchSizeBytes" to something much 
> smaller:
> {quote}mutations.apply("Write", SpannerIO
>    .write()
>    // Artificially reduce the max batch size b/c the batcher currently doesn't
>    // take into account the 2 mutation multiplicity limit
>    .withBatchSizeBytes(1024) // 1KB
>    .withProjectId("#PROJECTID#")
>    .withInstanceId("#INSTANCE#")
>    .withDatabaseId("#DATABASE#")
>  );
> {quote}
> While this is not as efficient, it at least allows it to work consistently



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88229=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88229
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 20:38
Start Date: 05/Apr/18 20:38
Worklog Time Spent: 10m 
  Work Description: tvalentyn commented on issue #4959: [BEAM-3956] 
Preserve stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379068995
 
 
   Jenkins issue is tracked here: 
https://issues.apache.org/jira/browse/BEAM-4022. Changes LGTM.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88229)
Time Spent: 5.5h  (was: 5h 20m)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Taylor Edmiston (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427764#comment-16427764
 ] 

Taylor Edmiston commented on BEAM-4025:
---

I saw that the quickstart docs list using Python 2 but didn't see it mentioned 
anywhere explicitly that Python 3 wasn't supported.  A customer came to us with 
the same confusion.

> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> config was fixed after 2.2.0 - 
> https://github.com/apache/beam/commit/807b4bcefd877440c12f7fc3b401446e21899904.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build is back to stable : beam_PostCommit_Java_MavenInstall #6387

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-3956) Stacktraces from exceptions in user code should be preserved in the Python SDK

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3956?focusedWorklogId=88237=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88237
 ]

ASF GitHub Bot logged work on BEAM-3956:


Author: ASF GitHub Bot
Created on: 05/Apr/18 21:03
Start Date: 05/Apr/18 21:03
Worklog Time Spent: 10m 
  Work Description: shoyer commented on issue #4959: [BEAM-3956] Preserve 
stacktraces for Python exceptions
URL: https://github.com/apache/beam/pull/4959#issuecomment-379076063
 
 
   OK, should I squash my commits or do you do that yourself as part of the 
merge process?


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88237)
Time Spent: 5h 40m  (was: 5.5h)

> Stacktraces from exceptions in user code should be preserved in the Python SDK
> --
>
> Key: BEAM-3956
> URL: https://issues.apache.org/jira/browse/BEAM-3956
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Stephan Hoyer
>Priority: Major
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> Currently, Beam's Python SDK loses stacktraces for exceptions. It does 
> helpfully add a tag like "[while running StageA]" to exception error 
> messages, but that doesn't include the stacktrace of Python functions being 
> called.
> Including the full stacktraces would make a big difference for the ease of 
> debugging Beam pipelines when things go wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-2103) Missing remark about Beam being incompatible to Python 3 in the documentation

[Taylor Edmiston (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-2103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427682#comment-16427682
 ] 

Taylor Edmiston commented on BEAM-2103:
---

I read this documentation but came here to echo the same sentiment.

The quickstart docs mention a Python 2 requirement, but they don't explicitly 
call out Python 3 incompatibility.  Since Python 3 incompatibility is quite a 
rare thing in 2018, I feel like this should be clarified.

This comes from the dev side but our customer using Dataflow via Airflow wasn't 
clear that it's incompatible.

> Missing remark about Beam being incompatible to Python 3 in the documentation
> -
>
> Key: BEAM-2103
> URL: https://issues.apache.org/jira/browse/BEAM-2103
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core, website
>Affects Versions: 0.6.0
>Reporter: Tobias Kaymak
>Priority: Minor
>
> Python quickstart and the frontpage documentation do not mention that the SDK 
> is currently only for Python 2.7



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3981) Futurize and fix python 2 compatibility for coders package

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3981?focusedWorklogId=88259=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88259
 ]

ASF GitHub Bot logged work on BEAM-3981:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:46
Start Date: 05/Apr/18 22:46
Worklog Time Spent: 10m 
  Work Description: asfgit commented on issue #4990: [BEAM-3981] [WIP] 
Futurize and fix python 2 compatibility for coders subpackage
URL: https://github.com/apache/beam/pull/4990#issuecomment-379098999
 
 
   FAILURE

   --none--


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88259)
Time Spent: 5h 20m  (was: 5h 10m)

> Futurize and fix python 2 compatibility for coders package
> --
>
> Key: BEAM-3981
> URL: https://issues.apache.org/jira/browse/BEAM-3981
> Project: Beam
>  Issue Type: Sub-task
>  Components: sdk-py-core
>Reporter: Robbe
>Assignee: Ahmet Altay
>Priority: Major
>  Time Spent: 5h 20m
>  Remaining Estimate: 0h
>
> Run automatic conversion with futurize tool on coders subpackage and fix 
> python 2 compatibility. This prepares the subpackage for python 3 support.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88284=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88284
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 06/Apr/18 00:03
Start Date: 06/Apr/18 00:03
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179632203
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -33,6 +33,16 @@ extras = test
 whitelist_externals =
   find
   time
+deps =
+  grpcio-tools==1.3.5
+  cython: cython==0.26.1
+
+# These 2 magic command overrides are required for Jenkins builds.
 
 Review comment:
   I believe this is already running using tox 3.0.0.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88284)
Time Spent: 1h 10m  (was: 1h)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 10m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> 

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #14

[Apache Jenkins Server]

See 


--
[...truncated 45.26 KB...]
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-hadoop-file-system:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-hdfs:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty:jar:6.1.26 from the shaded jar.
[INFO] Excluding org.mortbay.jetty:jetty-util:jar:6.1.26 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-core:jar:1.9 from the shaded jar.
[INFO] Excluding com.sun.jersey:jersey-server:jar:1.9 from the shaded jar.
[INFO] Excluding asm:asm:jar:3.1 from the shaded jar.
[INFO] Excluding commons-cli:commons-cli:jar:1.2 from the shaded jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.4 from the shaded jar.
[INFO] Excluding commons-io:commons-io:jar:2.4 from the shaded jar.
[INFO] Excluding commons-lang:commons-lang:jar:2.6 from the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.1.3 from the shaded jar.
[INFO] Excluding commons-daemon:commons-daemon:jar:1.0.13 from the shaded jar.
[INFO] Excluding log4j:log4j:jar:1.2.17 from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java:jar:3.2.0 from the shaded 
jar.
[INFO] Excluding javax.servlet:servlet-api:jar:2.5 from the shaded jar.
[INFO] Excluding xmlenc:xmlenc:jar:0.52 from the shaded jar.
[INFO] Excluding io.netty:netty-all:jar:4.0.23.Final from the shaded jar.
[INFO] Excluding xerces:xercesImpl:jar:2.9.1 from the shaded jar.
[INFO] Excluding xml-apis:xml-apis:jar:1.3.04 from the shaded jar.
[INFO] Excluding org.apache.htrace:htrace-core:jar:3.1.0-incubating from the 
shaded jar.
[INFO] Excluding org.fusesource.leveldbjni:leveldbjni-all:jar:1.8 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-client:jar:2.7.1 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-common:jar:2.7.3 from the shaded jar.
[INFO] Excluding org.apache.commons:commons-math3:jar:3.1.1 from the shaded jar.
[INFO] Excluding commons-httpclient:commons-httpclient:jar:3.1 from the shaded 
jar.
[INFO] Excluding commons-net:commons-net:jar:3.1 from the shaded jar.
[INFO] Excluding commons-collections:commons-collections:jar:3.2.2 from the 
shaded jar.
[INFO] Excluding javax.servlet.jsp:jsp-api:jar:2.1 from the shaded jar.
[INFO] Excluding commons-configuration:commons-configuration:jar:1.6 from the 
shaded jar.
[INFO] Excluding commons-digester:commons-digester:jar:1.8 from the shaded jar.
[INFO] Excluding commons-beanutils:commons-beanutils:jar:1.7.0 from the shaded 
jar.
[INFO] Excluding commons-beanutils:commons-beanutils-core:jar:1.8.0 from the 
shaded jar.
[INFO] Excluding org.slf4j:slf4j-log4j12:jar:1.7.10 from the shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.2.4 from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-auth:jar:2.7.3 from the shaded jar.
[INFO] Excluding 
org.apache.directory.server:apacheds-kerberos-codec:jar:2.0.0-M15 from the 
shaded jar.
[INFO] Excluding org.apache.directory.server:apacheds-i18n:jar:2.0.0-M15 from 
the shaded jar.
[INFO] Excluding org.apache.directory.api:api-asn1-api:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.directory.api:api-util:jar:1.0.0-M20 from the 
shaded jar.
[INFO] Excluding org.apache.curator:curator-framework:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.curator:curator-recipes:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.zookeeper:zookeeper:jar:3.4.6 from the shaded jar.
[INFO] Excluding io.netty:netty:jar:3.7.0.Final from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-app:jar:2.7.1 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-common:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-client:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-server-common:jar:2.7.1 from the 
shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-shuffle:jar:2.7.1 
from the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-api:jar:2.7.1 from the shaded 
jar.
[INFO] Excluding org.apache.hadoop:hadoop-mapreduce-client-core:jar:2.7.3 from 
the shaded jar.
[INFO] Excluding org.apache.hadoop:hadoop-yarn-common:jar:2.7.3 from the shaded 
jar.
[INFO] Excluding com.sun.jersey:jersey-client:jar:1.9 from the shaded jar.
[INFO] Excluding org.codehaus.jackson:jackson-jaxrs:jar:1.9.13 from the shaded 
jar.
[INFO] Excluding org.codehaus.jackson:jackson-xc:jar:1.9.13 from the shaded 

[jira] [Work logged] (BEAM-3437) Support schema in PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3437?focusedWorklogId=88206=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88206
 ]

ASF GitHub Bot logged work on BEAM-3437:


Author: ASF GitHub Bot
Created on: 05/Apr/18 19:41
Start Date: 05/Apr/18 19:41
Worklog Time Spent: 10m 
  Work Description: kennknowles commented on issue #4964: [BEAM-3437] 
Introduce Schema class, and use it in BeamSQL
URL: https://github.com/apache/beam/pull/4964#issuecomment-379053692
 
 
   OK - if it doesn't exist in the portable model and is just a feature of the 
Java SDK then that is a solid avenue for exploration. For SQL we actually want 
perhaps a lot more than this, as we'd like embedded SQL transforms to be able 
to input and output non-Row types generically based on schema.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88206)
Time Spent: 8h 10m  (was: 8h)

> Support schema in PCollections
> --
>
> Key: BEAM-3437
> URL: https://issues.apache.org/jira/browse/BEAM-3437
> Project: Beam
>  Issue Type: Wish
>  Components: beam-model
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 8h 10m
>  Remaining Estimate: 0h
>
> As discussed with some people in the team, it would be great to add schema 
> support in {{PCollections}}. It will allow us:
> 1. To expect some data type in {{PTransforms}}
> 2. Improve some runners with additional features (I'm thinking about Spark 
> runner with data frames for instance).
> A technical draft document has been created: 
> https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=BhykQIs=5a203b46=comment_email_document
> I also started a PoC on a branch, I will update this Jira with a "discussion" 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3437) Support schema in PCollections

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3437?focusedWorklogId=88217=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88217
 ]

ASF GitHub Bot logged work on BEAM-3437:


Author: ASF GitHub Bot
Created on: 05/Apr/18 19:51
Start Date: 05/Apr/18 19:51
Worklog Time Spent: 10m 
  Work Description: akedin commented on issue #4964: [BEAM-3437] Introduce 
Schema class, and use it in BeamSQL
URL: https://github.com/apache/beam/pull/4964#issuecomment-379056365
 
 
   Yes, lgtm from me, merge when ready


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88217)
Time Spent: 8.5h  (was: 8h 20m)

> Support schema in PCollections
> --
>
> Key: BEAM-3437
> URL: https://issues.apache.org/jira/browse/BEAM-3437
> Project: Beam
>  Issue Type: Wish
>  Components: beam-model
>Reporter: Jean-Baptiste Onofré
>Assignee: Jean-Baptiste Onofré
>Priority: Major
>  Time Spent: 8.5h
>  Remaining Estimate: 0h
>
> As discussed with some people in the team, it would be great to add schema 
> support in {{PCollections}}. It will allow us:
> 1. To expect some data type in {{PTransforms}}
> 2. Improve some runners with additional features (I'm thinking about Spark 
> runner with data frames for instance).
> A technical draft document has been created: 
> https://docs.google.com/document/d/1tnG2DPHZYbsomvihIpXruUmQ12pHGK0QIvXS1FOTgRc/edit?disco=BhykQIs=5a203b46=comment_email_document
> I also started a PoC on a branch, I will update this Jira with a "discussion" 
> PR.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[Charles Chen (JIRA)]

Charles Chen created BEAM-4024:
--

 Summary: BundleBasedDirectRunner fails with recent side input 
changes
 Key: BEAM-4024
 URL: https://issues.apache.org/jira/browse/BEAM-4024
 Project: Beam
  Issue Type: Improvement
  Components: sdk-py-core
Reporter: Charles Chen
Assignee: Charles Chen


Recently, changes were made to support streaming side inputs in the Beam Python 
DirectRunner.  However, these changes cause problems when the 
BundleBasedDirectRunner is explicitly used.  When Beam Python tests are run 
with the BundleBasedDirectRunner (i.e. when the FnApiRunner is disabled), the 
following errors occur:

{{==}}
{{FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)}}
{{--}}
{{Traceback (most recent call last):}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
 line 159, in test_empty_write}}
{{ p | beam.Create([]) | beam.io.Write(sink) # pylint: 
disable=expression-not-assigned}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", line 
409, in __exit__}}
{{ self.run().wait_until_finish()}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
 line 104, in run}}
{{ state = result.wait_until_finish()}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
 line 421, in wait_until_finish}}
{{ self._executor.await_completion()}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 360, in await_completion}}
{{ self._executor.await_completion()}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 406, in await_completion}}
{{ six.reraise(t, v, tb)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call}}
{{ side_input_values)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call}}
{{ self._completion_callback.handle_result(self, self._input_bundle, result)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result}}
{{ input_committed_bundle, self._timer_firings, transform_result)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result}}
{{ committed_bundle.get_elements_iterable(make_copy=True))}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values}}
{{ assert not view.has_result}}
{{AssertionError: }}
{{ >> begin captured logging << }}
{{root: INFO: Missing pipeline option (runner). Executing pipeline using the 
default runner: DirectRunner.}}
{{root: INFO: Running pipeline with DirectRunner.}}
{{root: DEBUG: Creating 1 empty shard(s).}}
{{root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
batches: 1, num_threads: 1}}
{{root: INFO: Renamed 1 shards in 0.11 seconds.}}
{{root: ERROR: Exception at bundle 
, 
due to an exception.}}
{{ Traceback (most recent call last):}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call}}
{{ side_input_values)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call}}
{{ self._completion_callback.handle_result(self, self._input_bundle, result)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 238, in handle_result}}
{{ input_committed_bundle, self._timer_firings, transform_result)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 242, in handle_result}}
{{ committed_bundle.get_elements_iterable(make_copy=True))}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
 line 97, in add_values}}
{{ assert not view.has_result}}
{{AssertionError}}{{root: ERROR: Exception at bundle 
, 
due to an exception.}}
{{ Traceback (most recent call last):}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 307, in call}}
{{ side_input_values)}}
{{ File 
"/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
 line 346, in attempt_call}}
{{ self._completion_callback.handle_result(self, self._input_bundle, result)}}
{{ File 

[jira] [Updated] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Taylor Edmiston (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Taylor Edmiston updated BEAM-4025:
--
Description: 
Beam does not currently support Python 3, however, it's possible to install 
_some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
config was fixed after 2.2.0 - 
https://github.com/apache/beam/commit/807b4bcefd877440c12f7fc3b401446e21899904.)

This affects the following 5 versions:
{code:java}
$ pip3 install apache-beam==-1
Collecting apache-beam==-1
 Could not find a version that satisfies the requirement apache-beam==-1 (from 
versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
No matching distribution found for apache-beam==-1{code}
This is a confusing experience for users (customers) to discover that it's 
incompatible only on import after installing the package (and not reading the 
docs) via the error message:
{code:java}
RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
releaselevel='final', serial=0)].{code}
Since it's not compatible anyway, it's confusing that the non-working package 
versions are installable.

I don't think there's an easy way to remove just the Python 3 build from the 
wheels or in the PyPI GUI.  As far as I can tell, this would require rebuilding 
+ re-uploading the Python 2 wheels for affected versions.

  was:
Beam does not currently support Python 3, however, it's possible to install 
_some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
PyPI configuration was fixed at some point after 2.2.0.)

This affects the following 5 versions:
{code:java}
$ pip3 install apache-beam==-1
Collecting apache-beam==-1
 Could not find a version that satisfies the requirement apache-beam==-1 (from 
versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
No matching distribution found for apache-beam==-1{code}
This is a confusing experience for users (customers) to discover that it's 
incompatible only on import after installing the package (and not reading the 
docs) via the error message:
{code:java}
RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
releaselevel='final', serial=0)].{code}
Since it's not compatible anyway, it's confusing that the non-working package 
versions are installable.

I don't think there's an easy way to remove just the Python 3 build from the 
wheels or in the PyPI GUI.  As far as I can tell, this would require rebuilding 
+ re-uploading the Python 2 wheels for affected versions.


> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> config was fixed after 2.2.0 - 
> https://github.com/apache/beam/commit/807b4bcefd877440c12f7fc3b401446e21899904.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88283=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88283
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:57
Start Date: 05/Apr/18 23:57
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179631403
 
 

 ##
 File path: sdks/python/build.gradle
 ##
 @@ -37,7 +38,10 @@ task setupVirtualenv {
   args '-c', ". ${envdir}/bin/activate && pip install --upgrade tox"
 }
   }
-  outputs.files("${envdir}/bin/tox")
+  // Gradle will delete outputs whenever it thinks they are stale. Putting a
 
 Review comment:
   I intended it as documentation about why we're using outputs.dirs vs 
outputs.files.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88283)
Time Spent: 1h  (was: 50m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> 

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88281
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 23:56
Start Date: 05/Apr/18 23:56
Worklog Time Spent: 10m 
  Work Description: udim commented on a change in pull request #5032: 
[BEAM-4021] Work around tox bug.
URL: https://github.com/apache/beam/pull/5032#discussion_r179631299
 
 

 ##
 File path: sdks/python/tox.ini
 ##
 @@ -33,6 +33,16 @@ extras = test
 whitelist_externals =
   find
   time
+deps =
+  grpcio-tools==1.3.5
+  cython: cython==0.26.1
+
+# These 2 magic command overrides are required for Jenkins builds.
+# Otherwise we get "OSError: [Errno 2] No such file or directory" errors.
+# Source:
+# https://github.com/tox-dev/tox/issues/123#issuecomment-284714629
+install_command = {envbindir}/python {envbindir}/pip install {opts} {packages}
 
 Review comment:
   Yes, I tried without it and it fails.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88281)
Time Spent: 50m  (was: 40m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 50m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #12

[Apache Jenkins Server]

See 


--
[...truncated 105.04 MB...]
org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(95f648bff24437939087dadfd073d400) switched from RUNNING to FINISHED.
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(95f648bff24437939087dadfd073d400).
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(95f648bff24437939087dadfd073d400) [FINISHED]
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: Combine.perKey(TestCombineFnWithContext) -> 
PAssert$167/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$167/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (1c07c281c0d981d964b40c16e8ba6844) switched from 
RUNNING to FINISHED.
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for Combine.perKey(TestCombineFnWithContext) 
-> PAssert$167/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$167/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (1c07c281c0d981d964b40c16e8ba6844).
Apr 06, 2018 12:14:27 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task ToKeyedWorkItem (78117b56ac816b7a8e3ab915e746c634)
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
Combine.perKey(TestCombineFnWithContext) -> 
PAssert$167/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$167/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (1c07c281c0d981d964b40c16e8ba6844) [FINISHED]
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (1/1) (b6c98bb27557a32bba36550b54eb4273) switched from 
RUNNING to FINISHED.
Apr 06, 2018 12:14:27 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task PAssert$168/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$168/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) 
(1e82505b65ddccfe5ab27738b60195ee)
Apr 06, 2018 12:14:27 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 

[jira] [Resolved] (BEAM-4022) beam_PreCommit_Python_MavenInstall failing with cython error

[Udi Meiri (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Udi Meiri resolved BEAM-4022.
-
   Resolution: Duplicate
Fix Version/s: Not applicable

> beam_PreCommit_Python_MavenInstall failing with cython error
> 
>
> Key: BEAM-4022
> URL: https://issues.apache.org/jira/browse/BEAM-4022
> Project: Beam
>  Issue Type: Bug
>  Components: testing
>Reporter: Ankur Goenka
>Assignee: Ankur Goenka
>Priority: Major
> Fix For: Not applicable
>
>
> beam_PreCommit_Python_MavenInstall is failing with (link to failing job 
> [https://builds.apache.org/view/A-D/view/ActiveMQ/job/beam_PreCommit_Python_MavenInstall/4425/console]
>  )
>  
> OK (skipped=44)
> py27-gcp runtests: commands[5] | 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/run_tox_cleanup.sh
> py27-cython2 create: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/.tox/py27-cython2
> py27-cython2 installdeps: cython==0.26.1
> ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python
> Traceback (most recent call last):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/bin/tox",
>  line 11, in 
> sys.exit(run_main())
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main
> main(args)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main
> retcode = Session(config).runcommand()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand
> return self.subcommand_test()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test
> if self.setupenv(venv):
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv
> status = venv.update(action=action)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update
> self.hook.tox_testenv_install_deps(action=action, venv=self)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__
> return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec
> return self._inner_hookexec(hook, methods, kwargs)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in 
> firstresult=hook.spec_opts.get('firstresult'),
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall
> return outcome.get_result()
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result
> _reraise(*ex)  # noqa
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/pluggy/callers.py",
>  line 180, in _multicall
> res = hook_impl.function(*args)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 452, in tox_testenv_install_deps
> venv._install(deps, action=action)
>   File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_MavenInstall@2/src/sdks/python/target/python/lib/python2.7/site-packages/tox/venv.py",
>  line 331, in _install
> 

[jira] [Updated] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[Luke Cwik (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Cwik updated BEAM-4021:

Issue Type: Sub-task  (was: Bug)
Parent: BEAM-3249

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 180, in _multicall}}
> {{ res = hook_impl.function(*args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 452, in tox_testenv_install_deps}}
> {{ venv._install(deps, action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 331, in _install}}
> {{ action=action)}}
> {{ File 
> 

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88250=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88250
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:07
Start Date: 05/Apr/18 22:07
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179614424
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Hooks allow runners to tailor execution of the worker to allow for 
customization
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88250)
Time Spent: 5h 40m  (was: 5.5h)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 5h 40m
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3355) Make Go SDK runtime harness hooks pluggable

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3355?focusedWorklogId=88249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88249
 ]

ASF GitHub Bot logged work on BEAM-3355:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:07
Start Date: 05/Apr/18 22:07
Worklog Time Spent: 10m 
  Work Description: wcn3 commented on a change in pull request #4311: 
[BEAM-3355] Diagnostic interfaces
URL: https://github.com/apache/beam/pull/4311#discussion_r179614299
 
 

 ##
 File path: sdks/go/pkg/beam/core/util/hooks/hooks.go
 ##
 @@ -0,0 +1,184 @@
+// Licensed to the Apache Software Foundation (ASF) under one or more
+// contributor license agreements.  See the NOTICE file distributed with
+// this work for additional information regarding copyright ownership.
+// The ASF licenses this file to You under the Apache License, Version 2.0
+// (the "License"); you may not use this file except in compliance with
+// the License.  You may obtain a copy of the License at
+//
+//http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// Hooks allow runners to tailor execution of the worker to allow for 
customization
+// of features used by the harness.
+//
+// Examples of customization:
+//
+// gRPC integration
+// session recording
+// profile recording
+//
+// Registration methods for hooks must be called prior to calling beam.Init()
+// Request methods for hooks must be called as part of building the pipeline
+// request for the runner's Execute method.
+
+package hooks
+
+import (
+   "bytes"
+   "context"
+   "encoding/csv"
+   "encoding/json"
+   "fmt"
+   "os"
+   "strings"
+
+   "github.com/apache/beam/sdks/go/pkg/beam/core/runtime"
+   "github.com/apache/beam/sdks/go/pkg/beam/log"
+   fnpb "github.com/apache/beam/sdks/go/pkg/beam/model/fnexecution_v1"
+)
+
+var (
+   hookRegistry = make(map[string]HookFactory)
+   enabledHooks = make(map[string][]string)
+   activeHooks  = make(map[string]Hook)
+)
+
+// A Hook is a set of hooks to run at various stages of executing a
+// pipelne.
+type Hook struct {
+   // Init is called once at the startup of the worker.
+   Init InitHook
+   // Req is called each time the worker handles a FnAPI instruction 
request.
+   Req RequestHook
+   // Resp is called each time the worker generates a FnAPI instruction 
response.
+   Resp ResponseHook
+}
+
+// InitHook is a hook that is called when the harness
+// initializes.
+type InitHook func(context.Context) error
+
+// HookFactory is a function that produces a Hook from the supplied arguments.
+type HookFactory func([]string) Hook
+
+// RegisterHook registers a Hook for the
+// supplied identifier.
+func RegisterHook(name string, h HookFactory) {
+   hookRegistry[name] = h
+}
+
+// RunInitHooks runs the init hooks.
+func RunInitHooks(ctx context.Context) error {
+   // If an init hook fails to complete, the invariants of the
+   // system are compromised and we can't run a workflow.
+   // The hooks can run in any order. They should not be
+   // interdependent or interfere with each other.
+   for _, h := range activeHooks {
+   if h.Init != nil {
+   if err := h.Init(ctx); err != nil {
+   return err
+   }
+   }
+   }
+   return nil
+}
+
+// RequestHook is called when handling a Fn API instruction.
 
 Review comment:
   Done.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88249)
Time Spent: 5.5h  (was: 5h 20m)

> Make Go SDK runtime harness hooks pluggable
> ---
>
> Key: BEAM-3355
> URL: https://issues.apache.org/jira/browse/BEAM-3355
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-go
>Reporter: Henning Rohde
>Assignee: Bill Neubauer
>Priority: Minor
>  Time Spent: 5.5h
>  Remaining Estimate: 0h
>
> We currently hardcode cpu profiling and session recording in the harness. We 
> should make it pluggable instead.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4024) BundleBasedDirectRunner fails with recent side input changes

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4024?focusedWorklogId=88263=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88263
 ]

ASF GitHub Bot logged work on BEAM-4024:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:52
Start Date: 05/Apr/18 22:52
Worklog Time Spent: 10m 
  Work Description: charlesccychen commented on issue #5035: [BEAM-4024] 
Fix AttributeError in BundleBasedDirectRunner
URL: https://github.com/apache/beam/pull/5035#issuecomment-379099988
 
 
   R: @aaltay 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88263)
Time Spent: 20m  (was: 10m)

> BundleBasedDirectRunner fails with recent side input changes
> 
>
> Key: BEAM-4024
> URL: https://issues.apache.org/jira/browse/BEAM-4024
> Project: Beam
>  Issue Type: Improvement
>  Components: sdk-py-core
>Reporter: Charles Chen
>Assignee: Charles Chen
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Recently, changes were made to support streaming side inputs in the Beam 
> Python DirectRunner ([https://github.com/apache/incubator-beam/pull/4838] and 
> [https://github.com/apache/beam/pull/4949]).  However, these changes cause 
> problems when the BundleBasedDirectRunner is explicitly used.  When Beam 
> Python tests are run with the BundleBasedDirectRunner (i.e. when the 
> FnApiRunner is disabled), the following errors occur:
> ==
>  FAIL: test_empty_write (apache_beam.io.filebasedsink_test.TestFileBasedSink)
>  --
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/io/filebasedsink_test.py",
>  line 159, in test_empty_write
>  p | beam.Create([]) | beam.io.Write(sink) # pylint: 
> disable=expression-not-assigned
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/pipeline.py", 
> line 409, in __exit__
>  self.run().wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/testing/test_pipeline.py",
>  line 104, in run
>  state = result.wait_until_finish()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
>  line 421, in wait_until_finish
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 360, in await_completion
>  self._executor.await_completion()
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 406, in await_completion
>  six.reraise(t, v, tb)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in attempt_call
>  self._completion_callback.handle_result(self, self._input_bundle, result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 238, in handle_result
>  input_committed_bundle, self._timer_firings, transform_result)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 242, in handle_result
>  committed_bundle.get_elements_iterable(make_copy=True))
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/evaluation_context.py",
>  line 97, in add_values
>  assert not view.has_result
>  AssertionError: 
>   >> begin captured logging << 
>  root: INFO: Missing pipeline option (runner). Executing pipeline using the 
> default runner: DirectRunner.
>  root: INFO: Running pipeline with DirectRunner.
>  root: DEBUG: Creating 1 empty shard(s).
>  root: INFO: Starting finalize_write threads with num_shards: 1 (skipped: 0), 
> batches: 1, num_threads: 1
>  root: INFO: Renamed 1 shards in 0.11 seconds.
>  root: ERROR: Exception at bundle 
> , 
> due to an exception.
>  Traceback (most recent call last):
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 307, in call
>  side_input_values)
>  File 
> "/usr/local/google/home/ccy/git/beam/sdks/python/apache_beam/runners/direct/executor.py",
>  line 346, in 

[jira] [Commented] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Taylor Edmiston (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427713#comment-16427713
 ] 

Taylor Edmiston commented on BEAM-4025:
---

Note that 2.1.1 is also an affected version, but the dropdown above says the 
version does not exist.

> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> PyPI configuration was fixed at some point after 2.2.0.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88265=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88265
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 05/Apr/18 22:52
Start Date: 05/Apr/18 22:52
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5032: [BEAM-4021] Work around 
tox bug.
URL: https://github.com/apache/beam/pull/5032#issuecomment-379100014
 
 
   R: @chamikaramj , @lukecwik 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88265)
Time Spent: 20m  (was: 10m)

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 
> 

Build failed in Jenkins: beam_PerformanceTests_HadoopInputFormat #106

[Apache Jenkins Server]

See 


--
[...truncated 40.55 KB...]
Progress (1): 1.7 kBDownloaded from central: 
https://repo.maven.apache.org/maven2/io/netty/netty-codec-http2/maven-metadata.xml
 (1.7 kB at 7.0 kB/s)
Downloading from Apache Snapshots Repository: 
https://repository.apache.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
Downloading from Sonatype Snapshots Repository: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
Progress (1): 1.1 kBDownloaded from Sonatype Snapshots 
Repository: 
https://oss.sonatype.org/content/repositories/snapshots/io/netty/netty-codec-http2/maven-metadata.xml
 (1.1 kB at 4.8 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-core-construction-java/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 2.9 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-core-construction-java/2.5.0-SNAPSHOT/maven-metadata.xml
 (2.9 kB at 18 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-core-construction-java/2.5.0-SNAPSHOT/beam-runners-core-construction-java-2.5.0-20180405.184646-35.pom
Progress (1): 4.1/5.0 kBProgress (1): 5.0 kBDownloaded 
from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-core-construction-java/2.5.0-SNAPSHOT/beam-runners-core-construction-java-2.5.0-20180405.184646-35.pom
 (5.0 kB at 32 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-model-job-management/2.5.0-SNAPSHOT/maven-metadata.xml
Progress (1): 2.9 kBDownloaded from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-model-job-management/2.5.0-SNAPSHOT/maven-metadata.xml
 (2.9 kB at 19 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-model-job-management/2.5.0-SNAPSHOT/beam-model-job-management-2.5.0-20180405.191032-41.pom
Progress (1): 4.1/6.9 kBProgress (1): 6.9 kBDownloaded 
from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-model-job-management/2.5.0-SNAPSHOT/beam-model-job-management-2.5.0-20180405.191032-41.pom
 (6.9 kB at 45 kB/s)
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-core/2.5.0-SNAPSHOT/beam-sdks-java-core-2.5.0-20180405.205225-38.jar
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-hadoop-common/2.5.0-SNAPSHOT/beam-sdks-java-io-hadoop-common-2.5.0-20180405.185457-37.jar
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-runners-direct-java/2.5.0-SNAPSHOT/beam-runners-direct-java-2.5.0-20180405.184717-35.jar
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-io-common/2.5.0-SNAPSHOT/beam-sdks-java-io-common-2.5.0-20180405.185320-37.jar
Downloading from Nexus: 
http://repository.apache.org/snapshots/org/apache/beam/beam-sdks-java-core/2.5.0-SNAPSHOT/beam-sdks-java-core-2.5.0-20180405.205225-38-tests.jar
Progress (1): 0/1.9 MBProgress (1): 0/1.9 MBProgress (1): 0/1.9 MBProgress (1): 
0/1.9 MBProgress (2): 0/1.9 MB | 4.1/8.2 kBProgress (2): 0/1.9 MB | 7.8/8.2 
kBProgress (2): 0/1.9 MB | 8.2 kBProgress (3): 0/1.9 MB | 8.2 kB | 4.1/342 
kBProgress (3): 0/1.9 MB | 8.2 kB | 7.8/342 kBProgress (3): 0/1.9 MB | 8.2 kB | 
12/342 kB Progress (3): 0/1.9 MB | 8.2 kB | 12/342 kBProgress (4): 0/1.9 MB | 
8.2 kB | 12/342 kB | 261 BProgress (5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 
0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress 
(5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 
8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 
kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 
0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress 
(5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 
8.2 kB | 12/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 
kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 12/342 kB | 261 B | 
0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 17/342 kB | 261 B | 0/1.8 MBProgress 
(5): 0/1.9 MB | 8.2 kB | 21/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 
8.2 kB | 25/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 29/342 
kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 33/342 kB | 261 B | 
0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB | 37/342 kB | 261 B | 0/1.8 MBProgress 
(5): 0/1.9 MB | 8.2 kB | 41/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 
8.2 kB | 41/342 kB | 261 B | 0/1.8 MBProgress (5): 0/1.9 MB | 8.2 kB 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #20

[Apache Jenkins Server]

See 


--
[...truncated 2.14 KB...]
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins336791189046647374.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1372198435436453814.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1522958475899
namespace "filebasedioithdfs-1522958475899" created
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins844294445450136491.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1522958475899
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1138386492758494297.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4053993786993707482.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4739951381466183528.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1913846070919237845.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4759637703230287665.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6318024894209514680.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached xmltodict-0.11.0-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #15

[Apache Jenkins Server]

See 


--
[...truncated 119.59 KB...]
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-instance-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-database-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding 
com.google.api.grpc:grpc-google-cloud-spanner-admin-instance-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-longrunning-v1:jar:0.1.11 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-longrunning-v1:jar:0.1.11 
from the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-protos:jar:1.0.0-pre3 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigtable:bigtable-client-core:jar:1.0.0 from 
the shaded jar.
[INFO] Excluding commons-logging:commons-logging:jar:1.2 from the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-appengine:jar:0.7.0 from 
the shaded jar.
[INFO] Excluding io.opencensus:opencensus-contrib-grpc-util:jar:0.7.0 from the 
shaded jar.
[INFO] Excluding io.opencensus:opencensus-api:jar:0.7.0 from the shaded jar.
[INFO] Excluding io.dropwizard.metrics:metrics-core:jar:3.1.2 from the shaded 
jar.
[INFO] Excluding 
com.google.api.grpc:proto-google-cloud-spanner-admin-database-v1:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding io.netty:netty-tcnative-boringssl-static:jar:1.1.33.Fork26 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-runners-core-construction-java:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding org.apache.beam:beam-model-job-management:jar:2.5.0-SNAPSHOT 
from the shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client:jar:1.22.0 from the 
shaded jar.
[INFO] Excluding org.apache.httpcomponents:httpclient:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding org.apache.httpcomponents:httpcore:jar:4.0.1 from the shaded 
jar.
[INFO] Excluding commons-codec:commons-codec:jar:1.3 from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-dataflow:jar:v1b3-rev221-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-clouddebugger:jar:v2-rev8-1.22.0 from the 
shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding 

Build failed in Jenkins: beam_PerformanceTests_JDBC #417

[Apache Jenkins Server]

See 


--
[...truncated 65 B...]
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 446a3752718dc3353832e64a38f25ee1803e75da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 446a3752718dc3353832e64a38f25ee1803e75da
Commit message: "Merge pull request #5033 from alanmyrvold/alan-delete-windows"
 > git rev-list --no-walk 446a3752718dc3353832e64a38f25ee1803e75da # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8791244105841228091.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6138766031662065037.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins920978106816378791.sh
+ kubectl 
--kubeconfig=
 create namespace jdbcioit-1522958492701
namespace "jdbcioit-1522958492701" created
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6958160223478163341.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=jdbcioit-1522958492701
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins5305607197346983162.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins1264820287778018323.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins4620711960928203747.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8351066500291904206.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6373799654841773620.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6624640635554282215.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #13

[Apache Jenkins Server]

See 


--
Started by timer
[EnvInject] - Loading node environment variables.
Building remotely on beam1 (beam) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 446a3752718dc3353832e64a38f25ee1803e75da (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 446a3752718dc3353832e64a38f25ee1803e75da
Commit message: "Merge pull request #5033 from alanmyrvold/alan-delete-windows"
 > git rev-list --no-walk 446a3752718dc3353832e64a38f25ee1803e75da # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4344626249108790369.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8178912743124294376.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8578198137246839382.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1522958475932
namespace "filebasedioithdfs-1522958475932" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1439931454846903619.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1522958475932
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5358322854318159789.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins531319458957143180.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2004934499781220866.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4736417345313863496.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8098491362112615560.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins107453402103077405.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 

Jenkins build is back to normal : beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #13

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-4011) Python SDK: add glob support for HDFS

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4011?focusedWorklogId=88292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88292
 ]

ASF GitHub Bot logged work on BEAM-4011:


Author: ASF GitHub Bot
Created on: 06/Apr/18 00:54
Start Date: 06/Apr/18 00:54
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #5024: [BEAM-4011] Unify Python 
IO glob implementation.
URL: https://github.com/apache/beam/pull/5024#issuecomment-379118600
 
 
   run python postcommit


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88292)
Time Spent: 0.5h  (was: 20m)

> Python SDK: add glob support for HDFS
> -
>
> Key: BEAM-4011
> URL: https://issues.apache.org/jira/browse/BEAM-4011
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4014) Migrate MavenInstall Jenkins PostCommits to Gradle

[Thomas Groh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427807#comment-16427807
 ] 

Thomas Groh commented on BEAM-4014:
---

There are a couple of difficulties here - the primary one is that with the 
existing Maven test, we execute all of the dependencies of the Core SDK and 
execute all the integration tests that are part of the classpath. I haven't 
found an obvious way from within Gradle to, within a top-level module, 
aggregate all of the produced artifacts (matching against a string, ideally, 
like {{"**/*IT.java"}}, and run a single execution against them.

 

My belief is that this likely requires each module that declares integration 
tests to declare an "integrationTest" task, which executes its integration test 
on some subset of runners where such a test is appropriate (beginning with only 
the Dataflow Runner), and have the top-level {{postCommit}} task depend on all 
of them. The actual jenkins task is then executing that postCommit, as is done 
in the gradle precommits.

 

The actual name {{MavenInstall}} is also generically misleading - this is an 
execution of the Java ITs on the Dataflow Runner, not a validation that 
{{MavenInstall}} produces the proper artifacts.

> Migrate MavenInstall Jenkins PostCommits to Gradle
> --
>
> Key: BEAM-4014
> URL: https://issues.apache.org/jira/browse/BEAM-4014
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Henning Rohde
>Assignee: Thomas Groh
>Priority: Major
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[Udi Meiri (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427792#comment-16427792
 ] 

Udi Meiri commented on BEAM-4021:
-

Hopefully this is resolved.

> "No such file or directory" in beam_PreCommit_Python_GradleBuild
> 
>
> Key: BEAM-4021
> URL: https://issues.apache.org/jira/browse/BEAM-4021
> Project: Beam
>  Issue Type: Sub-task
>  Components: testing
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>  Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> Seems to only happen in this working directory:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild}}
> but not this:
> {{/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild@2}}
> {{ERROR: invocation failed (errno 2), args: 
> ['/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/target/.tox/py27-cython2/bin/pip',
>  'install', 'cython==0.26.1'], cwd: 
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python}}
> {{ Traceback (most recent call last):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/bin/tox",
>  line 11, in }}
> {{ sys.exit(run_main())}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 40, in run_main}}
> {{ main(args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 46, in main}}
> {{ retcode = Session(config).runcommand()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 415, in runcommand}}
> {{ return self.subcommand_test()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 599, in subcommand_test}}
> {{ if self.setupenv(venv):}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/session.py",
>  line 491, in setupenv}}
> {{ status = venv.update(action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 171, in update}}
> {{ self.hook.tox_testenv_install_deps(action=action, venv=self)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 617, in __call__}}
> {{ return self._hookexec(self, self._nonwrappers + self._wrappers, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 222, in _hookexec}}
> {{ return self._inner_hookexec(hook, methods, kwargs)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/__init__.py",
>  line 216, in }}
> {{ firstresult=hook.spec_opts.get('firstresult'),}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 201, in _multicall}}
> {{ return outcome.get_result()}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 77, in get_result}}
> {{ _reraise(*ex) # noqa}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/pluggy/callers.py",
>  line 180, in _multicall}}
> {{ res = hook_impl.function(*args)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 452, in tox_testenv_install_deps}}
> {{ venv._install(deps, action=action)}}
> {{ File 
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_GradleBuild/src/sdks/python/build/gradleenv/local/lib/python2.7/site-packages/tox/venv.py",
>  line 331, in _install}}
> {{ action=action)}}
> {{ File 
> 

[jira] [Work logged] (BEAM-4021) "No such file or directory" in beam_PreCommit_Python_GradleBuild

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-4021?focusedWorklogId=88289=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88289
 ]

ASF GitHub Bot logged work on BEAM-4021:


Author: ASF GitHub Bot
Created on: 06/Apr/18 00:19
Start Date: 06/Apr/18 00:19
Worklog Time Spent: 10m 
  Work Description: aaltay closed pull request #5032: [BEAM-4021] Work 
around tox bug.
URL: https://github.com/apache/beam/pull/5032
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git a/sdks/python/build.gradle b/sdks/python/build.gradle
index be17644e8d3..6c0f7f8e45a 100644
--- a/sdks/python/build.gradle
+++ b/sdks/python/build.gradle
@@ -26,6 +26,7 @@ task test {}
 check.dependsOn test
 
 def envdir = "${project.buildDir}/gradleenv"
+def tox_opts = "-c tox.ini --recreate"
 
 task setupVirtualenv {
   doLast {
@@ -37,7 +38,10 @@ task setupVirtualenv {
   args '-c', ". ${envdir}/bin/activate && pip install --upgrade tox"
 }
   }
-  outputs.files("${envdir}/bin/tox")
+  // Gradle will delete outputs whenever it thinks they are stale. Putting a
+  // specific binary here could make gradle delete it while pip will believe
+  // the package is fully installed.
+  outputs.dirs(envdir)
 }
 
 task sdist(dependsOn: 'setupVirtualenv') {
@@ -77,7 +81,7 @@ task lintPy27(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e py27-lint -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py27-lint"
 }
   }
 }
@@ -87,7 +91,7 @@ task lintPy3(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e py3-lint -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py3-lint"
 }
   }
 }
@@ -97,7 +101,7 @@ task testGcp(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e py27-gcp -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py27-gcp"
 }
   }
 }
@@ -107,7 +111,7 @@ task testPython(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e py27 -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py27"
 }
   }
 }
@@ -117,7 +121,7 @@ task testCython(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e py27-cython2 -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e py27-cython"
 }
   }
 }
@@ -131,7 +135,7 @@ task docs(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e docs -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e docs"
 }
   }
 }
@@ -141,7 +145,7 @@ task cover(dependsOn: 'setupVirtualenv') {
   doLast {
 exec {
   executable 'sh'
-  args '-c', ". ${envdir}/bin/activate && tox -e cover -c tox.ini"
+  args '-c', ". ${envdir}/bin/activate && tox ${tox_opts} -e cover"
 }
   }
 }
diff --git a/sdks/python/tox.ini b/sdks/python/tox.ini
index add9bbfb2cc..ff88ac42fa8 100644
--- a/sdks/python/tox.ini
+++ b/sdks/python/tox.ini
@@ -17,7 +17,7 @@
 
 [tox]
 # new environments will be excluded by default unless explicitly added to 
envlist.
-envlist = py27,py27-{gcp,cython2,lint},py3-lint,docs
+envlist = py27,py27-{gcp,cython,lint},py3-lint,docs
 toxworkdir = {toxinidir}/target/.tox
 
 [pycodestyle]
@@ -33,6 +33,16 @@ extras = test
 whitelist_externals =
   find
   time
+deps =
+  grpcio-tools==1.3.5
+  cython: cython==0.26.1
+
+# These 2 magic command overrides are required for Jenkins builds.
+# Otherwise we get "OSError: [Errno 2] No such file or directory" errors.
+# Source:
+# https://github.com/tox-dev/tox/issues/123#issuecomment-284714629
+install_command = {envbindir}/python {envbindir}/pip install {opts} {packages}
+list_dependencies_command = {envbindir}/python {envbindir}/pip freeze
 
 [testenv:py27]
 commands =
@@ -43,15 +53,12 @@ commands =
   python setup.py test
   {toxinidir}/run_tox_cleanup.sh
 
-# This environment will fail in Jenkins if named "py27-cython".
-[testenv:py27-cython2]
+[testenv:py27-cython]
 # cython tests are only expected to work in linux (2.x and 3.x)
 # If we want to add other platforms in the future, it should be:
 # `platform = linux2|darwin|...`
 # See https://docs.python.org/2/library/sys.html#sys.platform for platform 
codes
 platform = linux2
-deps =
-  cython==0.26.1
 commands =
   python --version
  

Build failed in Jenkins: beam_PerformanceTests_Spark #1555

[Apache Jenkins Server]

See 


--
[...truncated 94.50 KB...]
'apache-beam-testing:bqjob_r2402881fa0f2ef40_01629852a3c7_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-06 00:19:37,298 6032a78c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 00:20:03,672 6032a78c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 00:20:05,871 6032a78c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r1d83c4ddbb82c50d_0162985312e9_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r1d83c4ddbb82c50d_0162985312e9_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r1d83c4ddbb82c50d_0162985312e9_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-06 00:20:05,871 6032a78c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 00:20:26,730 6032a78c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 00:20:28,786 6032a78c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r5ad530e8421231c4_016298536d05_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r5ad530e8421231c4_016298536d05_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r5ad530e8421231c4_016298536d05_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 
/usr/lib/google-cloud-sdk/platform/bq/third_party/oauth2client/contrib/gce.py:73:
 UserWarning: You have requested explicit scopes to be used with a GCE service 
account.
Using this argument will have no effect on the actual scopes for tokens
requested. These scopes are set at VM instance creation time and
can't be overridden in the request.

  warnings.warn(_SCOPES_WARNING)

2018-04-06 00:20:28,787 6032a78c MainThread INFO Retrying exception running 
IssueRetryableCommand: Command returned a non-zero exit code.

2018-04-06 00:20:56,648 6032a78c MainThread INFO Running: bq load 
--autodetect --source_format=NEWLINE_DELIMITED_JSON 
beam_performance.pkb_results 

2018-04-06 00:20:58,856 6032a78c MainThread INFO Ran: {bq load --autodetect 
--source_format=NEWLINE_DELIMITED_JSON beam_performance.pkb_results 

  ReturnCode:1
STDOUT: Upload complete.
Waiting on bqjob_r128e942b6141789d_01629853e1f0_1 ... (0s) Current status: 
RUNNING 
 Waiting on bqjob_r128e942b6141789d_01629853e1f0_1 ... (0s) 
Current status: DONE   
BigQuery error in load operation: Error processing job
'apache-beam-testing:bqjob_r128e942b6141789d_01629853e1f0_1': Invalid schema
update. Field timestamp has changed type from TIMESTAMP to FLOAT

STDERR: 

[jira] [Commented] (BEAM-4025) Non-working Python 3 versions are installable on PyPI

[Taylor Edmiston (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-4025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16427803#comment-16427803
 ] 

Taylor Edmiston commented on BEAM-4025:
---

That sounds great to me.  Thanks for the quick reply.  We will share the same 
with customers.

> Non-working Python 3 versions are installable on PyPI
> -
>
> Key: BEAM-4025
> URL: https://issues.apache.org/jira/browse/BEAM-4025
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-py-core
>Affects Versions: 0.6.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Taylor Edmiston
>Assignee: Ahmet Altay
>Priority: Major
>
> Beam does not currently support Python 3, however, it's possible to install 
> _some versions_ from PyPI / Warehouse under Python 3 today.  (It appears the 
> config was fixed after 2.2.0 - 
> https://github.com/apache/beam/commit/807b4bcefd877440c12f7fc3b401446e21899904.)
> This affects the following 5 versions:
> {code:java}
> $ pip3 install apache-beam==-1
> Collecting apache-beam==-1
>  Could not find a version that satisfies the requirement apache-beam==-1 
> (from versions: 0.6.0, 2.0.0, 2.1.0, 2.1.1, 2.2.0)
> No matching distribution found for apache-beam==-1{code}
> This is a confusing experience for users (customers) to discover that it's 
> incompatible only on import after installing the package (and not reading the 
> docs) via the error message:
> {code:java}
> RuntimeError: The Apache Beam SDK for Python is supported only on Python 2.7. 
> It is not supported on Python [sys.version_info(major=3, minor=6, micro=4, 
> releaselevel='final', serial=0)].{code}
> Since it's not compatible anyway, it's confusing that the non-working package 
> versions are installable.
> I don't think there's an easy way to remove just the Python 3 build from the 
> wheels or in the PyPI GUI.  As far as I can tell, this would require 
> rebuilding + re-uploading the Python 2 wheels for affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88299=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88299
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 01:44
Start Date: 06/Apr/18 01:44
Worklog Time Spent: 10m 
  Work Description: udim commented on issue #412: [BEAM-3985] Document 
Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412#issuecomment-379125265
 
 
   R: @melap 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88299)
Time Spent: 20m  (was: 10m)

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Python_Verify #4602

[Apache Jenkins Server]

See 


--
[...truncated 5.31 MB...]
component_coder_ids: "ref_Coder_TupleCoder_10"
component_coder_ids: "ref_Coder_GlobalWindowCoder_6"
  }
}
coders {
  key: "ref_Coder_WindowedValueCoder_9_bytes"
  value {
spec {
  spec {
urn: "beam:coder:windowed_value:v1"
  }
}
component_coder_ids: "ref_Coder_TupleCoder_10_bytes"
component_coder_ids: "ref_Coder_GlobalWindowCoder_6"
  }
}
coders {
  key: "ref_Coder_WindowedValueCoder_9_len_prefix"
  value {
spec {
  spec {
urn: "beam:coder:windowed_value:v1"
  }
}
component_coder_ids: "ref_Coder_TupleCoder_10_len_prefix"
component_coder_ids: "ref_Coder_GlobalWindowCoder_6"
  }
}
  }
}

root: DEBUG: CONTROL RESPONSE instruction_id: "control_2"
register {
}

root: DEBUG: CONTROL REQUEST instruction_id: "bundle_538"
process_bundle {
  process_bundle_descriptor_reference: "1"
}

root: INFO: start 
root: INFO: start 
root: INFO: start 
root: INFO: start 
root: INFO: start , 
start_position=None, stop_position=None)>
- >> end captured logging << -

==
ERROR: test_file_pattern_verify_details 
(apache_beam.io.vcfio_test.VcfSourceTest)
--
Traceback (most recent call last):
  File 
"
 line 317, in test_file_pattern_verify_details
read_data = self._read_records(os.path.join(tempdir.get_path(), '*.vcf'))
  File 
"
 line 102, in _read_records
VcfSource(file_or_pattern, **kwargs))
  File 
"
 line 244, in __init__
validate=validate)
  File 
"
 line 119, in __init__
self._validate()
  File 
"
 line 124, in _f
return fnc(self, *args, **kwargs)
  File 
"
 line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
  File 
"
 line 164, in match
return filesystem.match(patterns, limits)
  File 
"
 line 590, in match
raise BeamIOError("Match operation failed", exceptions)
BeamIOError: Match operation failed with exceptions {'/tmp/tmp5_aFqg/*.vcf': 
ValueError('Could not parse URL: %s', '/tmp/tmp5_aFqg/')}

==
ERROR: test_invalid_file (apache_beam.io.vcfio_test.VcfSourceTest)
--
Traceback (most recent call last):
  File 
"
 line 345, in test_invalid_file
self._read_records(os.path.join(tempdir.get_path(), '*.vcf'))
  File 
"
 line 102, in _read_records
VcfSource(file_or_pattern, **kwargs))
  File 
"
 line 244, in __init__
validate=validate)
  File 
"
 line 119, in __init__
self._validate()
  File 
"
 line 124, in _f
return fnc(self, *args, **kwargs)
  File 
"
 line 176, in _validate
match_result = FileSystems.match([pattern], limits=[1])[0]
  File 
"
 line 164, in match
return filesystem.match(patterns, limits)
  File 
"
 line 590, 

[jira] [Work logged] (BEAM-3985) Update developer guide to reference new Python linting enviroments

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3985?focusedWorklogId=88298=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-88298
 ]

ASF GitHub Bot logged work on BEAM-3985:


Author: ASF GitHub Bot
Created on: 06/Apr/18 01:43
Start Date: 06/Apr/18 01:43
Worklog Time Spent: 10m 
  Work Description: udim opened a new pull request #412: [BEAM-3985] 
Document Gradle commands for testing and linting.
URL: https://github.com/apache/beam-site/pull/412
 
 
   Also fixes outdated references to lint_py2 and lint_py3.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 88298)
Time Spent: 10m
Remaining Estimate: 0h

> Update developer guide to reference new Python linting enviroments
> --
>
> Key: BEAM-3985
> URL: https://issues.apache.org/jira/browse/BEAM-3985
> Project: Beam
>  Issue Type: Task
>  Components: sdk-py-core, website
>Reporter: holdenk
>Assignee: Udi Meiri
>Priority: Minor
>  Time Spent: 10m
>  Remaining Estimate: 0h
>
> tox.ini changed, but [https://beam.apache.org/contribute/contribution-guide/] 
> still references the told envs.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Jenkins build became unstable: beam_PostCommit_Java_ValidatesRunner_Dataflow #5289

[Apache Jenkins Server]

See 

Jenkins build is back to stable : beam_PostCommit_Java_ValidatesRunner_Dataflow #5290

[Apache Jenkins Server]

See 

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #7

[Apache Jenkins Server]

See 


--
[...truncated 97.44 MB...]
org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/05/2018 06:16:11 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Combine.perKey(Singleton)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/Combine.globally(Singleton)/Values/Values/Map/ParMultiDo(Anonymous)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 05, 2018 6:16:11 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Combine.perKey(Concatenate)
 -> 
View.AsSingleton/Combine.GloballyAsSingletonView/View.CreatePCollectionView/Combine.globally(Concatenate)/Values/Values/Map/ParMultiDo(Anonymous)
 -> (Map, Map) (f667b777e6769655c98ae977a99ee82a)
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: PAssert$168/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$168/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(ab99175be30f289dd80eb2ab7dfad4ab) switched from RUNNING to FINISHED.
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
PAssert$168/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$168/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(ab99175be30f289dd80eb2ab7dfad4ab).
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
PAssert$168/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$168/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(ab99175be30f289dd80eb2ab7dfad4ab) [FINISHED]
Apr 05, 2018 6:16:11 AM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 
Combine.globally(TestCombineFnWithContext)/Combine.perKey(TestCombineFnWithContext)
 -> 
Combine.globally(TestCombineFnWithContext)/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/Window.Into()/Window.Assign.out -> 
PAssert$168/GroupGlobally/GatherAllOutputs/Reify.Window/ParDo(Anonymous)/ParMultiDo(Anonymous)
 -> 
PAssert$168/GroupGlobally/GatherAllOutputs/WithKeys/AddKeys/Map/ParMultiDo(Anonymous)
 -> PAssert$168/GroupGlobally/GatherAllOutputs/Window.Into()/Window.Assign.out 
-> ToKeyedWorkItem (53b2be6ec6f062d593a34f99fc0c18a0)
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: ToKeyedWorkItem (1/1) (8217074cb7609fa010d69116aede5e69) switched 
from RUNNING to FINISHED.
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for ToKeyedWorkItem (1/1) 
(8217074cb7609fa010d69116aede5e69).
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task ToKeyedWorkItem 
(1/1) (8217074cb7609fa010d69116aede5e69) [FINISHED]
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: PAssert$166/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$166/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$166/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$166/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(0736f618cd7abbf0e660b6da3e7037dd) switched from RUNNING to FINISHED.
Apr 05, 2018 6:16:11 AM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/05/2018 06:16:11   
PAssert$166/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$166/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$166/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$166/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous)(1/1) 

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=87921=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87921
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 05/Apr/18 07:13
Start Date: 05/Apr/18 07:13
Worklog Time Spent: 10m 
  Work Description: herohde opened a new pull request #5030: [BEAM-3250] 
Migrate Spark ValidatesRunner tests to Gradle
URL: https://github.com/apache/beam/pull/5030
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 87921)
Time Spent: 2h 40m  (was: 2.5h)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work logged] (BEAM-2852) Add support for Kafka as source/sink on Nexmark

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2852?focusedWorklogId=87965=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87965
 ]

ASF GitHub Bot logged work on BEAM-2852:


Author: ASF GitHub Bot
Created on: 05/Apr/18 09:41
Start Date: 05/Apr/18 09:41
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5019: 
[BEAM-2852] Add support for Kafka as source/sink on Nexmark
URL: https://github.com/apache/beam/pull/5019#discussion_r179405497
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
 ##
 @@ -764,6 +774,68 @@ public void processElement(ProcessContext c) {
   }));
   }
 
+  static final DoFn EVENT_TO_BYTEARRAY =
+  new DoFn() {
+@ProcessElement
+public void processElement(ProcessContext c) {
+  try {
+byte[] encodedEvent = 
CoderUtils.encodeToByteArray(Event.CODER, c.element());
+c.output(encodedEvent);
+  } catch (CoderException e1) {
+LOG.error("Error while sending Event {} to Kafka: 
serialization error",
+c.element().toString());
+  }
+}
+  };
+
+  /**
+   * Send {@code events} to Kafka.
+   */
+  private void sinkEventsToKafka(PCollection events) {
 
 Review comment:
   what is important in the connection with MOM: 
   - have the ability to keep a track of the generated events that lead to a 
benchmark result
   - be able to read events from a topic
   - write benchmark results to topic


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 87965)
Time Spent: 2h 50m  (was: 2h 40m)

> Add support for Kafka as source/sink on Nexmark
> ---
>
> Key: BEAM-2852
> URL: https://issues.apache.org/jira/browse/BEAM-2852
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ismaël Mejía
>Assignee: Kai Jiang
>Priority: Minor
>  Labels: newbie, nexmark, starter
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (BEAM-3726) Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move backwards

[Alexey Romanenko (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Alexey Romanenko resolved BEAM-3726.

   Resolution: Fixed
Fix Version/s: 2.5.0

Fixed by [BEAM-3087|https://issues.apache.org/jira/browse/BEAM-3087]


> Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move 
> backwards
> 
>
> Key: BEAM-3726
> URL: https://issues.apache.org/jira/browse/BEAM-3726
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.5.0
>
> Attachments: KinesisIO-state.txt
>
>
> When the job is restored from savepoint Kinesis Reader throws almost always 
> {{java.lang.IllegalArgumentException: Attempting to move backwards}}
> After a few job restarts caused again by the same exception, job finally 
> starts up and continues to run with no further problems.
> Beam job is reading from 32 shards with parallelism set to 32. Using Flink 
> 1.3.2. But I have seen this exception also when using Beam 2.2 when Kinesis 
> client was refactored to use MovingFunction. I think this is a serious 
> regression bug introduced in Beam 2.2. 
>  
> {code:java}
> java.lang.IllegalArgumentException: Attempting to move backwards
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
> at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
> at 
> org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
> at 
> org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
> at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> at 
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
>  
> Kinesis Reader transformation configuration:
> {code:java}
> pipeline.apply("KINESIS READER", KinesisIO.read()
> .withStreamName(streamName)
> .withInitialPositionInStream(InitialPositionInStream.LATEST)
> .withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
>  
> When testing locally I managed to catch this exception. Just before executing 
> this 
> [link|https://github.com/apache/beam/blob/6c93105c2cb7be709c6b3e2e6cdcd09df2b48308/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java#L97]
>  that threw exception I captured the state of the class so that you can 
> replicate the issue
> {code:java}
> org.apache.beam.sdk.util.MovingFunction@71781a[sampleUpdateMs=5000,numSignificantBuckets=2,numSignificantSamples=10,function=org.apache.beam.sdk.transforms.Min$MinLongFn@7909d8d3,buckets={9223372036854775807,9223372036854775807,1519315344334,1519315343759,1519315343770,1519315344086,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807},numSamples={0,0,1,158,156,146,0,0,0,0,144,0},currentMsSinceEpoch=1519315585000,currentIndex=2]{code}
>  
> the add function of MovingFunction was called with nowMsSinceEpoch = 
> 1519315583591
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (BEAM-4015) GCS artifact proxy is breaking the build when credentials are not available

[JIRA]

Ismaël Mejía created BEAM-4015:
--

 Summary: GCS artifact proxy is breaking the build when credentials 
are not available
 Key: BEAM-4015
 URL: https://issues.apache.org/jira/browse/BEAM-4015
 Project: Beam
  Issue Type: Bug
  Components: sdk-go
Reporter: Ismaël Mejía
Assignee: Henning Rohde


When running the maven build in a machine without the valid auth credentials 
the module breaks like this:

{{{code:bash}}}

{{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
beam-runners-gcp-gcsproxy ---}}
{{[INFO] Prepared command line : bin/go build -buildmode=default -o 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
github.com/apache/beam/cmd/gcsproxy}}
{{[ERROR] }}
{{[ERROR] -Exec.Err-}}
{{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
{{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication}}
{{[ERROR] }}
{{}}

{{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy 
FAILURE [  1.038 s]}}
{{}}
{{[INFO] BUILD FAILURE}}
{{}}
{{[ERROR] Failed to execute goal 
com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
beam-runners-gcp-gcsproxy: Can't find generated target file : 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
{{[ERROR] }}

{{{code}}}{{}}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3954) Get Jenkins agents dockerized

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426674#comment-16426674
 ] 

Ismaël Mejía edited comment on BEAM-3954 at 4/5/18 9:40 AM:


Ok so it seems we are aligned then [~alanmyrvold]. i know docker well so I can 
help there, however I am a jenkins newbie, but I can try to help there also if 
needed.

[~yifanzou] The image adds whatever you pass as a parameter at build time and 
the default is github's  master, but you can pass a PR, a branch, etc for 
details take a look at the 
[documentation|[https://beam.apache.org/contribute/docker-images/].]

How do you guys plan to tackle the problem of downloading all the deps on each 
build ? I did this via volume mounting but still not convinced is the best way, 
if no the build takes ages. We have to also pay attention at how to tune the 
container execution, see [java in 
containers|https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/].

 


was (Author: iemejia):
Ok so it seems we are aligned then [~alanmyrvold]. i know docker well so I can 
help there, however I am a jenkins newbie, but I can try to help there also if 
needed.

[~yifanzou] The image adds whatever you pass as a parameter at build time and 
the default is github's  master, but you can pass a PR, a branch, etc for 
details take a look at the 
[documentation|[https://beam.apache.org/contribute/docker-images/].]

How do you guys plan to tackle the problem of downloading all the deps on each 
build ? I did this via volume mounting but still not convinced is the best way, 
if no the build takes ages. We have to also pay attention at how we run [java 
in 
containers|https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/].

 

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3726) Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move backwards

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426686#comment-16426686
 ] 

Alexey Romanenko edited comment on BEAM-3726 at 4/5/18 9:53 AM:


Fixed by BEAM-3087



was (Author: aromanenko):
Fixed by [BEAM-3087|https://issues.apache.org/jira/browse/BEAM-3087]


> Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move 
> backwards
> 
>
> Key: BEAM-3726
> URL: https://issues.apache.org/jira/browse/BEAM-3726
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.5.0
>
> Attachments: KinesisIO-state.txt
>
>
> When the job is restored from savepoint Kinesis Reader throws almost always 
> {{java.lang.IllegalArgumentException: Attempting to move backwards}}
> After a few job restarts caused again by the same exception, job finally 
> starts up and continues to run with no further problems.
> Beam job is reading from 32 shards with parallelism set to 32. Using Flink 
> 1.3.2. But I have seen this exception also when using Beam 2.2 when Kinesis 
> client was refactored to use MovingFunction. I think this is a serious 
> regression bug introduced in Beam 2.2. 
>  
> {code:java}
> java.lang.IllegalArgumentException: Attempting to move backwards
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
> at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
> at 
> org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
> at 
> org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
> at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> at 
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
>  
> Kinesis Reader transformation configuration:
> {code:java}
> pipeline.apply("KINESIS READER", KinesisIO.read()
> .withStreamName(streamName)
> .withInitialPositionInStream(InitialPositionInStream.LATEST)
> .withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
>  
> When testing locally I managed to catch this exception. Just before executing 
> this 
> [link|https://github.com/apache/beam/blob/6c93105c2cb7be709c6b3e2e6cdcd09df2b48308/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java#L97]
>  that threw exception I captured the state of the class so that you can 
> replicate the issue
> {code:java}
> org.apache.beam.sdk.util.MovingFunction@71781a[sampleUpdateMs=5000,numSignificantBuckets=2,numSignificantSamples=10,function=org.apache.beam.sdk.transforms.Min$MinLongFn@7909d8d3,buckets={9223372036854775807,9223372036854775807,1519315344334,1519315343759,1519315343770,1519315344086,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807},numSamples={0,0,1,158,156,146,0,0,0,0,144,0},currentMsSinceEpoch=1519315585000,currentIndex=2]{code}
>  
> the add function of MovingFunction was called with nowMsSinceEpoch = 
> 1519315583591
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3946) Python SDK tests are failing if no GOOGLE_APPLICATION_CREDENTIALS was set

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426717#comment-16426717
 ] 

Alexey Romanenko commented on BEAM-3946:


Thank you!

> Python SDK tests are failing if no GOOGLE_APPLICATION_CREDENTIALS was set
> -
>
> Key: BEAM-3946
> URL: https://issues.apache.org/jira/browse/BEAM-3946
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Alexey Romanenko
>Assignee: Mark Liu
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Running locally mvn clean install fails on following Apache Beam :: SDKs :: 
> Python tests:
> {{ERROR: test_message_matcher_mismatch 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  {{ERROR: test_message_matcher_success 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  {{ERROR: test_message_metcher_timeout 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  
> with an error:
> DefaultCredentialsError: Could not automatically determine credentials. 
> Please set GOOGLE_APPLICATION_CREDENTIALS or
>  explicitly create credential and re-run the application. For more
>  information, please see
>  
> [https://developers.google.com/accounts/docs/application-default-credentials].
>   >> begin captured logging << 
>  google.auth.transport._http_client: DEBUG: Making request: GET 
> [http://169.254.169.254|http://169.254.169.254/]
>  google.auth.compute_engine._metadata: INFO: Compute Engine Metadata server 
> unavailable.
>  - >> end captured logging << -
>  
> It looks like it's a regression and it was caused by this commit: 
> [301853647f2c726c04c5bdb02cab6ff6b39f09d0|https://github.com/apache/beam/commit/301853647f2c726c04c5bdb02cab6ff6b39f09d0]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PostCommit_Java_ValidatesRunner_Flink_Gradle #8

[Apache Jenkins Server]

See 


--
[...truncated 78.33 MB...]
Apr 05, 2018 12:13:31 PM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task PAssert$166/GroupGlobally/GroupDummyAndContents -> 
PAssert$166/GroupGlobally/Values/Values/Map/ParMultiDo(Anonymous) -> 
PAssert$166/GroupGlobally/ParDo(Concat)/ParMultiDo(Concat) -> 
PAssert$166/GetPane/Map/ParMultiDo(Anonymous) -> 
PAssert$166/RunChecks/ParMultiDo(GroupedValuesChecker) -> 
PAssert$166/VerifyAssertions/ParDo(DefaultConclude)/ParMultiDo(DefaultConclude) 
(331070c9286c306ede48770518b83476)
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: ToKeyedWorkItem (1/1) (fb078e9a74e02f3b6e5fd9555f9c9939) switched 
from RUNNING to FINISHED.
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for ToKeyedWorkItem (1/1) 
(fb078e9a74e02f3b6e5fd9555f9c9939).
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task ToKeyedWorkItem 
(1/1) (fb078e9a74e02f3b6e5fd9555f9c9939) [FINISHED]
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task 
transitionState
INFO: PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(87f31c6f08efe6bf3c704c6a561562db) switched from RUNNING to FINISHED.
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: PAssert$166/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$166/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$166/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$166/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(059d8449eee092a21afeb0776b365644) switched from RUNNING to FINISHED.
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Freeing task resources for 
PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(87f31c6f08efe6bf3c704c6a561562db).
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.taskmanager.Task run
INFO: Ensuring all FileSystem streams are closed for task 
PAssert$167/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$167/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$167/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$167/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous) (1/1) 
(87f31c6f08efe6bf3c704c6a561562db) [FINISHED]
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/05/2018 12:13:31   
PAssert$166/GroupGlobally/GatherAllOutputs/GroupByKey -> 
PAssert$166/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$166/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$166/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous)(1/1) 
switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/05/2018 12:13:31 PAssert$166/GroupGlobally/GatherAllOutputs/GroupByKey 
-> 
PAssert$166/GroupGlobally/GatherAllOutputs/Values/Values/Map/ParMultiDo(Anonymous)
 -> PAssert$166/GroupGlobally/RewindowActuals/Window.Assign.out -> 
PAssert$166/GroupGlobally/KeyForDummy/AddKeys/Map/ParMultiDo(Anonymous)(1/1) 
switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.executiongraph.Execution 
transitionState
INFO: ToKeyedWorkItem (1/1) (9a96dc6943f3f0a1bd4e6bfc68150307) switched 
from RUNNING to FINISHED.
Apr 05, 2018 12:13:31 PM org.apache.flink.runtime.client.JobClientActor 
logAndPrintMessage
INFO: 04/05/2018 12:13:31   ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_OUT
04/05/2018 12:13:31 ToKeyedWorkItem(1/1) switched to FINISHED 

org.apache.beam.sdk.transforms.CombineTest > testSimpleCombineWithContextEmpty 
STANDARD_ERROR
Apr 05, 2018 12:13:31 PM grizzled.slf4j.Logger info
INFO: Un-registering task and sending final execution state FINISHED to 
JobManager for task 

Jenkins build is back to normal : beam_PerformanceTests_HadoopInputFormat #104

[Apache Jenkins Server]

See 

[jira] [Work logged] (BEAM-2852) Add support for Kafka as source/sink on Nexmark

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-2852?focusedWorklogId=87960=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87960
 ]

ASF GitHub Bot logged work on BEAM-2852:


Author: ASF GitHub Bot
Created on: 05/Apr/18 09:28
Start Date: 05/Apr/18 09:28
Worklog Time Spent: 10m 
  Work Description: echauchot commented on a change in pull request #5019: 
[BEAM-2852] Add support for Kafka as source/sink on Nexmark
URL: https://github.com/apache/beam/pull/5019#discussion_r179402264
 
 

 ##
 File path: 
sdks/java/nexmark/src/main/java/org/apache/beam/sdk/nexmark/NexmarkLauncher.java
 ##
 @@ -764,6 +774,68 @@ public void processElement(ProcessContext c) {
   }));
   }
 
+  static final DoFn EVENT_TO_BYTEARRAY =
+  new DoFn() {
+@ProcessElement
+public void processElement(ProcessContext c) {
+  try {
+byte[] encodedEvent = 
CoderUtils.encodeToByteArray(Event.CODER, c.element());
+c.output(encodedEvent);
+  } catch (CoderException e1) {
+LOG.error("Error while sending Event {} to Kafka: 
serialization error",
+c.element().toString());
+  }
+}
+  };
+
+  /**
+   * Send {@code events} to Kafka.
+   */
+  private void sinkEventsToKafka(PCollection events) {
 
 Review comment:
   IMHO I think we should refactor the whole COMBINED mode:
   - See NexmarkLauncher#createSource: it does a switch on the source type to 
configure sink.
   - it sends synthetic events to sink when in COMBINED mode but 
NexmarkUtils#COMBINED states that combine modes is for "Both publish and 
consume, but as separate jobs".
   Once refactored to something more coherent, implement it for kafka.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 87960)
Time Spent: 2h 40m  (was: 2.5h)

> Add support for Kafka as source/sink on Nexmark
> ---
>
> Key: BEAM-2852
> URL: https://issues.apache.org/jira/browse/BEAM-2852
> Project: Beam
>  Issue Type: Improvement
>  Components: testing
>Reporter: Ismaël Mejía
>Assignee: Kai Jiang
>Priority: Minor
>  Labels: newbie, nexmark, starter
>  Time Spent: 2h 40m
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (BEAM-3726) Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move backwards

[Alexey Romanenko (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426686#comment-16426686
 ] 

Alexey Romanenko edited comment on BEAM-3726 at 4/5/18 9:55 AM:


Fixed by BEAM-3087
Thank you to [~pawelbartoszek] for reporting this and help with reproducing.


was (Author: aromanenko):
Fixed by BEAM-3087


> Kinesis Reader: java.lang.IllegalArgumentException: Attempting to move 
> backwards
> 
>
> Key: BEAM-3726
> URL: https://issues.apache.org/jira/browse/BEAM-3726
> Project: Beam
>  Issue Type: Bug
>  Components: io-java-kinesis
>Affects Versions: 2.2.0
>Reporter: Pawel Bartoszek
>Assignee: Alexey Romanenko
>Priority: Major
> Fix For: 2.5.0
>
> Attachments: KinesisIO-state.txt
>
>
> When the job is restored from savepoint Kinesis Reader throws almost always 
> {{java.lang.IllegalArgumentException: Attempting to move backwards}}
> After a few job restarts caused again by the same exception, job finally 
> starts up and continues to run with no further problems.
> Beam job is reading from 32 shards with parallelism set to 32. Using Flink 
> 1.3.2. But I have seen this exception also when using Beam 2.2 when Kinesis 
> client was refactored to use MovingFunction. I think this is a serious 
> regression bug introduced in Beam 2.2. 
>  
> {code:java}
> java.lang.IllegalArgumentException: Attempting to move backwards
> at 
> org.apache.beam.sdk.repackaged.com.google.common.base.Preconditions.checkArgument(Preconditions.java:122)
> at org.apache.beam.sdk.util.MovingFunction.flush(MovingFunction.java:97)
> at org.apache.beam.sdk.util.MovingFunction.add(MovingFunction.java:114)
> at 
> org.apache.beam.sdk.io.kinesis.KinesisReader.advance(KinesisReader.java:137)
> at 
> org.apache.beam.runners.flink.metrics.ReaderInvocationUtil.invokeAdvance(ReaderInvocationUtil.java:67)
> at 
> org.apache.beam.runners.flink.translation.wrappers.streaming.io.UnboundedSourceWrapper.run(UnboundedSourceWrapper.java:264)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:87)
> at 
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:55)
> at 
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask.run(SourceStreamTask.java:95)
> at 
> org.apache.flink.streaming.runtime.tasks.StoppableSourceStreamTask.run(StoppableSourceStreamTask.java:39)
> at 
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:263)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:702){code}
>  
> Kinesis Reader transformation configuration:
> {code:java}
> pipeline.apply("KINESIS READER", KinesisIO.read()
> .withStreamName(streamName)
> .withInitialPositionInStream(InitialPositionInStream.LATEST)
> .withAWSClientsProvider(awsAccessKey, awsSecretKey, EU_WEST_1)){code}
>  
> When testing locally I managed to catch this exception. Just before executing 
> this 
> [link|https://github.com/apache/beam/blob/6c93105c2cb7be709c6b3e2e6cdcd09df2b48308/sdks/java/core/src/main/java/org/apache/beam/sdk/util/MovingFunction.java#L97]
>  that threw exception I captured the state of the class so that you can 
> replicate the issue
> {code:java}
> org.apache.beam.sdk.util.MovingFunction@71781a[sampleUpdateMs=5000,numSignificantBuckets=2,numSignificantSamples=10,function=org.apache.beam.sdk.transforms.Min$MinLongFn@7909d8d3,buckets={9223372036854775807,9223372036854775807,1519315344334,1519315343759,1519315343770,1519315344086,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807,9223372036854775807},numSamples={0,0,1,158,156,146,0,0,0,0,144,0},currentMsSinceEpoch=1519315585000,currentIndex=2]{code}
>  
> the add function of MovingFunction was called with nowMsSinceEpoch = 
> 1519315583591
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_Compressed_TextIOIT_HDFS #12

[Apache Jenkins Server]

See 


--
[...truncated 175.86 KB...]
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:248)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:235)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
Caused by: java.net.ConnectException: Connection refused
at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
at 
sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:717)
at 
org.apache.hadoop.net.SocketIOWithTimeout.connect(SocketIOWithTimeout.java:206)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:531)
at org.apache.hadoop.net.NetUtils.connect(NetUtils.java:495)
at 
org.apache.hadoop.ipc.Client$Connection.setupConnection(Client.java:614)
at 
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:712)
at org.apache.hadoop.ipc.Client$Connection.access$2900(Client.java:375)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1528)
at org.apache.hadoop.ipc.Client.call(Client.java:1451)
at org.apache.hadoop.ipc.Client.call(Client.java:1412)
at 
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
at com.sun.proxy.$Proxy60.create(Unknown Source)
at 
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
at 
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy61.create(Unknown Source)
at 
org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1623)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1703)
at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1638)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
at 
org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
at 
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:109)
at 
org.apache.beam.sdk.io.hdfs.HadoopFileSystem.create(HadoopFileSystem.java:68)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:248)
at org.apache.beam.sdk.io.FileSystems.create(FileSystems.java:235)
at 
org.apache.beam.sdk.io.FileBasedSink$Writer.open(FileBasedSink.java:923)
at 
org.apache.beam.sdk.io.WriteFiles$WriteUnshardedTempFilesWithSpillingFn.processElement(WriteFiles.java:503)
at 

Build failed in Jenkins: beam_PostCommit_Python_ValidatesRunner_Dataflow #1262

[Apache Jenkins Server]

See 


--
[...truncated 182.18 KB...]
  "is_pair_like": true
}, 
{
  "@type": "kind:stream", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}
  ], 
  "is_stream_like": true
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": "assert:odd/Group/GroupByKey.out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s29"
}, 
"serialized_fn": 
"%0AD%22B%0A%1Dref_Coder_GlobalWindowCoder_1%12%21%0A%1F%0A%1D%0A%1Bbeam%3Acoder%3Aglobal_window%3Av1jT%0A%25%0A%23%0A%21beam%3Awindowfn%3Aglobal_windows%3Av0.1%10%01%1A%1Dref_Coder_GlobalWindowCoder_1%22%02%3A%00%28%010%018%01H%01",
 
"user_name": "assert:odd/Group/GroupByKey"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s31", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": "_merge_tagged_vals_under_key"
  }, 
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.ParDo", 
"shortValue": "CallableWrapperDoFn", 
"type": "STRING", 
"value": "apache_beam.transforms.core.CallableWrapperDoFn"
  }
], 
"non_parallel_inputs": {}, 
"output_info": [
  {
"encoding": {
  "@type": "kind:windowed_value", 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": [
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}, 
{
  "@type": 
"FastPrimitivesCoder$eNprYEpOLEhMzkiNT0pNzNVLzk9JLSqGUlxuicUlAUWZuZklmWWpxc4gQa5CBs3GQsbaQqZQ/vi0xJycpMTk7Hiw+kJmPEYFZCZn56RCjWABGsFaW8iWVJykBwDlGS3/",
 
  "component_encodings": []
}
  ], 
  "is_pair_like": true
}, 
{
  "@type": "kind:global_window"
}
  ], 
  "is_wrapper": true
}, 
"output_name": "out", 
"user_name": 
"assert:odd/Group/Map(_merge_tagged_vals_under_key).out"
  }
], 
"parallel_input": {
  "@type": "OutputReference", 
  "output_name": "out", 
  "step_name": "s30"
}, 
"serialized_fn": "", 
"user_name": "assert:odd/Group/Map(_merge_tagged_vals_under_key)"
  }
}, 
{
  "kind": "ParallelDo", 
  "name": "s32", 
  "properties": {
"display_data": [
  {
"key": "fn", 
"label": "Transform Function", 
"namespace": "apache_beam.transforms.core.CallableWrapperDoFn", 
"type": "STRING", 
"value": ""
  }, 
  {
"key": "fn", 
"label": "Transform Function", 

Build failed in Jenkins: beam_PerformanceTests_Python #1109

[Apache Jenkins Server]

See 


--
[...truncated 1.68 KB...]
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2591319722150719529.sh
+ rm -rf .env
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins2426024854102122007.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_Python] $ /bin/bash -xe /tmp/jenkins716054111847692302.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins3711440670367489711.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_Python] $ /bin/bash -xe 
/tmp/jenkins8866673851995322779.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18))
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19))
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21))
Requirement already satisfied: numpy==1.13.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 22))
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Requirement already satisfied: contextlib2>=0.5.1 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 24))
Requirement already satisfied: pywinrm in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Requirement already satisfied: xmltodict in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests-ntlm>=0.3.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: requests>=2.9.1 in 
/usr/local/lib/python2.7/dist-packages (from pywinrm->-r 
PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: ntlm-auth>=1.0.2 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: cryptography>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from 
requests-ntlm>=0.3.0->pywinrm->-r PerfKitBenchmarker/requirements.txt (line 25))
Requirement already satisfied: urllib3<1.23,>=1.21.1 in 
/usr/local/lib/python2.7/dist-packages (from requests>=2.9.1->pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT #107

[Apache Jenkins Server]

See 


--
[...truncated 25.80 KB...]
[INFO] Excluding 
com.google.apis:google-api-services-storage:jar:v1-rev71-1.22.0 from the shaded 
jar.
[INFO] Excluding com.google.auth:google-auth-library-credentials:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.auth:google-auth-library-oauth2-http:jar:0.7.1 from 
the shaded jar.
[INFO] Excluding com.google.cloud.bigdataoss:util:jar:1.4.5 from the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-java6:jar:1.22.0 from 
the shaded jar.
[INFO] Excluding com.google.api-client:google-api-client-jackson2:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.oauth-client:google-oauth-client-java6:jar:1.22.0 
from the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-io-google-cloud-platform:jar:2.5.0-SNAPSHOT from 
the shaded jar.
[INFO] Excluding 
org.apache.beam:beam-sdks-java-extensions-protobuf:jar:2.5.0-SNAPSHOT from the 
shaded jar.
[INFO] Excluding io.grpc:grpc-core:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.errorprone:error_prone_annotations:jar:2.0.15 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-context:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.instrumentation:instrumentation-api:jar:0.3.0 from 
the shaded jar.
[INFO] Excluding 
com.google.apis:google-api-services-bigquery:jar:v2-rev374-1.22.0 from the 
shaded jar.
[INFO] Excluding com.google.api:gax-grpc:jar:0.20.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.api:api-common:jar:1.0.0-rc2 from the shaded jar.
[INFO] Excluding com.google.auto.value:auto-value:jar:1.5.3 from the shaded jar.
[INFO] Excluding com.google.api:gax:jar:1.3.1 from the shaded jar.
[INFO] Excluding org.threeten:threetenbp:jar:1.3.3 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core-grpc:jar:1.2.0 from the 
shaded jar.
[INFO] Excluding com.google.protobuf:protobuf-java-util:jar:3.2.0 from the 
shaded jar.
[INFO] Excluding com.google.code.gson:gson:jar:2.7 from the shaded jar.
[INFO] Excluding com.google.apis:google-api-services-pubsub:jar:v1-rev10-1.22.0 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-pubsub-v1:jar:0.1.18 
from the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-iam-v1:jar:0.1.18 from the 
shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-proto-client:jar:1.4.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-protobuf:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.http-client:google-http-client-jackson:jar:1.22.0 
from the shaded jar.
[INFO] Excluding com.google.cloud.datastore:datastore-v1-protos:jar:1.3.0 from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:grpc-google-common-protos:jar:0.1.9 from 
the shaded jar.
[INFO] Excluding io.grpc:grpc-auth:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-netty:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http2:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec-http:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler-proxy:jar:4.1.8.Final from the shaded 
jar.
[INFO] Excluding io.netty:netty-codec-socks:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-handler:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-buffer:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-common:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-transport:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-resolver:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.netty:netty-codec:jar:4.1.8.Final from the shaded jar.
[INFO] Excluding io.grpc:grpc-stub:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-all:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-okhttp:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.squareup.okhttp:okhttp:jar:2.5.0 from the shaded jar.
[INFO] Excluding com.squareup.okio:okio:jar:1.6.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-lite:jar:1.2.0 from the shaded jar.
[INFO] Excluding io.grpc:grpc-protobuf-nano:jar:1.2.0 from the shaded jar.
[INFO] Excluding com.google.protobuf.nano:protobuf-javanano:jar:3.0.0-alpha-5 
from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-core:jar:1.0.2 from the shaded 
jar.
[INFO] Excluding org.json:json:jar:20160810 from the shaded jar.
[INFO] Excluding com.google.cloud:google-cloud-spanner:jar:0.20.0b-beta from 
the shaded jar.
[INFO] Excluding com.google.api.grpc:proto-google-cloud-spanner-v1:jar:0.1.11b 
from the shaded jar.
[INFO] Excluding 

[jira] [Commented] (BEAM-3946) Python SDK tests are failing if no GOOGLE_APPLICATION_CREDENTIALS was set

[Colm O hEigeartaigh (JIRA)]

[ 
https://issues.apache.org/jira/browse/BEAM-3946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426712#comment-16426712
 ] 

Colm O hEigeartaigh commented on BEAM-3946:
---

I can confirm the issue is fixed - thanks!

> Python SDK tests are failing if no GOOGLE_APPLICATION_CREDENTIALS was set
> -
>
> Key: BEAM-3946
> URL: https://issues.apache.org/jira/browse/BEAM-3946
> Project: Beam
>  Issue Type: Bug
>  Components: examples-python
>Reporter: Alexey Romanenko
>Assignee: Mark Liu
>Priority: Major
> Fix For: Not applicable
>
>  Time Spent: 20m
>  Remaining Estimate: 0h
>
> Running locally mvn clean install fails on following Apache Beam :: SDKs :: 
> Python tests:
> {{ERROR: test_message_matcher_mismatch 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  {{ERROR: test_message_matcher_success 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  {{ERROR: test_message_metcher_timeout 
> (apache_beam.io.gcp.tests.pubsub_matcher_test.PubSubMatcherTest)}}
>  
> with an error:
> DefaultCredentialsError: Could not automatically determine credentials. 
> Please set GOOGLE_APPLICATION_CREDENTIALS or
>  explicitly create credential and re-run the application. For more
>  information, please see
>  
> [https://developers.google.com/accounts/docs/application-default-credentials].
>   >> begin captured logging << 
>  google.auth.transport._http_client: DEBUG: Making request: GET 
> [http://169.254.169.254|http://169.254.169.254/]
>  google.auth.compute_engine._metadata: INFO: Compute Engine Metadata server 
> unavailable.
>  - >> end captured logging << -
>  
> It looks like it's a regression and it was caused by this commit: 
> [301853647f2c726c04c5bdb02cab6ff6b39f09d0|https://github.com/apache/beam/commit/301853647f2c726c04c5bdb02cab6ff6b39f09d0]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (BEAM-3954) Get Jenkins agents dockerized

[JIRA]

[ 
https://issues.apache.org/jira/browse/BEAM-3954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16426674#comment-16426674
 ] 

Ismaël Mejía commented on BEAM-3954:


Ok so it seems we are aligned then [~alanmyrvold]. i know docker well so I can 
help there, however I am a jenkins newbie, but I can try to help there also if 
needed.

[~yifanzou] The image adds whatever you pass as a parameter at build time and 
the default is github's  master, but you can pass a PR, a branch, etc for 
details take a look at the 
[documentation|[https://beam.apache.org/contribute/docker-images/].]

How do you guys plan to tackle the problem of downloading all the deps on each 
build ? I did this via volume mounting but still not convinced is the best way, 
if no the build takes ages. We have to also pay attention at how we run [java 
in 
containers|https://blog.csanchez.org/2017/05/31/running-a-jvm-in-a-container-without-getting-killed/].

 

> Get Jenkins agents dockerized 
> --
>
> Key: BEAM-3954
> URL: https://issues.apache.org/jira/browse/BEAM-3954
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system
>Reporter: yifan zou
>Assignee: yifan zou
>Priority: Major
>  Time Spent: 0.5h
>  Remaining Estimate: 0h
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (BEAM-4015) GCS artifact proxy is breaking the build when credentials are not available

[JIRA]

 [ 
https://issues.apache.org/jira/browse/BEAM-4015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismaël Mejía updated BEAM-4015:
---
Description: 
When running the maven build in a machine without the valid auth credentials 
the module breaks like this:

{code:bash}

{{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
beam-runners-gcp-gcsproxy ---}}
{{[INFO] Prepared command line : bin/go build -buildmode=default -o 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
github.com/apache/beam/cmd/gcsproxy}}
{{[ERROR] }}
{{[ERROR] -Exec.Err-}}
{{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
{{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication}}
{{[ERROR] }}
{{}}

{{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy 
FAILURE [  1.038 s]}}
{{}}
{{[INFO] BUILD FAILURE}}
{{}}
{{[ERROR] Failed to execute goal 
com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
beam-runners-gcp-gcsproxy: Can't find generated target file : 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
{{[ERROR] }}

{code}

  was:
When running the maven build in a machine without the valid auth credentials 
the module breaks like this:

{{{code:bash}}}

{{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
beam-runners-gcp-gcsproxy ---}}
{{[INFO] Prepared command line : bin/go build -buildmode=default -o 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
github.com/apache/beam/cmd/gcsproxy}}
{{[ERROR] }}
{{[ERROR] -Exec.Err-}}
{{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
{{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
undefined: option.WithoutAuthentication}}
{{[ERROR] }}
{{}}

{{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact proxy 
FAILURE [  1.038 s]}}
{{}}
{{[INFO] BUILD FAILURE}}
{{}}
{{[ERROR] Failed to execute goal 
com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
beam-runners-gcp-gcsproxy: Can't find generated target file : 
/home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
{{[ERROR] }}

{{{code}}}{{}}


> GCS artifact proxy is breaking the build when credentials are not available
> ---
>
> Key: BEAM-4015
> URL: https://issues.apache.org/jira/browse/BEAM-4015
> Project: Beam
>  Issue Type: Bug
>  Components: sdk-go
>Reporter: Ismaël Mejía
>Assignee: Henning Rohde
>Priority: Blocker
>
> When running the maven build in a machine without the valid auth credentials 
> the module breaks like this:
> {code:bash}
> {{[INFO] --- mvn-golang-wrapper:2.1.6:build (go-build) @ 
> beam-runners-gcp-gcsproxy ---}}
> {{[INFO] Prepared command line : bin/go build -buildmode=default -o 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy 
> github.com/apache/beam/cmd/gcsproxy}}
> {{[ERROR] }}
> {{[ERROR] -Exec.Err-}}
> {{[ERROR] # github.com/apache/beam/sdks/go/pkg/beam/util/gcsx}}
> {{[ERROR] github.com/apache/beam/sdks/go/pkg/beam/util/gcsx/gcs.go:46:37: 
> undefined: option.WithoutAuthentication}}
> {{[ERROR] }}
> {{}}
> {{[INFO] Apache Beam :: Runners :: Google Cloud Platform :: GCS artifact 
> proxy FAILURE [  1.038 s]}}
> {{}}
> {{[INFO] BUILD FAILURE}}
> {{}}
> {{[ERROR] Failed to execute goal 
> com.igormaznitsa:mvn-golang-wrapper:2.1.6:build (go-build) on project 
> beam-runners-gcp-gcsproxy: Can't find generated target file : 
> /home/ismael/upstream/beam/runners/gcp/gcsproxy/target/gcsproxy -> [Help 1]}}
> {{[ERROR] }}
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Build failed in Jenkins: beam_PerformanceTests_MongoDBIO_IT #13

[Apache Jenkins Server]

See 


--
[...truncated 3.06 KB...]
INFO: Display format "default".
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins2997631832766220765.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins4099298585065614089.sh
+ kubectl 
--kubeconfig=
 create namespace mongodbioit-1522922499749
namespace "mongodbioit-1522922499749" created
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7277287841497966238.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=mongodbioit-1522922499749
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins7464447287272266712.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins1428643769294819023.sh
+ rm -rf .env
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins6346696967223251575.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins9210029343468086157.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in 
./.env/lib/python2.7/site-packages (39.0.1)
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages 
(10.0.0b2)
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins5970347955523031897.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_MongoDBIO_IT] $ /bin/bash -xe 
/tmp/jenkins461847908127210411.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14)) (0.1.3)
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15)) (2.8)
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16)) (39.0.1)
Requirement already satisfied: colorlog[windows]==2.6.0 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 17)) (2.6.0)
Requirement already satisfied: blinker>=1.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 18)) (1.4)
Requirement already satisfied: futures>=3.0.3 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 19)) (3.1.1)
Requirement already satisfied: PyYAML==3.12 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20)) (3.12)
Requirement already satisfied: pint>=0.7 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 21)) (0.7.2)
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
:339:
 SNIMissingWarning: An HTTPS request has been made, but the SNI (Subject Name 
Indication) extension to TLS is not available on this platform. This may cause 
the server to present an incorrect TLS certificate, which can cause validation 
failures. You can upgrade to a newer version of Python to solve this. For more 
information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  SNIMissingWarning
:137:
 InsecurePlatformWarning: A true SSLContext object is not available. This 
prevents urllib3 from configuring SSL appropriately and may cause certain SSL 
connections to fail. You can upgrade to a newer version of Python to solve 
this. For more information, see 
https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings
  InsecurePlatformWarning
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23)) (3.2.3.post2)

Build failed in Jenkins: beam_PerformanceTests_XmlIOIT_HDFS #11

[Apache Jenkins Server]

See 


--
[...truncated 902 B...]
Checking out Revision 50139b4395584513099094445f57c495b515 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 50139b4395584513099094445f57c495b515
Commit message: "Merge pull request #4790 from 
rmannibucau/fix/BEAM-3409_wait-for-teardown-execution-in-direct-runner"
 > git rev-list --no-walk 50139b4395584513099094445f57c495b515 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8107580962859289087.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='Updates a kubeconfig file with appropriate 
credentials to point\nkubectl at a Container Engine Cluster. By default, 
credentials\nare written to HOME/.kube/config. You can provide an 
alternate\npath by setting the KUBECONFIG environment variable.\n\nSee 
[](https://cloud.google.com/container-engine/docs/kubectl) for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, 
calliope_command=, command_path=['gcloud', 'container', 'clusters', 
'get-credentials'], configuration=None, credential_file_override=None, 
document=None, flatten=None, format=None, h=None, help=None, http_timeout=None, 
log_http=None, name='io-datastores', project=None, quiet=None, 
trace_email=None, trace_log=None, trace_token=None, user_output_enabled=None, 
verbosity='debug', version=None, zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
DEBUG: SDK update checks are disabled.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5964627428376801548.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1981670355970295709.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1522922472868
namespace "filebasedioithdfs-1522922472868" created
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins9023884533833813066.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1522922472868
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins6660834373402104149.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins4759483930257616215.sh
+ rm -rf .env
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins2940782492509448902.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3183813551650708668.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_XmlIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins577656382049551751.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_XmlIOIT_HDFS] $ 

Build failed in Jenkins: beam_PerformanceTests_TextIOIT_HDFS #18

[Apache Jenkins Server]

See 


--
[...truncated 2.18 KB...]
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7775843329849416196.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5046890806552735551.sh
+ kubectl 
--kubeconfig=
 create namespace filebasedioithdfs-1522922472827
namespace "filebasedioithdfs-1522922472827" created
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins5706657386220660702.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=filebasedioithdfs-1522922472827
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7478023021936784740.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins3482195949834319812.sh
+ rm -rf .env
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1754758472120160473.sh
+ virtualenv .env --system-site-packages
New python executable in .env/bin/python
Installing setuptools, pip...done.
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins7071304881877035190.sh
+ .env/bin/pip install --upgrade setuptools pip
Downloading/unpacking setuptools from 
https://pypi.python.org/packages/20/d7/04a0b689d3035143e2ff288f4b9ee4bf6ed80585cc121c90bfd85a1a8c2e/setuptools-39.0.1-py2.py3-none-any.whl#md5=ca299c7acd13a72e1171a3697f2b99bc
Downloading/unpacking pip from 
https://pypi.python.org/packages/ac/95/a05b56bb975efa78d3557efa36acaf9cf5d2fd0ee0062060493687432e03/pip-9.0.3-py2.py3-none-any.whl#md5=d512ceb964f38ba31addb8142bc657cb
Installing collected packages: setuptools, pip
  Found existing installation: setuptools 2.2
Uninstalling setuptools:
  Successfully uninstalled setuptools
  Found existing installation: pip 1.5.4
Uninstalling pip:
  Successfully uninstalled pip
Successfully installed setuptools pip
Cleaning up...
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins8901272209584405548.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_TextIOIT_HDFS] $ /bin/bash -xe 
/tmp/jenkins1537694218695121896.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Collecting absl-py (from -r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r PerfKitBenchmarker/requirements.txt (line 16))
Collecting colorlog[windows]==2.6.0 (from -r 
PerfKitBenchmarker/requirements.txt (line 17))
  Using cached colorlog-2.6.0-py2.py3-none-any.whl
Collecting blinker>=1.3 (from -r PerfKitBenchmarker/requirements.txt (line 18))
Collecting futures>=3.0.3 (from -r PerfKitBenchmarker/requirements.txt (line 
19))
  Using cached futures-3.2.0-py2-none-any.whl
Requirement already satisfied: PyYAML==3.12 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 20))
Collecting pint>=0.7 (from -r PerfKitBenchmarker/requirements.txt (line 21))
Collecting numpy==1.13.3 (from -r PerfKitBenchmarker/requirements.txt (line 22))
  Using cached numpy-1.13.3-cp27-cp27mu-manylinux1_x86_64.whl
Requirement already satisfied: functools32 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 23))
Collecting contextlib2>=0.5.1 (from -r PerfKitBenchmarker/requirements.txt 
(line 24))
  Using cached contextlib2-0.5.5-py2.py3-none-any.whl
Collecting pywinrm (from -r PerfKitBenchmarker/requirements.txt (line 25))
  Using cached pywinrm-0.3.0-py2.py3-none-any.whl
Requirement already satisfied: six in /usr/local/lib/python2.7/dist-packages 
(from absl-py->-r PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: MarkupSafe>=0.23 in 
/usr/local/lib/python2.7/dist-packages (from jinja2>=2.7->-r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: colorama; extra == "windows" in 
/usr/lib/python2.7/dist-packages (from colorlog[windows]==2.6.0->-r 
PerfKitBenchmarker/requirements.txt (line 17))
Collecting xmltodict (from pywinrm->-r PerfKitBenchmarker/requirements.txt 
(line 25))
  Using cached xmltodict-0.11.0-py2.py3-none-any.whl
Collecting requests-ntlm>=0.3.0 (from pywinrm->-r 

Build failed in Jenkins: beam_PerformanceTests_JDBC #415

[Apache Jenkins Server]

See 


--
[...truncated 65 B...]
 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url https://github.com/apache/beam.git # timeout=10
Fetching upstream changes from https://github.com/apache/beam.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/beam.git 
 > +refs/heads/*:refs/remotes/origin/* 
 > +refs/pull/${ghprbPullId}/*:refs/remotes/origin/pr/${ghprbPullId}/*
 > git rev-parse origin/master^{commit} # timeout=10
Checking out Revision 50139b4395584513099094445f57c495b515 (origin/master)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 50139b4395584513099094445f57c495b515
Commit message: "Merge pull request #4790 from 
rmannibucau/fix/BEAM-3409_wait-for-teardown-execution-in-direct-runner"
 > git rev-list --no-walk 50139b4395584513099094445f57c495b515 # timeout=10
Cleaning workspace
 > git rev-parse --verify HEAD # timeout=10
Resetting working tree
 > git reset --hard # timeout=10
 > git clean -fdx # timeout=10
[EnvInject] - Executing scripts and injecting environment variables after the 
SCM step.
[EnvInject] - Injecting as environment variables the properties content 
SPARK_LOCAL_IP=127.0.0.1

[EnvInject] - Variables injected successfully.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins447134194987811133.sh
+ gcloud container clusters get-credentials io-datastores --zone=us-central1-a 
--verbosity=debug
DEBUG: Running gcloud.container.clusters.get-credentials with 
Namespace(__calliope_internal_deepest_parser=ArgumentParser(prog='gcloud.container.clusters.get-credentials',
 usage=None, description='See 
https://cloud.google.com/container-engine/docs/kubectl for\nkubectl 
documentation.', version=None, formatter_class=, conflict_handler='error', add_help=False), 
account=None, api_version=None, authority_selector=None, 
authorization_token_file=None, cmd_func=>, 
command_path=['gcloud', 'container', 'clusters', 'get-credentials'], 
configuration=None, credential_file_override=None, document=None, format=None, 
h=None, help=None, http_timeout=None, log_http=None, name='io-datastores', 
project=None, quiet=None, trace_email=None, trace_log=None, trace_token=None, 
user_output_enabled=None, verbosity='debug', version=None, 
zone='us-central1-a').
Fetching cluster endpoint and auth data.
DEBUG: Saved kubeconfig to /home/jenkins/.kube/config
kubeconfig entry generated for io-datastores.
INFO: Display format "default".
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins6415979489839681473.sh
+ cp /home/jenkins/.kube/config 

[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins3966548659024986477.sh
+ kubectl 
--kubeconfig=
 create namespace jdbcioit-1522922491509
namespace "jdbcioit-1522922491509" created
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8099819848604532427.sh
++ kubectl config current-context
+ kubectl 
--kubeconfig=
 config set-context gke_apache-beam-testing_us-central1-a_io-datastores 
--namespace=jdbcioit-1522922491509
Context "gke_apache-beam-testing_us-central1-a_io-datastores" modified.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins7840605553958493904.sh
+ rm -rf PerfKitBenchmarker
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins9210678640694792219.sh
+ rm -rf .env
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins2295903970342982059.sh
+ virtualenv .env --system-site-packages
New python executable in 

Installing setuptools, pip, wheel...done.
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins738287463579438513.sh
+ .env/bin/pip install --upgrade setuptools pip
Requirement already up-to-date: setuptools in ./.env/lib/python2.7/site-packages
Requirement already up-to-date: pip in ./.env/lib/python2.7/site-packages
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins8448110779262506156.sh
+ git clone https://github.com/GoogleCloudPlatform/PerfKitBenchmarker.git
Cloning into 'PerfKitBenchmarker'...
[beam_PerformanceTests_JDBC] $ /bin/bash -xe /tmp/jenkins469961488905201746.sh
+ .env/bin/pip install -r PerfKitBenchmarker/requirements.txt
Requirement already satisfied: absl-py in 
/home/jenkins/.local/lib/python2.7/site-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 14))
Requirement already satisfied: jinja2>=2.7 in 
/usr/local/lib/python2.7/dist-packages (from -r 
PerfKitBenchmarker/requirements.txt (line 15))
Requirement already satisfied: setuptools in ./.env/lib/python2.7/site-packages 
(from -r 

[jira] [Work logged] (BEAM-3250) Migrate ValidatesRunner Jenkins PostCommits to Gradle

[ASF GitHub Bot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/BEAM-3250?focusedWorklogId=87923=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-87923
 ]

ASF GitHub Bot logged work on BEAM-3250:


Author: ASF GitHub Bot
Created on: 05/Apr/18 07:16
Start Date: 05/Apr/18 07:16
Worklog Time Spent: 10m 
  Work Description: herohde commented on issue #5030: [BEAM-3250] Migrate 
Spark ValidatesRunner tests to Gradle
URL: https://github.com/apache/beam/pull/5030#issuecomment-378842371
 
 
   R: @lukecwik 
   
   The dependency failure that took me some time to figure out was that we used 
a wrong kafka version where a class was slightly renamed.


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


Issue Time Tracking
---

Worklog Id: (was: 87923)
Time Spent: 2h 50m  (was: 2h 40m)

> Migrate ValidatesRunner Jenkins PostCommits to Gradle
> -
>
> Key: BEAM-3250
> URL: https://issues.apache.org/jira/browse/BEAM-3250
> Project: Beam
>  Issue Type: Sub-task
>  Components: build-system, testing
>Reporter: Luke Cwik
>Assignee: Henning Rohde
>Priority: Major
>  Time Spent: 2h 50m
>  Remaining Estimate: 0h
>
> Update these targets to execute ValidatesRunner tests: 
> https://github.com/apache/beam/search?l=Groovy=ValidatesRunner==%E2%9C%93



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)