[jira] [Comment Edited] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017754#comment-17017754 ] sunjincheng edited comment on BEAM-9137 at 1/17/20 7:14 AM: I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. So, maybe we should focus on the commits of that day. was (Author: sunjincheng121): I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017754#comment-17017754 ] sunjincheng commented on BEAM-9137: --- I have a quick look at of this issue and it seems that the beam_PostCommit_Py_ValCont was broken on 20 Dec, 2019. We can see that it succeed on [19 Dec 2019 |http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c100795298.3239.1576775433670.JavaMail.jenkins@jenkins02%3e]and failed on [20 Dec, 2019|http://mail-archives.apache.org/mod_mbox/beam-builds/201912.mbox/%3c165865149.3447.1576823413985.JavaMail.jenkins@jenkins02%3e]. > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sunjincheng reassigned BEAM-9137: - Assignee: (was: sunjincheng) > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9137) PR10338 breaks beam_PostCommit_Py_ValCont
[ https://issues.apache.org/jira/browse/BEAM-9137?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017747#comment-17017747 ] sunjincheng commented on BEAM-9137: --- It seems that the test error is not caused by[1], more detail log can be found in [2]. [~boyuanz] [1] [https://github.com/apache/beam/pull/10338] [2] [https://github.com/apache/beam/pull/10338#issuecomment-575482269] > PR10338 breaks beam_PostCommit_Py_ValCont > - > > Key: BEAM-9137 > URL: https://issues.apache.org/jira/browse/BEAM-9137 > Project: Beam > Issue Type: Bug > Components: test-failures >Affects Versions: 2.19.0 >Reporter: Boyuan Zhang >Assignee: sunjincheng >Priority: Blocker > Fix For: 2.19.0 > > > For the first failure, please refer to > https://builds.apache.org/job/beam_PostCommit_Py_ValCont/5172/#showFailuresLink -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=373427=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373427 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 17/Jan/20 06:21 Start Date: 17/Jan/20 06:21 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10338: [BEAM-8935] Fail fast if sdk harness startup failed. URL: https://github.com/apache/beam/pull/10338#issuecomment-575482269 It seems that the test error is not caused by this commit. @boyuanzz This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373427) Time Spent: 2h 20m (was: 2h 10m) > Fail fast if sdk harness startup failed > --- > > Key: BEAM-8935 > URL: https://issues.apache.org/jira/browse/BEAM-8935 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently the runner waits for the sdk harness to startup blockingly until > the sdk harness is available or timeout occurs. The timeout is 1 or 2 > minutes. If the sdk harness startup failed for some reason, the runner may be > aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=373429=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373429 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 17/Jan/20 06:22 Start Date: 17/Jan/20 06:22 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10338: [BEAM-8935] Fail fast if sdk harness startup failed. URL: https://github.com/apache/beam/pull/10338#issuecomment-575482269 It seems that the test error is not caused by this commit. @boyuanzz https://builds.apache.org/job/beam_PostCommit_Py_ValCont_PR/107/consoleFull This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373429) Time Spent: 2.5h (was: 2h 20m) > Fail fast if sdk harness startup failed > --- > > Key: BEAM-8935 > URL: https://issues.apache.org/jira/browse/BEAM-8935 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2.5h > Remaining Estimate: 0h > > Currently the runner waits for the sdk harness to startup blockingly until > the sdk harness is available or timeout occurs. The timeout is 1 or 2 > minutes. If the sdk harness startup failed for some reason, the runner may be > aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017732#comment-17017732
]
Chad Dombrova commented on BEAM-9130:
-
I updated my PR. There are still other problems with it, but I solved the
google.protobuf import problem.
There are a few pieces to the solution:
* use a pyproject.toml file to specify build requirements:
[https://github.com/apache/beam/pull/10038/files#diff-71ede7e13a310beaedc4466b48096e26R18]
* use pep517 module to build source:
[https://github.com/apache/beam/pull/10038/files#diff-d3cc66016c37760e131f228abbee6667R34]
* add pep517's temp build directory to the site dir to allow the google
package's .pth file to work:
[https://github.com/apache/beam/pull/10038/commits/ed06bf1b17e5f60d7f6bcf7766b93e4eaf4b86ff]
I'm sure there are easier ways to hack this to work without such dramatic
changes, but this approach sets us on a course to untangle the mess that we
have now.
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017699#comment-17017699
]
Chad Dombrova commented on BEAM-9130:
-
Just in case it's not clear, I think you'd want to edit the Dockerfile to use
tox, here:
{code:java}
CMD holdup -t 45 http://namenode:50070 http://datanode:50075 && \
echo "Waiting for safe mode to end." && \
sleep 45 && \
hdfscli -v -v -v upload -f kinglear.txt / && \
python -m apache_beam.examples.wordcount \
--input hdfs://kinglear* \
--output hdfs://py-wordcount-integration \
--hdfs_host namenode --hdfs_port 50070 --hdfs_user root
{code}
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373391=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373391 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 17/Jan/20 04:27 Start Date: 17/Jan/20 04:27 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575378395 Java PreCommits failed because of test assertion change due to toString update. I missed this locally because I ran `./gradlew -p sdks/java check` (this does not run tests in runners). Python PreCommit failed: ``` E IndexError: index out of range ``` https://builds.apache.org/job/beam_PreCommit_Python_Commit/10812/testReport/junit/ Checking it with my local environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373391) Time Spent: 2h 40m (was: 2.5h) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 2h 40m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017691#comment-17017691
]
Chad Dombrova commented on BEAM-9130:
-
the protobuf package is a dependency of grpcio-tools, which is part of
build_requirements.txt. Did you include build_requirements.txt to the deps
for your tox env? that could help, but it also might not: I can't remember
if the deps are installed in time to affect setup.py (which calls gen_protos).
I think it may not be, which is where pyproject.toml and pep517 comes into
play: they specify the deps that need to be installed before setup.py runs.
I'm currently rebasing my old testing refactor branch to see if it helps.
can you push your current WIP solution for this so that I can try to help out.
I'm on vacation tomorrow and monday, but I may be able to chip away at some
things on the plane.
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=373390=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373390 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 17/Jan/20 04:13 Start Date: 17/Jan/20 04:13 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10338: [BEAM-8935] Fail fast if sdk harness startup failed. URL: https://github.com/apache/beam/pull/10338#issuecomment-575456506 Thanks for your info @boyuanzz , I would like to make a test for it. https://github.com/apache/beam/pull/10623 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373390) Time Spent: 2h 10m (was: 2h) > Fail fast if sdk harness startup failed > --- > > Key: BEAM-8935 > URL: https://issues.apache.org/jira/browse/BEAM-8935 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Currently the runner waits for the sdk harness to startup blockingly until > the sdk harness is available or timeout occurs. The timeout is 1 or 2 > minutes. If the sdk harness startup failed for some reason, the runner may be > aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=373389=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373389 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 17/Jan/20 04:12 Start Date: 17/Jan/20 04:12 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on issue #10623: [DO NOT REVIEW] Revert "[BEAM-8935] Fail fast if sdk harness startup failed." URL: https://github.com/apache/beam/pull/10623#issuecomment-575456329 Run Python Dataflow ValidatesContainer This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373389) Time Spent: 2h (was: 1h 50m) > Fail fast if sdk harness startup failed > --- > > Key: BEAM-8935 > URL: https://issues.apache.org/jira/browse/BEAM-8935 > Project: Beam > Issue Type: Improvement > Components: java-fn-execution >Reporter: sunjincheng >Assignee: sunjincheng >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h > Remaining Estimate: 0h > > Currently the runner waits for the sdk harness to startup blockingly until > the sdk harness is available or timeout occurs. The timeout is 1 or 2 > minutes. If the sdk harness startup failed for some reason, the runner may be > aware of it after 1 or 2 minutes. This is too long. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8935) Fail fast if sdk harness startup failed
[ https://issues.apache.org/jira/browse/BEAM-8935?focusedWorklogId=373388=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373388 ] ASF GitHub Bot logged work on BEAM-8935: Author: ASF GitHub Bot Created on: 17/Jan/20 04:11 Start Date: 17/Jan/20 04:11 Worklog Time Spent: 10m Work Description: sunjincheng121 commented on pull request #10623: [DO NOT REVIEW] Revert "[BEAM-8935] Fail fast if sdk harness startup failed." URL: https://github.com/apache/beam/pull/10623 DO NOT REVIEW. Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [![Build
[jira] [Work logged] (BEAM-7246) Create a Spanner IO for Python
[ https://issues.apache.org/jira/browse/BEAM-7246?focusedWorklogId=373384=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373384 ] ASF GitHub Bot logged work on BEAM-7246: Author: ASF GitHub Bot Created on: 17/Jan/20 03:51 Start Date: 17/Jan/20 03:51 Worklog Time Spent: 10m Work Description: chamikaramj commented on issue #9606: [BEAM-7246] Add Google Spanner IO Read on Python SDK URL: https://github.com/apache/beam/pull/9606#issuecomment-575452714 Any updates ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373384) Time Spent: 11h (was: 10h 50m) > Create a Spanner IO for Python > -- > > Key: BEAM-7246 > URL: https://issues.apache.org/jira/browse/BEAM-7246 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Reuven Lax >Assignee: Shehzaad Nakhoda >Priority: Major > Time Spent: 11h > Remaining Estimate: 0h > > Add I/O support for Google Cloud Spanner for the Python SDK (Batch Only). > Testing in this work item will be in the form of DirectRunner tests and > manual testing. > Integration and performance tests are a separate work item (not included > here). > See https://beam.apache.org/documentation/io/built-in/. The goal is to add > Google Clound Spanner to the Database column for the Python/Batch row. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017685#comment-17017685
]
Udi Meiri commented on BEAM-9130:
-
I was trying to debug this, converted it to a tox environment, but still got
the same errors.
Weird thing is, if I run a shell on the docker image, and run "tox -e
hdfs_integration_test" twice, it succeeds the second time.
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8625) Implement servlet in Dataflow runner for sdk status query endpoint
[ https://issues.apache.org/jira/browse/BEAM-8625?focusedWorklogId=373373=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373373 ] ASF GitHub Bot logged work on BEAM-8625: Author: ASF GitHub Bot Created on: 17/Jan/20 03:30 Start Date: 17/Jan/20 03:30 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10553: [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… URL: https://github.com/apache/beam/pull/10553 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373373) Time Spent: 2h 10m (was: 2h) > Implement servlet in Dataflow runner for sdk status query endpoint > -- > > Key: BEAM-8625 > URL: https://issues.apache.org/jira/browse/BEAM-8625 > Project: Beam > Issue Type: Sub-task > Components: runner-dataflow >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 2h 10m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9122) Add uses_keyed_state step property to python dataflow runner
[ https://issues.apache.org/jira/browse/BEAM-9122?focusedWorklogId=373372=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373372 ] ASF GitHub Bot logged work on BEAM-9122: Author: ASF GitHub Bot Created on: 17/Jan/20 03:26 Start Date: 17/Jan/20 03:26 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10596: [BEAM-9122] Add uses_keyed_state step property in python dataflow run… URL: https://github.com/apache/beam/pull/10596#discussion_r367754044 ## File path: sdks/python/apache_beam/runners/dataflow/dataflow_runner.py ## @@ -950,6 +950,10 @@ def run_ParDo(self, transform_node, options): step.add_property(PropertyNames.RESTRICTION_ENCODING, self._get_cloud_encoding(restriction_coder)) +from apache_beam.runners.common import DoFnSignature +if DoFnSignature(transform.dofn).is_stateful_dofn(): Review comment: Yes, java side we do the same. https://github.com/apache/beam/blob/2fd785dedb979a248e63c6385f978fd18fd2fbc4/runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowPipelineTranslator.java#L1260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373372) Time Spent: 1h 20m (was: 1h 10m) > Add uses_keyed_state step property to python dataflow runner > > > Key: BEAM-9122 > URL: https://issues.apache.org/jira/browse/BEAM-9122 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Add additional step property to dataflow job property when a DoFn is stateful > in python sdk. So that the backend runner can recognize stateful steps. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373369=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373369
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 03:23
Start Date: 17/Jan/20 03:23
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r367753262
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373370=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373370 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 03:23 Start Date: 17/Jan/20 03:23 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r367748598 ## File path: sdks/python/apache_beam/runners/worker/worker_status.py ## @@ -0,0 +1,139 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +"""Worker status api handler for reporting SDK harness debug info.""" + +from __future__ import absolute_import +from __future__ import division + +import queue +import sys +import threading +import traceback +from collections import defaultdict + +import grpc + +from apache_beam.portability.api import beam_fn_api_pb2 +from apache_beam.portability.api import beam_fn_api_pb2_grpc +from apache_beam.runners.worker.channel_factory import GRPCChannelFactory +from apache_beam.runners.worker.worker_id_interceptor import WorkerIdInterceptor + + +def thread_dump(): Review comment: We have thread dump code in `sdk_worker_main.py` under `get_thread_dump`. Shall we reuse it or move it here. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373370) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373366=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373366
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 03:23
Start Date: 17/Jan/20 03:23
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r367751685
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373368=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373368 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 03:23 Start Date: 17/Jan/20 03:23 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r367745344 ## File path: sdks/python/apache_beam/runners/test/utils.py ## @@ -0,0 +1,49 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +#http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +from __future__ import absolute_import + +import sys +import threading + +from future.utils import raise_ + + +def timeout(timeout_secs): Review comment: Let's add documentation and move it to `sdks/python/apache_beam/testing/util.py` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373368) Time Spent: 50m (was: 40m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373365=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373365 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 03:23 Start Date: 17/Jan/20 03:23 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r367746459 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -110,6 +112,15 @@ def __init__(self, data_channel_factory=self._data_channel_factory, fns=self._fns) +if status_address: Review comment: Let's move it in `sdk_worker_main` where we keep other reporting related code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373365) Time Spent: 40m (was: 0.5h) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[
https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373367=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373367
]
ASF GitHub Bot logged work on BEAM-8626:
Author: ASF GitHub Bot
Created on: 17/Jan/20 03:23
Start Date: 17/Jan/20 03:23
Worklog Time Spent: 10m
Work Description: angoenka commented on pull request #10598: [BEAM-8626]
Implement status fn api handler in python sdk
URL: https://github.com/apache/beam/pull/10598#discussion_r367752588
##
File path: sdks/python/apache_beam/runners/worker/worker_status.py
##
@@ -0,0 +1,139 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+#http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+"""Worker status api handler for reporting SDK harness debug info."""
+
+from __future__ import absolute_import
+from __future__ import division
+
+import queue
+import sys
+import threading
+import traceback
+from collections import defaultdict
+
+import grpc
+
+from apache_beam.portability.api import beam_fn_api_pb2
+from apache_beam.portability.api import beam_fn_api_pb2_grpc
+from apache_beam.runners.worker.channel_factory import GRPCChannelFactory
+from apache_beam.runners.worker.worker_id_interceptor import
WorkerIdInterceptor
+
+
+def thread_dump():
+ # deduplicate threads with same stack trace
+ stack_traces = defaultdict(list)
+ frames = sys._current_frames() # pylint: disable=protected-access
+
+ for t in threading.enumerate():
+stack_trace = ''.join(traceback.format_stack(frames[t.ident]))
+thread_ident_name = (t.ident, t.name)
+stack_traces[stack_trace].append(thread_ident_name)
+
+ all_traces = ['=' * 10 + 'THREAD DUMP' + '=' * 10]
+ for stack, identity in stack_traces.items():
+ident, name = identity[0]
+trace = '--- Thread #%s name: %s %s---\n' % (
+ident, name, 'and other %d threads' %
+(len(identity) - 1) if len(identity) > 1 else '')
+if len(identity) > 1:
+ trace += 'threads: %s\n' % identity
+trace += stack
+all_traces.append(trace)
+ all_traces.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in all_traces)
+
+
+def active_processing_bundles_state(bundle_process_cache):
+ active_bundles = ['=' * 10 + 'ACTIVE PROCESSING BUNDLES' + '=' * 10]
+ if not bundle_process_cache.active_bundle_processors:
+active_bundles.append("No active processing bundles.")
+ else:
+cache = []
+for instruction in list(
+bundle_process_cache.active_bundle_processors.keys()):
+ processor = bundle_process_cache.lookup(instruction)
+ if processor:
+info = processor.state_sampler.get_info()
+cache.append((instruction,
+ processor.process_bundle_descriptor.id,
+ info.tracked_thread, info.time_since_transition))
+# reverse sort active bundle by time since last transition, keep top 10.
+cache.sort(key=lambda x: x[-1], reverse=True)
+for s in cache[:10]:
+ state = '--- instruction %s ---\n' % s[0]
+ state += 'ProcessBundleDescriptorId: %s\n' % s[1]
+ state += "tracked thread: %s\n" % s[2]
+ state += "time since transition: %.2f seconds\n" % (s[3] / 1e9)
+ active_bundles.append(state)
+
+ active_bundles.append('=' * 30)
+ return '\n'.join(x.encode('utf-8') for x in active_bundles)
+
+
+DONE = object()
+
+
+class FnApiWorkerStatusHandler(object):
+ def __init__(self, status_address, bundle_process_cache=None):
+self._alive = True
+self._bundle_process_cache = bundle_process_cache
+ch = GRPCChannelFactory.insecure_channel(status_address)
+grpc.channel_ready_future(ch).result(timeout=60)
+self._status_channel = grpc.intercept_channel(ch, WorkerIdInterceptor())
+self._status_stub = beam_fn_api_pb2_grpc.BeamFnWorkerStatusStub(
+self._status_channel)
+self._responses = queue.Queue()
+self._server = threading.Thread(target=lambda: self._serve(),
+name='fn_api_status_handler')
+self._server.daemon = True
+self._server.start()
+
+ def _get_responses(self):
+while True:
+ response = self._responses.get()
+ if response is DONE:
+self._alive =
[jira] [Work logged] (BEAM-8626) Implement status api handler in python sdk harness
[ https://issues.apache.org/jira/browse/BEAM-8626?focusedWorklogId=373364=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373364 ] ASF GitHub Bot logged work on BEAM-8626: Author: ASF GitHub Bot Created on: 17/Jan/20 03:23 Start Date: 17/Jan/20 03:23 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10598: [BEAM-8626] Implement status fn api handler in python sdk URL: https://github.com/apache/beam/pull/10598#discussion_r367745627 ## File path: sdks/python/apache_beam/runners/worker/sdk_worker.py ## @@ -73,6 +74,7 @@ class SdkHarness(object): def __init__(self, control_address, # type: str + status_address=None, # type: Optional[str, unicode] Review comment: Let's add it as the last entry to keep the ordering of the arguments same and backward compatible, This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373364) Time Spent: 0.5h (was: 20m) > Implement status api handler in python sdk harness > -- > > Key: BEAM-8626 > URL: https://issues.apache.org/jira/browse/BEAM-8626 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-harness >Reporter: Yichi Zhang >Assignee: Yichi Zhang >Priority: Major > Time Spent: 0.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9141) Add test name label to pipelines created by tests
[ https://issues.apache.org/jira/browse/BEAM-9141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017681#comment-17017681 ] Udi Meiri commented on BEAM-9141: - [~tvalentyn] > Add test name label to pipelines created by tests > - > > Key: BEAM-9141 > URL: https://issues.apache.org/jira/browse/BEAM-9141 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Udi Meiri >Priority: Major > > This will make it easier to track down what test launched the pipeline (such > as when inspecting a Dataflow job). > In Python, this could be done via the --label pipeline option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373359=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373359
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 17/Jan/20 02:30
Start Date: 17/Jan/20 02:30
Worklog Time Spent: 10m
Work Description: boyuanzz commented on issue #10619: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619#issuecomment-575436945
Thanks Valentyn! I'll try to run python postcommit against release branch
again to verify.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373359)
Time Spent: 2h 40m (was: 2.5h)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 40m
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373356=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373356 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 17/Jan/20 02:07 Start Date: 17/Jan/20 02:07 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575378395 Java PreCommits failed because of test assertion change due to toString update. I missed this locally because I ran `./gradlew -p sdks/java check` (this does not run tests in runners). Python PreCommit failed: ``` E IndexError: index out of range ``` https://builds.apache.org/job/beam_PreCommit_Python_Commit/10812/testReport/junit/apache_beam.runners.portability.fn_api_runner_test/FnApiRunnerTestWithGrpc/test_large_elements_2/ Checking it with my local environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373356) Time Spent: 2.5h (was: 2h 20m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 2.5h > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide
[jira] [Work logged] (BEAM-9056) Staging artifacts from environment
[ https://issues.apache.org/jira/browse/BEAM-9056?focusedWorklogId=373354=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373354 ] ASF GitHub Bot logged work on BEAM-9056: Author: ASF GitHub Bot Created on: 17/Jan/20 02:03 Start Date: 17/Jan/20 02:03 Worklog Time Spent: 10m Work Description: ihji commented on pull request #10621: [WIP/BEAM-9056] Staging artifacts from environment URL: https://github.com/apache/beam/pull/10621 Work still in progress. Only for early review. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Updated] (BEAM-9142) Gradle command to run example on ZetaSQL
[ https://issues.apache.org/jira/browse/BEAM-9142?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kyle Weaver updated BEAM-9142: -- Status: Open (was: Triage Needed) > Gradle command to run example on ZetaSQL > > > Key: BEAM-9142 > URL: https://issues.apache.org/jira/browse/BEAM-9142 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: Major > > There should be an easy way to run a simple example on ZetaSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017633#comment-17017633
]
Chad Dombrova edited comment on BEAM-9130 at 1/17/20 1:37 AM:
--
IIRC, the reason it only happens for python2 is that the `google` package in
python2 uses .pth files to find sub-packages, and .pth files must be in a
"site" directory, so if it's installed in a non-standard way (into a directory
that isn't one of the python interpreter's official site-packages directories),
it won't be able to find the sub-packages. It is possible to mark a directory
as a "site" directory using site.addsitedir(). For more info see
[https://docs.python.org/2/library/site.html]
In python3 namespace packages were redesigned in order to avoid this brittle
workflow: [https://www.python.org/dev/peps/pep-0420/]
If this test went through tox, I'm pretty sure this error would not occur.
What are the chances that we can work on getting all tests running through tox
in the near term?
was (Author: chadrik):
IIRC, the reason it only happens for python2 is that the `google` package in
python2 uses .pth files to find sub-packages, and .pth files must be in a
"site" directory, so if it's installed in a non-standard way (one that doesn't
end up in the python interpreter's official site-packages directory), it won't
be able to find the sub-packages. It is possible to mark a directory as a
"site" directory using site.addsitedir(). For more info see
[https://docs.python.org/2/library/site.html]
In python3 namespace packages were redesigned in order to avoid this brittle
workflow: [https://www.python.org/dev/peps/pep-0420/]
If this test went through tox, I'm pretty sure this error would not occur.
What are the chances that we can work on getting all tests running through tox
in the near term?
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as
[jira] [Created] (BEAM-9142) Gradle command to run example on ZetaSQL
Kyle Weaver created BEAM-9142: - Summary: Gradle command to run example on ZetaSQL Key: BEAM-9142 URL: https://issues.apache.org/jira/browse/BEAM-9142 Project: Beam Issue Type: Improvement Components: dsl-sql-zetasql Reporter: Kyle Weaver Assignee: Kyle Weaver There should be an easy way to run a simple example on ZetaSQL. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017633#comment-17017633
]
Chad Dombrova commented on BEAM-9130:
-
IIRC, the reason it only happens for python2 is that the `google` package in
python2 uses .pth files to find sub-packages, and .pth files must be in a
"site" directory, so if it's installed in a non-standard way (one that doesn't
end up in the python interpreter's official site-packages directory), it won't
be able to find the sub-packages. It is possible to mark a directory as a
"site" directory using site.addsitedir(). For more info see
[https://docs.python.org/2/library/site.html]
In python3 namespace packages were redesigned in order to avoid this brittle
workflow: [https://www.python.org/dev/peps/pep-0420/]
If this test went through tox, I'm pretty sure this error would not occur.
What are the chances that we can work on getting all tests running through tox
in the near term?
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017621#comment-17017621
]
Udi Meiri commented on BEAM-9130:
-
Only Python 2 postcommit is affected, if that helps.
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9141) Add test name label to pipelines created by tests
Udi Meiri created BEAM-9141: --- Summary: Add test name label to pipelines created by tests Key: BEAM-9141 URL: https://issues.apache.org/jira/browse/BEAM-9141 Project: Beam Issue Type: Improvement Components: testing Reporter: Udi Meiri This will make it easier to track down what test launched the pipeline (such as when inspecting a Dataflow job). In Python, this could be done via the --label pipeline option. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017600#comment-17017600
]
Chad Dombrova commented on BEAM-9130:
-
It works for pre-commit, so I'm assuming that this goes back, once again, to
the inconsistent way that the beam python sdk is built.
Here's my old PR attempting to get everything using tox and pep517:
[https://github.com/apache/beam/pull/10038]
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (BEAM-8676) Beam Dependency Update Request: com.google.api:gax-grpc
[ https://issues.apache.org/jira/browse/BEAM-8676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomo Suzuki closed BEAM-8676. - Fix Version/s: 2.20.0 Not applicable Resolution: Fixed > Beam Dependency Update Request: com.google.api:gax-grpc > --- > > Key: BEAM-8676 > URL: https://issues.apache.org/jira/browse/BEAM-8676 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: Not applicable, 2.20.0 > > Time Spent: 20.5h > Remaining Estimate: 0h > > - 2019-11-15 19:38:32.410774 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.50.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:03:23.809273 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.50.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:08:16.165687 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.50.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:07:17.894174 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.51.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:07:26.564886 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.52.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:02:53.791811 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.52.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:06:28.934655 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.52.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:06:40.087956 > - > Please consider upgrading the dependency com.google.api:gax-grpc. > The current version is 1.38.0. The latest version is 1.53.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017595#comment-17017595 ] Tomo Suzuki commented on BEAM-8388: --- (Answering my question) bq. Why Aaron's Beam 2.16 was working? Was Avro's TimestampConversion introduced to Apache Beam after 2.16 release? Yes, this change was added since 2.17. https://github.com/apache/beam/commit/13cea03cb6e8f8f48e0ac0ae64d60928df1fd9d5 > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL: https://issues.apache.org/jira/browse/BEAM-8388 > Project: Beam > Issue Type: Improvement > Components: io-java-avro >Reporter: Jordanna Chord >Assignee: Jordanna Chord >Priority: Major > Original Estimate: 24h > Time Spent: 3h 20m > Remaining Estimate: 20h 40m > > Update build dependency to 1.9.1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373336=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373336 ] ASF GitHub Bot logged work on BEAM-9140: Author: ASF GitHub Bot Created on: 17/Jan/20 00:48 Start Date: 17/Jan/20 00:48 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10620: [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 URL: https://github.com/apache/beam/pull/10620#issuecomment-575415279 run zetasql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373336) Time Spent: 0.5h (was: 20m) > Update to ZetaSQL 2020.01.1 > --- > > Key: BEAM-9140 > URL: https://issues.apache.org/jira/browse/BEAM-9140 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Fix For: 2.20.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=373335=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373335 ] ASF GitHub Bot logged work on BEAM-9140: Author: ASF GitHub Bot Created on: 17/Jan/20 00:48 Start Date: 17/Jan/20 00:48 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10620: [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 URL: https://github.com/apache/beam/pull/10620#issuecomment-575415238 run sql postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373335) Time Spent: 20m (was: 10m) > Update to ZetaSQL 2020.01.1 > --- > > Key: BEAM-9140 > URL: https://issues.apache.org/jira/browse/BEAM-9140 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Fix For: 2.20.0 > > Time Spent: 20m > Remaining Estimate: 0h > > I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?focusedWorklogId=37=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-37 ] ASF GitHub Bot logged work on BEAM-9140: Author: ASF GitHub Bot Created on: 17/Jan/20 00:47 Start Date: 17/Jan/20 00:47 Worklog Time Spent: 10m Work Description: apilloud commented on pull request #10620: [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 URL: https://github.com/apache/beam/pull/10620 [BEAM-9140] Upgrade to ZetaSQL 2020.01.1 Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373332=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373332
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 17/Jan/20 00:44
Start Date: 17/Jan/20 00:44
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10619: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619#issuecomment-575414443
@boyuanzz also, heads-up that
https://issues.apache.org/jira/browse/BEAM-9127
https://issues.apache.org/jira/browse/BEAM-9130
may also be failing on the release branch, if the commits that triggered
these recent bugs are included in the release branch. I did not check that.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373332)
Time Spent: 2.5h (was: 2h 20m)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9140) Update to ZetaSQL 2020.01.1
Andrew Pilloud created BEAM-9140: Summary: Update to ZetaSQL 2020.01.1 Key: BEAM-9140 URL: https://issues.apache.org/jira/browse/BEAM-9140 Project: Beam Issue Type: Improvement Components: dsl-sql-zetasql Reporter: Andrew Pilloud Assignee: Andrew Pilloud Fix For: 2.20.0 I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9140) Update to ZetaSQL 2020.01.1
[ https://issues.apache.org/jira/browse/BEAM-9140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Pilloud updated BEAM-9140: - Status: Open (was: Triage Needed) > Update to ZetaSQL 2020.01.1 > --- > > Key: BEAM-9140 > URL: https://issues.apache.org/jira/browse/BEAM-9140 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Andrew Pilloud >Assignee: Andrew Pilloud >Priority: Major > Fix For: 2.20.0 > > > I hear ZetaSQL 2020.01.1 will be coming out in the next few hours. We should > upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=373331=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373331 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 17/Jan/20 00:34 Start Date: 17/Jan/20 00:34 Worklog Time Spent: 10m Work Description: milantracy commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-575412136 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373331) Time Spent: 1h 50m (was: 1h 40m) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 1h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8739) Consistently use with Pipeline(...) syntax
[ https://issues.apache.org/jira/browse/BEAM-8739?focusedWorklogId=373328=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373328 ] ASF GitHub Bot logged work on BEAM-8739: Author: ASF GitHub Bot Created on: 17/Jan/20 00:30 Start Date: 17/Jan/20 00:30 Worklog Time Spent: 10m Work Description: robertwb commented on pull request #10149: [BEAM-8739] Consistently use with Pipeline(...) syntax URL: https://github.com/apache/beam/pull/10149 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373328) Time Spent: 1.5h (was: 1h 20m) > Consistently use with Pipeline(...) syntax > -- > > Key: BEAM-8739 > URL: https://issues.apache.org/jira/browse/BEAM-8739 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Robert Bradshaw >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > > I've run into a couple of tests that forgot to do p.run(). In addition, I'm > seeing new tests written in this old style. We should consistently use the > with syntax where possible for our examples and tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017592#comment-17017592
]
Valentyn Tymofieiev commented on BEAM-9061:
---
Cleaning /tmp directory will not be sufficient, since we can download
pyhamcrest==2.0.0 again when running tests against branches that did not pick
up #10568 (such as 2.18.0, 2.19.0)
Sent CP for 2.19.0. Thanks for highlighting this.
https://github.com/apache/beam/pull/10619
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373326=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373326
]
ASF GitHub Bot logged work on BEAM-9061:
Author: ASF GitHub Bot
Created on: 17/Jan/20 00:25
Start Date: 17/Jan/20 00:25
Worklog Time Spent: 10m
Work Description: tvalentyn commented on issue #10619: [BEAM-9061] Add
version guards to requirements file for integration tests.
URL: https://github.com/apache/beam/pull/10619#issuecomment-575410118
R: @boyuanzz
cc: @udim
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373326)
Time Spent: 2h 20m (was: 2h 10m)
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h 20m
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[ https://issues.apache.org/jira/browse/BEAM-9061?focusedWorklogId=373325=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373325 ] ASF GitHub Bot logged work on BEAM-9061: Author: ASF GitHub Bot Created on: 17/Jan/20 00:24 Start Date: 17/Jan/20 00:24 Worklog Time Spent: 10m Work Description: tvalentyn commented on pull request #10619: [BEAM-9061] Add version guards to requirements file for integration tests. URL: https://github.com/apache/beam/pull/10619 This is a CP of edfc95bb65478619c49da1af5b4ac1db1cabd727 to 2.19.0 release branch. Without this change, Python 2 postcommits may be failing on the release branch due to BEAM-9061. Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [![Build
[jira] [Work logged] (BEAM-9083) PR9677 breaks ValidatesRunnerTest of open source runners
[ https://issues.apache.org/jira/browse/BEAM-9083?focusedWorklogId=373322=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373322 ] ASF GitHub Bot logged work on BEAM-9083: Author: ASF GitHub Bot Created on: 17/Jan/20 00:20 Start Date: 17/Jan/20 00:20 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10615: [BEAM-9083] Cherry-pick PR10610: Exclude testOutputTimestamp from flink streaming tests. URL: https://github.com/apache/beam/pull/10615 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373322) Time Spent: 1h 10m (was: 1h) > PR9677 breaks ValidatesRunnerTest of open source runners > - > > Key: BEAM-9083 > URL: https://issues.apache.org/jira/browse/BEAM-9083 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Shehzaad Nakhoda >Priority: Blocker > Fix For: 2.19.0 > > Time Spent: 1h 10m > Remaining Estimate: 0h > > Breakages in: > Java_ValidatesRunner_Samza: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/ > Java_ValidatesRunner_Spark: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/ > Java_PVR_Flink_Streaming: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming > Java_PVR_Flink_Batch: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017587#comment-17017587
]
Valentyn Tymofieiev commented on BEAM-9061:
---
Beam users will not be affected by the problem, but beam release managers may
be affected.
The solution is to cherrypick the changes in this bug to release branches, or
clean up jenkins tmp directory.
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017582#comment-17017582
]
Valentyn Tymofieiev edited comment on BEAM-9061 at 1/17/20 12:15 AM:
-
This is because we stage entire tmp directory to the Dataflow workers [1]
and pip does not check package metadata when installing dependencies with
--find-links command, such as:
pip install pyhamcrest --find-links
[1]
https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E
was (Author: tvalentyn):
This is because we stage entire tmp directory to the Dataflow workers [1]
and pip does not check package metadata when installing dependencies with
--find-links command, such as:
pip install pyhamcrest --find-links
[1]
https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017582#comment-17017582
]
Valentyn Tymofieiev commented on BEAM-9061:
---
This is because we stage entire tmp directory to the Dataflow workers [1]
and pip does not check package metadata when installing dependencies with
--find-links command, such as:
pip install pyhamcrest --find-links
[1]
https://lists.apache.org/thread.html/f99820489026a46160505ea9a4c8c4a0eabe8649143c1ee22b7088aa%40%3Cdev.beam.apache.org%3E
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=373321=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373321 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 17/Jan/20 00:14 Start Date: 17/Jan/20 00:14 Worklog Time Spent: 10m Work Description: milantracy commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-575407224 @amaliujia do you want to take a look? Feel free to tag anyone who is free for a code review This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373321) Time Spent: 1h 40m (was: 1.5h) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9139) release validation: Python Leaderboard with DirectRunner fails
[
https://issues.apache.org/jira/browse/BEAM-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri closed BEAM-9139.
---
Fix Version/s: Not applicable
Resolution: Won't Fix
> release validation: Python Leaderboard with DirectRunner fails
> --
>
> Key: BEAM-9139
> URL: https://issues.apache.org/jira/browse/BEAM-9139
> Project: Beam
> Issue Type: Bug
> Components: build-system, sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>
> {code}
> *
> * Running Python Leaderboard with DirectRunner
> *
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/__init__.py:84:
> UserWarning: You are using Apache Beam with Python 2. New releases of Apache
> Beam will soon support Python 3 only.
> 'You are using Apache Beam with Python 2. '
> INFO:root:Missing pipeline option (runner). Executing pipeline using the
> default runner: DirectRunner.
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1220:
> BeamDeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported
> experiments = p.options.view_as(DebugOptions).experiments or []
> INFO:root:Running pipeline with DirectRunner.
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/_default.py:69:
> UserWarning: Your application has authenticated using end user credentials
> from Google Cloud SDK. We recommend that most server applications use service
> accounts instead. If your application continues to use end user credentials
> from Cloud SDK, you might receive a "quota exceeded" or "API not enabled"
> error. For more information about service accounts, see
> https://cloud.google.com/docs/authentication/
> warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
> ERROR:concurrent.futures:exception calling callback for 0x7f7c6f37add0 state=finished raised IOError>
> Traceback (most recent call last):
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
> line 331, in _invoke_callbacks
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
> line 75, in wrapped
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
> line 455, in result
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/thread.py",
> line 63, in run
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
> line 67, in _get_authorization_headers
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/credentials.py",
> line 124, in before_request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/credentials.py",
> line 182, in refresh
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
> line 248, in refresh_grant
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
> line 105, in _token_endpoint_request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/requests.py",
> line 169, in __call__
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
> line 533, in request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
> line 646, in send
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
> line 416, in send
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
> line 228, in cert_verify
> IOError: Could not find a suitable TLS CA certificate bundle, invalid path:
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/certifi/cacert.pem
> {code}
> This IOError repeats until the job fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=373320=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373320 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 17/Jan/20 00:13 Start Date: 17/Jan/20 00:13 Worklog Time Spent: 10m Work Description: milantracy commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-575406833 Run SQL PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373320) Time Spent: 1.5h (was: 1h 20m) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 1.5h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9139) release validation: Python Leaderboard with DirectRunner fails
[
https://issues.apache.org/jira/browse/BEAM-9139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017581#comment-17017581
]
Udi Meiri commented on BEAM-9139:
-
I think this error may be due to teardown of the tmp directory, due to some
other test failure or other bug.
> release validation: Python Leaderboard with DirectRunner fails
> --
>
> Key: BEAM-9139
> URL: https://issues.apache.org/jira/browse/BEAM-9139
> Project: Beam
> Issue Type: Bug
> Components: build-system, sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
>
> {code}
> *
> * Running Python Leaderboard with DirectRunner
> *
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/__init__.py:84:
> UserWarning: You are using Apache Beam with Python 2. New releases of Apache
> Beam will soon support Python 3 only.
> 'You are using Apache Beam with Python 2. '
> INFO:root:Missing pipeline option (runner). Executing pipeline using the
> default runner: DirectRunner.
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1220:
> BeamDeprecationWarning: options is deprecated since First stable release.
> References to .options will not be supported
> experiments = p.options.view_as(DebugOptions).experiments or []
> INFO:root:Running pipeline with DirectRunner.
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/_default.py:69:
> UserWarning: Your application has authenticated using end user credentials
> from Google Cloud SDK. We recommend that most server applications use service
> accounts instead. If your application continues to use end user credentials
> from Cloud SDK, you might receive a "quota exceeded" or "API not enabled"
> error. For more information about service accounts, see
> https://cloud.google.com/docs/authentication/
> warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
> ERROR:concurrent.futures:exception calling callback for 0x7f7c6f37add0 state=finished raised IOError>
> Traceback (most recent call last):
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
> line 331, in _invoke_callbacks
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
> line 75, in wrapped
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
> line 455, in result
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/thread.py",
> line 63, in run
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
> line 67, in _get_authorization_headers
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/credentials.py",
> line 124, in before_request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/credentials.py",
> line 182, in refresh
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
> line 248, in refresh_grant
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
> line 105, in _token_endpoint_request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/requests.py",
> line 169, in __call__
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
> line 533, in request
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
> line 646, in send
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
> line 416, in send
> File
> "/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
> line 228, in cert_verify
> IOError: Could not find a suitable TLS CA certificate bundle, invalid path:
> /tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/certifi/cacert.pem
> {code}
> This IOError repeats until the job fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-5504) PubsubAvroTable
[ https://issues.apache.org/jira/browse/BEAM-5504?focusedWorklogId=373319=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373319 ] ASF GitHub Bot logged work on BEAM-5504: Author: ASF GitHub Bot Created on: 17/Jan/20 00:12 Start Date: 17/Jan/20 00:12 Worklog Time Spent: 10m Work Description: milantracy commented on issue #10487: [BEAM-5504] Introduce PubsubAvroTable URL: https://github.com/apache/beam/pull/10487#issuecomment-575406597 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373319) Time Spent: 1h 20m (was: 1h 10m) > PubsubAvroTable > --- > > Key: BEAM-5504 > URL: https://issues.apache.org/jira/browse/BEAM-5504 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Rui Wang >Assignee: Jing Chen >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7136) Beam documentation inconsistent
[ https://issues.apache.org/jira/browse/BEAM-7136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aizhamal Nurmamat kyzy reassigned BEAM-7136: Assignee: Rose Nguyen (was: Aizhamal Nurmamat kyzy) > Beam documentation inconsistent > --- > > Key: BEAM-7136 > URL: https://issues.apache.org/jira/browse/BEAM-7136 > Project: Beam > Issue Type: Bug > Components: beam-model, website >Reporter: Daniel Collins >Assignee: Rose Nguyen >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Apache beam documentation is inconsistent about Python usage of Pub/Sub. The > chart [here|[https://beam.apache.org/documentation/io/built-in/]] claims that > Pub/Sub is available for usage in Batch Pipelines, but the source [from > clicking on that > link|[https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/gcp/pubsub.py]] > explicitly says only streaming is supported. Both of these cannot be > correct. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9139) release validation: Python Leaderboard with DirectRunner fails
Udi Meiri created BEAM-9139:
---
Summary: release validation: Python Leaderboard with DirectRunner
fails
Key: BEAM-9139
URL: https://issues.apache.org/jira/browse/BEAM-9139
Project: Beam
Issue Type: Bug
Components: build-system, sdk-py-core
Reporter: Udi Meiri
{code}
*
* Running Python Leaderboard with DirectRunner
*
/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/__init__.py:84:
UserWarning: You are using Apache Beam with Python 2. New releases of Apache
Beam will soon support Python 3 only.
'You are using Apache Beam with Python 2. '
INFO:root:Missing pipeline option (runner). Executing pipeline using the
default runner: DirectRunner.
/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/apache_beam/io/gcp/bigquery.py:1220:
BeamDeprecationWarning: options is deprecated since First stable release.
References to .options will not be supported
experiments = p.options.view_as(DebugOptions).experiments or []
INFO:root:Running pipeline with DirectRunner.
/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/_default.py:69:
UserWarning: Your application has authenticated using end user credentials
from Google Cloud SDK. We recommend that most server applications use service
accounts instead. If your application continues to use end user credentials
from Cloud SDK, you might receive a "quota exceeded" or "API not enabled"
error. For more information about service accounts, see
https://cloud.google.com/docs/authentication/
warnings.warn(_CLOUD_SDK_CREDENTIALS_WARNING)
ERROR:concurrent.futures:exception calling callback for
Traceback (most recent call last):
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
line 331, in _invoke_callbacks
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
line 75, in wrapped
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/_base.py",
line 455, in result
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/concurrent/futures/thread.py",
line 63, in run
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/grpc.py",
line 67, in _get_authorization_headers
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/credentials.py",
line 124, in before_request
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/credentials.py",
line 182, in refresh
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
line 248, in refresh_grant
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/oauth2/_client.py",
line 105, in _token_endpoint_request
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/google/auth/transport/requests.py",
line 169, in __call__
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
line 533, in request
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/sessions.py",
line 646, in send
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
line 416, in send
File
"/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/requests/adapters.py",
line 228, in cert_verify
IOError: Could not find a suitable TLS CA certificate bundle, invalid path:
/tmp/beam-release.2.18.0.DCDdsy/beam_env_python2.7/local/lib/python2.7/site-packages/certifi/cacert.pem
{code}
This IOError repeats until the job fails.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017572#comment-17017572
]
Udi Meiri commented on BEAM-9061:
-
I've manually patching in these changes to run postcommits locally on the 2.18
release branch:
{code}
echo "pyhamcrest!=1.10.0,<2.0.0" > postcommit_requirements.txt
echo "mock<3.0.0" >> postcommit_requirements.txt
{code}
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017572#comment-17017572
]
Udi Meiri edited comment on BEAM-9061 at 1/16/20 11:48 PM:
---
I've manually patched in these changes to run postcommits locally on the 2.18
release branch:
{code}
echo "pyhamcrest!=1.10.0,<2.0.0" > postcommit_requirements.txt
echo "mock<3.0.0" >> postcommit_requirements.txt
{code}
was (Author: udim):
I've manually patching in these changes to run postcommits locally on the 2.18
release branch:
{code}
echo "pyhamcrest!=1.10.0,<2.0.0" > postcommit_requirements.txt
echo "mock<3.0.0" >> postcommit_requirements.txt
{code}
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9127) postcommit: suites:portable:py2:crossLanguagePortableWordCount failing
[
https://issues.apache.org/jira/browse/BEAM-9127?focusedWorklogId=373307=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373307
]
ASF GitHub Bot logged work on BEAM-9127:
Author: ASF GitHub Bot
Created on: 16/Jan/20 23:36
Start Date: 16/Jan/20 23:36
Worklog Time Spent: 10m
Work Description: robertwb commented on pull request #10605: [BEAM-9127]
Fix output type declaration in xlang wordcount.
URL: https://github.com/apache/beam/pull/10605
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373307)
Time Spent: 2h (was: 1h 50m)
> postcommit: suites:portable:py2:crossLanguagePortableWordCount failing
> --
>
> Key: BEAM-9127
> URL: https://issues.apache.org/jira/browse/BEAM-9127
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Log:
> https://builds.apache.org/job/beam_PostCommit_Python2/1445/timestamps/?time=HH:mm:ss=GMT-8=en_US
> {code}
> 11:32:07 RuntimeError: Pipeline
> BeamApp-root-0114193141-d68e793b_21a90e10-d9a1-41e8-bdba-154dddc532f3 failed
> in state FAILED: java.lang.RuntimeException: Error received from SDK harness
> for instruction 16: Traceback (most recent call last):
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 165, in _execute
> 11:32:07 response = task()
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 221, in
> 11:32:07 lambda: self.create_worker().do_instruction(request), request)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 350, in do_instruction
> 11:32:07 request.instruction_id)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 384, in process_bundle
> 11:32:07 bundle_processor.process_bundle(instruction_id))
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 811, in process_bundle
> 11:32:07 data.transform_id].process_encoded(data.data)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 206, in process_encoded
> 11:32:07 self.output(decoded_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 302, in
> apache_beam.runners.worker.operations.Operation.output
> 11:32:07 def output(self, windowed_value, output_index=0):
> 11:32:07File "apache_beam/runners/worker/operations.py", line 304, in
> apache_beam.runners.worker.operations.Operation.output
> 11:32:07 cython.cast(Receiver,
> self.receivers[output_index]).receive(windowed_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 178, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> 11:32:07 self.consumer.process(windowed_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 751, in
> apache_beam.runners.worker.operations.SdfProcessSizedElements.process
> 11:32:07 with self.scoped_process_state:
> 11:32:07File "apache_beam/runners/worker/operations.py", line 759, in
> apache_beam.runners.worker.operations.SdfProcessSizedElements.process
> 11:32:07 delayed_application =
> self.dofn_runner.process_with_sized_restriction(o)
> 11:32:07File "apache_beam/runners/common.py", line 888, in
> apache_beam.runners.common.DoFnRunner.process_with_sized_restriction
> 11:32:07 return self.do_fn_invoker.invoke_process(
> 11:32:07File "apache_beam/runners/common.py", line 657, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> 11:32:07 return self._invoke_process_per_window(
> 11:32:07File "apache_beam/runners/common.py", line 744, in
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
> 11:32:07 output_processor.process_outputs(
> 11:32:07File "apache_beam/runners/common.py", line 984, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> 11:32:07 def process_outputs(self, windowed_input_element, results):
> 11:32:07File "apache_beam/runners/common.py", line 1024, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> 11:32:07
[jira] [Commented] (BEAM-9130) sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with ImportError: No module named google.protobuf.message
[
https://issues.apache.org/jira/browse/BEAM-9130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017569#comment-17017569
]
Udi Meiri commented on BEAM-9130:
-
[~chadrik] This looks related to your recent PR:
https://github.com/apache/beam/pull/10367/files#diff-24277f98ee26157938eb64d5fe6e3c7eR65
> sdks:python:test-suites:direct:py2:hdfsIntegrationTest is failing with
> ImportError: No module named google.protobuf.message
> ---
>
> Key: BEAM-9130
> URL: https://issues.apache.org/jira/browse/BEAM-9130
> Project: Beam
> Issue Type: Improvement
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: Major
> Labels: currently-failing
>
> From logs:
> {noformat}
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 267, in _bootstrap
> 16:33:50 [0m[91mself.run()
> 16:33:50File "/usr/local/lib/python2.7/multiprocessing/process.py", line
> 114, in run
> 16:33:50 self._target(*self._args, **self._kwargs)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 357, in
> _install_grpcio_tools_and_generate_proto_files
> 16:33:50 generate_proto_files(force=force)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 324, in
> generate_proto_files
> 16:33:50 generate_urn_files(log, out_dir)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 65, in
> generate_urn_files
> 16:33:50 import google.protobuf.message as message
> 16:33:50 [0m[91mImportError: No module named google.protobuf.message
> 16:33:50 [0m[91mTraceback (most recent call last):
> 16:33:50File "setup.py", line 305, in
> 16:33:50 'mypy': generate_protos_first(mypy),
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/__init__.py", line 145, in
> setup
> 16:33:50 [0m[91mreturn distutils.core.setup(**attrs)
> 16:33:50File "/usr/local/lib/python2.7/distutils/core.py", line 151, in
> setup
> 16:33:50 [0m[91mdist.run_commands()
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 953, in
> run_commands
> 16:33:50 [0m[91mself.run_command(cmd)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 cmd_obj.run()
> 16:33:50File
> "/usr/local/lib/python2.7/site-packages/setuptools/command/sdist.py", line
> 44, in run
> 16:33:50 [0m[91mself.run_command('egg_info')
> 16:33:50File "/usr/local/lib/python2.7/distutils/cmd.py", line 326, in
> run_command
> 16:33:50 [0m[91mself.distribution.run_command(command)
> 16:33:50File "/usr/local/lib/python2.7/distutils/dist.py", line 972, in
> run_command
> 16:33:50 [0m[91mcmd_obj.run()
> 16:33:50File "setup.py", line 229, in run
> 16:33:50 [0m[91mgen_protos.generate_proto_files(log=log)
> 16:33:50File "/app/sdks/python/gen_protos.py", line 291, in
> generate_proto_files
> 16:33:50 raise ValueError("Proto generation failed (see log for
> details).")
> 16:33:50 [0m[91mValueError: [0m[91mProto generation failed (see log for
> details
> {noformat}
> {noformat}
> import google.protobuf.message as message
> ImportError: No module named google.protobuf.message
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9061) New release of pyhamcrest==1.10.0 breaks portable Python precommits.
[
https://issues.apache.org/jira/browse/BEAM-9061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017565#comment-17017565
]
Udi Meiri commented on BEAM-9061:
-
Note that pyhamcrest 2.0.0 is out now, which fails py2 postcommits
(worker-startup fails):
{code}
I 2020-01-16T22:53:58.107488Z Processing
/var/opt/google/dataflow/PyHamcrest-2.0.0.tar.gz
I 2020-01-16T22:53:58.384311Z ERROR: Package 'PyHamcrest' requires a different
Python: 2.7.13 not in '>=3.5'
I 2020-01-16T22:53:58.414867Z /usr/local/bin/pip failed with exit status 1
{code}
Would suggest cherrypicking into 2.19 if this is an issue, since
postcommit_requirements.txt has no version pinning.
> New release of pyhamcrest==1.10.0 breaks portable Python precommits.
>
>
> Key: BEAM-9061
> URL: https://issues.apache.org/jira/browse/BEAM-9061
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Valentyn Tymofieiev
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 2h
> Remaining Estimate: 0h
>
> {noformat}
> 08:35:26 File "apache_beam/runners/portability/fn_api_runner_test.py", line
> 38, in
> 08:35:26 import hamcrest # pylint: disable=ungrouped-imports
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/__init__.py",
> line 2, in
> 08:35:26 from hamcrest.library import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/__init__.py",
> line 7, in
> 08:35:26 from hamcrest.library.object import *
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/__init__.py",
> line 4, in
> 08:35:26 from .hasproperty import has_properties, has_property
> 08:35:26 File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python2_PVR_Flink_Commit/src/build/gradleenv/1866363813/local/lib/python2.7/site-packages/hamcrest/library/object/hasproperty.py",
> line 174
> 08:35:26 ),
> 08:35:26 ^
> {noformat}
> rootcause: https://github.com/hamcrest/PyHamcrest/issues/131
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9078) Large Tarball Artifacts Should Use GCS Resumable Upload
[ https://issues.apache.org/jira/browse/BEAM-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad West resolved BEAM-9078. - Resolution: Fixed > Large Tarball Artifacts Should Use GCS Resumable Upload > --- > > Key: BEAM-9078 > URL: https://issues.apache.org/jira/browse/BEAM-9078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Brad West >Assignee: Brad West >Priority: Major > Fix For: 2.19.0 > > Original Estimate: 1h > Time Spent: 40m > Remaining Estimate: 20m > > It's possible for the tarball uploaded to GCS to be quite large. An example > is a user vendoring multiple dependencies in their tarball so as to achieve a > more stable deployable artifact. > Before this change the GCS upload api call executed a multipart upload, which > Google > [documentation]([https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload)] > states should be used when the file is small enough to upload again when the > connection fails. For large tarballs, we will hit 60 second socket timeouts > before completing the multipart upload. By passing `total_size`, apitools > first checks if the size exceeds the resumable upload threshold, and executes > the more robust resumable upload rather than a multipart, avoiding > socket timeouts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-9078) Large Tarball Artifacts Should Use GCS Resumable Upload
[ https://issues.apache.org/jira/browse/BEAM-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brad West closed BEAM-9078. --- > Large Tarball Artifacts Should Use GCS Resumable Upload > --- > > Key: BEAM-9078 > URL: https://issues.apache.org/jira/browse/BEAM-9078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Brad West >Assignee: Brad West >Priority: Major > Fix For: 2.19.0 > > Original Estimate: 1h > Time Spent: 40m > Remaining Estimate: 20m > > It's possible for the tarball uploaded to GCS to be quite large. An example > is a user vendoring multiple dependencies in their tarball so as to achieve a > more stable deployable artifact. > Before this change the GCS upload api call executed a multipart upload, which > Google > [documentation]([https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload)] > states should be used when the file is small enough to upload again when the > connection fails. For large tarballs, we will hit 60 second socket timeouts > before completing the multipart upload. By passing `total_size`, apitools > first checks if the size exceeds the resumable upload threshold, and executes > the more robust resumable upload rather than a multipart, avoiding > socket timeouts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9138) beam_Release_Gradle_Build failure in Go
Udi Meiri created BEAM-9138:
---
Summary: beam_Release_Gradle_Build failure in Go
Key: BEAM-9138
URL: https://issues.apache.org/jira/browse/BEAM-9138
Project: Beam
Issue Type: Bug
Components: build-system, test-failures
Reporter: Udi Meiri
Phrase: "Run Release Gradle Build"
Recent failures:
https://builds.apache.org/job/beam_Release_Gradle_Build/24/
https://builds.apache.org/job/beam_Release_Gradle_Build/25/
Seems like all of these are Go-related, and it seems like the culprit might be
the failure in :sdks:go:resolveBuildDependencies (but I have very little
knowledge of these tasks).
CC: [~markflyhigh] who wrote the job, [~lostluck] who knows Go SDK
Please help in triaging and/or assigning to someone who can look at this.
{code}
13:00:15 Execution failed for task ':beam-test-tools:goVet'.
13:00:15 > Build failed due to return code 1 of:
13:00:15Command:
13:00:15 /usr/bin/go vet github.com/apache/beam/.test-infra/tools
{code}
{code}
13:00:15 Execution failed for task ':beam-test-tools:goTest'.
13:00:15 > There are 1 failed tests. Please see
/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_Build/src/.test-infra/tools/.gogradle/reports/test/index.html
for more details.
{code}
{code}
13:00:15 Execution failed for task ':beam-test-tools:buildLinuxAmd64'.
13:00:15 > Build failed due to return code 1 of:
13:00:15Command:
13:00:15 /usr/bin/go build -o ./.gogradle/beam-test-tools-linux-amd64
github.com/apache/beam/.test-infra/tools
{code}
{code}
13:00:15 Execution failed for task ':sdks:go:resolveBuildDependencies'.
13:00:15 > Exception in resolution, message is:
13:00:15Cannot resolve dependency:google.golang.org/api:
commit='386d4e5f4f92f86e6aec85985761bba4b938a2d5',
urls=[https://code.googlesource.com/google-api-go-client]
13:00:15Resolution stack is:
13:00:15+- github.com/apache/beam/sdks/go
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373297=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373297 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 22:45 Start Date: 16/Jan/20 22:45 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575378395 Java PreCommits failed because of test assertion change due to toString update. I missed this locally because I ran `./gradlew -p sdks/java check` (this does not run tests in runners). Python PreCommit failed: ``` E IndexError: index out of range ``` https://builds.apache.org/job/beam_PreCommit_Python_Commit/10812/testReport/junit/apache_beam.runners.portability.fn_api_runner_test/FnApiRunnerTestWithGrpc/test_large_elements/ Checking it with my local environment. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373297) Time Spent: 2h 20m (was: 2h 10m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 2h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373296=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373296 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 22:44 Start Date: 16/Jan/20 22:44 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575378395 Java PreCommits failed because of test assertion change due to toString update. I missed this locally because I ran `./gradlew -p sdks/java check` (this does not run tests in runners). Python PreCommit failed: ``` E IndexError: index out of range ``` https://builds.apache.org/job/beam_PreCommit_Python_Commit/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373296) Time Spent: 2h 10m (was: 2h) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 2h 10m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373292=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373292 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 22:33 Start Date: 16/Jan/20 22:33 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575378395 Java PreCommits failed because of test assertion change due to toString update. Python PreCommit failed: ``` E IndexError: index out of range ``` https://builds.apache.org/job/beam_PreCommit_Python_Commit/ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373292) Time Spent: 2h (was: 1h 50m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 2h > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency >
[jira] [Work logged] (BEAM-9127) postcommit: suites:portable:py2:crossLanguagePortableWordCount failing
[
https://issues.apache.org/jira/browse/BEAM-9127?focusedWorklogId=373290=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373290
]
ASF GitHub Bot logged work on BEAM-9127:
Author: ASF GitHub Bot
Created on: 16/Jan/20 22:30
Start Date: 16/Jan/20 22:30
Worklog Time Spent: 10m
Work Description: robertwb commented on issue #10605: [BEAM-9127] Fix
output type declaration in xlang wordcount.
URL: https://github.com/apache/beam/pull/10605#issuecomment-575377290
There's the HDFS failure, but should we get this merged in the meantime?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373290)
Time Spent: 1h 50m (was: 1h 40m)
> postcommit: suites:portable:py2:crossLanguagePortableWordCount failing
> --
>
> Key: BEAM-9127
> URL: https://issues.apache.org/jira/browse/BEAM-9127
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Udi Meiri
>Assignee: Robert Bradshaw
>Priority: Major
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Log:
> https://builds.apache.org/job/beam_PostCommit_Python2/1445/timestamps/?time=HH:mm:ss=GMT-8=en_US
> {code}
> 11:32:07 RuntimeError: Pipeline
> BeamApp-root-0114193141-d68e793b_21a90e10-d9a1-41e8-bdba-154dddc532f3 failed
> in state FAILED: java.lang.RuntimeException: Error received from SDK harness
> for instruction 16: Traceback (most recent call last):
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 165, in _execute
> 11:32:07 response = task()
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 221, in
> 11:32:07 lambda: self.create_worker().do_instruction(request), request)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 350, in do_instruction
> 11:32:07 request.instruction_id)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 384, in process_bundle
> 11:32:07 bundle_processor.process_bundle(instruction_id))
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 811, in process_bundle
> 11:32:07 data.transform_id].process_encoded(data.data)
> 11:32:07File
> "/usr/local/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 206, in process_encoded
> 11:32:07 self.output(decoded_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 302, in
> apache_beam.runners.worker.operations.Operation.output
> 11:32:07 def output(self, windowed_value, output_index=0):
> 11:32:07File "apache_beam/runners/worker/operations.py", line 304, in
> apache_beam.runners.worker.operations.Operation.output
> 11:32:07 cython.cast(Receiver,
> self.receivers[output_index]).receive(windowed_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 178, in
> apache_beam.runners.worker.operations.SingletonConsumerSet.receive
> 11:32:07 self.consumer.process(windowed_value)
> 11:32:07File "apache_beam/runners/worker/operations.py", line 751, in
> apache_beam.runners.worker.operations.SdfProcessSizedElements.process
> 11:32:07 with self.scoped_process_state:
> 11:32:07File "apache_beam/runners/worker/operations.py", line 759, in
> apache_beam.runners.worker.operations.SdfProcessSizedElements.process
> 11:32:07 delayed_application =
> self.dofn_runner.process_with_sized_restriction(o)
> 11:32:07File "apache_beam/runners/common.py", line 888, in
> apache_beam.runners.common.DoFnRunner.process_with_sized_restriction
> 11:32:07 return self.do_fn_invoker.invoke_process(
> 11:32:07File "apache_beam/runners/common.py", line 657, in
> apache_beam.runners.common.PerWindowInvoker.invoke_process
> 11:32:07 return self._invoke_process_per_window(
> 11:32:07File "apache_beam/runners/common.py", line 744, in
> apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
> 11:32:07 output_processor.process_outputs(
> 11:32:07File "apache_beam/runners/common.py", line 984, in
> apache_beam.runners.common._OutputProcessor.process_outputs
> 11:32:07 def process_outputs(self, windowed_input_element, results):
> 11:32:07File "apache_beam/runners/common.py",
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373287=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373287 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 16/Jan/20 22:28 Start Date: 16/Jan/20 22:28 Worklog Time Spent: 10m Work Description: 11moon11 commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575376707 Tests are running, but looks like GitHub UI did not catch that, links: - [beam_PreCommit_Java_Commit](https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_Java_Commit/9667/) - [beam_PreCommit_JavaBeamZetaSQL_Commit](https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_JavaBeamZetaSQL_Commit/882/) - [beam_PreCommit_SQL_Commit](https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_SQL_Commit/444/) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373287) Time Spent: 4h 10m (was: 4h) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h 10m > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7861) Make it easy to change between multi-process and multi-thread mode for Python Direct runners
[ https://issues.apache.org/jira/browse/BEAM-7861?focusedWorklogId=373282=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373282 ] ASF GitHub Bot logged work on BEAM-7861: Author: ASF GitHub Bot Created on: 16/Jan/20 22:26 Start Date: 16/Jan/20 22:26 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on pull request #10616: [WIP][BEAM-7861] update documentation URL: https://github.com/apache/beam/pull/10616 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[![Build
[jira] [Work logged] (BEAM-9072) [SQL] Add support for Datastore source
[ https://issues.apache.org/jira/browse/BEAM-9072?focusedWorklogId=373281=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373281 ] ASF GitHub Bot logged work on BEAM-9072: Author: ASF GitHub Bot Created on: 16/Jan/20 22:22 Start Date: 16/Jan/20 22:22 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10440: [BEAM-9072] [SQL] DataStoreV1 IO connector URL: https://github.com/apache/beam/pull/10440#issuecomment-575374615 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373281) Time Spent: 4h (was: 3h 50m) > [SQL] Add support for Datastore source > -- > > Key: BEAM-9072 > URL: https://issues.apache.org/jira/browse/BEAM-9072 > Project: Beam > Issue Type: New Feature > Components: dsl-sql >Reporter: Kirill Kozlov >Assignee: Kirill Kozlov >Priority: Major > Time Spent: 4h > Remaining Estimate: 0h > > * Create a Datastore table and table provider > * Conversion between Datastore and Beam data types > * Implement buildIOReader > * Implement buildIOWrite > * Implement getTableStatistics > Doc: > [https://docs.google.com/document/d/1FxuEGewJ3GPDl0IKglfOYf1edwa2m_wryFZYRMpRNbA/edit?pli=1] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519
]
Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM:
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
#
##
## ```
was (Author: atdixon):
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
```
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
## at org.apache.beam.sdk.coders.AvroCoder.
([AvroCoder.java:269|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=269=flow-p1-9c38=1])
## at org.apache.beam.sdk.coders.AvroCoder.of
([AvroCoder.java:121|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=121=flow-p1-9c38=1])
## at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
([KafkaUnboundedSource.java:131|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.io.kafka%2FKafkaUnboundedSource.java=131=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
([WorkerCustomSources.java:419|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FWorkerCustomSources.java=419=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
([ReadOperation.java:178|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=178=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
([ReadOperation.java:159|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=159=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
([MapTaskExecutor.java:77|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java=77=flow-p1-9c38=1])
## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
([StreamingDataflowWorker.java:1320|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1320=flow-p1-9c38=1])
## at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
[jira] [Commented] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[ https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519 ] Aaron Dixon commented on BEAM-8388: --- Just to add more context/help here. The exception I see after upgrading from Beam 2.16->2.17 is this (running in Dataflow): ``` java.lang.NoClassDefFoundError: org/apache/avro/data/TimeConversions$TimestampConversion # ## at org.apache.beam.sdk.coders.AvroCoder. ([AvroCoder.java:269|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=269=flow-p1-9c38=1]) ## at org.apache.beam.sdk.coders.AvroCoder.of ([AvroCoder.java:121|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.coders%2FAvroCoder.java=121=flow-p1-9c38=1]) ## at org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder ([KafkaUnboundedSource.java:131|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.sdk.io.kafka%2FKafkaUnboundedSource.java=131=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator ([WorkerCustomSources.java:419|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FWorkerCustomSources.java=419=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop ([ReadOperation.java:178|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=178=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start ([ReadOperation.java:159|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FReadOperation.java=159=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute ([MapTaskExecutor.java:77|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker.util.common.worker%2FMapTaskExecutor.java=77=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process ([StreamingDataflowWorker.java:1320|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1320=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000 ([StreamingDataflowWorker.java:151|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=151=flow-p1-9c38=1]) ## at org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run ([StreamingDataflowWorker.java:1053|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=org%2Fapache.beam.runners.dataflow.worker%2FStreamingDataflowWorker.java=1053=flow-p1-9c38=1]) # ## at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1142) ## at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:617) ## at java.lang.Thread.run (Thread.java:748) Caused by: java.lang.ClassNotFoundException: org.apache.avro.data.TimeConversions$TimestampConversion # ## at java.net.URLClassLoader.findClass ([URLClassLoader.java:381|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Fnet%2FURLClassLoader.java=381=flow-p1-9c38=1]) ## at java.lang.ClassLoader.loadClass ([ClassLoader.java:424|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Flang%2FClassLoader.java=424=flow-p1-9c38=1]) ## at sun.misc.Launcher$AppClassLoader.loadClass ([Launcher.java:335|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=sun%2Fmisc%2FLauncher.java=335=flow-p1-9c38=1]) ## at java.lang.ClassLoader.loadClass ([ClassLoader.java:357|https://console.cloud.google.com/debug/fromlog?appModule=Dataflow%20Jobs=2020-01-15_09_46_24-15077023656704045070=java%2Flang%2FClassLoader.java=357=flow-p1-9c38=1]) ## ``` > Update Avro to 1.9.1 from 1.8.2 > --- > > Key: BEAM-8388 > URL:
[jira] [Comment Edited] (BEAM-8388) Update Avro to 1.9.1 from 1.8.2
[
https://issues.apache.org/jira/browse/BEAM-8388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017519#comment-17017519
]
Aaron Dixon edited comment on BEAM-8388 at 1/16/20 9:51 PM:
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
was (Author: atdixon):
Just to add more context/help here. The exception I see after upgrading from
Beam 2.16->2.17 is this (running in Dataflow):
{code:java}
java.lang.NoClassDefFoundError:
org/apache/avro/data/TimeConversions$TimestampConversion #
at org.apache.beam.sdk.coders.AvroCoder. (AvroCoder.java:269) at
org.apache.beam.sdk.coders.AvroCoder.of (AvroCoder.java:121) at
org.apache.beam.sdk.io.kafka.KafkaUnboundedSource.getCheckpointMarkCoder
(KafkaUnboundedSource.java:131) at
org.apache.beam.runners.dataflow.worker.WorkerCustomSources$UnboundedReader.iterator
(WorkerCustomSources.java:419) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.runReadLoop
(ReadOperation.java:178) at
org.apache.beam.runners.dataflow.worker.util.common.worker.ReadOperation.start
(ReadOperation.java:159) at
org.apache.beam.runners.dataflow.worker.util.common.worker.MapTaskExecutor.execute
(MapTaskExecutor.java:77) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.process
(StreamingDataflowWorker.java:1320) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker.access$1000
(StreamingDataflowWorker.java:151) at
org.apache.beam.runners.dataflow.worker.StreamingDataflowWorker$6.run
(StreamingDataflowWorker.java:1053) at
java.util.concurrent.ThreadPoolExecutor.runWorker
(ThreadPoolExecutor.java:1142) at
java.util.concurrent.ThreadPoolExecutor$Worker.run
(ThreadPoolExecutor.java:617) at java.lang.Thread.run (Thread.java:748)
Caused by: java.lang.ClassNotFoundException:
org.apache.avro.data.TimeConversions$TimestampConversion # at
java.net.URLClassLoader.findClass (URLClassLoader.java:381) at
java.lang.ClassLoader.loadClass (ClassLoader.java:424) at
sun.misc.Launcher$AppClassLoader.loadClass (Launcher.java:335) at
java.lang.ClassLoader.loadClass (ClassLoader.java:357) {code}
#
##
## ```
> Update Avro to 1.9.1 from 1.8.2
> ---
>
> Key: BEAM-8388
> URL: https://issues.apache.org/jira/browse/BEAM-8388
> Project: Beam
> Issue Type: Improvement
> Components: io-java-avro
>Reporter: Jordanna Chord
>Assignee: Jordanna Chord
>Priority: Major
> Original Estimate: 24h
> Time Spent: 3h 20m
> Remaining Estimate: 20h 40m
>
> Update build dependency to 1.9.1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=373273=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373273 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 16/Jan/20 21:50 Start Date: 16/Jan/20 21:50 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10612: [NOT READY TO MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-575363276 To reviewers, please take additional care to review if I missed any places should be updated but missed from the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373273) Time Spent: 0.5h (was: 20m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 0.5h > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9063) Migrate docker images to apache namespace.
[ https://issues.apache.org/jira/browse/BEAM-9063?focusedWorklogId=373272=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373272 ] ASF GitHub Bot logged work on BEAM-9063: Author: ASF GitHub Bot Created on: 16/Jan/20 21:46 Start Date: 16/Jan/20 21:46 Worklog Time Spent: 10m Work Description: Hannah-Jiang commented on issue #10612: [NOT READY TO MERGE][BEAM-9063] migrate docker images to apache URL: https://github.com/apache/beam/pull/10612#issuecomment-575362138 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373272) Time Spent: 20m (was: 10m) > Migrate docker images to apache namespace. > -- > > Key: BEAM-9063 > URL: https://issues.apache.org/jira/browse/BEAM-9063 > Project: Beam > Issue Type: Task > Components: beam-community >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > https://hub.docker.com/u/apache -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-7926) Show PCollection with Interactive Beam in a data-centric user flow
[
https://issues.apache.org/jira/browse/BEAM-7926?focusedWorklogId=373271=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373271
]
ASF GitHub Bot logged work on BEAM-7926:
Author: ASF GitHub Bot
Created on: 16/Jan/20 21:43
Start Date: 16/Jan/20 21:43
Worklog Time Spent: 10m
Work Description: KevinGG commented on issue #10346: [BEAM-7926]
Data-centric Interactive Part2
URL: https://github.com/apache/beam/pull/10346#issuecomment-575360825
Rebased to upstream head.
retest this please.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373271)
Time Spent: 33.5h (was: 33h 20m)
> Show PCollection with Interactive Beam in a data-centric user flow
> --
>
> Key: BEAM-7926
> URL: https://issues.apache.org/jira/browse/BEAM-7926
> Project: Beam
> Issue Type: New Feature
> Components: runner-py-interactive
>Reporter: Ning Kang
>Assignee: Ning Kang
>Priority: Major
> Time Spent: 33.5h
> Remaining Estimate: 0h
>
> Support auto plotting / charting of materialized data of a given PCollection
> with Interactive Beam.
> Say an Interactive Beam pipeline defined as
>
> {code:java}
> p = beam.Pipeline(InteractiveRunner())
> pcoll = p | 'Transform' >> transform()
> pcoll2 = ...
> pcoll3 = ...{code}
> The use can call a single function and get auto-magical charting of the data.
> e.g.,
> {code:java}
> show(pcoll, pcoll2)
> {code}
> Throughout the process, a pipeline fragment is built to include only
> transforms necessary to produce the desired pcolls (pcoll and pcoll2) and
> execute that fragment.
> This makes the Interactive Beam user flow data-centric.
>
> Detailed
> [design|https://docs.google.com/document/d/1DYWrT6GL_qDCXhRMoxpjinlVAfHeVilK5Mtf8gO6zxQ/edit#heading=h.v6k2o3roarzz].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9083) PR9677 breaks ValidatesRunnerTest of open source runners
[ https://issues.apache.org/jira/browse/BEAM-9083?focusedWorklogId=373261=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373261 ] ASF GitHub Bot logged work on BEAM-9083: Author: ASF GitHub Bot Created on: 16/Jan/20 21:14 Start Date: 16/Jan/20 21:14 Worklog Time Spent: 10m Work Description: boyuanzz commented on pull request #10615: [BEAM-9083] Exclude testOutputTimestamp from flink streaming tests. URL: https://github.com/apache/beam/pull/10615 cherry picked from commit b0f1dcc50f10c647dfda0f4a2a545331bf9d789f: PR10610 R: @aaltay This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373261) Time Spent: 1h (was: 50m) > PR9677 breaks ValidatesRunnerTest of open source runners > - > > Key: BEAM-9083 > URL: https://issues.apache.org/jira/browse/BEAM-9083 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Shehzaad Nakhoda >Priority: Blocker > Fix For: 2.19.0 > > Time Spent: 1h > Remaining Estimate: 0h > > Breakages in: > Java_ValidatesRunner_Samza: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/ > Java_ValidatesRunner_Spark: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/ > Java_PVR_Flink_Streaming: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming > Java_PVR_Flink_Batch: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373258=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373258 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:11 Start Date: 16/Jan/20 21:11 Worklog Time Spent: 10m Work Description: suztomo commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575348722 @apilloud Thank you! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373258) Time Spent: 1h 50m (was: 1h 40m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1h 50m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373257=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373257 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:10 Start Date: 16/Jan/20 21:10 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575348093 Run SQL Postcommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373257) Time Spent: 1h 40m (was: 1.5h) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1h 40m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373255=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373255 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575348000 Run Dataflow ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373255) Time Spent: 1h 20m (was: 1h 10m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1h 20m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373253=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373253 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575347903 Run Java HadoopFormatIO Performance Test This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373253) Time Spent: 1h (was: 50m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1h > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373254=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373254 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575347943 Run BigQueryIO Streaming Performance Test Java This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373254) Time Spent: 1h 10m (was: 1h) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1h 10m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373256=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373256 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575348056 Run Spark ValidatesRunner This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373256) Time Spent: 1.5h (was: 1h 20m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 1.5h > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373252=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373252 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575347866 Run Java PostCommit This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373252) Time Spent: 50m (was: 40m) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 50m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373251=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373251 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 21:09 Start Date: 16/Jan/20 21:09 Worklog Time Spent: 10m Work Description: apilloud commented on issue #10614: [BEAM-8695] Upgrade google-http-client to 1.34.0 URL: https://github.com/apache/beam/pull/10614#issuecomment-575347827 retest this please This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373251) Time Spent: 40m (was: 0.5h) > Beam Dependency Update Request: com.google.http-client:google-http-client > - > > Key: BEAM-8695 > URL: https://issues.apache.org/jira/browse/BEAM-8695 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Priority: Major > Attachments: > LengthPrefixUnknownCodersTest.testLengthPrefixUnknownCoders.png, > image-2019-12-19-17-03-44-487.png > > Time Spent: 40m > Remaining Estimate: 0h > > - 2019-11-15 19:40:13.570557 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:20.477284 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:12.146269 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:24.693912 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.33.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:18.407974 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:06:51.597773 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:26.135863 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:09:56.623584 > - > Please consider upgrading the dependency > com.google.http-client:google-http-client. > The current version is 1.28.0. The latest version is 1.34.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for
[jira] [Commented] (BEAM-9078) Large Tarball Artifacts Should Use GCS Resumable Upload
[ https://issues.apache.org/jira/browse/BEAM-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017481#comment-17017481 ] Boyuan Zhang commented on BEAM-9078: Thanks Brad! unfortunately, I don't thin k the issue will be marked as resolved automatically. If you think this issue has been addressed, please close it manually. Thanks for your contribution! > Large Tarball Artifacts Should Use GCS Resumable Upload > --- > > Key: BEAM-9078 > URL: https://issues.apache.org/jira/browse/BEAM-9078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Brad West >Assignee: Brad West >Priority: Major > Fix For: 2.19.0 > > Original Estimate: 1h > Time Spent: 40m > Remaining Estimate: 20m > > It's possible for the tarball uploaded to GCS to be quite large. An example > is a user vendoring multiple dependencies in their tarball so as to achieve a > more stable deployable artifact. > Before this change the GCS upload api call executed a multipart upload, which > Google > [documentation]([https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload)] > states should be used when the file is small enough to upload again when the > connection fails. For large tarballs, we will hit 60 second socket timeouts > before completing the multipart upload. By passing `total_size`, apitools > first checks if the size exceeds the resumable upload threshold, and executes > the more robust resumable upload rather than a multipart, avoiding > socket timeouts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9083) PR9677 breaks ValidatesRunnerTest of open source runners
[ https://issues.apache.org/jira/browse/BEAM-9083?focusedWorklogId=373249=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373249 ] ASF GitHub Bot logged work on BEAM-9083: Author: ASF GitHub Bot Created on: 16/Jan/20 21:01 Start Date: 16/Jan/20 21:01 Worklog Time Spent: 10m Work Description: boyuanzz commented on issue #10610: [BEAM-9083] Exclude testOutputTimestamp from flink streaming tests. URL: https://github.com/apache/beam/pull/10610#issuecomment-575344691 Thank @mxm for your quick review! I'm going to cherry-pick this commit into release branch now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 373249) Time Spent: 50m (was: 40m) > PR9677 breaks ValidatesRunnerTest of open source runners > - > > Key: BEAM-9083 > URL: https://issues.apache.org/jira/browse/BEAM-9083 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Shehzaad Nakhoda >Priority: Blocker > Fix For: 2.19.0 > > Time Spent: 50m > Remaining Estimate: 0h > > Breakages in: > Java_ValidatesRunner_Samza: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/ > Java_ValidatesRunner_Spark: > https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/ > Java_PVR_Flink_Streaming: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming > Java_PVR_Flink_Batch: > https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9078) Large Tarball Artifacts Should Use GCS Resumable Upload
[ https://issues.apache.org/jira/browse/BEAM-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017479#comment-17017479 ] Brad West commented on BEAM-9078: - Hmm, I assumed the Github/Jira integration (is there not one?) would automatically update this ticket. PR was merged and fix is included in the release-2.19.0 branch. Do I mark as resolved or wait for automation to take care of this ticket? First time contributer, so please advise. Thanks > Large Tarball Artifacts Should Use GCS Resumable Upload > --- > > Key: BEAM-9078 > URL: https://issues.apache.org/jira/browse/BEAM-9078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Brad West >Assignee: Brad West >Priority: Major > Fix For: 2.19.0 > > Original Estimate: 1h > Time Spent: 40m > Remaining Estimate: 20m > > It's possible for the tarball uploaded to GCS to be quite large. An example > is a user vendoring multiple dependencies in their tarball so as to achieve a > more stable deployable artifact. > Before this change the GCS upload api call executed a multipart upload, which > Google > [documentation]([https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload)] > states should be used when the file is small enough to upload again when the > connection fails. For large tarballs, we will hit 60 second socket timeouts > before completing the multipart upload. By passing `total_size`, apitools > first checks if the size exceeds the resumable upload threshold, and executes > the more robust resumable upload rather than a multipart, avoiding > socket timeouts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8696) Beam Dependency Update Request: com.google.protobuf:protobuf-java
[ https://issues.apache.org/jira/browse/BEAM-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tomo Suzuki closed BEAM-8696. - Resolution: Fixed [~boyuanz] Resolved in master! https://github.com/apache/beam/pull/10498. Marking 2.19.0 tentatively. > Beam Dependency Update Request: com.google.protobuf:protobuf-java > - > > Key: BEAM-8696 > URL: https://issues.apache.org/jira/browse/BEAM-8696 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h > Remaining Estimate: 0h > > - 2019-11-15 19:40:18.666442 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.10.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:31.077226 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.10.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:27.069873 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:39.779191 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.1 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:45.319428 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:07:08.620760 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:42.949907 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:10:14.224482 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-6857) Support dynamic timers
[
https://issues.apache.org/jira/browse/BEAM-6857?focusedWorklogId=373244=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373244
]
ASF GitHub Bot logged work on BEAM-6857:
Author: ASF GitHub Bot
Created on: 16/Jan/20 20:23
Start Date: 16/Jan/20 20:23
Worklog Time Spent: 10m
Work Description: reuvenlax commented on pull request #10316: [BEAM-6857]
Support Dynamic Timers
URL: https://github.com/apache/beam/pull/10316
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373244)
Time Spent: 22h 50m (was: 22h 40m)
> Support dynamic timers
> --
>
> Key: BEAM-6857
> URL: https://issues.apache.org/jira/browse/BEAM-6857
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Reuven Lax
>Assignee: Shehzaad Nakhoda
>Priority: Major
> Time Spent: 22h 50m
> Remaining Estimate: 0h
>
> The Beam timers API currently requires each timer to be statically specified
> in the DoFn. The user must provide a separate callback method per timer. For
> example:
>
> {code:java}
> DoFn()
> {
> @TimerId("timer1")
> private final TimerSpec timer1 = TimerSpecs.timer(...);
> @TimerId("timer2")
> private final TimerSpec timer2 = TimerSpecs.timer(...);
> .. set timers in processElement
> @OnTimer("timer1")
> public void onTimer1() { .}
> @OnTimer("timer2")
> public void onTimer2() {}
> }
> {code}
>
> However there are many cases where the user does not know the set of timers
> statically when writing their code. This happens when the timer tag should be
> based on the data. It also happens when writing a DSL on top of Beam, where
> the DSL author has to create DoFns but does not know statically which timers
> their users will want to set (e.g. Scio).
>
> The goal is to support dynamic timers. Something as follows;
>
> {code:java}
> DoFn()
> {
> @TimerId("timer")
> private final TimerSpec timer1 = TimerSpecs.dynamicTimer(...);
> @ProcessElement process(@TimerId("timer") DynamicTimer timer)
> {
> timer.set("tag1'", ts);
>timer.set("tag2", ts);
> }
> @OnTimer("timer")
> public void onTimer1(@TimerTag String tag) { .}
> }
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-8695) Beam Dependency Update Request: com.google.http-client:google-http-client
[ https://issues.apache.org/jira/browse/BEAM-8695?focusedWorklogId=373243=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373243 ] ASF GitHub Bot logged work on BEAM-8695: Author: ASF GitHub Bot Created on: 16/Jan/20 20:22 Start Date: 16/Jan/20 20:22 Worklog Time Spent: 10m Work Description: suztomo commented on pull request #10614: [BEAM-8695] google-http-client 1.34.0 URL: https://github.com/apache/beam/pull/10614 https://issues.apache.org/jira/browse/BEAM-8695 Upgrading google-http-client dependency to 1.34.0 (latest). # Linkage Check result Following modules use google-http-client. ## beam-sdks-java-io-google-cloud-platform before: after: ## beam-sdks-java-testing-test-utils before: after: ## beam-sdks-extensions-google-cloud-platform-core before: after: ## beam-runners-google-cloud-dataflow-java before: after: ## beam-runners-google-cloud-dataflow-java-worker-legacy-worker before: after: ## beam-runners-google-cloud-dataflow-java-worker before: after: Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[![Build
[jira] [Commented] (BEAM-9078) Large Tarball Artifacts Should Use GCS Resumable Upload
[ https://issues.apache.org/jira/browse/BEAM-9078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017453#comment-17017453 ] Boyuan Zhang commented on BEAM-9078: Hey, 2.19.0 release branch has been cut. Any updates on this issue? > Large Tarball Artifacts Should Use GCS Resumable Upload > --- > > Key: BEAM-9078 > URL: https://issues.apache.org/jira/browse/BEAM-9078 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Affects Versions: 2.17.0 >Reporter: Brad West >Assignee: Brad West >Priority: Major > Fix For: 2.19.0 > > Original Estimate: 1h > Time Spent: 40m > Remaining Estimate: 20m > > It's possible for the tarball uploaded to GCS to be quite large. An example > is a user vendoring multiple dependencies in their tarball so as to achieve a > more stable deployable artifact. > Before this change the GCS upload api call executed a multipart upload, which > Google > [documentation]([https://cloud.google.com/storage/docs/json_api/v1/how-tos/upload)] > states should be used when the file is small enough to upload again when the > connection fails. For large tarballs, we will hit 60 second socket timeouts > before completing the multipart upload. By passing `total_size`, apitools > first checks if the size exceeds the resumable upload threshold, and executes > the more robust resumable upload rather than a multipart, avoiding > socket timeouts. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9065) Spark runner accumulates metrics (incorrectly) between runs
[ https://issues.apache.org/jira/browse/BEAM-9065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017450#comment-17017450 ] Boyuan Zhang commented on BEAM-9065: Hey, just wanna let you know that 2.19.0 release branch has been cut. If any changes applied to 2.19.0, please cherry-pick it into release branch. > Spark runner accumulates metrics (incorrectly) between runs > --- > > Key: BEAM-9065 > URL: https://issues.apache.org/jira/browse/BEAM-9065 > Project: Beam > Issue Type: Bug > Components: runner-spark >Reporter: Etienne Chauchot >Assignee: Etienne Chauchot >Priority: Major > Fix For: 2.19.0 > > Time Spent: 40m > Remaining Estimate: 0h > > When pipeline.run() is called, MetricsAccumulator (wrapper of > MetricsContainerStepMap spark accumulator) is initialized. Spark needs this > class to be a singleton for failover. The problem is that when several > pipelines are run inside the same JVM, the initialization of > MetricsAccumulator singleton does not reset the underlying spark accumulator > causing metrics to be accumulated between runs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8696) Beam Dependency Update Request: com.google.protobuf:protobuf-java
[ https://issues.apache.org/jira/browse/BEAM-8696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17017442#comment-17017442 ] Boyuan Zhang commented on BEAM-8696: Hey Tomo, any updates on this issue? > Beam Dependency Update Request: com.google.protobuf:protobuf-java > - > > Key: BEAM-8696 > URL: https://issues.apache.org/jira/browse/BEAM-8696 > Project: Beam > Issue Type: Sub-task > Components: dependencies >Reporter: Beam JIRA Bot >Assignee: Tomo Suzuki >Priority: Major > Fix For: 2.19.0 > > Time Spent: 2h > Remaining Estimate: 0h > > - 2019-11-15 19:40:18.666442 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.10.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-11-19 21:06:31.077226 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.10.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-02 12:12:27.069873 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.0 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-09 12:11:39.779191 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.1 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-23 12:11:45.319428 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-12-30 14:07:08.620760 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-06 12:10:42.949907 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2020-01-13 12:10:14.224482 > - > Please consider upgrading the dependency > com.google.protobuf:protobuf-java. > The current version is 3.7.1. The latest version is 3.11.2 > cc: [~chamikara], > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[
https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=373241=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373241
]
ASF GitHub Bot logged work on BEAM-9093:
Author: ASF GitHub Bot
Created on: 16/Jan/20 20:02
Start Date: 16/Jan/20 20:02
Worklog Time Spent: 10m
Work Description: angoenka commented on issue #10613: [BEAM-9093] Log
invalid overwrites in pipeline options
URL: https://github.com/apache/beam/pull/10613#issuecomment-575321578
R: @robertwb @ihji
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 373241)
Time Spent: 20m (was: 10m)
> Pipeline options which with different underlying store variable does not get
> over written
> -
>
> Key: BEAM-9093
> URL: https://issues.apache.org/jira/browse/BEAM-9093
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Ankur Goenka
>Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Example:
> PipelineOptions(flags=[],**\{'no_use_public_ips': True,})
> Expectation: use_public_ips should be set False.
> Actual: the value is not used as its not passed through argparser
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Work logged] (BEAM-9093) Pipeline options which with different underlying store variable does not get over written
[ https://issues.apache.org/jira/browse/BEAM-9093?focusedWorklogId=373240=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-373240 ] ASF GitHub Bot logged work on BEAM-9093: Author: ASF GitHub Bot Created on: 16/Jan/20 20:02 Start Date: 16/Jan/20 20:02 Worklog Time Spent: 10m Work Description: angoenka commented on pull request #10613: [BEAM-9093] Log invalid overwrites in pipeline options URL: https://github.com/apache/beam/pull/10613 **Please** add a meaningful description for your change here Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily: - [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`). - [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue. - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf). See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier). Post-Commit Tests Status (on master branch) Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark --- | --- | --- | --- | --- | --- | --- | --- Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/) Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/) Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)[