[jira] [Assigned] (BEAM-10217) Suboptimal type inference on Python 3.6
[
https://issues.apache.org/jira/browse/BEAM-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-10217:
Assignee: Udi Meiri
> Suboptimal type inference on Python 3.6
> ---
>
> Key: BEAM-10217
> URL: https://issues.apache.org/jira/browse/BEAM-10217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: P2
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Following trivial_inference_test fails for me with AssertionError: ::
> List[Tuple[Person, int]] != List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6
> and 3.7.3.
> {noformat}
> def testMethodReturnsTuple(self):
> class Person(object):
> def __init__(self, name):
> self.name = name
> def parse_record(name_and_age):
> name, age = name_and_age.split(',')
> return Person(name), int(age)
> self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
> lambda record: [parse_record(record)], [str])
> {noformat}
> See: https://github.com/apache/beam/pull/11954.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-10217) Suboptimal type inference on Python 3.6
[
https://issues.apache.org/jira/browse/BEAM-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-10217:
-
Status: Open (was: Triage Needed)
> Suboptimal type inference on Python 3.6
> ---
>
> Key: BEAM-10217
> URL: https://issues.apache.org/jira/browse/BEAM-10217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: P2
> Time Spent: 40m
> Remaining Estimate: 0h
>
> Following trivial_inference_test fails for me with AssertionError: ::
> List[Tuple[Person, int]] != List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6
> and 3.7.3.
> {noformat}
> def testMethodReturnsTuple(self):
> class Person(object):
> def __init__(self, name):
> self.name = name
> def parse_record(name_and_age):
> name, age = name_and_age.split(',')
> return Person(name), int(age)
> self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
> lambda record: [parse_record(record)], [str])
> {noformat}
> See: https://github.com/apache/beam/pull/11954.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-8153) PubSubIntegrationTest failing in post-commit
[
https://issues.apache.org/jira/browse/BEAM-8153?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-8153.
-
Fix Version/s: Not applicable
Resolution: Fixed
> PubSubIntegrationTest failing in post-commit
>
>
> Key: BEAM-8153
> URL: https://issues.apache.org/jira/browse/BEAM-8153
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Udi Meiri
>Assignee: Matthew Darwin
>Priority: P2
> Labels: stale-assigned
> Fix For: Not applicable
>
>
> Most likely due to: https://github.com/apache/beam/pull/9232
> {code}
> 11:44:31
> ==
> 11:44:31 ERROR: test_streaming_with_attributes
> (apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest)
> 11:44:31
> --
> 11:44:31 Traceback (most recent call last):
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py",
> line 199, in test_streaming_with_attributes
> 11:44:31 self._test_streaming(with_attributes=True)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/io/gcp/pubsub_integration_test.py",
> line 191, in _test_streaming
> 11:44:31 timestamp_attribute=self.TIMESTAMP_ATTRIBUTE)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/io/gcp/pubsub_it_pipeline.py",
> line 91, in run_pipeline
> 11:44:31 result = p.run()
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/pipeline.py",
> line 420, in run
> 11:44:31 return self.runner.run_pipeline(self, self._options)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 51, in run_pipeline
> 11:44:31 hc_assert_that(self.result, pickler.loads(on_success_matcher))
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/site-packages/hamcrest/core/assert_that.py",
> line 43, in assert_that
> 11:44:31 _assert_match(actual=arg1, matcher=arg2, reason=arg3)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/site-packages/hamcrest/core/assert_that.py",
> line 49, in _assert_match
> 11:44:31 if not matcher.matches(actual):
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/site-packages/hamcrest/core/core/allof.py",
> line 16, in matches
> 11:44:31 if not matcher.matches(item):
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/site-packages/hamcrest/core/base_matcher.py",
> line 28, in matches
> 11:44:31 match_result = self._matches(item)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/io/gcp/tests/pubsub_matcher.py",
> line 91, in _matches
> 11:44:31 return Counter(self.messages) == Counter(self.expected_msg)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/collections/__init__.py",
> line 566, in __init__
> 11:44:31 self.update(*args, **kwds)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/build/gradleenv/1398941891/lib/python3.7/collections/__init__.py",
> line 653, in update
> 11:44:31 _count_elements(self, iterable)
> 11:44:31 File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37_PR/src/sdks/python/apache_beam/io/gcp/pubsub.py",
> line 83, in __hash__
> 11:44:31 self.message_id, self.publish_time.seconds,
> 11:44:31 AttributeError: 'NoneType' object has no attribute 'seconds'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-10193) Update Jenkins VMs with docker-credential-gcloud
[ https://issues.apache.org/jira/browse/BEAM-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126125#comment-17126125 ] Udi Meiri commented on BEAM-10193: -- cc: [~tysonjh] > Update Jenkins VMs with docker-credential-gcloud > > > Key: BEAM-10193 > URL: https://issues.apache.org/jira/browse/BEAM-10193 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Brian Hulette >Priority: P3 > > See BEAM-7405 (test failure currently resolved with an inelegant workaround) > for motivation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125481#comment-17125481
]
Udi Meiri commented on BEAM-7405:
-
Well there was the workaround I put in place, but I left the bug open for
option no. 5 which required a VM image update.
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Udi Meiri
>Priority: P2
> Labels: stale-assigned
> Fix For: 2.14.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10191) _find_protoc_gen_mypy: Windows support
Udi Meiri created BEAM-10191: Summary: _find_protoc_gen_mypy: Windows support Key: BEAM-10191 URL: https://issues.apache.org/jira/browse/BEAM-10191 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Assignee: Chad Dombrova "pip install -e .[gcp, test]" fails. The script is looking for protoc-gen-mypy when it should be looking for protoc_gen_mypy.bat. https://github.com/dropbox/mypy-protobuf#python-implementation -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8473) Python typehints: spurious warnings for Generator None args
[ https://issues.apache.org/jira/browse/BEAM-8473?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125157#comment-17125157 ] Udi Meiri commented on BEAM-8473: - I'll work on it. > Python typehints: spurious warnings for Generator None args > --- > > Key: BEAM-8473 > URL: https://issues.apache.org/jira/browse/BEAM-8473 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: P2 > Labels: stale-assigned > > GeneratorHint outputs warnings if the 2 last args of typing.Generator are not > None. > 1. Compare to type(None) instead of None, since that's what the code converts > those to. (assert on logging output) > 2. Bug: comparing send_type in both cases. > https://github.com/apache/beam/blob/06fa1cc2a92a6f4f0e05f162ee2eb78b072920d9/sdks/python/apache_beam/typehints/typehints.py#L1088-L1091 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8843) Limit bigquery.py exponential backoff to 60s
[ https://issues.apache.org/jira/browse/BEAM-8843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124306#comment-17124306 ] Udi Meiri commented on BEAM-8843: - Was this fixed? > Limit bigquery.py exponential backoff to 60s > > > Key: BEAM-8843 > URL: https://issues.apache.org/jira/browse/BEAM-8843 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Pablo Estrada >Priority: P2 > Labels: stale-assigned > > Seen here: > https://issues.apache.org/jira/browse/BEAM-8842?focusedCommentId=16983995=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16983995 > retry delays grow quickly into the 1000s range. > cc [~chamikara] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9064) Add pytype to lint checks
[ https://issues.apache.org/jira/browse/BEAM-9064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124305#comment-17124305 ] Udi Meiri commented on BEAM-9064: - Might add this (there was a pending PR that got stale), as long as it doesn't affect lint precommit times too greatly. > Add pytype to lint checks > - > > Key: BEAM-9064 > URL: https://issues.apache.org/jira/browse/BEAM-9064 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core, testing >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: P2 > Labels: stale-assigned > Time Spent: 1h > Remaining Estimate: 0h > > [~chadrik] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8156) Finish migration to standard Python typing
[ https://issues.apache.org/jira/browse/BEAM-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124303#comment-17124303 ] Udi Meiri edited comment on BEAM-8156 at 6/2/20, 8:47 PM: -- Current state is that internal types will continue to be used in the IOTypeHints container and the 3 places in the original description. Perhaps this will change if we implement type checking on Python's typing module types, or to use an external type checking library (such as pytypes). In this case, we would have to solve the pickling issue somehow (either write a serializer or omit them -- how does standard pickling handle __annotations__?). was (Author: udim): Current thoughts are that internal types will continue to be used in the IOTypeHints container and the 3 places in the original description. Perhaps this will change if we rewrite these 3 places to work on Python's typing module types, or to use an external type checking library (such as pytypes). In this case, we would have to solve the pickling issue somehow (either write a serializer or omit them -- how does standard pickling handle __annotations__?). > Finish migration to standard Python typing > -- > > Key: BEAM-8156 > URL: https://issues.apache.org/jira/browse/BEAM-8156 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: P2 > Labels: stale-assigned > Original Estimate: 504h > Time Spent: 1h 40m > Remaining Estimate: 502h 20m > > We should migrate all Python uses of types to the standard typing module, and > make the typehints.* ones aliases of the Python ones. > > There are three places where we use custom typehints behavior: > (1) is_compatible_with > (2) bind_type_variables/match_type_variables > (3) trivial type inference. > > I would propose that each of these be adapted to a (internal) public > interface that accepts and returns standard typing types, and internally > converts to our (nowhere else exposed) typehints types, performs the logic, > and converts back. Each of these in turn can then be updated, as needed and > orthogonally, to operate on the typing types natively (possibly via deference > to a third-party library). > > I think coder inference could be easily adopted to use typing types directly, > but it may be a fourth place where we do internal conversion first. Another > gotcha is special care may need to be taken if we ever need to pickle these > types (which IIRC may have issues). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8156) Finish migration to standard Python typing
[ https://issues.apache.org/jira/browse/BEAM-8156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124303#comment-17124303 ] Udi Meiri commented on BEAM-8156: - Current thoughts are that internal types will continue to be used in the IOTypeHints container and the 3 places in the original description. Perhaps this will change if we rewrite these 3 places to work on Python's typing module types, or to use an external type checking library (such as pytypes). In this case, we would have to solve the pickling issue somehow (either write a serializer or omit them -- how does standard pickling handle __annotations__?). > Finish migration to standard Python typing > -- > > Key: BEAM-8156 > URL: https://issues.apache.org/jira/browse/BEAM-8156 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: P2 > Labels: stale-assigned > Original Estimate: 504h > Time Spent: 1h 40m > Remaining Estimate: 502h 20m > > We should migrate all Python uses of types to the standard typing module, and > make the typehints.* ones aliases of the Python ones. > > There are three places where we use custom typehints behavior: > (1) is_compatible_with > (2) bind_type_variables/match_type_variables > (3) trivial type inference. > > I would propose that each of these be adapted to a (internal) public > interface that accepts and returns standard typing types, and internally > converts to our (nowhere else exposed) typehints types, performs the logic, > and converts back. Each of these in turn can then be updated, as needed and > orthogonally, to operate on the typing types natively (possibly via deference > to a third-party library). > > I think coder inference could be easily adopted to use typing types directly, > but it may be a fourth place where we do internal conversion first. Another > gotcha is special care may need to be taken if we ever need to pickle these > types (which IIRC may have issues). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-5538) Beam Dependency Update Request: google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri closed BEAM-5538. --- Fix Version/s: Not applicable Resolution: Duplicate > Beam Dependency Update Request: google-cloud-core > - > > Key: BEAM-5538 > URL: https://issues.apache.org/jira/browse/BEAM-5538 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > Fix For: Not applicable > > > - 2018-10-01 19:15:15.957395 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:10:56.757120 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:09:43.192530 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:10.435397 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 16:36:39.492851 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 16:52:36.748339 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 17:36:04.816416 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-27 12:05:03.024147 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-03 12:02:38.435149 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-10 12:02:28.002973 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-17 12:31:46.478123 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.2 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Commented] (BEAM-5538) Beam Dependency Update Request: google-cloud-core
[ https://issues.apache.org/jira/browse/BEAM-5538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17124160#comment-17124160 ] Udi Meiri commented on BEAM-5538: - Obsoleted by https://issues.apache.org/jira/browse/BEAM-8756 > Beam Dependency Update Request: google-cloud-core > - > > Key: BEAM-5538 > URL: https://issues.apache.org/jira/browse/BEAM-5538 > Project: Beam > Issue Type: Bug > Components: dependencies >Reporter: Beam JIRA Bot >Priority: P2 > Labels: stale-P2 > > - 2018-10-01 19:15:15.957395 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-08 12:10:56.757120 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-15 12:09:43.192530 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2018-10-22 12:10:10.435397 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.25.0. The latest version is 0.28.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 16:36:39.492851 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 16:52:36.748339 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-20 17:36:04.816416 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-05-27 12:05:03.024147 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.0 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-03 12:02:38.435149 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-10 12:02:28.002973 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.1 > cc: > Please refer to [Beam Dependency Guide > |https://beam.apache.org/contribute/dependencies/]for more information. > Do Not Modify The Description Above. > - 2019-06-17 12:31:46.478123 > - > Please consider upgrading the dependency google-cloud-core. > The current version is 0.29.1. The latest version is 1.0.2 > cc: > Please refer to [Beam Dependency Guide >
[jira] [Commented] (BEAM-10168) Add Github "publish release" to release guide
[ https://issues.apache.org/jira/browse/BEAM-10168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17121291#comment-17121291 ] Udi Meiri commented on BEAM-10168: -- I uploaded my GPG public key used to sign the release and I get the same unverified message for 2.18. Probably because the key is tied to my @apache.org addr but I create commits using my @google.com addr. > Add Github "publish release" to release guide > - > > Key: BEAM-10168 > URL: https://issues.apache.org/jira/browse/BEAM-10168 > Project: Beam > Issue Type: Improvement > Components: website >Reporter: Kyle Weaver >Assignee: Kyle Weaver >Priority: P2 > > Github does not recognize tags as full-fledged releases unless they are > published through the Github API/UI. We need to add this step to the release > guide. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10145) Kafka IO performance tests leaving behind unused disks on apache-beam-testing
[
https://issues.apache.org/jira/browse/BEAM-10145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-10145:
-
Attachment: VfRJduZigE1.png
> Kafka IO performance tests leaving behind unused disks on apache-beam-testing
> -
>
> Key: BEAM-10145
> URL: https://issues.apache.org/jira/browse/BEAM-10145
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Udi Meiri
>Priority: P2
> Attachments: VfRJduZigE1.png
>
>
> Sample disk description:
> {"kubernetes.io/created-for/pv/name":"pvc-97dd8abb-a0ac-11ea-aa65-42010a80013b","kubernetes.io/created-for/pvc/name":"data-pzoo-0","kubernetes.io/created-for/pvc/namespace":"beam-performancetests-kafka-io-826"}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-10145) Kafka IO performance tests leaving behind unused disks on apache-beam-testing
Udi Meiri created BEAM-10145:
Summary: Kafka IO performance tests leaving behind unused disks on
apache-beam-testing
Key: BEAM-10145
URL: https://issues.apache.org/jira/browse/BEAM-10145
Project: Beam
Issue Type: Bug
Components: testing
Reporter: Udi Meiri
Sample disk description:
{"kubernetes.io/created-for/pv/name":"pvc-97dd8abb-a0ac-11ea-aa65-42010a80013b","kubernetes.io/created-for/pvc/name":"data-pzoo-0","kubernetes.io/created-for/pvc/namespace":"beam-performancetests-kafka-io-826"}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9355) Python typehints: support NewType
[ https://issues.apache.org/jira/browse/BEAM-9355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17108770#comment-17108770 ] Udi Meiri commented on BEAM-9355: - Yes, it needs an actual implementation. Current support just converts to Any. > Python typehints: support NewType > - > > Key: BEAM-9355 > URL: https://issues.apache.org/jira/browse/BEAM-9355 > Project: Beam > Issue Type: New Feature > Components: sdk-py-core >Reporter: Udi Meiri >Priority: Minor > Time Spent: 40m > Remaining Estimate: 0h > > https://docs.python.org/3/library/typing.html#newtype -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10013) Python type hints: fix arbitrary argument hints
Udi Meiri created BEAM-10013:
Summary: Python type hints: fix arbitrary argument hints
Key: BEAM-10013
URL: https://issues.apache.org/jira/browse/BEAM-10013
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Udi Meiri
Fix annotation import in from_callable to support annotations on arbitrary
(variadic) arguments.
For example, in IOTypeHintsTest.test_from_callable, the annotation for args and
kwargs should be:
{code}
*args: T
**kwargs: str
{code}
and converted to this in from_callable:
{code}
Tuple[T, ...]
Dict[str, str]
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-10013) Python type hints: fix arbitrary argument hints
[
https://issues.apache.org/jira/browse/BEAM-10013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-10013:
-
Status: Open (was: Triage Needed)
> Python type hints: fix arbitrary argument hints
> ---
>
> Key: BEAM-10013
> URL: https://issues.apache.org/jira/browse/BEAM-10013
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
>
> Fix annotation import in from_callable to support annotations on arbitrary
> (variadic) arguments.
> For example, in IOTypeHintsTest.test_from_callable, the annotation for args
> and kwargs should be:
> {code}
> *args: T
> **kwargs: str
> {code}
> and converted to this in from_callable:
> {code}
> Tuple[T, ...]
> Dict[str, str]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9865) Clean up jenkins workspaces for successful jobs
[ https://issues.apache.org/jira/browse/BEAM-9865?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-9865: --- Assignee: (was: Udi Meiri) > Clean up jenkins workspaces for successful jobs > --- > > Key: BEAM-9865 > URL: https://issues.apache.org/jira/browse/BEAM-9865 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Udi Meiri >Priority: Major > > Two recent bugs (and many more in the past) citing lack of disk space: > https://issues.apache.org/jira/browse/BEAM-9854 > https://issues.apache.org/jira/browse/BEAM-9462 > There are around 150 workspaces on each Jenkins machine: > apache-beam-jenkins-1..15. > Total size: > 1: 175G > 7: 158G > 8: 173G > The majority of jobs use a clone of the Beam read/write files under src/, > which is wiped out at the start of the job (wipeOutWorkspace()), so there is > really no point in keeping workspace files around after the job has completed > successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression
[
https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri closed BEAM-8964.
---
Fix Version/s: Not applicable
Resolution: Fixed
> DeprecationWarning: Flags not at the start of the expression
> -
>
> Key: BEAM-8964
> URL: https://issues.apache.org/jira/browse/BEAM-8964
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Labels: beginner, easy, newbie, starter
> Fix For: Not applicable
>
>
> I see lots of these warnings in our precommits.
> {code}
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp2jslxh39\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> 19:09:37
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp03vpdu3z\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression
[
https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-8964:
---
Assignee: Udi Meiri
> DeprecationWarning: Flags not at the start of the expression
> -
>
> Key: BEAM-8964
> URL: https://issues.apache.org/jira/browse/BEAM-8964
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Labels: beginner, easy, newbie, starter
>
> I see lots of these warnings in our precommits.
> {code}
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp2jslxh39\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> 19:09:37
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp03vpdu3z\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8964) DeprecationWarning: Flags not at the start of the expression
[
https://issues.apache.org/jira/browse/BEAM-8964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17105915#comment-17105915
]
Udi Meiri commented on BEAM-8964:
-
Oops I fixed this in https://github.com/apache/beam/pull/11016, but forgot to
close the bug.
I don't this warning at all in
https://builds.apache.org/job/beam_PreCommit_Python_Cron/2746/
> DeprecationWarning: Flags not at the start of the expression
> -
>
> Key: BEAM-8964
> URL: https://issues.apache.org/jira/browse/BEAM-8964
> Project: Beam
> Issue Type: Bug
> Components: io-py-files
>Reporter: Udi Meiri
>Priority: Major
> Labels: beginner, easy, newbie, starter
>
> I see lots of these warnings in our precommits.
> {code}
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp2jslxh39\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> 19:09:37
> 19:09:37 apache_beam/io/filesystem.py:583
> 19:09:37
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Commit/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/apache_beam/io/filesystem.py:583:
> DeprecationWarning: Flags not at the start of the expression
> '\\/tmp\\/tmp03vpdu3z\\/' (truncated)
> 19:09:37 re_pattern = re.compile(self.translate_pattern(pattern))
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-3713) Consider moving away from nose to nose2 or pytest.
[ https://issues.apache.org/jira/browse/BEAM-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-3713: Comment: was deleted (was: Yes, see #7949) > Consider moving away from nose to nose2 or pytest. > -- > > Key: BEAM-3713 > URL: https://issues.apache.org/jira/browse/BEAM-3713 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Minor > Time Spent: 18h > Remaining Estimate: 0h > > Per > [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,] > , nose is in maintenance mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3713) Consider moving away from nose to nose2 or pytest.
[ https://issues.apache.org/jira/browse/BEAM-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102780#comment-17102780 ] Udi Meiri commented on BEAM-3713: - Yes, see #7949 > Consider moving away from nose to nose2 or pytest. > -- > > Key: BEAM-3713 > URL: https://issues.apache.org/jira/browse/BEAM-3713 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Minor > Time Spent: 17h 50m > Remaining Estimate: 0h > > Per > [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,] > , nose is in maintenance mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3713) Consider moving away from nose to nose2 or pytest.
[ https://issues.apache.org/jira/browse/BEAM-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102781#comment-17102781 ] Udi Meiri commented on BEAM-3713: - Yes, see #7949 > Consider moving away from nose to nose2 or pytest. > -- > > Key: BEAM-3713 > URL: https://issues.apache.org/jira/browse/BEAM-3713 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Robert Bradshaw >Assignee: Udi Meiri >Priority: Minor > Time Spent: 17h 50m > Remaining Estimate: 0h > > Per > [https://nose.readthedocs.io/en/latest/|https://nose.readthedocs.io/en/latest/,] > , nose is in maintenance mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8389) Code Velocity dashboard lists closed PR as open
[ https://issues.apache.org/jira/browse/BEAM-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-8389: Attachment: KwGhjsOPxqc.png > Code Velocity dashboard lists closed PR as open > --- > > Key: BEAM-8389 > URL: https://issues.apache.org/jira/browse/BEAM-8389 > Project: Beam > Issue Type: Bug > Components: community-metrics >Reporter: Ahmet Altay >Assignee: Mikhail Gryzykhin >Priority: Minor > Attachments: KwGhjsOPxqc.png > > > https://github.com/apache/beam/pull/8430 is listed as a currently open PR in > http://metrics.beam.apache.org/d/code_velocity/code-velocity?orgId=1 even > though it has been merged. > cc: [~amyrvold] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8389) Code Velocity dashboard lists closed PR as open
[ https://issues.apache.org/jira/browse/BEAM-8389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17097575#comment-17097575 ] Udi Meiri commented on BEAM-8389: - Seeing the same thing: !KwGhjsOPxqc.png! > Code Velocity dashboard lists closed PR as open > --- > > Key: BEAM-8389 > URL: https://issues.apache.org/jira/browse/BEAM-8389 > Project: Beam > Issue Type: Bug > Components: community-metrics >Reporter: Ahmet Altay >Assignee: Mikhail Gryzykhin >Priority: Minor > Attachments: KwGhjsOPxqc.png > > > https://github.com/apache/beam/pull/8430 is listed as a currently open PR in > http://metrics.beam.apache.org/d/code_velocity/code-velocity?orgId=1 even > though it has been merged. > cc: [~amyrvold] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-8078. - Fix Version/s: Not applicable Resolution: Fixed > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Fix For: Not applicable > > Time Spent: 10h 10m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9462) No space left on device: apache-beam-1 and apache-beam-7
[ https://issues.apache.org/jira/browse/BEAM-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-9462. - Fix Version/s: Not applicable Resolution: Fixed > No space left on device: apache-beam-1 and apache-beam-7 > > > Key: BEAM-9462 > URL: https://issues.apache.org/jira/browse/BEAM-9462 > Project: Beam > Issue Type: Bug > Components: build-system, test-failures >Reporter: Michał Walenia >Priority: Blocker > Labels: currently-failing > Fix For: Not applicable > > > Jenkins workers 1 and 7 are failing their jobs since 2020-03-05 due to lack > of disk space left. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9462) No space left on device: apache-beam-1 and apache-beam-7
[ https://issues.apache.org/jira/browse/BEAM-9462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17096992#comment-17096992 ] Udi Meiri commented on BEAM-9462: - I believe this is obsolete (over 200G free on each right now). Longer term idea: https://issues.apache.org/jira/browse/BEAM-9865 > No space left on device: apache-beam-1 and apache-beam-7 > > > Key: BEAM-9462 > URL: https://issues.apache.org/jira/browse/BEAM-9462 > Project: Beam > Issue Type: Bug > Components: build-system, test-failures >Reporter: Michał Walenia >Priority: Blocker > Labels: currently-failing > > Jenkins workers 1 and 7 are failing their jobs since 2020-03-05 due to lack > of disk space left. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9865) Clean up jenkins workspaces for successful jobs
Udi Meiri created BEAM-9865: --- Summary: Clean up jenkins workspaces for successful jobs Key: BEAM-9865 URL: https://issues.apache.org/jira/browse/BEAM-9865 Project: Beam Issue Type: Bug Components: build-system Reporter: Udi Meiri Assignee: Udi Meiri Two recent bugs (and many more in the past) citing lack of disk space: https://issues.apache.org/jira/browse/BEAM-9854 https://issues.apache.org/jira/browse/BEAM-9462 There are around 150 workspaces on each Jenkins machine: apache-beam-jenkins-1..15. Total size: 1: 175G 7: 158G 8: 173G The majority of jobs use a clone of the Beam read/write files under src/, which is wiped out at the start of the job (wipeOutWorkspace()), so there is really no point in keeping workspace files around after the job has completed successfully. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-58) Support Google Cloud Storage encryption keys
[ https://issues.apache.org/jira/browse/BEAM-58?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094931#comment-17094931 ] Udi Meiri commented on BEAM-58: --- I have not worked on this, only customer *managed* keys, not *supplied* ones. > Support Google Cloud Storage encryption keys > > > Key: BEAM-58 > URL: https://issues.apache.org/jira/browse/BEAM-58 > Project: Beam > Issue Type: New Feature > Components: io-java-gcp >Reporter: Dan Halperin >Assignee: Udi Meiri >Priority: Minor > > Customer-supplied encryption keys are now in Beta. > https://cloud.google.com/compute/docs/disks/customer-supplied-encryption -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-8280) re-enable IOTypeHints.from_callable
[ https://issues.apache.org/jira/browse/BEAM-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-8280. - Fix Version/s: 2.21.0 Resolution: Fixed > re-enable IOTypeHints.from_callable > --- > > Key: BEAM-8280 > URL: https://issues.apache.org/jira/browse/BEAM-8280 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 12.5h > Remaining Estimate: 0h > > See https://issues.apache.org/jira/browse/BEAM-8279 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17091071#comment-17091071
]
Udi Meiri commented on BEAM-7405:
-
workaround merged: https://github.com/apache/beam/pull/11512
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h 40m
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9809) standardize docker and docker-compose usage in ITs
[ https://issues.apache.org/jira/browse/BEAM-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-9809: Description: Opening this against Python ITs, but may apply to other SDKs (I'm not aware if they use docker in their tests). - hdfs_integration_test: uses docker-compose in a shell script - beam-test-infra-metrics: uses docker-compose via gradle plugin - sdks:python:container:pyXX: use docker via a different gradle plugin was: Opening this against Python ITs, but may apply to other SDKs (I'm not aware if they might use docker). - hdfs_integration_test: uses docker-compose in a shell script - beam-test-infra-metrics: uses docker-compose via gradle plugin - sdks:python:container:pyXX: use docker via a different gradle plugin > standardize docker and docker-compose usage in ITs > -- > > Key: BEAM-9809 > URL: https://issues.apache.org/jira/browse/BEAM-9809 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Priority: Major > > Opening this against Python ITs, but may apply to other SDKs (I'm not aware > if they use docker in their tests). > - hdfs_integration_test: uses docker-compose in a shell script > - beam-test-infra-metrics: uses docker-compose via gradle plugin > - sdks:python:container:pyXX: use docker via a different gradle plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090148#comment-17090148
]
Udi Meiri commented on BEAM-7405:
-
Apparently not all docker-based tests are failing. The
sdks:python:containers:pyXX tests seem to ignore the config file (not sure how).
We are using at least 3 different methods of launching docker containers in our
tests.
Opened https://issues.apache.org/jira/browse/BEAM-9809 to standardize
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9809) standardize docker and docker-compose usage in ITs
Udi Meiri created BEAM-9809: --- Summary: standardize docker and docker-compose usage in ITs Key: BEAM-9809 URL: https://issues.apache.org/jira/browse/BEAM-9809 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri Opening this against Python ITs, but may apply to other SDKs (I'm not aware if they might use docker). - hdfs_integration_test: uses docker-compose in a shell script - beam-test-infra-metrics: uses docker-compose via gradle plugin - sdks:python:container:pyXX: use docker via a different gradle plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090130#comment-17090130
]
Udi Meiri commented on BEAM-7405:
-
I'm going with option 1, so we can get postcommits green again.
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17090010#comment-17090010
]
Udi Meiri commented on BEAM-7405:
-
5. Install docker-credential-gcloud.
This requires updating the VM image, which requires too much effort for me at
this time.
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-7981) ParDo function wrapper doesn't support Iterable output types
[
https://issues.apache.org/jira/browse/BEAM-7981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-7981.
-
Resolution: Fixed
> ParDo function wrapper doesn't support Iterable output types
>
>
> Key: BEAM-7981
> URL: https://issues.apache.org/jira/browse/BEAM-7981
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 50m
> Remaining Estimate: 0h
>
> I believe the bug is in CallableWrapperDoFn.default_type_hints, which
> converts Iterable[str] to str.
> This test will be included (commented out) in
> https://github.com/apache/beam/pull/9283
> {code}
> def test_typed_callable_iterable_output(self):
> @typehints.with_input_types(int)
> @typehints.with_output_types(typehints.Iterable[str])
> def do_fn(element):
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> Result:
> {code}
> ==
> ERROR: test_typed_callable_iterable_output
> (apache_beam.typehints.typed_pipeline_test.MainInputTest)
> --
> Traceback (most recent call last):
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/typehints/typed_pipeline_test.py",
> line 104, in test_typed_callable_iterable_output
> result = [1, 2] | beam.ParDo(do_fn)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py",
> line 519, in __ror__
> p.run().wait_until_finish()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 366, in run_pipeline
> default_environment=self._default_environment))
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 373, in run_via_runner_api
> return self.run_stages(stage_context, stages)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 455, in run_stages
> stage_context.safe_coders)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 733, in _run_stage
> result, splits = bundle_manager.process_bundle(data_input, data_output)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1663, in process_bundle
> part, expected_outputs), part_inputs):
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 586, in
> result_iterator
> yield fs.pop().result()
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
> return self.__get_result()
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in
> __get_result
> raise self._exception
> File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
> result = self.fn(*self.args, **self.kwargs)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1663, in
> part, expected_outputs), part_inputs):
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1601, in process_bundle
> result_future = self._worker_handler.control_conn.push(process_bundle_req)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1080, in push
> response = self.worker.do_instruction(request)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 343, in do_instruction
> request.instruction_id)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 369, in process_bundle
> bundle_processor.process_bundle(instruction_id))
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 593, in process_bundle
>
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089230#comment-17089230
]
Udi Meiri commented on BEAM-7405:
-
Possible solutions:
1. Use a custom docker config.json, for example:
{code}
$ export DOCKER_CONFIG=.
$ touch config.json
{code}
This has no race conditions, but is the least elegant.
2. Delete ~/.docker/config.json on Jenkins nodes.
May break things.
3. Upgrade docker-credential-gcr to 2.0.1 on Jenkins nodes and run
docker-credential-gcr configure-docker to replace "gcloud" with "gcr".
Alternatively, sed "s/gcloud/gcr/" in the config.
May race with whatever is setting "gcloud".
4. Figure out what is adding the offending config entry. (perhaps a non-Beam
job?)
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089215#comment-17089215
]
Udi Meiri commented on BEAM-7405:
-
The issue seems to be with this specific line in ~/.docker/config.json:
{code}
"marketplace.gcr.io": "gcloud",
{code}
There was a concern that running docker-credential-gcr configure-docker might
be adding the "gcloud" entry:
https://github.com/apache/beam/pull/8669#issuecomment-495519671
but the version we use (1.5.0) seems to ignore the marketplace entry. This is
the code that may be running docker-credential-gcr:
https://github.com/apache/beam/pull/7848/files#diff-422efc26e0cc609fd5488d55c01ad2c6R69
The same version (1.5.0) is installed on the jenkins node I'm looking at (10).
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7405) Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits - docker-credential-gcloud not installed
[
https://issues.apache.org/jira/browse/BEAM-7405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17089183#comment-17089183
]
Udi Meiri commented on BEAM-7405:
-
Same issue in CommunityMetrics:
https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/2138/console
> Task :sdks:python:hdfsIntegrationTest is failing in Python PostCommits -
> docker-credential-gcloud not installed
> ---
>
> Key: BEAM-7405
> URL: https://issues.apache.org/jira/browse/BEAM-7405
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Assignee: Yifan Zou
>Priority: Major
> Fix For: 2.14.0
>
> Time Spent: 3h
> Remaining Estimate: 0h
>
> This failure happened on apache-beam-jenkins-14.
> {noformat}
> 18:47:03 > Task :sdks:python:hdfsIntegrationTest
> 18:47:03 ++ dirname
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> 18:47:03 + TEST_DIR=./apache_beam/io/hdfs_integration_test
> 18:47:03 + ROOT_DIR=./apache_beam/io/hdfs_integration_test/../../../../..
> 18:47:03 +
> CONTEXT_DIR=./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + rm -r
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 rm: cannot remove
> './apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration':
> No such file or directory
> 18:47:03 + true
> 18:47:03 + mkdir -p
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks
> 18:47:03 + cp ./apache_beam/io/hdfs_integration_test/docker-compose.yml
> ./apache_beam/io/hdfs_integration_test/Dockerfile
> ./apache_beam/io/hdfs_integration_test/hdfscli.cfg
> ./apache_beam/io/hdfs_integration_test/hdfs_integration_test.sh
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 + cp -r
> ./apache_beam/io/hdfs_integration_test/../../../../../sdks/python
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/sdks/
> 18:47:03 + cp -r ./apache_beam/io/hdfs_integration_test/../../../../../model
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration/
> 18:47:03 ++ echo hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + PROJECT_NAME=hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714
> 18:47:03 + '[' -z jenkins-beam_PostCommit_Python_Verify_PR-714 ']'
> 18:47:03 + COLOR_OPT=--no-ansi
> 18:47:03 + COMPOSE_OPT='-p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi'
> 18:47:03 + cd
> ./apache_beam/io/hdfs_integration_test/../../../../../build/hdfs_integration
> 18:47:03 + docker network prune --force
> 18:47:03 + trap finally EXIT
> 18:47:03 + docker-compose -p
> hdfs_IT-jenkins-beam_PostCommit_Python_Verify_PR-714 --no-ansi build
> 18:47:03 namenode uses an image, skipping
> 18:47:03 datanode uses an image, skipping
> 18:47:03 Building test
> 18:47:03 [29234] Failed to execute script docker-compose
> 18:47:03 Traceback (most recent call last):
> 18:47:03 File "bin/docker-compose", line 6, in
> 18:47:03 File "compose/cli/main.py", line 71, in main
> 18:47:03 File "compose/cli/main.py", line 127, in perform_command
> 18:47:03 File "compose/cli/main.py", line 287, in build
> 18:47:03 File "compose/project.py", line 386, in build
> 18:47:03 File "compose/project.py", line 368, in build_service
> 18:47:03 File "compose/service.py", line 1084, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 260, in build
> 18:47:03 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 18:47:03 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 18:47:03 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 18:47:03 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 18:47:03 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 18:47:03 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9797) license_script.sh calls pip install/uninstall in local env
[ https://issues.apache.org/jira/browse/BEAM-9797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088956#comment-17088956 ] Udi Meiri commented on BEAM-9797: - Another thing you can do in tox is specify the Python version, removing the need to check for versions in pull_licenses_java.py. > license_script.sh calls pip install/uninstall in local env > -- > > Key: BEAM-9797 > URL: https://issues.apache.org/jira/browse/BEAM-9797 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Udi Meiri >Priority: Major > > File is: > https://github.com/apache/beam/blob/master/sdks/java/container/license_scripts/license_script.sh > The problem is with the code that does pip install and uninstall. > 1. It is not okay to modify the local environment. > 2. Running this script in parallel with itself (on Jenkins) has a chance to > cause a race. > The solution is to use a tox environment to run this script in. Tox will take > care of creating a virtualenv with the required dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9797) license_script.sh calls pip install/uninstall in local env
Udi Meiri created BEAM-9797: --- Summary: license_script.sh calls pip install/uninstall in local env Key: BEAM-9797 URL: https://issues.apache.org/jira/browse/BEAM-9797 Project: Beam Issue Type: Bug Components: build-system Reporter: Udi Meiri File is: https://github.com/apache/beam/blob/master/sdks/java/container/license_scripts/license_script.sh The problem is with the code that does pip install and uninstall. 1. It is not okay to modify the local environment. 2. Running this script in parallel with itself (on Jenkins) has a chance to cause a race. The solution is to use a tox environment to run this script in. Tox will take care of creating a virtualenv with the required dependencies. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088934#comment-17088934
]
Udi Meiri commented on BEAM-9755:
-
Let me know if you want me to take a look
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088930#comment-17088930
]
Udi Meiri edited comment on BEAM-9755 at 4/21/20, 6:10 PM:
---
Not surprising, as trivial_inference relies on bytecode implementation details
(CPython).
was (Author: udim):
Not surprising, as trivial_inference is tightly coupled to the bytecode
implementation (CPython).
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9755) infer_return_type does not infer as expected in Python 3.8
[
https://issues.apache.org/jira/browse/BEAM-9755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088930#comment-17088930
]
Udi Meiri commented on BEAM-9755:
-
Not surprising, as trivial_inference is tightly coupled to the bytecode
implementation (CPython).
> infer_return_type does not infer as expected in Python 3.8
> --
>
> Key: BEAM-9755
> URL: https://issues.apache.org/jira/browse/BEAM-9755
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: yoshiki obata
>Assignee: yoshiki obata
>Priority: Major
>
> apache_beam.typehints.trivial_inference.infer_return_type does not return
> type as expected in Python 3.8.
> And this cause testDictComprehensionSimple
> (apache_beam.typehints.trivial_inference_test.TrivialInferenceTest) FAIL.
> - Python 3.7.7
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[str, int]{code}
> - Python 3.8.2
> {code:python}
> >> from apache_beam.typehints.trivial_inference import infer_return_type
> >> infer_return_type(lambda _list: {'a': 1 for _ in _list}, [], debug=False,
> depth=5)
> Dict[int, str]{code}
>
> CPython bytecode changes[1] may affect.
> [1] https://docs.python.org/3/whatsnew/3.8.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
[
https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-6860.
-
Resolution: Fixed
> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -
>
> Key: BEAM-6860
> URL: https://issues.apache.org/jira/browse/BEAM-6860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: macOS, DirectRunner, python 2.7.15 via
> pyenv/pyenv-virtualenv
>Reporter: Henrik
>Assignee: Udi Meiri
>Priority: Major
> Labels: newbie
> Fix For: 2.22.0, 2.16.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Main error:
> > Cannot convert GlobalWindow to
> > apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the
> input, gives me just what I want; so the lines of data to serialise are
> "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
> File "./okr_end_ride.py", line 254, in
> run()
> File "./okr_end_ride.py", line 250, in run
> run_pipeline(pipeline_options, known_args)
> File "./okr_end_ride.py", line 198, in run_pipeline
> | 'write_all' >> WriteToText(known_args.output,
> file_name_suffix=".txt")
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 132, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 275, in run_pipeline
> default_environment=self._default_environment))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 278, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 354, in run_stages
> stage_context.safe_coders)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 509, in run_stage
> data_input, data_output)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1206, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 821, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 265, in do_instruction
> request.instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 281, in process_bundle
> delayed_applications = bundle_processor.process_bundle(instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 552, in process_bundle
> op.finish()
> File "apache_beam/runners/worker/operations.py", line 549, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 550, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 551, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/common.py", line 758, in
> apache_beam.runners.common.DoFnRunner.finish
> File "apache_beam/runners/common.py", line 752, in
> apache_beam.runners.common.DoFnRunner._invoke_bundle_method
> File "apache_beam/runners/common.py", line 777, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
> File
[jira] [Commented] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
[
https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17088097#comment-17088097
]
Udi Meiri commented on BEAM-6860:
-
Fix should go into 2.22 release.
Meanwhile, I believe a possible workaround is to window into global windows
before WriteToText.
Please let me know if that doesn't work.
> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -
>
> Key: BEAM-6860
> URL: https://issues.apache.org/jira/browse/BEAM-6860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: macOS, DirectRunner, python 2.7.15 via
> pyenv/pyenv-virtualenv
>Reporter: Henrik
>Assignee: Udi Meiri
>Priority: Major
> Labels: newbie
> Fix For: 2.16.0, 2.22.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Main error:
> > Cannot convert GlobalWindow to
> > apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the
> input, gives me just what I want; so the lines of data to serialise are
> "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
> File "./okr_end_ride.py", line 254, in
> run()
> File "./okr_end_ride.py", line 250, in run
> run_pipeline(pipeline_options, known_args)
> File "./okr_end_ride.py", line 198, in run_pipeline
> | 'write_all' >> WriteToText(known_args.output,
> file_name_suffix=".txt")
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 132, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 275, in run_pipeline
> default_environment=self._default_environment))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 278, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 354, in run_stages
> stage_context.safe_coders)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 509, in run_stage
> data_input, data_output)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1206, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 821, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 265, in do_instruction
> request.instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 281, in process_bundle
> delayed_applications = bundle_processor.process_bundle(instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 552, in process_bundle
> op.finish()
> File "apache_beam/runners/worker/operations.py", line 549, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 550, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 551, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/common.py", line 758, in
> apache_beam.runners.common.DoFnRunner.finish
> File "apache_beam/runners/common.py", line 752,
[jira] [Updated] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
[
https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-6860:
Fix Version/s: 2.22.0
> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -
>
> Key: BEAM-6860
> URL: https://issues.apache.org/jira/browse/BEAM-6860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: macOS, DirectRunner, python 2.7.15 via
> pyenv/pyenv-virtualenv
>Reporter: Henrik
>Assignee: Udi Meiri
>Priority: Major
> Labels: newbie
> Fix For: 2.16.0, 2.22.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Main error:
> > Cannot convert GlobalWindow to
> > apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the
> input, gives me just what I want; so the lines of data to serialise are
> "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
> File "./okr_end_ride.py", line 254, in
> run()
> File "./okr_end_ride.py", line 250, in run
> run_pipeline(pipeline_options, known_args)
> File "./okr_end_ride.py", line 198, in run_pipeline
> | 'write_all' >> WriteToText(known_args.output,
> file_name_suffix=".txt")
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 132, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 275, in run_pipeline
> default_environment=self._default_environment))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 278, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 354, in run_stages
> stage_context.safe_coders)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 509, in run_stage
> data_input, data_output)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1206, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 821, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 265, in do_instruction
> request.instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 281, in process_bundle
> delayed_applications = bundle_processor.process_bundle(instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 552, in process_bundle
> op.finish()
> File "apache_beam/runners/worker/operations.py", line 549, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 550, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 551, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/common.py", line 758, in
> apache_beam.runners.common.DoFnRunner.finish
> File "apache_beam/runners/common.py", line 752, in
> apache_beam.runners.common.DoFnRunner._invoke_bundle_method
> File "apache_beam/runners/common.py", line 777, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
>
[jira] [Resolved] (BEAM-9737) beam_PostCommit_Website_Test failing
[
https://issues.apache.org/jira/browse/BEAM-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-9737.
-
Fix Version/s: Not applicable
Resolution: Fixed
> beam_PostCommit_Website_Test failing
>
>
> Key: BEAM-9737
> URL: https://issues.apache.org/jira/browse/BEAM-9737
> Project: Beam
> Issue Type: Bug
> Components: test-failures, website
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Also failing: beam_PostCommit_Website_Publish (same failure)
> {code}
> > Task :website:buildLocalWebsite
> `/` is not writable.
> Bundler will use `/tmp/bundler/home/unknown' as your home directory
> temporarily.
> Configuration file: /repo/website/_config.yml
> Configuration file: /repo/website/_config_test.yml
> Configuration file: /tmp/_config_branch_repo.yml
> Source: /repo/website/src
>Destination: generated-local-content
> Incremental build: enabled
> Generating...
> jekyll 3.6.3 | Error: Permission denied @ dir_s_mkdir -
> /repo/build/website/generated-local-content/security
> {code}
> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Website_Test/3676/console
> Possible culprit: https://github.com/apache/beam/pull/11232/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (BEAM-7981) ParDo function wrapper doesn't support Iterable output types
[
https://issues.apache.org/jira/browse/BEAM-7981?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reopened BEAM-7981:
-
Missed a TODO in typed_pipeline_test
> ParDo function wrapper doesn't support Iterable output types
>
>
> Key: BEAM-7981
> URL: https://issues.apache.org/jira/browse/BEAM-7981
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: 2.18.0
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> I believe the bug is in CallableWrapperDoFn.default_type_hints, which
> converts Iterable[str] to str.
> This test will be included (commented out) in
> https://github.com/apache/beam/pull/9283
> {code}
> def test_typed_callable_iterable_output(self):
> @typehints.with_input_types(int)
> @typehints.with_output_types(typehints.Iterable[str])
> def do_fn(element):
> return [[str(element)] * 2]
> result = [1, 2] | beam.ParDo(do_fn)
> self.assertEqual([['1', '1'], ['2', '2']], sorted(result))
> {code}
> Result:
> {code}
> ==
> ERROR: test_typed_callable_iterable_output
> (apache_beam.typehints.typed_pipeline_test.MainInputTest)
> --
> Traceback (most recent call last):
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/typehints/typed_pipeline_test.py",
> line 104, in test_typed_callable_iterable_output
> result = [1, 2] | beam.ParDo(do_fn)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/transforms/ptransform.py",
> line 519, in __ror__
> p.run().wait_until_finish()
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 366, in run_pipeline
> default_environment=self._default_environment))
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 373, in run_via_runner_api
> return self.run_stages(stage_context, stages)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 455, in run_stages
> stage_context.safe_coders)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 733, in _run_stage
> result, splits = bundle_manager.process_bundle(data_input, data_output)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1663, in process_bundle
> part, expected_outputs), part_inputs):
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 586, in
> result_iterator
> yield fs.pop().result()
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 432, in result
> return self.__get_result()
> File "/usr/lib/python3.7/concurrent/futures/_base.py", line 384, in
> __get_result
> raise self._exception
> File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
> result = self.fn(*self.args, **self.kwargs)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1663, in
> part, expected_outputs), part_inputs):
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1601, in process_bundle
> result_future = self._worker_handler.control_conn.push(process_bundle_req)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/portability/fn_api_runner.py",
> line 1080, in push
> response = self.worker.do_instruction(request)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 343, in do_instruction
> request.instruction_id)
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 369, in process_bundle
> bundle_processor.process_bundle(instruction_id))
> File
> "/usr/local/google/home/ehudm/src/beam/sdks/python/apache_beam/runners/worker/bundle_processor.py",
> line 593, in process_bundle
>
[jira] [Updated] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
[
https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-6860:
Status: Open (was: Triage Needed)
> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -
>
> Key: BEAM-6860
> URL: https://issues.apache.org/jira/browse/BEAM-6860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: macOS, DirectRunner, python 2.7.15 via
> pyenv/pyenv-virtualenv
>Reporter: Henrik
>Assignee: Udi Meiri
>Priority: Major
> Labels: newbie
> Fix For: 2.16.0
>
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Main error:
> > Cannot convert GlobalWindow to
> > apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the
> input, gives me just what I want; so the lines of data to serialise are
> "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
> File "./okr_end_ride.py", line 254, in
> run()
> File "./okr_end_ride.py", line 250, in run
> run_pipeline(pipeline_options, known_args)
> File "./okr_end_ride.py", line 198, in run_pipeline
> | 'write_all' >> WriteToText(known_args.output,
> file_name_suffix=".txt")
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 132, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 275, in run_pipeline
> default_environment=self._default_environment))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 278, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 354, in run_stages
> stage_context.safe_coders)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 509, in run_stage
> data_input, data_output)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1206, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 821, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 265, in do_instruction
> request.instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 281, in process_bundle
> delayed_applications = bundle_processor.process_bundle(instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 552, in process_bundle
> op.finish()
> File "apache_beam/runners/worker/operations.py", line 549, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 550, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 551, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/common.py", line 758, in
> apache_beam.runners.common.DoFnRunner.finish
> File "apache_beam/runners/common.py", line 752, in
> apache_beam.runners.common.DoFnRunner._invoke_bundle_method
> File "apache_beam/runners/common.py", line 777, in
> apache_beam.runners.common.DoFnRunner._reraise_augmented
>
[jira] [Commented] (BEAM-9737) beam_PostCommit_Website_Test failing
[
https://issues.apache.org/jira/browse/BEAM-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17085285#comment-17085285
]
Udi Meiri commented on BEAM-9737:
-
Magics:
https://lists.apache.org/thread.html/r9f9b7d4c502141e2e973f23d9e7329dcbd11fb64487b77a02d42e144%40%3Cbuilds.apache.org%3E
> beam_PostCommit_Website_Test failing
>
>
> Key: BEAM-9737
> URL: https://issues.apache.org/jira/browse/BEAM-9737
> Project: Beam
> Issue Type: Bug
> Components: test-failures, website
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 3h
> Remaining Estimate: 0h
>
> Also failing: beam_PostCommit_Website_Publish (same failure)
> {code}
> > Task :website:buildLocalWebsite
> `/` is not writable.
> Bundler will use `/tmp/bundler/home/unknown' as your home directory
> temporarily.
> Configuration file: /repo/website/_config.yml
> Configuration file: /repo/website/_config_test.yml
> Configuration file: /tmp/_config_branch_repo.yml
> Source: /repo/website/src
>Destination: generated-local-content
> Incremental build: enabled
> Generating...
> jekyll 3.6.3 | Error: Permission denied @ dir_s_mkdir -
> /repo/build/website/generated-local-content/security
> {code}
> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Website_Test/3676/console
> Possible culprit: https://github.com/apache/beam/pull/11232/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Reopened] (BEAM-6860) WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
[
https://issues.apache.org/jira/browse/BEAM-6860?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reopened BEAM-6860:
-
This is still an issue. In the latest Beam it manifests as:
{code}
AttributeError: 'GlobalWindow' object has no attribute '_end_micros' [while
running 'Write/Write/WriteImpl/WriteBundles']
{code}
> WriteToText crash with "GlobalWindow -> ._IntervalWindowBase"
> -
>
> Key: BEAM-6860
> URL: https://issues.apache.org/jira/browse/BEAM-6860
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Affects Versions: 2.11.0
> Environment: macOS, DirectRunner, python 2.7.15 via
> pyenv/pyenv-virtualenv
>Reporter: Henrik
>Assignee: Udi Meiri
>Priority: Major
> Labels: newbie
> Fix For: 2.16.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> Main error:
> > Cannot convert GlobalWindow to
> > apache_beam.utils.windowed_value._IntervalWindowBase
> This is very hard for me to debug. Doing a DoPar call before, printing the
> input, gives me just what I want; so the lines of data to serialise are
> "alright"; just JSON strings, in fact.
> Stacktrace:
> {code:java}
> Traceback (most recent call last):
> File "./okr_end_ride.py", line 254, in
> run()
> File "./okr_end_ride.py", line 250, in run
> run_pipeline(pipeline_options, known_args)
> File "./okr_end_ride.py", line 198, in run_pipeline
> | 'write_all' >> WriteToText(known_args.output,
> file_name_suffix=".txt")
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 426, in __exit__
> self.run().wait_until_finish()
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/direct/direct_runner.py",
> line 132, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 275, in run_pipeline
> default_environment=self._default_environment))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 278, in run_via_runner_api
> return self.run_stages(*self.create_stages(pipeline_proto))
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 354, in run_stages
> stage_context.safe_coders)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 509, in run_stage
> data_input, data_output)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 1206, in process_bundle
> result_future = self._controller.control_handler.push(process_bundle)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/portability/fn_api_runner.py",
> line 821, in push
> response = self.worker.do_instruction(request)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 265, in do_instruction
> request.instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 281, in process_bundle
> delayed_applications = bundle_processor.process_bundle(instruction_id)
> File
> "/Users/h/.pyenv/versions/2.7.15/envs/log-analytics/lib/python2.7/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 552, in process_bundle
> op.finish()
> File "apache_beam/runners/worker/operations.py", line 549, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 550, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/worker/operations.py", line 551, in
> apache_beam.runners.worker.operations.DoOperation.finish
> File "apache_beam/runners/common.py", line 758, in
> apache_beam.runners.common.DoFnRunner.finish
> File "apache_beam/runners/common.py", line 752, in
>
[jira] [Commented] (BEAM-9764) :sdks:java:container:generateThirdPartyLicenses failing
[
https://issues.apache.org/jira/browse/BEAM-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084398#comment-17084398
]
Udi Meiri commented on BEAM-9764:
-
Or was that intentional?
> :sdks:java:container:generateThirdPartyLicenses failing
> ---
>
> Key: BEAM-9764
> URL: https://issues.apache.org/jira/browse/BEAM-9764
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, test-failures
>Reporter: Udi Meiri
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.21.0
>
>
> https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/774/console
> The traceback is interspersed with other logs:
> {code}
> Traceback (most recent call last):
> Successfully pulled
> java_third_party_licenses/protobuf-java-util-3.11.1.jar/LICENSE from
> https://opensource.org/licenses/BSD-3-Clause
> Successfully pulled java_third_party_licenses/protoc-3.11.0.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> File "sdks/java/container/license_scripts/pull_licenses_java.py", line 138,
> in
> Successfully pulled java_third_party_licenses/protoc-3.11.1.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> license_url = dep['moduleLicenseUrl']
> Successfully pulled java_third_party_licenses/zetasketch-0.1.0.jar/LICENSE
> from http://www.apache.org/licenses/LICENSE-2.0.txt
> KeyError: 'moduleLicenseUrl'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9764) :sdks:java:container:generateThirdPartyLicenses failing
[
https://issues.apache.org/jira/browse/BEAM-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084392#comment-17084392
]
Udi Meiri commented on BEAM-9764:
-
I think one issue is that pull_from_url doesn't retry on "Invalid url:" errors
(missing raise at the end of the branch).
> :sdks:java:container:generateThirdPartyLicenses failing
> ---
>
> Key: BEAM-9764
> URL: https://issues.apache.org/jira/browse/BEAM-9764
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, test-failures
>Reporter: Udi Meiri
>Assignee: Hannah Jiang
>Priority: Major
> Fix For: 2.21.0
>
>
> https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/774/console
> The traceback is interspersed with other logs:
> {code}
> Traceback (most recent call last):
> Successfully pulled
> java_third_party_licenses/protobuf-java-util-3.11.1.jar/LICENSE from
> https://opensource.org/licenses/BSD-3-Clause
> Successfully pulled java_third_party_licenses/protoc-3.11.0.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> File "sdks/java/container/license_scripts/pull_licenses_java.py", line 138,
> in
> Successfully pulled java_third_party_licenses/protoc-3.11.1.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> license_url = dep['moduleLicenseUrl']
> Successfully pulled java_third_party_licenses/zetasketch-0.1.0.jar/LICENSE
> from http://www.apache.org/licenses/LICENSE-2.0.txt
> KeyError: 'moduleLicenseUrl'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9767) test_streaming_wordcount flaky timeouts
Udi Meiri created BEAM-9767:
---
Summary: test_streaming_wordcount flaky timeouts
Key: BEAM-9767
URL: https://issues.apache.org/jira/browse/BEAM-9767
Project: Beam
Issue Type: Bug
Components: sdk-py-core, test-failures
Reporter: Udi Meiri
Assignee: Sam Rohde
Timed out after 600s, typically completes in 2.8s on my workstation.
https://builds.apache.org/job/beam_PreCommit_Python_Commit/12376/
{code}
self =
@unittest.skipIf(
sys.version_info < (3, 5, 3),
'The tests require at least Python 3.6 to work.')
def test_streaming_wordcount(self):
class WordExtractingDoFn(beam.DoFn):
def process(self, element):
text_line = element.strip()
words = text_line.split()
return words
# Add the TestStream so that it can be cached.
ib.options.capturable_sources.add(TestStream)
ib.options.capture_duration = timedelta(seconds=5)
p = beam.Pipeline(
runner=interactive_runner.InteractiveRunner(),
options=StandardOptions(streaming=True))
data = (
p
| TestStream()
.advance_watermark_to(0)
.advance_processing_time(1)
.add_elements(['to', 'be', 'or', 'not', 'to', 'be'])
.advance_watermark_to(20)
.advance_processing_time(1)
.add_elements(['that', 'is', 'the', 'question'])
| beam.WindowInto(beam.window.FixedWindows(10))) # yapf: disable
counts = (
data
| 'split' >> beam.ParDo(WordExtractingDoFn())
| 'pair_with_one' >> beam.Map(lambda x: (x, 1))
| 'group' >> beam.GroupByKey()
| 'count' >> beam.Map(lambda wordones: (wordones[0],
sum(wordones[1]
# Watch the local scope for Interactive Beam so that referenced
PCollections
# will be cached.
ib.watch(locals())
# This is normally done in the interactive_utils when a transform is
# applied but needs an IPython environment. So we manually run this here.
ie.current_env().track_user_pipelines()
# Create a fake limiter that cancels the BCJ once the main job receives
the
# expected amount of results.
class FakeLimiter:
def __init__(self, p, pcoll):
self.p = p
self.pcoll = pcoll
def is_triggered(self):
result = ie.current_env().pipeline_result(self.p)
if result:
try:
results = result.get(self.pcoll)
except ValueError:
return False
return len(results) >= 10
return False
# This sets the limiters to stop reading when the test receives 10
elements
# or after 5 seconds have elapsed (to eliminate the possibility of
hanging).
ie.current_env().options.capture_control.set_limiters_for_test(
[FakeLimiter(p, data), DurationLimiter(timedelta(seconds=5))])
# This tests that the data was correctly cached.
pane_info = PaneInfo(True, True, PaneInfoTiming.UNKNOWN, 0, 0)
expected_data_df = pd.DataFrame([
('to', 0, [IntervalWindow(0, 10)], pane_info),
('be', 0, [IntervalWindow(0, 10)], pane_info),
('or', 0, [IntervalWindow(0, 10)], pane_info),
('not', 0, [IntervalWindow(0, 10)], pane_info),
('to', 0, [IntervalWindow(0, 10)], pane_info),
('be', 0, [IntervalWindow(0, 10)], pane_info),
('that', 2000, [IntervalWindow(20, 30)], pane_info),
('is', 2000, [IntervalWindow(20, 30)], pane_info),
('the', 2000, [IntervalWindow(20, 30)], pane_info),
('question', 2000, [IntervalWindow(20, 30)], pane_info)
], columns=[0, 'event_time', 'windows', 'pane_info']) # yapf: disable
> data_df = ib.collect(data, include_window_info=True)
apache_beam/runners/interactive/interactive_runner_test.py:237:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
apache_beam/runners/interactive/interactive_beam.py:451: in collect
return head(pcoll, n=-1, include_window_info=include_window_info)
apache_beam/runners/interactive/utils.py:204: in run_within_progress_indicator
return func(*args, **kwargs)
apache_beam/runners/interactive/interactive_beam.py:515: in head
result.wait_until_finish()
apache_beam/runners/interactive/interactive_runner.py:250: in wait_until_finish
self._underlying_result.wait_until_finish()
apache_beam/runners/direct/direct_runner.py:455: in wait_until_finish
self._executor.await_completion()
apache_beam/runners/direct/executor.py:439: in await_completion
self._executor.await_completion()
apache_beam/runners/direct/executor.py:484: in await_completion
update = self.visible_updates.take()
apache_beam/runners/direct/executor.py:557: in take
[jira] [Updated] (BEAM-9764) :sdks:java:container:generateThirdPartyLicenses failing
[
https://issues.apache.org/jira/browse/BEAM-9764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-9764:
Component/s: test-failures
> :sdks:java:container:generateThirdPartyLicenses failing
> ---
>
> Key: BEAM-9764
> URL: https://issues.apache.org/jira/browse/BEAM-9764
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, test-failures
>Reporter: Udi Meiri
>Assignee: Hannah Jiang
>Priority: Major
>
> https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/774/console
> The traceback is interspersed with other logs:
> {code}
> Traceback (most recent call last):
> Successfully pulled
> java_third_party_licenses/protobuf-java-util-3.11.1.jar/LICENSE from
> https://opensource.org/licenses/BSD-3-Clause
> Successfully pulled java_third_party_licenses/protoc-3.11.0.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> File "sdks/java/container/license_scripts/pull_licenses_java.py", line 138,
> in
> Successfully pulled java_third_party_licenses/protoc-3.11.1.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> license_url = dep['moduleLicenseUrl']
> Successfully pulled java_third_party_licenses/zetasketch-0.1.0.jar/LICENSE
> from http://www.apache.org/licenses/LICENSE-2.0.txt
> KeyError: 'moduleLicenseUrl'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9136) Add LICENSES and NOTICES to docker images
[ https://issues.apache.org/jira/browse/BEAM-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084279#comment-17084279 ] Udi Meiri commented on BEAM-9136: - :sdks:java:container:generateThirdPartyLicenses is failing: https://issues.apache.org/jira/browse/BEAM-9764 > Add LICENSES and NOTICES to docker images > - > > Key: BEAM-9136 > URL: https://issues.apache.org/jira/browse/BEAM-9136 > Project: Beam > Issue Type: Task > Components: build-system >Reporter: Hannah Jiang >Assignee: Hannah Jiang >Priority: Major > Fix For: 2.21.0 > > Time Spent: 22h 10m > Remaining Estimate: 0h > > Scan dependencies and add licenses and notices of the dependencies to SDK > docker images. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9764) :sdks:java:container:generateThirdPartyLicenses failing
Udi Meiri created BEAM-9764:
---
Summary: :sdks:java:container:generateThirdPartyLicenses failing
Key: BEAM-9764
URL: https://issues.apache.org/jira/browse/BEAM-9764
Project: Beam
Issue Type: Bug
Components: sdk-java-core
Reporter: Udi Meiri
Assignee: Hannah Jiang
https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/774/console
The traceback is interspersed with other logs:
{code}
Traceback (most recent call last):
Successfully pulled
java_third_party_licenses/protobuf-java-util-3.11.1.jar/LICENSE from
https://opensource.org/licenses/BSD-3-Clause
Successfully pulled java_third_party_licenses/protoc-3.11.0.jar/LICENSE from
http://www.apache.org/licenses/LICENSE-2.0.txt
File "sdks/java/container/license_scripts/pull_licenses_java.py", line 138,
in
Successfully pulled java_third_party_licenses/protoc-3.11.1.jar/LICENSE from
http://www.apache.org/licenses/LICENSE-2.0.txt
license_url = dep['moduleLicenseUrl']
Successfully pulled java_third_party_licenses/zetasketch-0.1.0.jar/LICENSE from
http://www.apache.org/licenses/LICENSE-2.0.txt
KeyError: 'moduleLicenseUrl'
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17084267#comment-17084267
]
Udi Meiri commented on BEAM-9119:
-
https://builds.apache.org/job/beam_PreCommit_Python_Cron/2631/
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value = self._value_coder.decode_from_stream(in_stream, nested)
>
[jira] [Commented] (BEAM-9737) beam_PostCommit_Website_Test failing
[
https://issues.apache.org/jira/browse/BEAM-9737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17083535#comment-17083535
]
Udi Meiri commented on BEAM-9737:
-
It seems like I'm seeing this behavior: https://github.com/moby/moby/issues/2259
where a bind mount inside a container is seen as owned by root:root.
Except this doesn't happen on my workstation, and I don't see why.
> beam_PostCommit_Website_Test failing
>
>
> Key: BEAM-9737
> URL: https://issues.apache.org/jira/browse/BEAM-9737
> Project: Beam
> Issue Type: Bug
> Components: test-failures, website
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Also failing: beam_PostCommit_Website_Publish (same failure)
> {code}
> > Task :website:buildLocalWebsite
> `/` is not writable.
> Bundler will use `/tmp/bundler/home/unknown' as your home directory
> temporarily.
> Configuration file: /repo/website/_config.yml
> Configuration file: /repo/website/_config_test.yml
> Configuration file: /tmp/_config_branch_repo.yml
> Source: /repo/website/src
>Destination: generated-local-content
> Incremental build: enabled
> Generating...
> jekyll 3.6.3 | Error: Permission denied @ dir_s_mkdir -
> /repo/build/website/generated-local-content/security
> {code}
> https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Website_Test/3676/console
> Possible culprit: https://github.com/apache/beam/pull/11232/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9737) beam_PostCommit_Website_Test failing
Udi Meiri created BEAM-9737:
---
Summary: beam_PostCommit_Website_Test failing
Key: BEAM-9737
URL: https://issues.apache.org/jira/browse/BEAM-9737
Project: Beam
Issue Type: Bug
Components: test-failures, website
Reporter: Udi Meiri
Assignee: Udi Meiri
Also failing: beam_PostCommit_Website_Publish (same failure)
{code}
> Task :website:buildLocalWebsite
`/` is not writable.
Bundler will use `/tmp/bundler/home/unknown' as your home directory temporarily.
Configuration file: /repo/website/_config.yml
Configuration file: /repo/website/_config_test.yml
Configuration file: /tmp/_config_branch_repo.yml
Source: /repo/website/src
Destination: generated-local-content
Incremental build: enabled
Generating...
jekyll 3.6.3 | Error: Permission denied @ dir_s_mkdir -
/repo/build/website/generated-local-content/security
{code}
https://builds.apache.org/view/A-D/view/Beam/view/PostCommit/job/beam_PostCommit_Website_Test/3676/console
Possible culprit: https://github.com/apache/beam/pull/11232/files
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-5422) Update BigQueryIO DynamicDestinations documentation to clarify usage of getDestination() and getTable()
[ https://issues.apache.org/jira/browse/BEAM-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-5422: --- Assignee: Udi Meiri (was: Chamikara Madhusanka Jayalath) > Update BigQueryIO DynamicDestinations documentation to clarify usage of > getDestination() and getTable() > --- > > Key: BEAM-5422 > URL: https://issues.apache.org/jira/browse/BEAM-5422 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, there are some details related to these methods that should be > further clarified. For example, getTable() is expected to return a unique > value for each destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-5422) Update BigQueryIO DynamicDestinations documentation to clarify usage of getDestination() and getTable()
[ https://issues.apache.org/jira/browse/BEAM-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-5422. - Fix Version/s: 2.21.0 Resolution: Fixed > Update BigQueryIO DynamicDestinations documentation to clarify usage of > getDestination() and getTable() > --- > > Key: BEAM-5422 > URL: https://issues.apache.org/jira/browse/BEAM-5422 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1.5h > Remaining Estimate: 0h > > Currently, there are some details related to these methods that should be > further clarified. For example, getTable() is expected to return a unique > value for each destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9529) Remove googledatastore package
[ https://issues.apache.org/jira/browse/BEAM-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-9529. - Resolution: Fixed > Remove googledatastore package > -- > > Key: BEAM-9529 > URL: https://issues.apache.org/jira/browse/BEAM-9529 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Affects Versions: 2.21.0 >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 1h 40m > Remaining Estimate: 0h > > Followup to https://issues.apache.org/jira/browse/BEAM-4543 > Reasons: > 1. v1/datastoreio.py (which uses googledatastore) has had a deprecation > notice since 2.13 (admittedly citing a 3.0 Beam release). > v1new/datastoreio.py is the replacement, using the newer package. > 2. googledatastore does not support Python 3, and 2.7 support is going away > on Beam. > 3. googledatastore depends on an old version of httplib2 [1], which is > starting to conflict with other packages using new versions [2]. > [1] > https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/85e3856fa0a851ba5fc47525aab7fa8b7c711146/python/setup.py#L31 > [2] https://github.com/googleapis/google-api-python-client/pull/841/files -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8979) protoc-gen-mypy: program not found or is not executable
[
https://issues.apache.org/jira/browse/BEAM-8979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17076696#comment-17076696
]
Udi Meiri commented on BEAM-8979:
-
Have you tried running "pip install -r build-requirements.txt" before running
setup.py?
> protoc-gen-mypy: program not found or is not executable
> ---
>
> Key: BEAM-8979
> URL: https://issues.apache.org/jira/browse/BEAM-8979
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core, test-failures
>Reporter: Kamil Wasilewski
>Assignee: Chad Dombrova
>Priority: Major
> Fix For: Not applicable
>
> Time Spent: 12h 10m
> Remaining Estimate: 0h
>
> In some tests, `:sdks:python:sdist:` task fails due to problems in finding
> protoc-gen-mypy. The following tests are affected (there might be more):
> *
> [https://builds.apache.org/job/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/]
> *
> [https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/
>
> |https://builds.apache.org/job/beam_BiqQueryIO_Write_Performance_Test_Python_Batch/]
> Relevant logs:
> {code:java}
> 10:46:32 > Task :sdks:python:sdist FAILED
> 10:46:32 Requirement already satisfied: mypy-protobuf==1.12 in
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages
> (1.12)
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/descriptor.proto
> but not used.
> 10:46:32 beam_fn_api.proto: warning: Import google/protobuf/wrappers.proto
> but not used.
> 10:46:32 protoc-gen-mypy: program not found or is not executable
> 10:46:32 --mypy_out: protoc-gen-mypy: Plugin failed with status code 1.
> 10:46:32
> /home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/dist.py:476:
> UserWarning: Normalizing '2.19.0.dev' to '2.19.0.dev0'
> 10:46:32 normalized_version,
> 10:46:32 Traceback (most recent call last):
> 10:46:32 File "setup.py", line 295, in
> 10:46:32 'mypy': generate_protos_first(mypy),
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/__init__.py",
> line 145, in setup
> 10:46:32 return distutils.core.setup(**attrs)
> 10:46:32 File "/usr/lib/python3.7/distutils/core.py", line 148, in setup
> 10:46:32 dist.run_commands()
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 966, in
> run_commands
> 10:46:32 self.run_command(cmd)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/build/gradleenv/192237/lib/python3.7/site-packages/setuptools/command/sdist.py",
> line 44, in run
> 10:46:32 self.run_command('egg_info')
> 10:46:32 File "/usr/lib/python3.7/distutils/cmd.py", line 313, in
> run_command
> 10:46:32 self.distribution.run_command(command)
> 10:46:32 File "/usr/lib/python3.7/distutils/dist.py", line 985, in
> run_command
> 10:46:32 cmd_obj.run()
> 10:46:32 File "setup.py", line 220, in run
> 10:46:32 gen_protos.generate_proto_files(log=log)
> 10:46:32 File
> "/home/jenkins/jenkins-slave/workspace/beam_LoadTests_Python_37_ParDo_Dataflow_Batch_PR/src/sdks/python/gen_protos.py",
> line 144, in generate_proto_files
> 10:46:32 '%s' % ret_code)
> 10:46:32 RuntimeError: Protoc returned non-zero status (see logs for
> details): 1
> {code}
>
> This is what I have tried so far to resolve this (without being successful):
> * Including _--plugin=protoc-gen-mypy=\{abs_path_to_executable}_ parameter
> to the _protoc_ call ingen_protos.py:131
> * Appending protoc-gen-mypy's directory to the PATH variable
> I wasn't able to reproduce this error locally.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9695) beam_PostCommit_Go failing
[
https://issues.apache.org/jira/browse/BEAM-9695?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri updated BEAM-9695:
Description:
[~lostluck]
I'm seeing this in stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
was:
[~lostluck]
I'm seeing this in the stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
> beam_PostCommit_Go failing
> --
>
> Key: BEAM-9695
> URL: https://issues.apache.org/jira/browse/BEAM-9695
> Project: Beam
> Issue Type: Bug
> Components: sdk-go, test-failures
>Reporter: Udi Meiri
>Assignee: Robert Bradshaw
>Priority: Major
>
> [~lostluck]
> I'm seeing this in stackdriver logs:
> {code}
> rpc error: code = Unimplemented desc = unknown service
> org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
> {code}
> First failure:
> https://builds.apache.org/job/beam_PostCommit_Go/6624/
> which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9695) beam_PostCommit_Go failing
Udi Meiri created BEAM-9695:
---
Summary: beam_PostCommit_Go failing
Key: BEAM-9695
URL: https://issues.apache.org/jira/browse/BEAM-9695
Project: Beam
Issue Type: Bug
Components: sdk-go, test-failures
Reporter: Udi Meiri
Assignee: Robert Bradshaw
[~lostluck]
I'm seeing this in the stackdriver logs:
{code}
rpc error: code = Unimplemented desc = unknown service
org.apache.beam.model.job_management.v1.LegacyArtifactRetrievalService
{code}
First failure:
https://builds.apache.org/job/beam_PostCommit_Go/6624/
which lists commits from https://issues.apache.org/jira/browse/BEAM-9577
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9313) beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException: org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
[
https://issues.apache.org/jira/browse/BEAM-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074840#comment-17074840
]
Udi Meiri commented on BEAM-9313:
-
Or perhaps Gradle is prematurely cleaning up the tmpdir (such as
/tmp/groovy-generated-8526855374001831465-tmpdir)
> beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException:
> org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
>
>
> Key: BEAM-9313
> URL: https://issues.apache.org/jira/browse/BEAM-9313
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Labels: currently-failing
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Jenkins: https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/885/
> Gradle: https://scans.gradle.com/s/wbwr4nzluxtlc
> :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow and
> :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow are broken
> with ClassDefNotFound errors like:
> {code}
> INFO: 2020-02-08T11:05:44.038Z: Finished operation
> WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close
>
> Feb 08, 2020 11:05:45 AM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> INFO: 2020-02-08T11:05:44.119Z: Executing operation
> WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
>
> Feb 08, 2020 11:05:47 AM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> SEVERE: 2020-02-08T11:05:46.096Z: java.lang.NoClassDefFoundError:
> org/apache/beam/model/pipeline/v1/StandardWindowFns$SessionsPayload$Enum
> at
> org.apache.beam.runners.dataflow.worker.repackaged.org.apache.beam.runners.core.construction.WindowingStrategyTranslation.(WindowingStrategyTranslation.java:211)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowParDoFnFactory.deserializeWindowingStrategy(GroupAlsoByWindowParDoFnFactory.java:234)
>
> at
> org.apache.beam.runners.dataflow.worker.GroupAlsoByWindowParDoFnFactory.create(GroupAlsoByWindowParDoFnFactory.java:99)
>
> at
> org.apache.beam.runners.dataflow.worker.DefaultParDoFnFactory.create(DefaultParDoFnFactory.java:75)
>
> at
> org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.createParDoOperation(IntrinsicMapTaskExecutorFactory.java:264)
>
> at
> org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.access$000(IntrinsicMapTaskExecutorFactory.java:86)
>
> at
> org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:183)
>
> at
> org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory$1.typedApply(IntrinsicMapTaskExecutorFactory.java:165)
>
> at
> org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:63)
>
> at
> org.apache.beam.runners.dataflow.worker.graph.Networks$TypeSafeNodeFunction.apply(Networks.java:50)
>
> at
> org.apache.beam.runners.dataflow.worker.graph.Networks.replaceDirectedNetworkNodes(Networks.java:87)
>
> at
> org.apache.beam.runners.dataflow.worker.IntrinsicMapTaskExecutorFactory.create(IntrinsicMapTaskExecutorFactory.java:125)
>
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.doWork(BatchDataflowWorker.java:353)
>
> at
> org.apache.beam.runners.dataflow.worker.BatchDataflowWorker.getAndPerformWork(BatchDataflowWorker.java:306)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.doWork(DataflowBatchWorkerHarness.java:140)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:120)
>
> at
> org.apache.beam.runners.dataflow.worker.DataflowBatchWorkerHarness$WorkerThread.call(DataflowBatchWorkerHarness.java:107)
>
> at
[jira] [Commented] (BEAM-9313) beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException: org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
[
https://issues.apache.org/jira/browse/BEAM-9313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074832#comment-17074832
]
Udi Meiri commented on BEAM-9313:
-
Seeing something similar:
{code}
[INFO] Changes detected - recompiling the module!
[WARNING] File encoding has not been set, using platform encoding UTF-8, i.e.
build is platform dependent!
[INFO] Compiling 31 source files to
/tmp/groovy-generated-8526855374001831465-tmpdir/word-count-beam/target/classes
Waiting for pipeline to produce more results...
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java
(default-cli) on project word-count-beam: An exception occured while executing
the Java class. org.apache.beam.examples.complete.game.LeaderBoard -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.6.0:java
(default-cli) on project word-count-beam: An exception occured while executing
the Java class. org.apache.beam.examples.complete.game.LeaderBoard -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e
switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR]
[ERROR] For more information about the errors and possible solutions, please
read the following articles:
[ERROR] [Help 1]
http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[INFO] -
[ERROR] COMPILATION ERROR :
[INFO] -
[ERROR]
/tmp/groovy-generated-8526855374001831465-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/common/ExamplePubsubTopicOptions.java:
error reading
/tmp/groovy-generated-8526855374001831465-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/common/ExamplePubsubTopicOptions.java;
/tmp/groovy-generated-8526855374001831465-tmpdir/word-count-beam/src/main/java/org/apache/beam/examples/common/ExamplePubsubTopicOptions.java
(No such file or directory)
... (lots of file not found errors)
{code}
https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/937/consoleFull
The failed task is sometimes
:runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow (above) and
sometimes :runners:direct-java:runMobileGamingJavaDirect and sometimes both.
Since these tasks run in parallel, perhaps they're overwriting each others'
output files.
> beam_PostRelease_NightlySnapshot failure due to ClassNotFoundException:
> org.apache.beam.model.pipeline.v1.StandardWindowFns$SessionsPayload$Enum
>
>
> Key: BEAM-9313
> URL: https://issues.apache.org/jira/browse/BEAM-9313
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Brian Hulette
>Assignee: Brian Hulette
>Priority: Major
> Labels: currently-failing
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Jenkins: https://builds.apache.org/job/beam_PostRelease_NightlySnapshot/885/
> Gradle: https://scans.gradle.com/s/wbwr4nzluxtlc
> :runners:google-cloud-dataflow-java:runQuickstartJavaDataflow and
> :runners:google-cloud-dataflow-java:runMobileGamingJavaDataflow are broken
> with ClassDefNotFound errors like:
> {code}
> INFO: 2020-02-08T11:05:44.038Z: Finished operation
> WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Close
>
> Feb 08, 2020 11:05:45 AM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> INFO: 2020-02-08T11:05:44.119Z: Executing operation
> WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/GroupByKey/Read+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues+WordCount.CountWords/Count.PerElement/Combine.perKey(Count)/Combine.GroupedValues/Extract+MapElements/Map+WriteCounts/WriteFiles/RewindowIntoGlobal/Window.Assign+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/WriteUnshardedBundles+WriteCounts/WriteFiles/GatherTempFileResults/View.AsList/ParDo(ToIsmRecordForGlobalWindow)+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Reify+WriteCounts/WriteFiles/WriteUnshardedBundlesToTempFiles/GroupUnwritten/Write
>
> Feb 08, 2020 11:05:47 AM
> org.apache.beam.runners.dataflow.util.MonitoringUtil$LoggingHandler process
> SEVERE: 2020-02-08T11:05:46.096Z:
[jira] [Commented] (BEAM-8409) docker-credential-gcloud not installed or not available in PATH
[
https://issues.apache.org/jira/browse/BEAM-8409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17074818#comment-17074818
]
Udi Meiri commented on BEAM-8409:
-
This is now happening on multiple machines (7, 8, 11, 12) for
https://builds.apache.org/job/beam_PreCommit_CommunityMetrics_Cron/
> docker-credential-gcloud not installed or not available in PATH
> ---
>
> Key: BEAM-8409
> URL: https://issues.apache.org/jira/browse/BEAM-8409
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Kamil Wasilewski
>Assignee: Yifan Zou
>Priority: Major
> Labels: currently-failing
>
> _Use this form to file an issue for test failure:_
> *
> [beam_PreCommit_CommunityMetrics_Commit|https://builds.apache.org/view/A-D/view/Beam/view/All/job/beam_PreCommit_CommunityMetrics_Commit/1355/]
> *
> [beam_PostCommit_Python2_PR|https://builds.apache.org/job/beam_PostCommit_Python2_PR]
> Initial investigation:
> Jenkins job fails when executing docker-compose script.
> It seems the only Jenkins worker affected is *apache-beam-jenkins-15.*
>
> Relevant logs:
> 1)
>
> {code:java}
> 11:56:24 Execution failed for task ':beam-test-infra-metrics:composeUp'.
> 11:56:24 > Exit-code 255 when calling docker-compose, stdout: postgresql uses
> an image, skipping
> 11:56:24 prometheus uses an image, skipping
> 11:56:24 pushgateway uses an image, skipping
> 11:56:24 alertmanager uses an image, skipping
> 11:56:24 Building grafana
> 11:56:24 [17038] Failed to execute script docker-compose
> 11:56:24 Traceback (most recent call last):
> 11:56:24 File "bin/docker-compose", line 6, in
> 11:56:24 File "compose/cli/main.py", line 71, in main
> 11:56:24 File "compose/cli/main.py", line 127, in perform_command
> 11:56:24 File "compose/cli/main.py", line 287, in build
> 11:56:24 File "compose/project.py", line 386, in build
> 11:56:24 File "compose/project.py", line 368, in build_service
> 11:56:24 File "compose/service.py", line 1084, in build
> 11:56:24 File "site-packages/docker/api/build.py", line 260, in build
> 11:56:24 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 11:56:24 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 11:56:24 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 11:56:24 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 11:56:24 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 11:56:24 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {code}
> 2)
> {code:java}
> 16:26:08 [9316] Failed to execute script docker-compose
> 16:26:08 Traceback (most recent call last):
> 16:26:08 File "bin/docker-compose", line 6, in
> 16:26:08 File "compose/cli/main.py", line 71, in main
> 16:26:08 File "compose/cli/main.py", line 127, in perform_command
> 16:26:08 File "compose/cli/main.py", line 287, in build
> 16:26:08 File "compose/project.py", line 386, in build
> 16:26:08 File "compose/project.py", line 368, in build_service
> 16:26:08 File "compose/service.py", line 1084, in build
> 16:26:08 File "site-packages/docker/api/build.py", line 260, in build
> 16:26:08 File "site-packages/docker/api/build.py", line 307, in
> _set_auth_headers
> 16:26:08 File "site-packages/docker/auth.py", line 310, in
> get_all_credentials
> 16:26:08 File "site-packages/docker/auth.py", line 262, in
> _resolve_authconfig_credstore
> 16:26:08 File "site-packages/docker/auth.py", line 287, in
> _get_store_instance
> 16:26:08 File "site-packages/dockerpycreds/store.py", line 25, in __init__
> 16:26:08 dockerpycreds.errors.InitializationError: docker-credential-gcloud
> not installed or not available in PATH
> {code}
> **
>
>
> _After you've filled out the above details, pl__ease [assign the issue to an
> individual|https://beam.apache.org/contribute/postcommits-guides/index.html#find_specialist].
> Assignee should [treat test failures as
> high-priority|https://beam.apache.org/contribute/postcommits-policies/#assigned-failing-test],
> helping to fix the issue or find a more appropriate owner. See [Apache Beam
> Post-Commit
> Policies|https://beam.apache.org/contribute/postcommits-policies]._
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9620) textio (and fileio in general) takes too long to estimate sizes of large globs
[ https://issues.apache.org/jira/browse/BEAM-9620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068187#comment-17068187 ] Udi Meiri commented on BEAM-9620: - Since this is an estimation, perhaps there should be limits on how much it samples or a maximum amount of time it can spend sampling (overall). > textio (and fileio in general) takes too long to estimate sizes of large globs > -- > > Key: BEAM-9620 > URL: https://issues.apache.org/jira/browse/BEAM-9620 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Chamikara Madhusanka Jayalath >Priority: Major > > As a workaround we could introduce a way to not perform size estimation when > reading large globs. For example Java SDK has withHintMatchesManyFiles() > option. > > [https://github.com/apache/beam/blob/850e8469de798d45ec535fe90cb2dc5dbda4974a/sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextIO.java#L371] > > Additionally, seems like we are repeating the size estimation where the same > PCollection read from a file-based source is applied to multiple PTransforms. > > See following for more details. > [https://stackoverflow.com/questions/60874942/avoid-recomputing-size-of-all-cloud-storage-files-in-gcsio-beam-python-sdk] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-1894) Race conditions in python direct runner eager mode
[ https://issues.apache.org/jira/browse/BEAM-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri reassigned BEAM-1894: --- Assignee: Udi Meiri > Race conditions in python direct runner eager mode > -- > > Key: BEAM-1894 > URL: https://issues.apache.org/jira/browse/BEAM-1894 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Vikas Kedigehalli >Assignee: Udi Meiri >Priority: Major > > test_eager_pipeline > (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline_test.py#L283) > fails with the following error: > ERROR: test_eager_pipeline (apache_beam.pipeline_test.PipelineTest) > -- > Traceback (most recent call last): > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline_test.py", > line 285, in test_eager_pipeline > self.assertEqual([1, 4, 9], p | Create([1, 2, 3]) | Map(lambda x: x*x)) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/transforms/ptransform.py", > line 387, in __ror__ > p.run().wait_until_finish() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", > line 160, in run > self.to_runner_api(), self.runner, self.options).run(False) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", > line 169, in run > return self.runner.run(self) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 99, in run > result.wait_until_finish() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 166, in wait_until_finish > self._executor.await_completion() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", > line 336, in await_completion > self._executor.await_completion() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", > line 308, in __call__ > uncommitted_bundle.get_elements_iterable()) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/evaluation_context.py", > line 176, in append_to_cache > self._cache.append(applied_ptransform, tag, elements) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 138, in append > self._cache[(applied_ptransform, tag)].extend(elements) > TypeError: 'NoneType' object has no attribute '__getitem__' > This is triggered when Create is changed to a custom source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-9377) Python typehints: Map wrapper prevents Optional stripping
[
https://issues.apache.org/jira/browse/BEAM-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri resolved BEAM-9377.
-
Fix Version/s: Not applicable
Resolution: Won't Fix
> Python typehints: Map wrapper prevents Optional stripping
> -
>
> Key: BEAM-9377
> URL: https://issues.apache.org/jira/browse/BEAM-9377
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
> Fix For: Not applicable
>
>
> This existing test is wrong:
> {code}
> def test_map_wrapper_optional_output(self):
> # Optional does affect output type (Nones are NOT ignored).
> def map_fn(unused_element: int) -> typehints.Optional[int]:
> return 1
> th = beam.Map(map_fn).get_type_hints()
> self.assertEqual(th.input_types, ((int, ), {}))
> self.assertEqual(th.output_types, ((typehints.Optional[int], ), {}))
> {code}
> The resulting output type should be int.
> {code}
> inital output hint:
> Optional[int]
> with wrapper:
> Iterable[Optional[int]]
> with DoFn.default_type_hints:
> Optional[int]
> {code}
> However any Nones returned by a DoFn's process method are dropped, so the
> actual element_type returned is plain int.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9377) Python typehints: Map wrapper prevents Optional stripping
[
https://issues.apache.org/jira/browse/BEAM-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9377:
---
Assignee: Udi Meiri
> Python typehints: Map wrapper prevents Optional stripping
> -
>
> Key: BEAM-9377
> URL: https://issues.apache.org/jira/browse/BEAM-9377
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Udi Meiri
>Priority: Major
>
> This existing test is wrong:
> {code}
> def test_map_wrapper_optional_output(self):
> # Optional does affect output type (Nones are NOT ignored).
> def map_fn(unused_element: int) -> typehints.Optional[int]:
> return 1
> th = beam.Map(map_fn).get_type_hints()
> self.assertEqual(th.input_types, ((int, ), {}))
> self.assertEqual(th.output_types, ((typehints.Optional[int], ), {}))
> {code}
> The resulting output type should be int.
> {code}
> inital output hint:
> Optional[int]
> with wrapper:
> Iterable[Optional[int]]
> with DoFn.default_type_hints:
> Optional[int]
> {code}
> However any Nones returned by a DoFn's process method are dropped, so the
> actual element_type returned is plain int.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9377) Python typehints: Map wrapper prevents Optional stripping
[
https://issues.apache.org/jira/browse/BEAM-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068131#comment-17068131
]
Udi Meiri commented on BEAM-9377:
-
Verified for myself that Nones returned from map_fn indeed appear in the
PCollection:
{code}
def test_typed_map_optional(self):
# Optional does affect output type (Nones are NOT ignored).
def map_fn(element: int) -> typehints.Optional[int]:
if element == 1:
return None
else:
return element
result = [1, 2, 3] | beam.Map(map_fn)
self.assertCountEqual([None, 2, 3], result)
{code}
> Python typehints: Map wrapper prevents Optional stripping
> -
>
> Key: BEAM-9377
> URL: https://issues.apache.org/jira/browse/BEAM-9377
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
>
> This existing test is wrong:
> {code}
> def test_map_wrapper_optional_output(self):
> # Optional does affect output type (Nones are NOT ignored).
> def map_fn(unused_element: int) -> typehints.Optional[int]:
> return 1
> th = beam.Map(map_fn).get_type_hints()
> self.assertEqual(th.input_types, ((int, ), {}))
> self.assertEqual(th.output_types, ((typehints.Optional[int], ), {}))
> {code}
> The resulting output type should be int.
> {code}
> inital output hint:
> Optional[int]
> with wrapper:
> Iterable[Optional[int]]
> with DoFn.default_type_hints:
> Optional[int]
> {code}
> However any Nones returned by a DoFn's process method are dropped, so the
> actual element_type returned is plain int.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9377) Python typehints: Map wrapper prevents Optional stripping
[
https://issues.apache.org/jira/browse/BEAM-9377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068132#comment-17068132
]
Udi Meiri commented on BEAM-9377:
-
Nothing to do, closing
> Python typehints: Map wrapper prevents Optional stripping
> -
>
> Key: BEAM-9377
> URL: https://issues.apache.org/jira/browse/BEAM-9377
> Project: Beam
> Issue Type: New Feature
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
>
> This existing test is wrong:
> {code}
> def test_map_wrapper_optional_output(self):
> # Optional does affect output type (Nones are NOT ignored).
> def map_fn(unused_element: int) -> typehints.Optional[int]:
> return 1
> th = beam.Map(map_fn).get_type_hints()
> self.assertEqual(th.input_types, ((int, ), {}))
> self.assertEqual(th.output_types, ((typehints.Optional[int], ), {}))
> {code}
> The resulting output type should be int.
> {code}
> inital output hint:
> Optional[int]
> with wrapper:
> Iterable[Optional[int]]
> with DoFn.default_type_hints:
> Optional[int]
> {code}
> However any Nones returned by a DoFn's process method are dropped, so the
> actual element_type returned is plain int.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-1894) Race conditions in python direct runner eager mode
[ https://issues.apache.org/jira/browse/BEAM-1894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068130#comment-17068130 ] Udi Meiri commented on BEAM-1894: - EagerRunner was removed in https://github.com/apache/beam/pull/4492 > Race conditions in python direct runner eager mode > -- > > Key: BEAM-1894 > URL: https://issues.apache.org/jira/browse/BEAM-1894 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Vikas Kedigehalli >Priority: Major > > test_eager_pipeline > (https://github.com/apache/beam/blob/master/sdks/python/apache_beam/pipeline_test.py#L283) > fails with the following error: > ERROR: test_eager_pipeline (apache_beam.pipeline_test.PipelineTest) > -- > Traceback (most recent call last): > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline_test.py", > line 285, in test_eager_pipeline > self.assertEqual([1, 4, 9], p | Create([1, 2, 3]) | Map(lambda x: x*x)) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/transforms/ptransform.py", > line 387, in __ror__ > p.run().wait_until_finish() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", > line 160, in run > self.to_runner_api(), self.runner, self.options).run(False) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/pipeline.py", > line 169, in run > return self.runner.run(self) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 99, in run > result.wait_until_finish() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 166, in wait_until_finish > self._executor.await_completion() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", > line 336, in await_completion > self._executor.await_completion() > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/executor.py", > line 308, in __call__ > uncommitted_bundle.get_elements_iterable()) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/evaluation_context.py", > line 176, in append_to_cache > self._cache.append(applied_ptransform, tag, elements) > File > "/usr/local/google/home/vikasrk/work/incubator-beam/sdks/python/apache_beam/runners/direct/direct_runner.py", > line 138, in append > self._cache[(applied_ptransform, tag)].extend(elements) > TypeError: 'NoneType' object has no attribute '__getitem__' > This is triggered when Create is changed to a custom source. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9471) Error "No files to stage has been found" on Windows machines running 2.19.0
[
https://issues.apache.org/jira/browse/BEAM-9471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17068010#comment-17068010
]
Udi Meiri commented on BEAM-9471:
-
[~tvalentyn]
> Error "No files to stage has been found" on Windows machines running 2.19.0
> ---
>
> Key: BEAM-9471
> URL: https://issues.apache.org/jira/browse/BEAM-9471
> Project: Beam
> Issue Type: Bug
> Components: runner-core, runner-dataflow
>Affects Versions: 2.19.0
> Environment: Microsoft Windows 10, Java 8
>Reporter: Samuel Waggoner
>Priority: Major
>
> Windows users on our team have started encountering "No files to stage has
> been found" errors when attempting to run Beam pipelines on Dataflow. They
> are still able to run on the direct runner fine.
> It's difficult for me to provide a minimal reproduction case because I don't
> use a Windows machine. But I believe the failure happens when running _any_
> pipeline on Dataflow from a Windows machine without explicitly setting
> --filesToStage.
> I suspect this change is the cause, because it seems to completely change the
> code for gathering files to stage from the classpath:
> [https://github.com/apache/beam/commit/9629ff6c5bfd70770046468adf44e2eeb69e606d]
> I believe this stackoverflow is about the same issue:
> [https://stackoverflow.com/questions/60586141/dataflowrunner-exits-with-no-files-to-stage-has-been-found]
>
> {code:java}
> Caused by: java.lang.IllegalArgumentException: No files to stage has been
> found.
> at
> org.apache.beam.runners.dataflow.DataflowRunner.fromOptions(DataflowRunner.java:281)
> ... 30 more{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[
https://issues.apache.org/jira/browse/BEAM-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067281#comment-17067281
]
Udi Meiri edited comment on BEAM-8078 at 3/26/20, 1:35 AM:
---
Test was merged, but fails on Jenkins. I am making a PR to disable it until
it's fixed.
Error is:
{code}
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 100
messages.)
but: Expected 100 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: dict_items([(b'0: 1', 1), (b'1: 1', 1), (b'2: 1',
1), (b'3: 1', 1), (b'4: 1', 1), (b'5: 1', 1), (b'6: 1', 1), (b'7: 1', 1), (b'8:
1', 1), (b'9: 1', 1), (b'10: 1', 1), (b'11: 1', 1), (b'12: 1', 1), (b'13: 1',
1), (b'14: 1', 1), (b'15: 1', 1), (b'16: 1', 1), (b'17: 1', 1), (b'18: 1', 1),
(b'19: 1', 1), (b'20: 1', 1), (b'21: 1', 1), (b'22: 1', 1), (b'23: 1', 1),
(b'24: 1', 1), (b'25: 1', 1), (b'26: 1', 1), (b'27: 1', 1), (b'28: 1', 1),
(b'29: 1', 1), (b'30: 1', 1), (b'31: 1', 1), (b'32: 1', 1), (b'33: 1', 1),
(b'34: 1', 1), (b'35: 1', 1), (b'36: 1', 1), (b'37: 1', 1), (b'38: 1', 1),
(b'39: 1', 1), (b'40: 1', 1), (b'41: 1', 1), (b'42: 1', 1), (b'43: 1', 1),
(b'44: 1', 1), (b'45: 1', 1), (b'46: 1', 1), (b'47: 1', 1), (b'48: 1', 1),
(b'49: 1', 1), (b'50: 1', 1), (b'51: 1', 1), (b'52: 1', 1), (b'53: 1', 1),
(b'54: 1', 1), (b'55: 1', 1), (b'56: 1', 1), (b'57: 1', 1), (b'58: 1', 1),
(b'59: 1', 1), (b'60: 1', 1), (b'61: 1', 1), (b'62: 1', 1), (b'63: 1', 1),
(b'64: 1', 1), (b'65: 1', 1), (b'66: 1', 1), (b'67: 1', 1), (b'68: 1', 1),
(b'69: 1', 1), (b'70: 1', 1), (b'71: 1', 1), (b'72: 1', 1), (b'73: 1', 1),
(b'74: 1', 1), (b'75: 1', 1), (b'76: 1', 1), (b'77: 1', 1), (b'78: 1', 1),
(b'79: 1', 1), (b'80: 1', 1), (b'81: 1', 1), (b'82: 1', 1), (b'83: 1', 1),
(b'84: 1', 1), (b'85: 1', 1), (b'86: 1', 1), (b'87: 1', 1), (b'88: 1', 1),
(b'89: 1', 1), (b'90: 1', 1), (b'91: 1', 1), (b'92: 1', 1), (b'93: 1', 1),
(b'94: 1', 1), (b'95: 1', 1), (b'96: 1', 1), (b'97: 1', 1), (b'98: 1', 1),
(b'99: 1', 1)])
Unexpected: dict_items([])
{code}
https://builds.apache.org/job/beam_PostCommit_Python37/1985/
was (Author: udim):
Test was merged, but fails on Jenkins. I am making a PR to disable it until
it's fixed.
Error is:
{code}
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 100
messages.)
but: Expected 100 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: dict_items([(b'0: 1', 1), (b'1: 1', 1), (b'2: 1',
1), (b'3: 1', 1), (b'4: 1', 1), (b'5: 1', 1), (b'6: 1', 1), (b'7: 1', 1), (b'8:
1', 1), (b'9: 1', 1), (b'10: 1', 1), (b'11: 1', 1), (b'12: 1', 1), (b'13: 1',
1), (b'14: 1', 1), (b'15: 1', 1), (b'16: 1', 1), (b'17: 1', 1), (b'18: 1', 1),
(b'19: 1', 1), (b'20: 1', 1), (b'21: 1', 1), (b'22: 1', 1), (b'23: 1', 1),
(b'24: 1', 1), (b'25: 1', 1), (b'26: 1', 1), (b'27: 1', 1), (b'28: 1', 1),
(b'29: 1', 1), (b'30: 1', 1), (b'31: 1', 1), (b'32: 1', 1), (b'33: 1', 1),
(b'34: 1', 1), (b'35: 1', 1), (b'36: 1', 1), (b'37: 1', 1), (b'38: 1', 1),
(b'39: 1', 1), (b'40: 1', 1), (b'41: 1', 1), (b'42: 1', 1), (b'43: 1', 1),
(b'44: 1', 1), (b'45: 1', 1), (b'46: 1', 1), (b'47: 1', 1), (b'48: 1', 1),
(b'49: 1', 1), (b'50: 1', 1), (b'51: 1', 1), (b'52: 1', 1), (b'53: 1', 1),
(b'54: 1', 1), (b'55: 1', 1), (b'56: 1', 1), (b'57: 1', 1), (b'58: 1', 1),
(b'59: 1', 1), (b'60: 1', 1), (b'61: 1', 1), (b'62: 1', 1), (b'63: 1', 1),
(b'64: 1', 1), (b'65: 1', 1), (b'66: 1', 1), (b'67: 1', 1), (b'68: 1', 1),
(b'69: 1', 1), (b'70: 1', 1), (b'71: 1', 1), (b'72: 1', 1), (b'73: 1', 1),
(b'74: 1', 1), (b'75: 1', 1), (b'76: 1', 1), (b'77: 1', 1), (b'78: 1', 1),
(b'79: 1', 1), (b'80: 1', 1), (b'81: 1', 1), (b'82: 1', 1), (b'83: 1', 1),
(b'84: 1', 1), (b'85: 1', 1), (b'86: 1', 1), (b'87: 1', 1), (b'88: 1', 1),
(b'89: 1', 1), (b'90: 1', 1), (b'91: 1', 1), (b'92: 1', 1), (b'93: 1', 1),
(b'94: 1', 1), (b'95: 1', 1), (b'96: 1', 1), (b'97: 1', 1), (b'98: 1', 1),
(b'99: 1', 1)])
Unexpected: dict_items([])
{code}
> streaming_wordcount_debugging.py is missing a test
> --
>
> Key: BEAM-8078
> URL: https://issues.apache.org/jira/browse/BEAM-8078
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Aleksey Vysotin
>Priority: Minor
> Labels: beginner, easy, newbie, starter
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> It's example code and should have a basic_test (like the other wordcount
> variants in [1]) to at least verify that it runs in the latest Beam release.
> [1]
> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[
https://issues.apache.org/jira/browse/BEAM-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067281#comment-17067281
]
Udi Meiri edited comment on BEAM-8078 at 3/26/20, 1:34 AM:
---
Test was merged, but fails on Jenkins. I am making a PR to disable it until
it's fixed.
Error is:
{code}
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 100
messages.)
but: Expected 100 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: dict_items([(b'0: 1', 1), (b'1: 1', 1), (b'2: 1',
1), (b'3: 1', 1), (b'4: 1', 1), (b'5: 1', 1), (b'6: 1', 1), (b'7: 1', 1), (b'8:
1', 1), (b'9: 1', 1), (b'10: 1', 1), (b'11: 1', 1), (b'12: 1', 1), (b'13: 1',
1), (b'14: 1', 1), (b'15: 1', 1), (b'16: 1', 1), (b'17: 1', 1), (b'18: 1', 1),
(b'19: 1', 1), (b'20: 1', 1), (b'21: 1', 1), (b'22: 1', 1), (b'23: 1', 1),
(b'24: 1', 1), (b'25: 1', 1), (b'26: 1', 1), (b'27: 1', 1), (b'28: 1', 1),
(b'29: 1', 1), (b'30: 1', 1), (b'31: 1', 1), (b'32: 1', 1), (b'33: 1', 1),
(b'34: 1', 1), (b'35: 1', 1), (b'36: 1', 1), (b'37: 1', 1), (b'38: 1', 1),
(b'39: 1', 1), (b'40: 1', 1), (b'41: 1', 1), (b'42: 1', 1), (b'43: 1', 1),
(b'44: 1', 1), (b'45: 1', 1), (b'46: 1', 1), (b'47: 1', 1), (b'48: 1', 1),
(b'49: 1', 1), (b'50: 1', 1), (b'51: 1', 1), (b'52: 1', 1), (b'53: 1', 1),
(b'54: 1', 1), (b'55: 1', 1), (b'56: 1', 1), (b'57: 1', 1), (b'58: 1', 1),
(b'59: 1', 1), (b'60: 1', 1), (b'61: 1', 1), (b'62: 1', 1), (b'63: 1', 1),
(b'64: 1', 1), (b'65: 1', 1), (b'66: 1', 1), (b'67: 1', 1), (b'68: 1', 1),
(b'69: 1', 1), (b'70: 1', 1), (b'71: 1', 1), (b'72: 1', 1), (b'73: 1', 1),
(b'74: 1', 1), (b'75: 1', 1), (b'76: 1', 1), (b'77: 1', 1), (b'78: 1', 1),
(b'79: 1', 1), (b'80: 1', 1), (b'81: 1', 1), (b'82: 1', 1), (b'83: 1', 1),
(b'84: 1', 1), (b'85: 1', 1), (b'86: 1', 1), (b'87: 1', 1), (b'88: 1', 1),
(b'89: 1', 1), (b'90: 1', 1), (b'91: 1', 1), (b'92: 1', 1), (b'93: 1', 1),
(b'94: 1', 1), (b'95: 1', 1), (b'96: 1', 1), (b'97: 1', 1), (b'98: 1', 1),
(b'99: 1', 1)])
Unexpected: dict_items([])
{code}
was (Author: udim):
Test was merged, but fails on Jenkins. I am making a PR to disable it until
it's fixed.
> streaming_wordcount_debugging.py is missing a test
> --
>
> Key: BEAM-8078
> URL: https://issues.apache.org/jira/browse/BEAM-8078
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Udi Meiri
>Assignee: Aleksey Vysotin
>Priority: Minor
> Labels: beginner, easy, newbie, starter
> Time Spent: 6h 40m
> Remaining Estimate: 0h
>
> It's example code and should have a basic_test (like the other wordcount
> variants in [1]) to at least verify that it runs in the latest Beam release.
> [1]
> https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-8078) streaming_wordcount_debugging.py is missing a test
[ https://issues.apache.org/jira/browse/BEAM-8078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067281#comment-17067281 ] Udi Meiri commented on BEAM-8078: - Test was merged, but fails on Jenkins. I am making a PR to disable it until it's fixed. > streaming_wordcount_debugging.py is missing a test > -- > > Key: BEAM-8078 > URL: https://issues.apache.org/jira/browse/BEAM-8078 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Aleksey Vysotin >Priority: Minor > Labels: beginner, easy, newbie, starter > Time Spent: 6h 40m > Remaining Estimate: 0h > > It's example code and should have a basic_test (like the other wordcount > variants in [1]) to at least verify that it runs in the latest Beam release. > [1] > https://github.com/apache/beam/tree/master/sdks/python/apache_beam/examples -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9610) BigQueryWriteIntegrationTests.test_big_query_write flaky
[
https://issues.apache.org/jira/browse/BEAM-9610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Udi Meiri reassigned BEAM-9610:
---
Assignee: Juta Staes
> BigQueryWriteIntegrationTests.test_big_query_write flaky
>
>
> Key: BEAM-9610
> URL: https://issues.apache.org/jira/browse/BEAM-9610
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp, test-failures
>Reporter: Udi Meiri
>Assignee: Juta Staes
>Priority: Major
>
> {code}
> Expected: (Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4,
> 'привет')])
> but: Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')]
> Actual data is []
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37_PR/118/
> https://builds.apache.org/job/beam_PostCommit_Python35/2081/
> https://builds.apache.org/job/beam_PostCommit_Python35/2079/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9610) BigQueryWriteIntegrationTests.test_big_query_write flaky
Udi Meiri created BEAM-9610:
---
Summary: BigQueryWriteIntegrationTests.test_big_query_write flaky
Key: BEAM-9610
URL: https://issues.apache.org/jira/browse/BEAM-9610
Project: Beam
Issue Type: Bug
Components: io-py-gcp, test-failures
Reporter: Udi Meiri
{code}
Expected: (Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')])
but: Expected data is [(1, 'abc'), (2, 'def'), (3, '你好'), (4, 'привет')]
Actual data is []
{code}
https://builds.apache.org/job/beam_PostCommit_Python37_PR/118/
https://builds.apache.org/job/beam_PostCommit_Python35/2081/
https://builds.apache.org/job/beam_PostCommit_Python35/2079/
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (BEAM-9398) Python type hints: AbstractDoFnWrapper does not wrap setup
[ https://issues.apache.org/jira/browse/BEAM-9398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri resolved BEAM-9398. - Fix Version/s: 2.21.0 Resolution: Fixed > Python type hints: AbstractDoFnWrapper does not wrap setup > -- > > Key: BEAM-9398 > URL: https://issues.apache.org/jira/browse/BEAM-9398 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 40m > Remaining Estimate: 0h > > And possibly other methods. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9028) bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
[
https://issues.apache.org/jira/browse/BEAM-9028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17066802#comment-17066802
]
Udi Meiri commented on BEAM-9028:
-
Sure, np
> bigquery_read_it_test.ReadTests.test_iobase_source: grpc error
> --
>
> Key: BEAM-9028
> URL: https://issues.apache.org/jira/browse/BEAM-9028
> Project: Beam
> Issue Type: Bug
> Components: io-py-gcp
>Reporter: Udi Meiri
>Assignee: Kamil Wasilewski
>Priority: Major
>
> It's not clear what the root cause is (there's also a "cannot allocate
> memory" error message).
> {code}
> 14:16:06 ERROR: test_iobase_source
> (apache_beam.io.gcp.bigquery_read_it_test.ReadTests)
> 14:16:06
> --
> 14:16:06 Traceback (most recent call last):
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py",
> line 154, in test_iobase_source
> 14:16:06 assert_that(result, equal_to(self.TABLE_DATA))
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 480, in __exit__
> 14:16:06 self.run().wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 460, in run
> 14:16:06 self._options).run(False)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/pipeline.py",
> line 473, in run
> 14:16:06 return self.runner.run_pipeline(self, self._options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/flink_runner.py",
> line 45, in run_pipeline
> 14:16:06 return super(FlinkRunner, self).run_pipeline(pipeline, options)
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 330, in run_pipeline
> 14:16:06 result.wait_until_finish()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/sdks/python/apache_beam/runners/portability/portable_runner.py",
> line 438, in wait_until_finish
> 14:16:06 for state_response in self._state_stream:
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 416, in __next__
> 14:16:06 return self._next()
> 14:16:06File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python37/src/build/gradleenv/2022703441/lib/python3.7/site-packages/grpc/_channel.py",
> line 703, in _next
> 14:16:06 raise self
> 14:16:06 grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous
> of RPC that terminated with:
> 14:16:06 status = StatusCode.UNAVAILABLE
> 14:16:06 details = "Socket closed"
> 14:16:06 debug_error_string =
> "{"created":"@1577139361.436477926","description":"Error received from peer
> ipv4:127.0.0.1:51557","file":"src/core/lib/surface/call.cc","file_line":1056,"grpc_message":"Socket
> closed","grpc_status":14}"
> 14:16:06 >
> {code}
> https://builds.apache.org/job/beam_PostCommit_Python37/1210/timestamps/?time=HH:mm:ss=GMT-8=en_US
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-5422) Update BigQueryIO DynamicDestinations documentation to clarify usage of getDestination() and getTable()
[ https://issues.apache.org/jira/browse/BEAM-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065147#comment-17065147 ] Udi Meiri commented on BEAM-5422: - There's nothing I can look into myself - I don't know what to look for except what I've already mentioned. I'm not sure what to ask on the dev list either. > Update BigQueryIO DynamicDestinations documentation to clarify usage of > getDestination() and getTable() > --- > > Key: BEAM-5422 > URL: https://issues.apache.org/jira/browse/BEAM-5422 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > > Currently, there are some details related to these methods that should be > further clarified. For example, getTable() is expected to return a unique > value for each destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-5422) Update BigQueryIO DynamicDestinations documentation to clarify usage of getDestination() and getTable()
[ https://issues.apache.org/jira/browse/BEAM-5422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17065102#comment-17065102 ] Udi Meiri commented on BEAM-5422: - Ping on this. I independently discovered the issue that getTable should return a unique value. Do you recall what the other details are that need to be documented? > Update BigQueryIO DynamicDestinations documentation to clarify usage of > getDestination() and getTable() > --- > > Key: BEAM-5422 > URL: https://issues.apache.org/jira/browse/BEAM-5422 > Project: Beam > Issue Type: Improvement > Components: io-java-gcp >Reporter: Chamikara Madhusanka Jayalath >Assignee: Chamikara Madhusanka Jayalath >Priority: Major > > Currently, there are some details related to these methods that should be > further clarified. For example, getTable() is expected to return a unique > value for each destination. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17064995#comment-17064995
]
Udi Meiri commented on BEAM-9119:
-
Here it is timing out after 600s:
https://builds.apache.org/job/beam_PreCommit_Python_Commit/11867/
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value =
[jira] [Commented] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17063546#comment-17063546
]
Udi Meiri commented on BEAM-9119:
-
https://builds.apache.org/job/beam_PreCommit_Python_Cron/2532/console
Yes let's do something. Either disable or fix
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Assignee: Robert Bradshaw
>Priority: Major
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value =
[jira] [Updated] (BEAM-9529) Remove googledatastore package
[ https://issues.apache.org/jira/browse/BEAM-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-9529: Affects Version/s: 2.21.0 > Remove googledatastore package > -- > > Key: BEAM-9529 > URL: https://issues.apache.org/jira/browse/BEAM-9529 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Affects Versions: 2.21.0 >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Followup to https://issues.apache.org/jira/browse/BEAM-4543 > Reasons: > 1. v1/datastoreio.py (which uses googledatastore) has had a deprecation > notice since 2.13 (admittedly citing a 3.0 Beam release). > v1new/datastoreio.py is the replacement, using the newer package. > 2. googledatastore does not support Python 3, and 2.7 support is going away > on Beam. > 3. googledatastore depends on an old version of httplib2 [1], which is > starting to conflict with other packages using new versions [2]. > [1] > https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/85e3856fa0a851ba5fc47525aab7fa8b7c711146/python/setup.py#L31 > [2] https://github.com/googleapis/google-api-python-client/pull/841/files -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9529) Remove googledatastore package
[ https://issues.apache.org/jira/browse/BEAM-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Udi Meiri updated BEAM-9529: Fix Version/s: 2.21.0 > Remove googledatastore package > -- > > Key: BEAM-9529 > URL: https://issues.apache.org/jira/browse/BEAM-9529 > Project: Beam > Issue Type: Bug > Components: io-py-gcp >Reporter: Udi Meiri >Assignee: Udi Meiri >Priority: Major > Fix For: 2.21.0 > > Time Spent: 10m > Remaining Estimate: 0h > > Followup to https://issues.apache.org/jira/browse/BEAM-4543 > Reasons: > 1. v1/datastoreio.py (which uses googledatastore) has had a deprecation > notice since 2.13 (admittedly citing a 3.0 Beam release). > v1new/datastoreio.py is the replacement, using the newer package. > 2. googledatastore does not support Python 3, and 2.7 support is going away > on Beam. > 3. googledatastore depends on an old version of httplib2 [1], which is > starting to conflict with other packages using new versions [2]. > [1] > https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/85e3856fa0a851ba5fc47525aab7fa8b7c711146/python/setup.py#L31 > [2] https://github.com/googleapis/google-api-python-client/pull/841/files -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9529) Remove googledatastore package
Udi Meiri created BEAM-9529: --- Summary: Remove googledatastore package Key: BEAM-9529 URL: https://issues.apache.org/jira/browse/BEAM-9529 Project: Beam Issue Type: Bug Components: io-py-gcp Reporter: Udi Meiri Assignee: Udi Meiri Followup to https://issues.apache.org/jira/browse/BEAM-4543 Reasons: 1. v1/datastoreio.py (which uses googledatastore) has had a deprecation notice since 2.13 (admittedly citing a 3.0 Beam release). v1new/datastoreio.py is the replacement, using the newer package. 2. googledatastore does not support Python 3, and 2.7 support is going away on Beam. 3. googledatastore depends on an old version of httplib2 [1], which is starting to conflict with other packages using new versions [2]. [1] https://github.com/GoogleCloudPlatform/google-cloud-datastore/blob/85e3856fa0a851ba5fc47525aab7fa8b7c711146/python/setup.py#L31 [2] https://github.com/googleapis/google-api-python-client/pull/841/files -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-9489) python typehints: reduce the number of runtime-type-check wrappers
Udi Meiri created BEAM-9489: --- Summary: python typehints: reduce the number of runtime-type-check wrappers Key: BEAM-9489 URL: https://issues.apache.org/jira/browse/BEAM-9489 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Udi Meiri The --runtime_type_check flag is considered to be a performance penalty. One of the possible slowdowns is the wrapper around DoFn.process(), which adds an extra call to the wrapped process() method. Currently, there are 2 wrappers applied: OutputCheckWrapperDoFn and TypeCheckWrapperDoFn. These could be merged into one to possibly save on overhead. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9487) GBKs on unbounded pcolls with global windows and no triggers should fail
[
https://issues.apache.org/jira/browse/BEAM-9487?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17057217#comment-17057217
]
Udi Meiri commented on BEAM-9487:
-
Note that the above unit test doesn't hang as expected, probably due to how
TestStream is implemented.
> GBKs on unbounded pcolls with global windows and no triggers should fail
>
>
> Key: BEAM-9487
> URL: https://issues.apache.org/jira/browse/BEAM-9487
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Udi Meiri
>Priority: Major
>
> This, according to "4.2.2.1 GroupByKey and unbounded PCollections" in
> https://beam.apache.org/documentation/programming-guide/.
> bq. If you do apply GroupByKey or CoGroupByKey to a group of unbounded
> PCollections without setting either a non-global windowing strategy, a
> trigger strategy, or both for each collection, Beam generates an
> IllegalStateException error at pipeline construction time.
> Example where this doesn't happen in Python SDK:
> https://stackoverflow.com/questions/60623246/merge-pcollection-with-apache-beam
> I also believe that this unit test should fail, since test_stream is
> unbounded, uses global window, and has no triggers.
> {code}
> def test_global_window_gbk_fail(self):
> with TestPipeline() as p:
> test_stream = TestStream()
> _ = p | test_stream | GroupByKey()
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (BEAM-9487) GBKs on unbounded pcolls with global windows and no triggers should fail
Udi Meiri created BEAM-9487:
---
Summary: GBKs on unbounded pcolls with global windows and no
triggers should fail
Key: BEAM-9487
URL: https://issues.apache.org/jira/browse/BEAM-9487
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Udi Meiri
This, according to "4.2.2.1 GroupByKey and unbounded PCollections" in
https://beam.apache.org/documentation/programming-guide/.
bq. If you do apply GroupByKey or CoGroupByKey to a group of unbounded
PCollections without setting either a non-global windowing strategy, a trigger
strategy, or both for each collection, Beam generates an IllegalStateException
error at pipeline construction time.
Example where this doesn't happen in Python SDK:
https://stackoverflow.com/questions/60623246/merge-pcollection-with-apache-beam
I also believe that this unit test should fail, since test_stream is unbounded,
uses global window, and has no triggers.
{code}
def test_global_window_gbk_fail(self):
with TestPipeline() as p:
test_stream = TestStream()
_ = p | test_stream | GroupByKey()
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
