[jira] [Updated] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
[ https://issues.apache.org/jira/browse/BEAM-10220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reuben van Ammers updated BEAM-10220: - Description: Currently, RowJson.java doesn't have support for deserialising JSON with fields missing - it fails, even if the beam schema is nullable. This is often a problem for reading JSON as nulls are very commonly represented by omitting the field. Additionally, for this reason, not writing nulls is a reasonable when converting from Beam rows to JSON. This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to be implemented. > Add support for implicit nulls in de/serializing JSON > - > > Key: BEAM-10220 > URL: https://issues.apache.org/jira/browse/BEAM-10220 > Project: Beam > Issue Type: New Feature > Components: extensions-java-json >Reporter: Reuben van Ammers >Priority: P3 > > Currently, RowJson.java doesn't have support for deserialising JSON with > fields missing - it fails, even if the beam schema is nullable. This is often > a problem for reading JSON as nulls are very commonly represented by omitting > the field. Additionally, for this reason, not writing nulls is a reasonable > when converting from Beam rows to JSON. > This would help to allow https://issues.apache.org/jira/browse/BEAM-7624 to > be implemented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10220) Add support for implicit nulls in de/serializing JSON
Reuben van Ammers created BEAM-10220: Summary: Add support for implicit nulls in de/serializing JSON Key: BEAM-10220 URL: https://issues.apache.org/jira/browse/BEAM-10220 Project: Beam Issue Type: New Feature Components: extensions-java-json Reporter: Reuben van Ammers -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3573) Test jars should export only tests, and only be exported for select modules
[ https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3573: Labels: beam-fixit stale-P2 (was: beam-fixit) > Test jars should export only tests, and only be exported for select modules > --- > > Key: BEAM-3573 > URL: https://issues.apache.org/jira/browse/BEAM-3573 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 1h 20m > Remaining Estimate: 0h > > Today, we have test-jars that are used as libraries for testing. That is not > what "test jar" means, and dependency management actually does not work > correctly for this. It is OK to depend on a test jar in order to run the > tests therein, and not really OK to depend on one for another reason. > This ticket is a bucket ticket for fixes to this situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2339) Jenkins cross JDK version test on Windows
[ https://issues.apache.org/jira/browse/BEAM-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2339: Labels: beam-fixit stale-P2 (was: beam-fixit) > Jenkins cross JDK version test on Windows > - > > Key: BEAM-2339 > URL: https://issues.apache.org/jira/browse/BEAM-2339 > Project: Beam > Issue Type: Task > Components: build-system, testing >Reporter: Mark Liu >Priority: P2 > Labels: beam-fixit, stale-P2 > > We can set os variant to choose windows for Jenkins test, which can be > combined with JDK version test. So that we can have cross OS / cross JDK > version test. > This discussion came from > https://github.com/apache/beam/pull/3184#pullrequestreview-39303400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-177) Integrate code coverage to build and review process
[ https://issues.apache.org/jira/browse/BEAM-177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-177: --- Labels: beam-fixit stale-P2 (was: beam-fixit) > Integrate code coverage to build and review process > --- > > Key: BEAM-177 > URL: https://issues.apache.org/jira/browse/BEAM-177 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-P2 > > We cannot use codecov, but we can use coveralls. We have the maven plugin > included in the pom and need to invoke it appropriately in our various > builds, and disseminate knowledge about browser extensions to get it into the > pull request UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3573) Test jars should export only tests, and only be exported for select modules
[ https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128855#comment-17128855 ] Beam JIRA Bot commented on BEAM-3573: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Test jars should export only tests, and only be exported for select modules > --- > > Key: BEAM-3573 > URL: https://issues.apache.org/jira/browse/BEAM-3573 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 1h 20m > Remaining Estimate: 0h > > Today, we have test-jars that are used as libraries for testing. That is not > what "test jar" means, and dependency management actually does not work > correctly for this. It is OK to depend on a test jar in order to run the > tests therein, and not really OK to depend on one for another reason. > This ticket is a bucket ticket for fixes to this situation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3215) Add a performance test for HBaseIO
[ https://issues.apache.org/jira/browse/BEAM-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128852#comment-17128852 ] Beam JIRA Bot commented on BEAM-3215: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Add a performance test for HBaseIO > -- > > Key: BEAM-3215 > URL: https://issues.apache.org/jira/browse/BEAM-3215 > Project: Beam > Issue Type: Test > Components: io-java-hbase >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: beam-fixit, stale-P2 > > We should add a large scale performance test for HBaseIO. We could use > PerfKitBenchmarker based performance testing framework [1] to manage a > Kubernetes based muti-node HBase cluster and to publish benchmark results. > Example docker image to use: https://hub.docker.com/r/dajobe/hbase/ > [1] https://beam.apache.org/documentation/io/testing/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4029) Test ValidatesRunner tests for BundleBasedDirectRunner
[ https://issues.apache.org/jira/browse/BEAM-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128851#comment-17128851 ] Beam JIRA Bot commented on BEAM-4029: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Test ValidatesRunner tests for BundleBasedDirectRunner > -- > > Key: BEAM-4029 > URL: https://issues.apache.org/jira/browse/BEAM-4029 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Charles Chen >Priority: P2 > Labels: beam-fixit, stale-P2 > > We currently only run tests for the BundleBasedDirectRunner for streaming > tests. We should also run them for ValidatesRunner tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2313) Continuously execute CassandraIOIT
[ https://issues.apache.org/jira/browse/BEAM-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2313: Labels: beam-fixit stale-P2 (was: beam-fixit) > Continuously execute CassandraIOIT > -- > > Key: BEAM-2313 > URL: https://issues.apache.org/jira/browse/BEAM-2313 > Project: Beam > Issue Type: Task > Components: io-java-cassandra >Reporter: Jean-Baptiste Onofré >Priority: P2 > Labels: beam-fixit, stale-P2 > > It would be great to establishing continuously running IT coverage for > CassandraIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2637) Post commit test for mobile gaming examples
[ https://issues.apache.org/jira/browse/BEAM-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128856#comment-17128856 ] Beam JIRA Bot commented on BEAM-2637: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Post commit test for mobile gaming examples > --- > > Key: BEAM-2637 > URL: https://issues.apache.org/jira/browse/BEAM-2637 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: P2 > Labels: beam-fixit, stale-P2 > > We need post commit test for mobile gaming examples to prevent failures > beyond DirectRunner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2339) Jenkins cross JDK version test on Windows
[ https://issues.apache.org/jira/browse/BEAM-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128857#comment-17128857 ] Beam JIRA Bot commented on BEAM-2339: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Jenkins cross JDK version test on Windows > - > > Key: BEAM-2339 > URL: https://issues.apache.org/jira/browse/BEAM-2339 > Project: Beam > Issue Type: Task > Components: build-system, testing >Reporter: Mark Liu >Priority: P2 > Labels: beam-fixit, stale-P2 > > We can set os variant to choose windows for Jenkins test, which can be > combined with JDK version test. So that we can have cross OS / cross JDK > version test. > This discussion came from > https://github.com/apache/beam/pull/3184#pullrequestreview-39303400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4037) Add Python streaming wordcount snippets and test
[ https://issues.apache.org/jira/browse/BEAM-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128853#comment-17128853 ] Beam JIRA Bot commented on BEAM-4037: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Add Python streaming wordcount snippets and test > > > Key: BEAM-4037 > URL: https://issues.apache.org/jira/browse/BEAM-4037 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Charles Chen >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > We should add Python streaming wordcount snippets and tests. The > documentation will refer to these snippets. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4029) Test ValidatesRunner tests for BundleBasedDirectRunner
[ https://issues.apache.org/jira/browse/BEAM-4029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4029: Labels: beam-fixit stale-P2 (was: beam-fixit) > Test ValidatesRunner tests for BundleBasedDirectRunner > -- > > Key: BEAM-4029 > URL: https://issues.apache.org/jira/browse/BEAM-4029 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Charles Chen >Priority: P2 > Labels: beam-fixit, stale-P2 > > We currently only run tests for the BundleBasedDirectRunner for streaming > tests. We should also run them for ValidatesRunner tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-2313) Continuously execute CassandraIOIT
[ https://issues.apache.org/jira/browse/BEAM-2313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128858#comment-17128858 ] Beam JIRA Bot commented on BEAM-2313: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Continuously execute CassandraIOIT > -- > > Key: BEAM-2313 > URL: https://issues.apache.org/jira/browse/BEAM-2313 > Project: Beam > Issue Type: Task > Components: io-java-cassandra >Reporter: Jean-Baptiste Onofré >Priority: P2 > Labels: beam-fixit, stale-P2 > > It would be great to establishing continuously running IT coverage for > CassandraIO. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3138: Labels: beam-fixit stale-P2 (was: beam-fixit) > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2637) Post commit test for mobile gaming examples
[ https://issues.apache.org/jira/browse/BEAM-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-2637: Labels: beam-fixit stale-P2 (was: beam-fixit) > Post commit test for mobile gaming examples > --- > > Key: BEAM-2637 > URL: https://issues.apache.org/jira/browse/BEAM-2637 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: P2 > Labels: beam-fixit, stale-P2 > > We need post commit test for mobile gaming examples to prevent failures > beyond DirectRunner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4037) Add Python streaming wordcount snippets and test
[ https://issues.apache.org/jira/browse/BEAM-4037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4037: Labels: beam-fixit stale-P2 (was: beam-fixit) > Add Python streaming wordcount snippets and test > > > Key: BEAM-4037 > URL: https://issues.apache.org/jira/browse/BEAM-4037 > Project: Beam > Issue Type: Improvement > Components: sdk-py-core >Reporter: Charles Chen >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 1.5h > Remaining Estimate: 0h > > We should add Python streaming wordcount snippets and tests. The > documentation will refer to these snippets. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-4214) Python ValidatesRunner test coverage is very poor
[ https://issues.apache.org/jira/browse/BEAM-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-4214: Labels: beam-fixit stale-P2 (was: beam-fixit) > Python ValidatesRunner test coverage is very poor > - > > Key: BEAM-4214 > URL: https://issues.apache.org/jira/browse/BEAM-4214 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Eugene Kirpichov >Priority: P2 > Labels: beam-fixit, stale-P2 > > Java has 357 ValidatesRunner tests, Python has 16. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-3215) Add a performance test for HBaseIO
[ https://issues.apache.org/jira/browse/BEAM-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-3215: Labels: beam-fixit stale-P2 (was: beam-fixit) > Add a performance test for HBaseIO > -- > > Key: BEAM-3215 > URL: https://issues.apache.org/jira/browse/BEAM-3215 > Project: Beam > Issue Type: Test > Components: io-java-hbase >Reporter: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: beam-fixit, stale-P2 > > We should add a large scale performance test for HBaseIO. We could use > PerfKitBenchmarker based performance testing framework [1] to manage a > Kubernetes based muti-node HBase cluster and to publish benchmark results. > Example docker image to use: https://hub.docker.com/r/dajobe/hbase/ > [1] https://beam.apache.org/documentation/io/testing/ -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-4214) Python ValidatesRunner test coverage is very poor
[ https://issues.apache.org/jira/browse/BEAM-4214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128850#comment-17128850 ] Beam JIRA Bot commented on BEAM-4214: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Python ValidatesRunner test coverage is very poor > - > > Key: BEAM-4214 > URL: https://issues.apache.org/jira/browse/BEAM-4214 > Project: Beam > Issue Type: Test > Components: sdk-py-core, testing >Reporter: Eugene Kirpichov >Priority: P2 > Labels: beam-fixit, stale-P2 > > Java has 357 ValidatesRunner tests, Python has 16. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128854#comment-17128854 ] Beam JIRA Bot commented on BEAM-3138: - This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-177) Integrate code coverage to build and review process
[ https://issues.apache.org/jira/browse/BEAM-177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128859#comment-17128859 ] Beam JIRA Bot commented on BEAM-177: This issue is P2 but has been unassigned without any comment for 60 days so it has been labeled "stale-P2". If this issue is still affecting you, we care! Please comment and remove the label. Otherwise, in 14 days the issue will be moved to P3. Please see https://beam.apache.org/contribute/jira-priorities/ for a detailed explanation of what these priorities mean. > Integrate code coverage to build and review process > --- > > Key: BEAM-177 > URL: https://issues.apache.org/jira/browse/BEAM-177 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-P2 > > We cannot use codecov, but we can use coveralls. We have the maven plugin > included in the pom and need to invoke it appropriately in our various > builds, and disseminate knowledge about browser extensions to get it into the > pull request UI. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9527) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf is flaky
[
https://issues.apache.org/jira/browse/BEAM-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-9527:
---
Assignee: (was: Boyuan Zhang)
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf
> is flaky
> -
>
> Key: BEAM-9527
> URL: https://issues.apache.org/jira/browse/BEAM-9527
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake, stale-assigned
>
> {noformat}
> self = 0x7fe494edb450>
> split_manager =
> inputs = {'ref_PCollection_PCollection_3_split/Read':
> ['\x7f\xdf;dZ\x1c\xac\t\x00\x00\x00\x01\x0f\x08V\xff\x80\x02capache_beam\nOffsetRange\nq\x01)\x81q\x02}q\x03(U\x04stopq\x04K\x05U\x05startq\x05K\x00ub.\x01\x00@\x14\x00\x00\x00\x00\x00\x00']}
> process_bundle_id = 'bundle_2575'
> def _generate_splits_for_testing(self,
> split_manager,
> inputs, # type: Mapping[str,
> PartitionableBuffer]
> process_bundle_id):
> # type: (...) -> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> split_results = [] # type:
> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> read_transform_id, buffer_data = only_element(inputs.items())
> byte_stream = b''.join(buffer_data)
> num_elements = len(
> list(
> self._get_input_coder_impl(read_transform_id).decode_all(
> byte_stream)))
>
> # Start the split manager in case it wants to set any breakpoints.
> split_manager_generator = split_manager(num_elements)
> try:
> split_fraction = next(split_manager_generator)
> done = False
> except StopIteration:
> done = True
>
> # Send all the data.
> self._send_input_to_worker(
> process_bundle_id, read_transform_id, [byte_stream])
>
> assert self._worker_handler is not None
>
> # Execute the requested splits.
> while not done:
> if split_fraction is None:
> split_result = None
> else:
> split_request = beam_fn_api_pb2.InstructionRequest(
> process_bundle_split=beam_fn_api_pb2.ProcessBundleSplitRequest(
> instruction_id=process_bundle_id,
> desired_splits={
> read_transform_id: beam_fn_api_pb2.
> ProcessBundleSplitRequest.DesiredSplit(
> fraction_of_remainder=split_fraction,
> estimated_input_elements=num_elements)
> }))
> split_response = self._worker_handler.control_conn.push(
> split_request).get() # type:
> beam_fn_api_pb2.InstructionResponse
> for t in (0.05, 0.1, 0.2):
> waiting = ('Instruction not running', 'not yet scheduled')
> if any(msg in split_response.error for msg in waiting):
> time.sleep(t)
> split_response = self._worker_handler.control_conn.push(
> split_request).get()
> if 'Unknown process bundle' in split_response.error:
> # It may have finished too fast.
> split_result = None
> elif split_response.error:
> > raise RuntimeError(split_response.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 190, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 229, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 416, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 479, in process_bundle_split
> E process_bundle_split=processor.try_split(request))
> E File
>
[jira] [Updated] (BEAM-9002) test_flatten_same_pcollections (apache_beam.transforms.ptransform_test.PTransformTest) does not work in Streaming VR suite on Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9002: Labels: beam-fixit (was: beam-fixit stale-assigned) > test_flatten_same_pcollections > (apache_beam.transforms.ptransform_test.PTransformTest) does not work in > Streaming VR suite on Dataflow > -- > > Key: BEAM-9002 > URL: https://issues.apache.org/jira/browse/BEAM-9002 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Valentyn Tymofieiev >Priority: P2 > Labels: beam-fixit > Time Spent: 50m > Remaining Estimate: 0h > > Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the > test times out and was recently added to VR test suite. > [~liumomo315], I will sickbay this test for streaming, could you please help > triage the failure? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9559) Remove smoke load test for Java and Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9559: --- Assignee: (was: Michał Walenia) > Remove smoke load test for Java and Python SDK > -- > > Key: BEAM-9559 > URL: https://issues.apache.org/jira/browse/BEAM-9559 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Priority: P3 > Labels: beam-fixit, stale-assigned > > As discussed in PR: > [https://github.com/apache/beam/pull/11135#discussion_r392852028] > No one seems to use them and regular load test will fail too whenever > something is wrong. There's plenty of load tests now and they are smaller > than they used to be at the time of creating the smoke tests. If something is > wrong there will be quite quick feedback from them. All that makes the smoke > test redundant imo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-9119:
---
Assignee: (was: Robert Bradshaw)
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake, stale-assigned
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value =
[jira] [Updated] (BEAM-9118) apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses is flaky
[
https://issues.apache.org/jira/browse/BEAM-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-9118:
Labels: beam-fixit flake (was: beam-fixit flake stale-assigned)
> apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
> is flaky
>
>
> Key: BEAM-9118
> URL: https://issues.apache.org/jira/browse/BEAM-9118
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
>
> Sample errors:
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1373
> {noformat}
> 4:30:12 self =
> testMethod=test_pardo_unfusable_side_inputs>
> 14:30:12
> 14:30:12 def test_pardo_unfusable_side_inputs(self):
> 14:30:12def cross_product(elem, sides):
> 14:30:12 for side in sides:
> 14:30:12yield elem, side
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(pcoll)),
> 14:30:12 equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 derived = ((pcoll,) | beam.Flatten()
> 14:30:12 | beam.Map(lambda x: (x, x))
> 14:30:12 | beam.GroupByKey()
> 14:30:12 | 'Unkey' >> beam.Map(lambda kv: kv[0]))
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(derived)),
> 14:30:12 > equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12 apache_beam/runners/portability/fn_api_runner_test.py:258:
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12 apache_beam/pipeline.py:481: in __exit__
> 14:30:12 self.run().wait_until_finish()
> 14:30:12 apache_beam/runners/portability/portable_runner.py:445: in
> wait_until_finish
> 14:30:12 for state_response in self._state_stream:
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:416:
> in __next__
> 14:30:12 return self._next()
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:694:
> in _next
> 14:30:12 _common.wait(self._state.condition.wait, _response_ready)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:140:
> in wait
> 14:30:12 _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:105:
> in _wait_once
> 14:30:12 wait_fn(timeout=timeout)
> 14:30:12 /usr/lib/python3.6/threading.py:299: in wait
> 14:30:12 gotit = waiter.acquire(True, timeout)
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12
> 14:30:12 signum = 14, frame =
> 14:30:12
> 14:30:12 def handler(signum, frame):
> 14:30:12msg = 'Timed out after %s seconds.' % self.TIMEOUT_SECS
> 14:30:12print('=' * 20, msg, '=' * 20)
> 14:30:12traceback.print_stack(frame)
> 14:30:12threads_by_id = {th.ident: th for th in threading.enumerate()}
> 14:30:12for thread_id, stack in sys._current_frames().items():
> 14:30:12 th = threads_by_id.get(thread_id)
> 14:30:12 print()
> 14:30:12 print('# Thread:', th or thread_id)
> 14:30:12 traceback.print_stack(stack)
> 14:30:12 > raise BaseException(msg)
> 14:30:12 E BaseException: Timed out after 60 seconds.
> 14:30:12
> 14:30:12 apache_beam/runners/portability/portable_runner_test.py:77:
> BaseException
> {noformat}
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1366/
> {noformat}
> 09:06:01 self =
> testMethod=test_assert_that>
> 09:06:01
> 09:06:01 def test_assert_that(self):
> 09:06:01# TODO: figure out a way for fn_api_runner to parse and raise
> the
> 09:06:01# underlying exception.
> 09:06:01with self.assertRaisesRegex(Exception, 'Failed assert'):
> 09:06:01 with self.create_pipeline() as p:
> 09:06:01 > assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
> 09:06:01 E AssertionError: "Failed assert" does not match "Pipeline
> timed out waiting for job service subprocess."
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (BEAM-9118) apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses is flaky
[
https://issues.apache.org/jira/browse/BEAM-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-9118:
---
Assignee: (was: Robert Bradshaw)
> apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
> is flaky
>
>
> Key: BEAM-9118
> URL: https://issues.apache.org/jira/browse/BEAM-9118
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake, stale-assigned
>
> Sample errors:
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1373
> {noformat}
> 4:30:12 self =
> testMethod=test_pardo_unfusable_side_inputs>
> 14:30:12
> 14:30:12 def test_pardo_unfusable_side_inputs(self):
> 14:30:12def cross_product(elem, sides):
> 14:30:12 for side in sides:
> 14:30:12yield elem, side
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(pcoll)),
> 14:30:12 equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 derived = ((pcoll,) | beam.Flatten()
> 14:30:12 | beam.Map(lambda x: (x, x))
> 14:30:12 | beam.GroupByKey()
> 14:30:12 | 'Unkey' >> beam.Map(lambda kv: kv[0]))
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(derived)),
> 14:30:12 > equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12 apache_beam/runners/portability/fn_api_runner_test.py:258:
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12 apache_beam/pipeline.py:481: in __exit__
> 14:30:12 self.run().wait_until_finish()
> 14:30:12 apache_beam/runners/portability/portable_runner.py:445: in
> wait_until_finish
> 14:30:12 for state_response in self._state_stream:
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:416:
> in __next__
> 14:30:12 return self._next()
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:694:
> in _next
> 14:30:12 _common.wait(self._state.condition.wait, _response_ready)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:140:
> in wait
> 14:30:12 _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:105:
> in _wait_once
> 14:30:12 wait_fn(timeout=timeout)
> 14:30:12 /usr/lib/python3.6/threading.py:299: in wait
> 14:30:12 gotit = waiter.acquire(True, timeout)
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12
> 14:30:12 signum = 14, frame =
> 14:30:12
> 14:30:12 def handler(signum, frame):
> 14:30:12msg = 'Timed out after %s seconds.' % self.TIMEOUT_SECS
> 14:30:12print('=' * 20, msg, '=' * 20)
> 14:30:12traceback.print_stack(frame)
> 14:30:12threads_by_id = {th.ident: th for th in threading.enumerate()}
> 14:30:12for thread_id, stack in sys._current_frames().items():
> 14:30:12 th = threads_by_id.get(thread_id)
> 14:30:12 print()
> 14:30:12 print('# Thread:', th or thread_id)
> 14:30:12 traceback.print_stack(stack)
> 14:30:12 > raise BaseException(msg)
> 14:30:12 E BaseException: Timed out after 60 seconds.
> 14:30:12
> 14:30:12 apache_beam/runners/portability/portable_runner_test.py:77:
> BaseException
> {noformat}
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1366/
> {noformat}
> 09:06:01 self =
> testMethod=test_assert_that>
> 09:06:01
> 09:06:01 def test_assert_that(self):
> 09:06:01# TODO: figure out a way for fn_api_runner to parse and raise
> the
> 09:06:01# underlying exception.
> 09:06:01with self.assertRaisesRegex(Exception, 'Failed assert'):
> 09:06:01 with self.create_pipeline() as p:
> 09:06:01 > assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
> 09:06:01 E AssertionError: "Failed assert" does not match "Pipeline
> timed out waiting for job service subprocess."
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9002) test_flatten_same_pcollections (apache_beam.transforms.ptransform_test.PTransformTest) does not work in Streaming VR suite on Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128842#comment-17128842 ] Beam JIRA Bot commented on BEAM-9002: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > test_flatten_same_pcollections > (apache_beam.transforms.ptransform_test.PTransformTest) does not work in > Streaming VR suite on Dataflow > -- > > Key: BEAM-9002 > URL: https://issues.apache.org/jira/browse/BEAM-9002 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Valentyn Tymofieiev >Priority: P2 > Labels: beam-fixit > Time Spent: 50m > Remaining Estimate: 0h > > Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the > test times out and was recently added to VR test suite. > [~liumomo315], I will sickbay this test for streaming, could you please help > triage the failure? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7914) Add python 3 test in crossLanguageValidateRunner task
[ https://issues.apache.org/jira/browse/BEAM-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7914: --- Assignee: (was: Chamikara Madhusanka Jayalath) > Add python 3 test in crossLanguageValidateRunner task > - > > Key: BEAM-7914 > URL: https://issues.apache.org/jira/browse/BEAM-7914 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Priority: P2 > Labels: beam-fixit, portability, stale-assigned > > add python 3 test in crossLanguageValidateRunner task -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-9119:
Labels: beam-fixit flake (was: beam-fixit flake stale-assigned)
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in append
> 12:10:27 windowed_key_value =
> coder_impl.decode_from_stream(input_stream, True)
> 12:10:27 apache_beam/coders/coder_impl.py:1153: in decode_from_stream
> 12:10:27 value =
[jira] [Updated] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
[
https://issues.apache.org/jira/browse/BEAM-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-6419:
Labels: beam-fixit (was: beam-fixit stale-assigned)
> [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data.
> --
>
> Key: BEAM-6419
> URL: https://issues.apache.org/jira/browse/BEAM-6419
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: beam-fixit
>
> {code}
> [ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data. For report generation the same class files must be
> used as at runtime.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
> does not match.
> {code}
> ... and so on.
> There's some discussion of similar-sounding issues at
> https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
> If JaCoCo is looking at the class files, but tests run against the shaded
> jar, this would be expected because only byte-for-byte identical class files
> will match.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-9527) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf is flaky
[
https://issues.apache.org/jira/browse/BEAM-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128839#comment-17128839
]
Beam JIRA Bot commented on BEAM-9527:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf
> is flaky
> -
>
> Key: BEAM-9527
> URL: https://issues.apache.org/jira/browse/BEAM-9527
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
>
> {noformat}
> self = 0x7fe494edb450>
> split_manager =
> inputs = {'ref_PCollection_PCollection_3_split/Read':
> ['\x7f\xdf;dZ\x1c\xac\t\x00\x00\x00\x01\x0f\x08V\xff\x80\x02capache_beam\nOffsetRange\nq\x01)\x81q\x02}q\x03(U\x04stopq\x04K\x05U\x05startq\x05K\x00ub.\x01\x00@\x14\x00\x00\x00\x00\x00\x00']}
> process_bundle_id = 'bundle_2575'
> def _generate_splits_for_testing(self,
> split_manager,
> inputs, # type: Mapping[str,
> PartitionableBuffer]
> process_bundle_id):
> # type: (...) -> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> split_results = [] # type:
> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> read_transform_id, buffer_data = only_element(inputs.items())
> byte_stream = b''.join(buffer_data)
> num_elements = len(
> list(
> self._get_input_coder_impl(read_transform_id).decode_all(
> byte_stream)))
>
> # Start the split manager in case it wants to set any breakpoints.
> split_manager_generator = split_manager(num_elements)
> try:
> split_fraction = next(split_manager_generator)
> done = False
> except StopIteration:
> done = True
>
> # Send all the data.
> self._send_input_to_worker(
> process_bundle_id, read_transform_id, [byte_stream])
>
> assert self._worker_handler is not None
>
> # Execute the requested splits.
> while not done:
> if split_fraction is None:
> split_result = None
> else:
> split_request = beam_fn_api_pb2.InstructionRequest(
> process_bundle_split=beam_fn_api_pb2.ProcessBundleSplitRequest(
> instruction_id=process_bundle_id,
> desired_splits={
> read_transform_id: beam_fn_api_pb2.
> ProcessBundleSplitRequest.DesiredSplit(
> fraction_of_remainder=split_fraction,
> estimated_input_elements=num_elements)
> }))
> split_response = self._worker_handler.control_conn.push(
> split_request).get() # type:
> beam_fn_api_pb2.InstructionResponse
> for t in (0.05, 0.1, 0.2):
> waiting = ('Instruction not running', 'not yet scheduled')
> if any(msg in split_response.error for msg in waiting):
> time.sleep(t)
> split_response = self._worker_handler.control_conn.push(
> split_request).get()
> if 'Unknown process bundle' in split_response.error:
> # It may have finished too fast.
> split_result = None
> elif split_response.error:
> > raise RuntimeError(split_response.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 190, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 229, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 416, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
>
[jira] [Commented] (BEAM-6335) GroupByKey uses data insertion pipeline in streaming tests
[ https://issues.apache.org/jira/browse/BEAM-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128845#comment-17128845 ] Beam JIRA Bot commented on BEAM-6335: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > GroupByKey uses data insertion pipeline in streaming tests > -- > > Key: BEAM-6335 > URL: https://issues.apache.org/jira/browse/BEAM-6335 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kasia Kucharczyk >Priority: P2 > Labels: beam-fixit > Time Spent: 50m > Remaining Estimate: 0h > > Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to > load test streaming. > This task contains following steps: > # Create GroupByKey streaming test that accepts bytes > # To stop test after arrived messages it is required to add a matcher. The > matcher should work on number of messages because in case of load testing it > would be difficult to compare big load of bytes (also casting bytes to string > to compare load would be difficult). > # All data is generated by SyntheticDataPublisher.java, which is sending > bytes produced by synthetic source to PubSub. PubSub is used as a streaming > source for the Python test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7794) DynamoDBIOTest is blocking forever
[ https://issues.apache.org/jira/browse/BEAM-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7794: Labels: beam-fixit (was: beam-fixit stale-assigned) > DynamoDBIOTest is blocking forever > -- > > Key: BEAM-7794 > URL: https://issues.apache.org/jira/browse/BEAM-7794 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.15.0 >Reporter: Ismaël Mejía >Priority: P2 > Labels: beam-fixit > Time Spent: 3.5h > Remaining Estimate: 0h > > It was reported in the mailing list that there is a problem in some > environments with the test container. Until we can reproduce it maybe is a > good idea to add the @Ignore annotation to the test class. (in the aws2 > module probably too) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-6335) GroupByKey uses data insertion pipeline in streaming tests
[ https://issues.apache.org/jira/browse/BEAM-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-6335: Labels: beam-fixit (was: beam-fixit stale-assigned) > GroupByKey uses data insertion pipeline in streaming tests > -- > > Key: BEAM-6335 > URL: https://issues.apache.org/jira/browse/BEAM-6335 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kasia Kucharczyk >Priority: P2 > Labels: beam-fixit > Time Spent: 50m > Remaining Estimate: 0h > > Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to > load test streaming. > This task contains following steps: > # Create GroupByKey streaming test that accepts bytes > # To stop test after arrived messages it is required to add a matcher. The > matcher should work on number of messages because in case of load testing it > would be difficult to compare big load of bytes (also casting bytes to string > to compare load would be difficult). > # All data is generated by SyntheticDataPublisher.java, which is sending > bytes produced by synthetic source to PubSub. PubSub is used as a streaming > source for the Python test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-7798) After changes in type inference, apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python is failing in Python 3.5 postcommits
[
https://issues.apache.org/jira/browse/BEAM-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-7798:
---
Assignee: (was: Robert Bradshaw)
> After changes in type inference,
> apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python
> is failing in Python 3.5 postcommits
> ---
>
> Key: BEAM-7798
> URL: https://issues.apache.org/jira/browse/BEAM-7798
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, stale-assigned
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> {noformat}
> Error Message
> Tuple[t0, t1, ...]: each t must be a type. Got Any.
> Stacktrace
> Traceback (most recent call last):
> File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
> yield
> File "/usr/lib/python3.5/unittest/case.py", line 600, in run
> testMethod()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 58, in test_bigquery_read_1M_python
> self.run_bigquery_io_read_pipeline('1M')
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 54, in run_bigquery_io_read_pipeline
> **extra_opts))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py",
> line 74, in run
> p.run()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> else test_runner_api))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 43, in run_pipeline
> self.result = super(TestDirectRunner, self).run_pipeline(pipeline,
> options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 355, in run_pipeline
> pipeline.replace_all(_get_transform_overrides(options))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 389, in replace_all
> self._replace(override)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 300, in _replace
> self.visit(TransformUpdater(self))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 447, in visit
> self._root_transform().visit(visitor, self, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 822, in visit
> visitor.enter_composite_transform(self)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 295, in enter_composite_transform
> self._replace_if_needed(transform_node)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 264, in _replace_if_needed
> new_output = replacement_transform.expand(input_node)
> File
>
[jira] [Assigned] (BEAM-7794) DynamoDBIOTest is blocking forever
[ https://issues.apache.org/jira/browse/BEAM-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-7794: --- Assignee: (was: Cam Mach) > DynamoDBIOTest is blocking forever > -- > > Key: BEAM-7794 > URL: https://issues.apache.org/jira/browse/BEAM-7794 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.15.0 >Reporter: Ismaël Mejía >Priority: P2 > Labels: beam-fixit, stale-assigned > Time Spent: 3.5h > Remaining Estimate: 0h > > It was reported in the mailing list that there is a problem in some > environments with the test container. Until we can reproduce it maybe is a > good idea to add the @Ignore annotation to the test class. (in the aws2 > module probably too) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9559) Remove smoke load test for Java and Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128838#comment-17128838 ] Beam JIRA Bot commented on BEAM-9559: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Remove smoke load test for Java and Python SDK > -- > > Key: BEAM-9559 > URL: https://issues.apache.org/jira/browse/BEAM-9559 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Priority: P3 > Labels: beam-fixit > > As discussed in PR: > [https://github.com/apache/beam/pull/11135#discussion_r392852028] > No one seems to use them and regular load test will fail too whenever > something is wrong. There's plenty of load tests now and they are smaller > than they used to be at the time of creating the smoke tests. If something is > wrong there will be quite quick feedback from them. All that makes the smoke > test redundant imo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7798) After changes in type inference, apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python is failing in Python 3.5 postcommits
[
https://issues.apache.org/jira/browse/BEAM-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-7798:
Labels: beam-fixit (was: beam-fixit stale-assigned)
> After changes in type inference,
> apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python
> is failing in Python 3.5 postcommits
> ---
>
> Key: BEAM-7798
> URL: https://issues.apache.org/jira/browse/BEAM-7798
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> {noformat}
> Error Message
> Tuple[t0, t1, ...]: each t must be a type. Got Any.
> Stacktrace
> Traceback (most recent call last):
> File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
> yield
> File "/usr/lib/python3.5/unittest/case.py", line 600, in run
> testMethod()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 58, in test_bigquery_read_1M_python
> self.run_bigquery_io_read_pipeline('1M')
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 54, in run_bigquery_io_read_pipeline
> **extra_opts))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py",
> line 74, in run
> p.run()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> else test_runner_api))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 43, in run_pipeline
> self.result = super(TestDirectRunner, self).run_pipeline(pipeline,
> options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 355, in run_pipeline
> pipeline.replace_all(_get_transform_overrides(options))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 389, in replace_all
> self._replace(override)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 300, in _replace
> self.visit(TransformUpdater(self))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 447, in visit
> self._root_transform().visit(visitor, self, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 822, in visit
> visitor.enter_composite_transform(self)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 295, in enter_composite_transform
> self._replace_if_needed(transform_node)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 264, in _replace_if_needed
> new_output = replacement_transform.expand(input_node)
> File
>
[jira] [Assigned] (BEAM-6335) GroupByKey uses data insertion pipeline in streaming tests
[ https://issues.apache.org/jira/browse/BEAM-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-6335: --- Assignee: (was: Kasia Kucharczyk) > GroupByKey uses data insertion pipeline in streaming tests > -- > > Key: BEAM-6335 > URL: https://issues.apache.org/jira/browse/BEAM-6335 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kasia Kucharczyk >Priority: P2 > Labels: beam-fixit, stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > > Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to > load test streaming. > This task contains following steps: > # Create GroupByKey streaming test that accepts bytes > # To stop test after arrived messages it is required to add a matcher. The > matcher should work on number of messages because in case of load testing it > would be difficult to compare big load of bytes (also casting bytes to string > to compare load would be difficult). > # All data is generated by SyntheticDataPublisher.java, which is sending > bytes produced by synthetic source to PubSub. PubSub is used as a streaming > source for the Python test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-7798) After changes in type inference, apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python is failing in Python 3.5 postcommits
[
https://issues.apache.org/jira/browse/BEAM-7798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128847#comment-17128847
]
Beam JIRA Bot commented on BEAM-7798:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> After changes in type inference,
> apache_beam.io.gcp.bigquery_io_read_it_test.BigqueryIOReadIT.test_bigquery_read_1M_python
> is failing in Python 3.5 postcommits
> ---
>
> Key: BEAM-7798
> URL: https://issues.apache.org/jira/browse/BEAM-7798
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit
> Time Spent: 4h 10m
> Remaining Estimate: 0h
>
> {noformat}
> Error Message
> Tuple[t0, t1, ...]: each t must be a type. Got Any.
> Stacktrace
> Traceback (most recent call last):
> File "/usr/lib/python3.5/unittest/case.py", line 58, in testPartExecutor
> yield
> File "/usr/lib/python3.5/unittest/case.py", line 600, in run
> testMethod()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 58, in test_bigquery_read_1M_python
> self.run_bigquery_io_read_pipeline('1M')
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_it_test.py",
> line 54, in run_bigquery_io_read_pipeline
> **extra_opts))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/io/gcp/bigquery_io_read_pipeline.py",
> line 74, in run
> p.run()
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/testing/test_pipeline.py",
> line 107, in run
> else test_runner_api))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 406, in run
> self._options).run(False)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 419, in run
> return self.runner.run_pipeline(self, self._options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/test_direct_runner.py",
> line 43, in run_pipeline
> self.result = super(TestDirectRunner, self).run_pipeline(pipeline,
> options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 129, in run_pipeline
> return runner.run_pipeline(pipeline, options)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/runners/direct/direct_runner.py",
> line 355, in run_pipeline
> pipeline.replace_all(_get_transform_overrides(options))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 389, in replace_all
> self._replace(override)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 300, in _replace
> self.visit(TransformUpdater(self))
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 447, in visit
> self._root_transform().visit(visitor, self, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 824, in visit
> part.visit(visitor, pipeline, visited)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 822, in visit
> visitor.enter_composite_transform(self)
> File
> "/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python35_PR/src/sdks/python/apache_beam/pipeline.py",
> line 295, in enter_composite_transform
> self._replace_if_needed(transform_node)
> File
>
[jira] [Commented] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
[
https://issues.apache.org/jira/browse/BEAM-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128844#comment-17128844
]
Beam JIRA Bot commented on BEAM-6419:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data.
> --
>
> Key: BEAM-6419
> URL: https://issues.apache.org/jira/browse/BEAM-6419
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: beam-fixit
>
> {code}
> [ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data. For report generation the same class files must be
> used as at runtime.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
> does not match.
> {code}
> ... and so on.
> There's some discussion of similar-sounding issues at
> https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
> If JaCoCo is looking at the class files, but tests run against the shaded
> jar, this would be expected because only byte-for-byte identical class files
> will match.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-9527) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf is flaky
[
https://issues.apache.org/jira/browse/BEAM-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-9527:
Labels: beam-fixit flake (was: beam-fixit flake stale-assigned)
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerSplitTest.test_split_crazy_sdf
> is flaky
> -
>
> Key: BEAM-9527
> URL: https://issues.apache.org/jira/browse/BEAM-9527
> Project: Beam
> Issue Type: Bug
> Components: test-failures
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
>
> {noformat}
> self = 0x7fe494edb450>
> split_manager =
> inputs = {'ref_PCollection_PCollection_3_split/Read':
> ['\x7f\xdf;dZ\x1c\xac\t\x00\x00\x00\x01\x0f\x08V\xff\x80\x02capache_beam\nOffsetRange\nq\x01)\x81q\x02}q\x03(U\x04stopq\x04K\x05U\x05startq\x05K\x00ub.\x01\x00@\x14\x00\x00\x00\x00\x00\x00']}
> process_bundle_id = 'bundle_2575'
> def _generate_splits_for_testing(self,
> split_manager,
> inputs, # type: Mapping[str,
> PartitionableBuffer]
> process_bundle_id):
> # type: (...) -> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> split_results = [] # type:
> List[beam_fn_api_pb2.ProcessBundleSplitResponse]
> read_transform_id, buffer_data = only_element(inputs.items())
> byte_stream = b''.join(buffer_data)
> num_elements = len(
> list(
> self._get_input_coder_impl(read_transform_id).decode_all(
> byte_stream)))
>
> # Start the split manager in case it wants to set any breakpoints.
> split_manager_generator = split_manager(num_elements)
> try:
> split_fraction = next(split_manager_generator)
> done = False
> except StopIteration:
> done = True
>
> # Send all the data.
> self._send_input_to_worker(
> process_bundle_id, read_transform_id, [byte_stream])
>
> assert self._worker_handler is not None
>
> # Execute the requested splits.
> while not done:
> if split_fraction is None:
> split_result = None
> else:
> split_request = beam_fn_api_pb2.InstructionRequest(
> process_bundle_split=beam_fn_api_pb2.ProcessBundleSplitRequest(
> instruction_id=process_bundle_id,
> desired_splits={
> read_transform_id: beam_fn_api_pb2.
> ProcessBundleSplitRequest.DesiredSplit(
> fraction_of_remainder=split_fraction,
> estimated_input_elements=num_elements)
> }))
> split_response = self._worker_handler.control_conn.push(
> split_request).get() # type:
> beam_fn_api_pb2.InstructionResponse
> for t in (0.05, 0.1, 0.2):
> waiting = ('Instruction not running', 'not yet scheduled')
> if any(msg in split_response.error for msg in waiting):
> time.sleep(t)
> split_response = self._worker_handler.control_conn.push(
> split_request).get()
> if 'Unknown process bundle' in split_response.error:
> # It may have finished too fast.
> split_result = None
> elif split_response.error:
> > raise RuntimeError(split_response.error)
> E RuntimeError: Traceback (most recent call last):
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 190, in _execute
> E response = task()
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 229, in
> E lambda: self.create_worker().do_instruction(request), request)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 416, in do_instruction
> E getattr(request, request_type), request.instruction_id)
> E File
> "/home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py2/build/srcs/sdks/python/apache_beam/runners/worker/sdk_worker.py",
> line 479, in process_bundle_split
> E process_bundle_split=processor.try_split(request))
> E File
>
[jira] [Assigned] (BEAM-9002) test_flatten_same_pcollections (apache_beam.transforms.ptransform_test.PTransformTest) does not work in Streaming VR suite on Dataflow
[ https://issues.apache.org/jira/browse/BEAM-9002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot reassigned BEAM-9002: --- Assignee: (was: Ankur Goenka) > test_flatten_same_pcollections > (apache_beam.transforms.ptransform_test.PTransformTest) does not work in > Streaming VR suite on Dataflow > -- > > Key: BEAM-9002 > URL: https://issues.apache.org/jira/browse/BEAM-9002 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Valentyn Tymofieiev >Priority: P2 > Labels: beam-fixit, stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > > Per investigation in https://issues.apache.org/jira/browse/BEAM-8877, the > test times out and was recently added to VR test suite. > [~liumomo315], I will sickbay this test for streaming, could you please help > triage the failure? > Thank you! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-6419) [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no match with execution data.
[
https://issues.apache.org/jira/browse/BEAM-6419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot reassigned BEAM-6419:
---
Assignee: (was: Kenneth Knowles)
> [SQL] Jacoco error: Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data.
> --
>
> Key: BEAM-6419
> URL: https://issues.apache.org/jira/browse/BEAM-6419
> Project: Beam
> Issue Type: Bug
> Components: dsl-sql
>Reporter: Kenneth Knowles
>Priority: P2
> Labels: beam-fixit, stale-assigned
>
> {code}
> [ant:jacocoReport] Classes in bundle 'beam-sdks-java-extensions-sql' do no
> match with execution data. For report generation the same class files must be
> used as at runtime.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/udf/IsNan does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamUnnestRel$Transform does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSetOperatorRelBase$OpType
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/text/TextTableProvider$RowToCsv
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamSortRel does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/agg/CovarianceFn does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/parser/impl/ParseException does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamMinusRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/transform/BeamBuiltinAggregations$IntegerAvg
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProvider
> does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rule/BeamUncollectRule does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRel$1 does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/rel/BeamCalcRel$CalcFn does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/impl/ParseException does not match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTable does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/UdfUdafProvider does not
> match.
> [ant:jacocoReport] Execution data for class
> org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaCSVTable$CsvRecorderDecoder
> does not match.
> {code}
> ... and so on.
> There's some discussion of similar-sounding issues at
> https://stackoverflow.com/questions/31720139/jacoco-code-coverage-report-generator-showing-error-classes-in-bundle-code-c
> If JaCoCo is looking at the class files, but tests run against the shaded
> jar, this would be expected because only byte-for-byte identical class files
> will match.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-7794) DynamoDBIOTest is blocking forever
[ https://issues.apache.org/jira/browse/BEAM-7794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128846#comment-17128846 ] Beam JIRA Bot commented on BEAM-7794: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > DynamoDBIOTest is blocking forever > -- > > Key: BEAM-7794 > URL: https://issues.apache.org/jira/browse/BEAM-7794 > Project: Beam > Issue Type: Bug > Components: io-java-aws >Affects Versions: 2.15.0 >Reporter: Ismaël Mejía >Priority: P2 > Labels: beam-fixit > Time Spent: 3.5h > Remaining Estimate: 0h > > It was reported in the mailing list that there is a problem in some > environments with the test container. Until we can reproduce it maybe is a > good idea to add the @Ignore annotation to the test class. (in the aws2 > module probably too) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9118) apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses is flaky
[
https://issues.apache.org/jira/browse/BEAM-9118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128841#comment-17128841
]
Beam JIRA Bot commented on BEAM-9118:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> apache_beam.runners.portability.portable_runner_test.PortableRunnerTestWithSubprocesses
> is flaky
>
>
> Key: BEAM-9118
> URL: https://issues.apache.org/jira/browse/BEAM-9118
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
>
> Sample errors:
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1373
> {noformat}
> 4:30:12 self =
> testMethod=test_pardo_unfusable_side_inputs>
> 14:30:12
> 14:30:12 def test_pardo_unfusable_side_inputs(self):
> 14:30:12def cross_product(elem, sides):
> 14:30:12 for side in sides:
> 14:30:12yield elem, side
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(pcoll)),
> 14:30:12 equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12with self.create_pipeline() as p:
> 14:30:12 pcoll = p | beam.Create(['a', 'b'])
> 14:30:12 derived = ((pcoll,) | beam.Flatten()
> 14:30:12 | beam.Map(lambda x: (x, x))
> 14:30:12 | beam.GroupByKey()
> 14:30:12 | 'Unkey' >> beam.Map(lambda kv: kv[0]))
> 14:30:12 assert_that(
> 14:30:12 pcoll | beam.FlatMap(cross_product,
> beam.pvalue.AsList(derived)),
> 14:30:12 > equal_to([('a', 'a'), ('a', 'b'), ('b', 'a'), ('b',
> 'b')]))
> 14:30:12
> 14:30:12 apache_beam/runners/portability/fn_api_runner_test.py:258:
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12 apache_beam/pipeline.py:481: in __exit__
> 14:30:12 self.run().wait_until_finish()
> 14:30:12 apache_beam/runners/portability/portable_runner.py:445: in
> wait_until_finish
> 14:30:12 for state_response in self._state_stream:
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:416:
> in __next__
> 14:30:12 return self._next()
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_channel.py:694:
> in _next
> 14:30:12 _common.wait(self._state.condition.wait, _response_ready)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:140:
> in wait
> 14:30:12 _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
> 14:30:12
> target/.tox-py36-gcp-pytest/py36-gcp-pytest/lib/python3.6/site-packages/grpc/_common.py:105:
> in _wait_once
> 14:30:12 wait_fn(timeout=timeout)
> 14:30:12 /usr/lib/python3.6/threading.py:299: in wait
> 14:30:12 gotit = waiter.acquire(True, timeout)
> 14:30:12 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 14:30:12
> 14:30:12 signum = 14, frame =
> 14:30:12
> 14:30:12 def handler(signum, frame):
> 14:30:12msg = 'Timed out after %s seconds.' % self.TIMEOUT_SECS
> 14:30:12print('=' * 20, msg, '=' * 20)
> 14:30:12traceback.print_stack(frame)
> 14:30:12threads_by_id = {th.ident: th for th in threading.enumerate()}
> 14:30:12for thread_id, stack in sys._current_frames().items():
> 14:30:12 th = threads_by_id.get(thread_id)
> 14:30:12 print()
> 14:30:12 print('# Thread:', th or thread_id)
> 14:30:12 traceback.print_stack(stack)
> 14:30:12 > raise BaseException(msg)
> 14:30:12 E BaseException: Timed out after 60 seconds.
> 14:30:12
> 14:30:12 apache_beam/runners/portability/portable_runner_test.py:77:
> BaseException
> {noformat}
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1366/
> {noformat}
> 09:06:01 self =
> testMethod=test_assert_that>
> 09:06:01
> 09:06:01 def test_assert_that(self):
> 09:06:01# TODO: figure out a way for fn_api_runner to parse and raise
> the
> 09:06:01# underlying exception.
> 09:06:01with self.assertRaisesRegex(Exception, 'Failed assert'):
> 09:06:01 with self.create_pipeline() as p:
> 09:06:01 > assert_that(p | beam.Create(['a', 'b']), equal_to(['a']))
> 09:06:01 E
[jira] [Updated] (BEAM-7914) Add python 3 test in crossLanguageValidateRunner task
[ https://issues.apache.org/jira/browse/BEAM-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-7914: Labels: beam-fixit portability (was: beam-fixit portability stale-assigned) > Add python 3 test in crossLanguageValidateRunner task > - > > Key: BEAM-7914 > URL: https://issues.apache.org/jira/browse/BEAM-7914 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Priority: P2 > Labels: beam-fixit, portability > > add python 3 test in crossLanguageValidateRunner task -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9119) apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements is flaky
[
https://issues.apache.org/jira/browse/BEAM-9119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128843#comment-17128843
]
Beam JIRA Bot commented on BEAM-9119:
-
This issue was marked "stale-assigned" and has not received a public comment in
7 days. It is now automatically unassigned. If you are still working on it, you
can assign it to yourself again. Please also give an update about the status of
the work.
> apache_beam.runners.portability.fn_api_runner_test.FnApiRunnerTest[...].test_large_elements
> is flaky
>
>
> Key: BEAM-9119
> URL: https://issues.apache.org/jira/browse/BEAM-9119
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P1
> Labels: beam-fixit, flake
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Saw 3 errors today, all manifest with:
> IndexError: index out of range in apache_beam/coders/slow_stream.py", line
> 169, in read_byte_py3.
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1369
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1365
> https://builds.apache.org/job/beam_PreCommit_Python_Phrase/1370
> Sample logs:
> {noformat}
> 12:10:27 === FAILURES
> ===
> 12:10:27 FnApiRunnerTestWithDisabledCaching.test_large_elements
>
> 12:10:27 [gw0] linux -- Python 3.6.8
> /home/jenkins/jenkins-slave/workspace/beam_PreCommit_Python_Phrase/src/sdks/python/test-suites/tox/py36/build/srcs/sdks/python/target/.tox-py36-gcp-pytest/py36-gcp-pytest/bin/python
> 12:10:27
> 12:10:27 self =
> testMethod=test_large_elements>
> 12:10:27
> 12:10:27 def test_large_elements(self):
> 12:10:27with self.create_pipeline() as p:
> 12:10:27 big = (p
> 12:10:27 | beam.Create(['a', 'a', 'b'])
> 12:10:27 | beam.Map(lambda x: (
> 12:10:27 x, x *
> data_plane._DEFAULT_SIZE_FLUSH_THRESHOLD)))
> 12:10:27
> 12:10:27 side_input_res = (
> 12:10:27 big
> 12:10:27 | beam.Map(lambda x, side: (x[0], side.count(x[0])),
> 12:10:27 beam.pvalue.AsList(big | beam.Map(lambda x:
> x[0]
> 12:10:27 assert_that(side_input_res,
> 12:10:27 equal_to([('a', 2), ('a', 2), ('b', 1)]),
> label='side')
> 12:10:27
> 12:10:27 gbk_res = (
> 12:10:27 big
> 12:10:27 | beam.GroupByKey()
> 12:10:27 | beam.Map(lambda x: x[0]))
> 12:10:27 > assert_that(gbk_res, equal_to(['a', 'b']), label='gbk')
> 12:10:27
> 12:10:27 apache_beam/runners/portability/fn_api_runner_test.py:617:
> 12:10:27 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _
> 12:10:27 apache_beam/pipeline.py:479: in __exit__
> 12:10:27 self.run().wait_until_finish()
> 12:10:27 apache_beam/pipeline.py:459: in run
> 12:10:27 self._options).run(False)
> 12:10:27 apache_beam/pipeline.py:472: in run
> 12:10:27 return self.runner.run_pipeline(self, self._options)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:472: in
> run_pipeline
> 12:10:27 default_environment=self._default_environment))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:480: in
> run_via_runner_api
> 12:10:27 return self.run_stages(stage_context, stages)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:569: in run_stages
> 12:10:27 stage_context.safe_coders)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:889: in _run_stage
> 12:10:27 result, splits = bundle_manager.process_bundle(data_input,
> data_output)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> process_bundle
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:586: in
> result_iterator
> 12:10:27 yield fs.pop().result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:432: in result
> 12:10:27 return self.__get_result()
> 12:10:27 /usr/lib/python3.6/concurrent/futures/_base.py:384: in __get_result
> 12:10:27 raise self._exception
> 12:10:27 apache_beam/utils/thread_pool_executor.py:44: in run
> 12:10:27 self._future.set_result(self._fn(*self._fn_args,
> **self._fn_kwargs))
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2076: in
> 12:10:27 part, expected_outputs), part_inputs):
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:2020: in
> process_bundle
> 12:10:27 expected_outputs[output.transform_id]).append(output.data)
> 12:10:27 apache_beam/runners/portability/fn_api_runner.py:285: in
[jira] [Commented] (BEAM-7914) Add python 3 test in crossLanguageValidateRunner task
[ https://issues.apache.org/jira/browse/BEAM-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128840#comment-17128840 ] Beam JIRA Bot commented on BEAM-7914: - This issue was marked "stale-assigned" and has not received a public comment in 7 days. It is now automatically unassigned. If you are still working on it, you can assign it to yourself again. Please also give an update about the status of the work. > Add python 3 test in crossLanguageValidateRunner task > - > > Key: BEAM-7914 > URL: https://issues.apache.org/jira/browse/BEAM-7914 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Priority: P2 > Labels: beam-fixit, portability > > add python 3 test in crossLanguageValidateRunner task -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9559) Remove smoke load test for Java and Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-9559: Labels: beam-fixit (was: beam-fixit stale-assigned) > Remove smoke load test for Java and Python SDK > -- > > Key: BEAM-9559 > URL: https://issues.apache.org/jira/browse/BEAM-9559 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Priority: P3 > Labels: beam-fixit > > As discussed in PR: > [https://github.com/apache/beam/pull/11135#discussion_r392852028] > No one seems to use them and regular load test will fail too whenever > something is wrong. There's plenty of load tests now and they are smaller > than they used to be at the time of creating the smoke tests. If something is > wrong there will be quite quick feedback from them. All that makes the smoke > test redundant imo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-1681) Add Unit Tests for fixes in BEAM-1649
[
https://issues.apache.org/jira/browse/BEAM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128826#comment-17128826
]
Beam JIRA Bot commented on BEAM-1681:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Add Unit Tests for fixes in BEAM-1649
> -
>
> Key: BEAM-1681
> URL: https://issues.apache.org/jira/browse/BEAM-1681
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> BEAM-1649 was delivered without UTs included.
> This is a follow-up to add UT coverage for the following functions:
> {noformat}
> OrderedPositionRangeTracker.stop_position()
> ValueStateTag.__repr__()
> typehints.decorators._unpack_positional_arg_hints()
> WindowedTypeConstraint.type_check()
> PipelineOptions.__getattr__()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-1683) Add unit tests for counters.py
[
https://issues.apache.org/jira/browse/BEAM-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-1683:
Labels: beam-fixit stale-assigned (was: beam-fixit)
> Add unit tests for counters.py
> --
>
> Key: BEAM-1683
> URL: https://issues.apache.org/jira/browse/BEAM-1683
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Rahul Sabbineni
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> Python-SDK's {{apache_beam/utils/counters.py}} does not have associated unit
> tests and have low (indirect) test coverage.
> Create the respective tests to ensure code quality and increase test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-1620) Add streaming Dataflow ValidatesRunner coverage
[ https://issues.apache.org/jira/browse/BEAM-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128829#comment-17128829 ] Beam JIRA Bot commented on BEAM-1620: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Add streaming Dataflow ValidatesRunner coverage > --- > > Key: BEAM-1620 > URL: https://issues.apache.org/jira/browse/BEAM-1620 > Project: Beam > Issue Type: Test > Components: runner-dataflow, testing >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-assigned > > Currently, the runner validation test suite is not run on Dataflow in > streaming mode. In fact, it should be able to run - all the functionality is > in place. I think this is just a matter of maven + Jenkins + making sure not > to leak a bunch of streaming jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1680) Uber-JIRA: Extend Python-SDK's unit test coverage
[
https://issues.apache.org/jira/browse/BEAM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-1680:
Labels: beam-fixit stale-assigned (was: beam-fixit)
> Uber-JIRA: Extend Python-SDK's unit test coverage
> -
>
> Key: BEAM-1680
> URL: https://issues.apache.org/jira/browse/BEAM-1680
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> Some components of the Python-SDK has low coverage rate.
> As of now ([ea33e33|https://github.com/apache/beam/tree/ea33e33]) the test
> coverage looks as follows (excluding autocomplete_test.py):
> {code}
> Ran 1050 tests in 145.653s
> OK (skipped=14)
> py27gcp runtests: commands[5] | coverage report
> Name
> Stmts Miss Cover
> --
> apache_beam/__init__.py
> 12 192%
> apache_beam/coders/__init__.py
> 2 0 100%
> apache_beam/coders/coder_impl.py
>414 2394%
> apache_beam/coders/coders.py
>392 6883%
> apache_beam/coders/observable.py
> 10 0 100%
> apache_beam/coders/slow_stream.py
>105 694%
> apache_beam/coders/typecoders.py
> 69 691%
> apache_beam/error.py
> 7 0 100%
> apache_beam/internal/__init__.py
> 0 0 100%
> apache_beam/internal/gcp/__init__.py
> 0 0 100%
> apache_beam/internal/gcp/auth.py
> 75 3948%
> apache_beam/internal/gcp/json_value.py
> 64 888%
> apache_beam/internal/pickler.py
> 96 1090%
> apache_beam/internal/util.py
> 30 0 100%
> apache_beam/io/__init__.py
> 16 288%
> apache_beam/io/avroio.py
>139 1490%
> apache_beam/io/concat_source.py
>134 1093%
> apache_beam/io/filebasedsource.py
>112 397%
> apache_beam/io/fileio.py
>188 5571%
> apache_beam/io/filesystem.py
>184 2388%
> apache_beam/io/filesystems_util.py
> 7 0 100%
> apache_beam/io/gcp/__init__.py
> 0 0 100%
> apache_beam/io/gcp/bigquery.py
>418 2993%
> apache_beam/io/gcp/datastore/__init__.py
> 0 0 100%
> apache_beam/io/gcp/datastore/v1/__init__.py
> 0 0 100%
> apache_beam/io/gcp/datastore/v1/datastoreio.py
>170 3381%
> apache_beam/io/gcp/datastore/v1/fake_datastore.py
> 38 295%
> apache_beam/io/gcp/datastore/v1/helper.py
>126 894%
> apache_beam/io/gcp/datastore/v1/query_splitter.py
>102 793%
> apache_beam/io/gcp/gcsfilesystem.py
>109 1190%
> apache_beam/io/gcp/gcsio.py
>413 4689%
> apache_beam/io/gcp/internal/__init__.py
> 0 0 100%
> apache_beam/io/gcp/internal/clients/__init__.py
> 0 0 100%
> apache_beam/io/gcp/internal/clients/bigquery/__init__.py
> 9 278%
> apache_beam/io/gcp/internal/clients/bigquery/bigquery_v2_client.py
>116 65
[jira] [Commented] (BEAM-1680) Uber-JIRA: Extend Python-SDK's unit test coverage
[
https://issues.apache.org/jira/browse/BEAM-1680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128830#comment-17128830
]
Beam JIRA Bot commented on BEAM-1680:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Uber-JIRA: Extend Python-SDK's unit test coverage
> -
>
> Key: BEAM-1680
> URL: https://issues.apache.org/jira/browse/BEAM-1680
> Project: Beam
> Issue Type: Improvement
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> Some components of the Python-SDK has low coverage rate.
> As of now ([ea33e33|https://github.com/apache/beam/tree/ea33e33]) the test
> coverage looks as follows (excluding autocomplete_test.py):
> {code}
> Ran 1050 tests in 145.653s
> OK (skipped=14)
> py27gcp runtests: commands[5] | coverage report
> Name
> Stmts Miss Cover
> --
> apache_beam/__init__.py
> 12 192%
> apache_beam/coders/__init__.py
> 2 0 100%
> apache_beam/coders/coder_impl.py
>414 2394%
> apache_beam/coders/coders.py
>392 6883%
> apache_beam/coders/observable.py
> 10 0 100%
> apache_beam/coders/slow_stream.py
>105 694%
> apache_beam/coders/typecoders.py
> 69 691%
> apache_beam/error.py
> 7 0 100%
> apache_beam/internal/__init__.py
> 0 0 100%
> apache_beam/internal/gcp/__init__.py
> 0 0 100%
> apache_beam/internal/gcp/auth.py
> 75 3948%
> apache_beam/internal/gcp/json_value.py
> 64 888%
> apache_beam/internal/pickler.py
> 96 1090%
> apache_beam/internal/util.py
> 30 0 100%
> apache_beam/io/__init__.py
> 16 288%
> apache_beam/io/avroio.py
>139 1490%
> apache_beam/io/concat_source.py
>134 1093%
> apache_beam/io/filebasedsource.py
>112 397%
> apache_beam/io/fileio.py
>188 5571%
> apache_beam/io/filesystem.py
>184 2388%
> apache_beam/io/filesystems_util.py
> 7 0 100%
> apache_beam/io/gcp/__init__.py
> 0 0 100%
> apache_beam/io/gcp/bigquery.py
>418 2993%
> apache_beam/io/gcp/datastore/__init__.py
> 0 0 100%
> apache_beam/io/gcp/datastore/v1/__init__.py
> 0 0 100%
> apache_beam/io/gcp/datastore/v1/datastoreio.py
>170 3381%
> apache_beam/io/gcp/datastore/v1/fake_datastore.py
> 38 295%
> apache_beam/io/gcp/datastore/v1/helper.py
>126 894%
> apache_beam/io/gcp/datastore/v1/query_splitter.py
>102 793%
> apache_beam/io/gcp/gcsfilesystem.py
>109 1190%
> apache_beam/io/gcp/gcsio.py
>413 4689%
> apache_beam/io/gcp/internal/__init__.py
> 0 0
[jira] [Updated] (BEAM-1685) Measure and report code coverage in Python-SDK's unit tests
[ https://issues.apache.org/jira/browse/BEAM-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1685: Labels: beam-fixit stale-assigned (was: beam-fixit) > Measure and report code coverage in Python-SDK's unit tests > --- > > Key: BEAM-1685 > URL: https://issues.apache.org/jira/browse/BEAM-1685 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Tibor Kiss >Assignee: Tibor Kiss >Priority: P3 > Labels: beam-fixit, stale-assigned > > During the execution of the Python UTs the test coverage should be measured. > The results should be shown on screen & posted to coveralls.io page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-1685) Measure and report code coverage in Python-SDK's unit tests
[ https://issues.apache.org/jira/browse/BEAM-1685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128828#comment-17128828 ] Beam JIRA Bot commented on BEAM-1685: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > Measure and report code coverage in Python-SDK's unit tests > --- > > Key: BEAM-1685 > URL: https://issues.apache.org/jira/browse/BEAM-1685 > Project: Beam > Issue Type: Sub-task > Components: sdk-py-core >Reporter: Tibor Kiss >Assignee: Tibor Kiss >Priority: P3 > Labels: beam-fixit, stale-assigned > > During the execution of the Python UTs the test coverage should be measured. > The results should be shown on screen & posted to coveralls.io page. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1620) Add streaming Dataflow ValidatesRunner coverage
[ https://issues.apache.org/jira/browse/BEAM-1620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1620: Labels: beam-fixit stale-assigned (was: beam-fixit) > Add streaming Dataflow ValidatesRunner coverage > --- > > Key: BEAM-1620 > URL: https://issues.apache.org/jira/browse/BEAM-1620 > Project: Beam > Issue Type: Test > Components: runner-dataflow, testing >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: P2 > Labels: beam-fixit, stale-assigned > > Currently, the runner validation test suite is not run on Dataflow in > streaming mode. In fact, it should be able to run - all the functionality is > in place. I think this is just a matter of maven + Jenkins + making sure not > to leak a bunch of streaming jobs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-1683) Add unit tests for counters.py
[
https://issues.apache.org/jira/browse/BEAM-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128827#comment-17128827
]
Beam JIRA Bot commented on BEAM-1683:
-
This issue is assigned but has not received an update in 30 days so it has been
labeled "stale-assigned". If you are still working on the issue, please give an
update and remove the label. If you are no longer working on the issue, please
unassign so someone else may work on it. In 7 days the issue will be
automatically unassigned.
> Add unit tests for counters.py
> --
>
> Key: BEAM-1683
> URL: https://issues.apache.org/jira/browse/BEAM-1683
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Rahul Sabbineni
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> Python-SDK's {{apache_beam/utils/counters.py}} does not have associated unit
> tests and have low (indirect) test coverage.
> Create the respective tests to ensure code quality and increase test coverage.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (BEAM-1890) No INFO/DEBUG log if Python ValidatesRunner test timeout
[ https://issues.apache.org/jira/browse/BEAM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128825#comment-17128825 ] Beam JIRA Bot commented on BEAM-1890: - This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned. > No INFO/DEBUG log if Python ValidatesRunner test timeout > > > Key: BEAM-1890 > URL: https://issues.apache.org/jira/browse/BEAM-1890 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Mark Liu >Assignee: Mark Liu >Priority: P3 > Labels: beam-fixit, stale-assigned, triaged > > Python service test (ValidatesRunner test and Integration test with service > runner) enabled multiprocess execution in Postcommit through Nose. When > TimedOutException happens, only stack trace are printed out which contains > very little information about job info. Printing whatever logs until timeout > happens, which is captured by Nose, will make debugging on service more > easier. > This is a issue on Nose framework side and there is a link to track: > https://github.com/nose-devs/nose/issues/1044 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1681) Add Unit Tests for fixes in BEAM-1649
[
https://issues.apache.org/jira/browse/BEAM-1681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Beam JIRA Bot updated BEAM-1681:
Labels: beam-fixit stale-assigned (was: beam-fixit)
> Add Unit Tests for fixes in BEAM-1649
> -
>
> Key: BEAM-1681
> URL: https://issues.apache.org/jira/browse/BEAM-1681
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-py-core
>Reporter: Tibor Kiss
>Assignee: Tibor Kiss
>Priority: P3
> Labels: beam-fixit, stale-assigned
>
> BEAM-1649 was delivered without UTs included.
> This is a follow-up to add UT coverage for the following functions:
> {noformat}
> OrderedPositionRangeTracker.stop_position()
> ValueStateTag.__repr__()
> typehints.decorators._unpack_positional_arg_hints()
> WindowedTypeConstraint.type_check()
> PipelineOptions.__getattr__()
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-1890) No INFO/DEBUG log if Python ValidatesRunner test timeout
[ https://issues.apache.org/jira/browse/BEAM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Beam JIRA Bot updated BEAM-1890: Labels: beam-fixit stale-assigned triaged (was: beam-fixit triaged) > No INFO/DEBUG log if Python ValidatesRunner test timeout > > > Key: BEAM-1890 > URL: https://issues.apache.org/jira/browse/BEAM-1890 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Mark Liu >Assignee: Mark Liu >Priority: P3 > Labels: beam-fixit, stale-assigned, triaged > > Python service test (ValidatesRunner test and Integration test with service > runner) enabled multiprocess execution in Postcommit through Nose. When > TimedOutException happens, only stack trace are printed out which contains > very little information about job info. Printing whatever logs until timeout > happens, which is captured by Nose, will make debugging on service more > easier. > This is a issue on Nose framework side and there is a link to track: > https://github.com/nose-devs/nose/issues/1044 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10219) Support ZetaSQL TIME functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128811#comment-17128811 ] Robin Qiu commented on BEAM-10219: -- FYI, [https://github.com/apache/beam/pull/11272] is a PR that implements the DATE functions in BeamSQL > Support ZetaSQL TIME functions in BeamSQL > - > > Key: BEAM-10219 > URL: https://issues.apache.org/jira/browse/BEAM-10219 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See > the list of functions below: > [https://github.com/google/zetasql/blob/master/docs/time_functions.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10219) Support ZetaSQL TIME functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10219: - Description: Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See the list of functions below: [https://github.com/google/zetasql/blob/master/docs/time_functions.md] was: Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See the list of functions below: [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] (and EXTRACT, which is not included in the documentation) > Support ZetaSQL TIME functions in BeamSQL > - > > Key: BEAM-10219 > URL: https://issues.apache.org/jira/browse/BEAM-10219 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See > the list of functions below: > [https://github.com/google/zetasql/blob/master/docs/time_functions.md] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10219) Support ZetaSQL TIME functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10219: - Status: Open (was: Triage Needed) > Support ZetaSQL TIME functions in BeamSQL > - > > Key: BEAM-10219 > URL: https://issues.apache.org/jira/browse/BEAM-10219 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See > the list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] > (and EXTRACT, which is not included in the documentation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10219) Support ZetaSQL TIME functions in BeamSQL
[ https://issues.apache.org/jira/browse/BEAM-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robin Qiu updated BEAM-10219: - Description: Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See the list of functions below: [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] (and EXTRACT, which is not included in the documentation) was: Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See the full list of functions below: [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] > Support ZetaSQL TIME functions in BeamSQL > - > > Key: BEAM-10219 > URL: https://issues.apache.org/jira/browse/BEAM-10219 > Project: Beam > Issue Type: New Feature > Components: dsl-sql-zetasql >Reporter: Robin Qiu >Assignee: Zijie Song >Priority: P2 > > Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See > the list of functions below: > [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] > (and EXTRACT, which is not included in the documentation) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10219) Support ZetaSQL TIME functions in BeamSQL
Robin Qiu created BEAM-10219: Summary: Support ZetaSQL TIME functions in BeamSQL Key: BEAM-10219 URL: https://issues.apache.org/jira/browse/BEAM-10219 Project: Beam Issue Type: New Feature Components: dsl-sql-zetasql Reporter: Robin Qiu Assignee: Zijie Song Support *all* TIME functions defined in ZetaSQL (BigQuery Standard SQL). See the full list of functions below: [https://cloud.google.com/bigquery/docs/reference/standard-sql/time_functions] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10218) Add FnApiRunner to cross-language validate runner test suite
Heejong Lee created BEAM-10218: -- Summary: Add FnApiRunner to cross-language validate runner test suite Key: BEAM-10218 URL: https://issues.apache.org/jira/browse/BEAM-10218 Project: Beam Issue Type: Improvement Components: cross-language, runner-direct Reporter: Heejong Lee Assignee: Heejong Lee Add FnApiRunner to cross-language validate runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10218) Add FnApiRunner to cross-language validate runner test suite
[ https://issues.apache.org/jira/browse/BEAM-10218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-10218: --- Status: Open (was: Triage Needed) > Add FnApiRunner to cross-language validate runner test suite > > > Key: BEAM-10218 > URL: https://issues.apache.org/jira/browse/BEAM-10218 > Project: Beam > Issue Type: Improvement > Components: cross-language, runner-direct >Reporter: Heejong Lee >Assignee: Heejong Lee >Priority: P2 > > Add FnApiRunner to cross-language validate runner test suite -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10217) Suboptimal type inference on Python 3.6
Valentyn Tymofieiev created BEAM-10217:
--
Summary: Suboptimal type inference on Python 3.6
Key: BEAM-10217
URL: https://issues.apache.org/jira/browse/BEAM-10217
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Valentyn Tymofieiev
Following test fails for me with AssertionError: :: List[Tuple[Person, int]]
!= List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6 and 3.7.3.
{noformat}
def testMethodReturnsTuple(self):
class Person(object):
def __init__(self, name):
self.name = name
def parse_record(name_and_age):
name, age = name_and_age.split(',')
return Person(name), int(age)
self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
lambda record: [parse_record(record)], [str])
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-10217) Suboptimal type inference on Python 3.6
[
https://issues.apache.org/jira/browse/BEAM-10217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Valentyn Tymofieiev updated BEAM-10217:
---
Description:
Following trivial_inference_test fails for me with AssertionError: ::
List[Tuple[Person, int]] != List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6
and 3.7.3.
{noformat}
def testMethodReturnsTuple(self):
class Person(object):
def __init__(self, name):
self.name = name
def parse_record(name_and_age):
name, age = name_and_age.split(',')
return Person(name), int(age)
self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
lambda record: [parse_record(record)], [str])
{noformat}
was:
Following test fails for me with AssertionError: :: List[Tuple[Person, int]]
!= List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6 and 3.7.3.
{noformat}
def testMethodReturnsTuple(self):
class Person(object):
def __init__(self, name):
self.name = name
def parse_record(name_and_age):
name, age = name_and_age.split(',')
return Person(name), int(age)
self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
lambda record: [parse_record(record)], [str])
{noformat}
> Suboptimal type inference on Python 3.6
> ---
>
> Key: BEAM-10217
> URL: https://issues.apache.org/jira/browse/BEAM-10217
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Valentyn Tymofieiev
>Priority: P2
>
> Following trivial_inference_test fails for me with AssertionError: ::
> List[Tuple[Person, int]] != List[Any] on Python 3.6.8; Passed on 3.5.2, 3.5.6
> and 3.7.3.
> {noformat}
> def testMethodReturnsTuple(self):
> class Person(object):
> def __init__(self, name):
> self.name = name
> def parse_record(name_and_age):
> name, age = name_and_age.split(',')
> return Person(name), int(age)
> self.assertReturnType(typehints.List[typehints.Tuple[Person, int]],
> lambda record: [parse_record(record)], [str])
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Issue Comment Deleted] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] josson paul kalapparambath updated BEAM-9977: - Comment: was deleted (was: [~boyuanz] . Looks like I don't need BEAM-10123. But I need few more clarifications. Let us say the pipeline failed and restarted. 1) WatchGrothFn will fetch *ALL* the Kafka partitions again and sends it further down. Now, does ReadFromKafkaViaSDF get each partition and start from where it check pointed last?. ) > Build Kafka Read on top of Java SplittableDoFn > -- > > Key: BEAM-9977 > URL: https://issues.apache.org/jira/browse/BEAM-9977 > Project: Beam > Issue Type: New Feature > Components: io-java-kafka >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (BEAM-10215) @Ignore: Concat now works with varargs
[ https://issues.apache.org/jira/browse/BEAM-10215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang resolved BEAM-10215. - Fix Version/s: Not applicable Resolution: Fixed > @Ignore: Concat now works with varargs > -- > > Key: BEAM-10215 > URL: https://issues.apache.org/jira/browse/BEAM-10215 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: P2 > Labels: beam-fixit > Fix For: Not applicable > > > Will fix this ignored test: > testConcatWithSixParameters() -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (BEAM-10213) @Ignore: fix the test for testCastToDateWithCase
[ https://issues.apache.org/jira/browse/BEAM-10213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang closed BEAM-10213. --- Fix Version/s: Not applicable Resolution: Fixed > @Ignore: fix the test for testCastToDateWithCase > > > Key: BEAM-10213 > URL: https://issues.apache.org/jira/browse/BEAM-10213 > Project: Beam > Issue Type: Bug > Components: dsl-sql-zetasql >Reporter: Rui Wang >Assignee: Rui Wang >Priority: P2 > Labels: beam-fixit > Fix For: Not applicable > > > Fix this ignored test: testCastToDateWithCase() > There are two improvements: > 1. SUBSTR's usage is wrong in the original test. E.g. ZetaSQL SUBSTR count > position from 1. > 2. Expected output row was't used DATE as a logical type. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-10216) SESSION TVF Optimizations
[ https://issues.apache.org/jira/browse/BEAM-10216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Wang updated BEAM-10216: Status: Open (was: Triage Needed) > SESSION TVF Optimizations > - > > Key: BEAM-10216 > URL: https://issues.apache.org/jira/browse/BEAM-10216 > Project: Beam > Issue Type: Improvement > Components: dsl-sql-zetasql >Reporter: Rui Wang >Priority: P2 > > Current TVF Session has the following limitation: > Use a "dummy" key to go through a GBK to compute window_start and window_end. > This introduces an extra shuffle so it costs more. > One optimizations we can do is always merge TableFunctionScanRel with > AggregationRel, thus we can use only one GBK to compute both SESSION > window_start and window_end, and aggregations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (BEAM-10216) SESSION TVF Optimizations
Rui Wang created BEAM-10216: --- Summary: SESSION TVF Optimizations Key: BEAM-10216 URL: https://issues.apache.org/jira/browse/BEAM-10216 Project: Beam Issue Type: Improvement Components: dsl-sql-zetasql Reporter: Rui Wang Current TVF Session has the following limitation: Use a "dummy" key to go through a GBK to compute window_start and window_end. This introduces an extra shuffle so it costs more. One optimizations we can do is always merge TableFunctionScanRel with AggregationRel, thus we can use only one GBK to compute both SESSION window_start and window_end, and aggregations. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (BEAM-2762) Coverage report for Python code
[ https://issues.apache.org/jira/browse/BEAM-2762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Saavan Nanavati reassigned BEAM-2762: - Assignee: Saavan Nanavati > Coverage report for Python code > --- > > Key: BEAM-2762 > URL: https://issues.apache.org/jira/browse/BEAM-2762 > Project: Beam > Issue Type: Task > Components: sdk-py-core, testing >Reporter: Mark Liu >Assignee: Saavan Nanavati >Priority: P2 > Labels: beam-fixit > Fix For: Not applicable > > > It's good to have code coverage in Python SDK to show the test coverage. Java > is using jacoco-maven-plugin to generate coverage report and coveralls > service to manage/display data. > Python have similar tool called coverage.py for report generation and > coveralls-python to send report to api of coveralls service. > It's nice to have one place (like coveralls service) to manage and show data > from different SDKs together/separately. However, there are still some > problems due to the fact of Beam CI system and multi languages in Beam. The > coveralls service doesn't have a good way to collect data separately from > different projects. But the postcommit builds are separated by sdks. > As the first step, I think it's good to have the python coverage report > printout in the build to give people a idea of the current coverage status at > lease. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-10079) Samza tests failing [Java 11]
[
https://issues.apache.org/jira/browse/BEAM-10079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128704#comment-17128704
]
Yixing Zhang commented on BEAM-10079:
-
[~pawel.pasterz] [~kenn], I don't see the error by running:
./gradlew :runners:samza:test
How can I run it with Java 11?
> Samza tests failing [Java 11]
> -
>
> Key: BEAM-10079
> URL: https://issues.apache.org/jira/browse/BEAM-10079
> Project: Beam
> Issue Type: Sub-task
> Components: runner-samza
>Reporter: Pawel Pasterz
>Assignee: Yixing Zhang
>Priority: P2
>
> Gradle task *_:runners:samza:test_* fails during Java 11 Precommit job
>
> Example stack trace:
> {code:java}
> > Task :runners:samza:test
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadEventTimeTimers
> INFO: Loaded 0 event time timers in memory
> May 26, 2020 7:33:55 AM
> org.apache.beam.runners.samza.runtime.SamzaTimerInternalsFactory$SamzaTimerState
> loadProcessingTimeTimers
> INFO: Loaded 0 processing time timers in memory
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Registering task instances with consumers.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Starting consumer multiplexer.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: Entering run loop.
> May 26, 2020 7:33:56 AM
> org.apache.samza.processor.StreamProcessor$ContainerListener afterStart
> WARNING: Received container start notification for container:
> org.apache.samza.container.SamzaContainer@af148fb in stream processor: 1.
> May 26, 2020 7:33:56 AM org.apache.samza.util.Logging$class info
> INFO: End of stream reached for partition: SystemStreamPartition
> [11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
>
> 11-PAssert_0_GroupGlobally_Create_Values_Read_CreateSource__out__PCollection_,
> 0]
> May 26, 2020 7:33:56 AM org.apache.beam.runners.samza.runtime.OpAdapter
> processWatermark
> SEVERE: Op org.apache.beam.runners.samza.runtime.OpAdapter threw an
> exception during processing watermark
> java.lang.RuntimeException: org.apache.beam.sdk.util.UserCodeException:
> java.lang.IncompatibleClassChangeError: Method
> org.apache.samza.storage.kv.KeyValueStore.deleteAll(Ljava/util/List;)V must
> be InterfaceMethodref constant
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.withMetrics(DoFnRunnerWithMetrics.java:98)
> at
> org.apache.beam.runners.samza.metrics.DoFnRunnerWithMetrics.processElement(DoFnRunnerWithMetrics.java:55)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.fireTimer(GroupByKeyOp.java:225)
> at
> org.apache.beam.runners.samza.runtime.GroupByKeyOp.processWatermark(GroupByKeyOp.java:203)
> at
> org.apache.beam.runners.samza.runtime.OpAdapter.processWatermark(OpAdapter.java:109)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:399)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
> java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
> at
> java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
> at
> java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:550)
> at
> java.base/java.util.stream.AbstractPipeline.evaluateToArrayNode(AbstractPipeline.java:260)
> at
> java.base/java.util.stream.ReferencePipeline.toArray(ReferencePipeline.java:517)
> at
> org.apache.samza.operators.impl.OperatorImpl.propagateWatermark(OperatorImpl.java:434)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$onWatermark$25(OperatorImpl.java:416)
> at
> java.base/java.util.concurrent.CompletableFuture.uniComposeStage(CompletableFuture.java:1106)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:2235)
> at
> java.base/java.util.concurrent.CompletableFuture.thenCompose(CompletableFuture.java:143)
> at
> org.apache.samza.operators.impl.OperatorImpl.onWatermark(OperatorImpl.java:416)
> at
> org.apache.samza.operators.impl.OperatorImpl.lambda$propagateWatermark$26(OperatorImpl.java:433)
> at
> java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
> at
> java.base/java.util.HashMap$KeySpliterator.forEachRemaining(HashMap.java:1603)
> at
>
[jira] [Comment Edited] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128687#comment-17128687 ] josson paul kalapparambath edited comment on BEAM-9977 at 6/8/20, 10:21 PM: [~boyuanz] . Looks like I don't need BEAM-10123. But I need few more clarifications. Let us say the pipeline failed and restarted. 1) WatchGrothFn will fetch *ALL* the Kafka partitions again and sends it further down. Now, does ReadFromKafkaViaSDF get each partition and start from where it check pointed last?. was (Author: josson): [~boyuanz] . Looks like I don't need BEAM-10123. But I need few more clarifiations. Let us say the pipeline failed and restarted. 1) WatchGrothFn will fetch *ALL* the Kafka partitions again and sends it further down. Now, does ReadFromKafkaViaSDF get each partition and start from where it check pointed last?. > Build Kafka Read on top of Java SplittableDoFn > -- > > Key: BEAM-9977 > URL: https://issues.apache.org/jira/browse/BEAM-9977 > Project: Beam > Issue Type: New Feature > Components: io-java-kafka >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-9977) Build Kafka Read on top of Java SplittableDoFn
[ https://issues.apache.org/jira/browse/BEAM-9977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128687#comment-17128687 ] josson paul kalapparambath commented on BEAM-9977: -- [~boyuanz] . Looks like I don't need BEAM-10123. But I need few more clarifiations. Let us say the pipeline failed and restarted. 1) WatchGrothFn will fetch *ALL* the Kafka partitions again and sends it further down. Now, does ReadFromKafkaViaSDF get each partition and start from where it check pointed last?. > Build Kafka Read on top of Java SplittableDoFn > -- > > Key: BEAM-9977 > URL: https://issues.apache.org/jira/browse/BEAM-9977 > Project: Beam > Issue Type: New Feature > Components: io-java-kafka >Reporter: Boyuan Zhang >Assignee: Boyuan Zhang >Priority: P2 > Time Spent: 10h > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-1890) No INFO/DEBUG log if Python ValidatesRunner test timeout
[ https://issues.apache.org/jira/browse/BEAM-1890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-1890: -- Labels: beam-fixit triaged (was: triaged) > No INFO/DEBUG log if Python ValidatesRunner test timeout > > > Key: BEAM-1890 > URL: https://issues.apache.org/jira/browse/BEAM-1890 > Project: Beam > Issue Type: Bug > Components: sdk-py-core, testing >Reporter: Mark Liu >Assignee: Mark Liu >Priority: P3 > Labels: beam-fixit, triaged > > Python service test (ValidatesRunner test and Integration test with service > runner) enabled multiprocess execution in Postcommit through Nose. When > TimedOutException happens, only stack trace are printed out which contains > very little information about job info. Printing whatever logs until timeout > happens, which is captured by Nose, will make debugging on service more > easier. > This is a issue on Nose framework side and there is a link to track: > https://github.com/nose-devs/nose/issues/1044 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5758) Load tests for SyntheticSources in Python
[ https://issues.apache.org/jira/browse/BEAM-5758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5758: -- Labels: beam-fixit stale-P2 (was: stale-P2) > Load tests for SyntheticSources in Python > - > > Key: BEAM-5758 > URL: https://issues.apache.org/jira/browse/BEAM-5758 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Kasia Kucharczyk >Priority: P2 > Labels: beam-fixit, stale-P2 > Time Spent: 3h > Remaining Estimate: 0h > > For purpose of load testing the SyntheticSources there should be created > tests with metrics which will be sent to BigQuery. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7789) :beam-test-tools project fails to build locally
[
https://issues.apache.org/jira/browse/BEAM-7789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-7789:
--
Labels: beam-fixit stale-P2 (was: stale-P2)
> :beam-test-tools project fails to build locally
> ---
>
> Key: BEAM-7789
> URL: https://issues.apache.org/jira/browse/BEAM-7789
> Project: Beam
> Issue Type: Bug
> Components: testing
>Reporter: Anton Kedin
>Priority: P2
> Labels: beam-fixit, stale-P2
>
> Running the release-verification build (global build of everything) in turn
> triggers the build of the `beam-test-tools` project, which has some test
> infrastructure scripts that we run on Jenkins. It seems to work fine on
> jenkins. However running the build of the project locally fails:
> https://scans.gradle.com/s/kqhkzyozbpiua/console-log#L6
> What seems to happen is the gradle vendoring plugin caches the dependencies
> locally, but fails to cache simplelru.
> One workaround (based on ./gradlew :beam-test-tools:showGopathGoroot)
> {code}
> export GOPATH=$PWD/.test-infra/tools/.gogradle/project_gopath
> go get github.com/hashicorp/golang-lru/simplelru
> ./gradlew :beam-test-tools:build
> {code}
> It is able to find the `lrumap` and `simplelru` during the dependency
> resolution step, and I can see it mentioned in couple of artifacts produced
> by the `gogradle` plugin. But when it does `:installDepedencies` to actually
> copy them to `vendor` directory, this specific package is missing. This
> reproduces for me on a couple of different machines I tried, both on release
> and master branches
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-6335) GroupByKey uses data insertion pipeline in streaming tests
[ https://issues.apache.org/jira/browse/BEAM-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-6335: -- Labels: beam-fixit stale-assigned (was: stale-assigned) > GroupByKey uses data insertion pipeline in streaming tests > -- > > Key: BEAM-6335 > URL: https://issues.apache.org/jira/browse/BEAM-6335 > Project: Beam > Issue Type: Sub-task > Components: testing >Reporter: Kasia Kucharczyk >Assignee: Kasia Kucharczyk >Priority: P2 > Labels: beam-fixit, stale-assigned > Time Spent: 50m > Remaining Estimate: 0h > > Uses prepared Java Data Insertion Pipeline to update GroupByKey in Python to > load test streaming. > This task contains following steps: > # Create GroupByKey streaming test that accepts bytes > # To stop test after arrived messages it is required to add a matcher. The > matcher should work on number of messages because in case of load testing it > would be difficult to compare big load of bytes (also casting bytes to string > to compare load would be difficult). > # All data is generated by SyntheticDataPublisher.java, which is sending > bytes produced by synthetic source to PubSub. PubSub is used as a streaming > source for the Python test. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9559) Remove smoke load test for Java and Python SDK
[ https://issues.apache.org/jira/browse/BEAM-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9559: -- Summary: Remove smoke load test for Java and Python SDK (was: Remove smoke load test for Java and Python dsk) > Remove smoke load test for Java and Python SDK > -- > > Key: BEAM-9559 > URL: https://issues.apache.org/jira/browse/BEAM-9559 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Assignee: Michał Walenia >Priority: P3 > Labels: beam-fixit, stale-assigned > > As discussed in PR: > [https://github.com/apache/beam/pull/11135#discussion_r392852028] > No one seems to use them and regular load test will fail too whenever > something is wrong. There's plenty of load tests now and they are smaller > than they used to be at the time of creating the smoke tests. If something is > wrong there will be quite quick feedback from them. All that makes the smoke > test redundant imo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (BEAM-8130) Storing, displaying and detecting anomalies in test results
[ https://issues.apache.org/jira/browse/BEAM-8130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17128660#comment-17128660 ] Kenneth Knowles commented on BEAM-8130: --- [~tysonjh] pinging because we discussed the issue of regression detection > Storing, displaying and detecting anomalies in test results > --- > > Key: BEAM-8130 > URL: https://issues.apache.org/jira/browse/BEAM-8130 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Kamil Wasilewski >Assignee: Kamil Wasilewski >Priority: P3 > > An implementation of the following proposal: > https://s.apache.org/test-metrics-storage-corrected -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2637) Post commit test for mobile gaming examples
[ https://issues.apache.org/jira/browse/BEAM-2637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-2637: -- Labels: beam-fixit (was: ) > Post commit test for mobile gaming examples > --- > > Key: BEAM-2637 > URL: https://issues.apache.org/jira/browse/BEAM-2637 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Ahmet Altay >Priority: P2 > Labels: beam-fixit > > We need post commit test for mobile gaming examples to prevent failures > beyond DirectRunner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-9559) Remove smoke load test for Java and Python dsk
[ https://issues.apache.org/jira/browse/BEAM-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-9559: -- Labels: beam-fixit stale-assigned (was: stale-assigned) > Remove smoke load test for Java and Python dsk > -- > > Key: BEAM-9559 > URL: https://issues.apache.org/jira/browse/BEAM-9559 > Project: Beam > Issue Type: Wish > Components: testing >Reporter: Lukasz Gajowy >Assignee: Michał Walenia >Priority: P3 > Labels: beam-fixit, stale-assigned > > As discussed in PR: > [https://github.com/apache/beam/pull/11135#discussion_r392852028] > No one seems to use them and regular load test will fail too whenever > something is wrong. There's plenty of load tests now and they are smaller > than they used to be at the time of creating the smoke tests. If something is > wrong there will be quite quick feedback from them. All that makes the smoke > test redundant imo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7316) FileIOTest.testFileIoDynamicNaming breaks on Spark runner
[ https://issues.apache.org/jira/browse/BEAM-7316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7316: -- Labels: beam-fixit (was: ) > FileIOTest.testFileIoDynamicNaming breaks on Spark runner > - > > Key: BEAM-7316 > URL: https://issues.apache.org/jira/browse/BEAM-7316 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Ismaël Mejía >Priority: P3 > Labels: beam-fixit > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7914) Add python 3 test in crossLanguageValidateRunner task
[ https://issues.apache.org/jira/browse/BEAM-7914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7914: -- Labels: beam-fixit portability stale-assigned (was: portability stale-assigned) > Add python 3 test in crossLanguageValidateRunner task > - > > Key: BEAM-7914 > URL: https://issues.apache.org/jira/browse/BEAM-7914 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Heejong Lee >Assignee: Chamikara Madhusanka Jayalath >Priority: P2 > Labels: beam-fixit, portability, stale-assigned > > add python 3 test in crossLanguageValidateRunner task -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7314) ViewTest.testEmptySingletonSideInput breaks on Spark runner
[ https://issues.apache.org/jira/browse/BEAM-7314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7314: -- Labels: beam-fixit (was: ) > ViewTest.testEmptySingletonSideInput breaks on Spark runner > --- > > Key: BEAM-7314 > URL: https://issues.apache.org/jira/browse/BEAM-7314 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Ismaël Mejía >Priority: P3 > Labels: beam-fixit > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7315) GatherAllPanesTest.multiplePanesMultipleReifiedPane breaks on Spark runner
[ https://issues.apache.org/jira/browse/BEAM-7315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7315: -- Labels: beam-fixit (was: ) > GatherAllPanesTest.multiplePanesMultipleReifiedPane breaks on Spark runner > -- > > Key: BEAM-7315 > URL: https://issues.apache.org/jira/browse/BEAM-7315 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Ismaël Mejía >Priority: P3 > Labels: beam-fixit > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7237) Run NeedsRunner test category with Spark Runner
[
https://issues.apache.org/jira/browse/BEAM-7237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-7237:
--
Labels: beam-fixit stale-P2 (was: stale-P2)
> Run NeedsRunner test category with Spark Runner
> ---
>
> Key: BEAM-7237
> URL: https://issues.apache.org/jira/browse/BEAM-7237
> Project: Beam
> Issue Type: Test
> Components: runner-spark
>Reporter: Ismaël Mejía
>Priority: P2
> Labels: beam-fixit, stale-P2
>
> The {{:validatesRunner}} task uses the {{ValidatesRunner}} test category
> which is a subtype of {{NeedsRunner}}. It would be good to expand the scope
> of the tests to {{NeedsRunner}} because there are many additional tests which
> currently excluded.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (BEAM-7313) PipelineRunnerTest testRunPTransform breaks on Spark runner
[ https://issues.apache.org/jira/browse/BEAM-7313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7313: -- Labels: beam-fixit (was: ) > PipelineRunnerTest testRunPTransform breaks on Spark runner > --- > > Key: BEAM-7313 > URL: https://issues.apache.org/jira/browse/BEAM-7313 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Ismaël Mejía >Priority: P3 > Labels: beam-fixit > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-2339) Jenkins cross JDK version test on Windows
[ https://issues.apache.org/jira/browse/BEAM-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-2339: -- Labels: beam-fixit (was: ) > Jenkins cross JDK version test on Windows > - > > Key: BEAM-2339 > URL: https://issues.apache.org/jira/browse/BEAM-2339 > Project: Beam > Issue Type: Task > Components: build-system, testing >Reporter: Mark Liu >Priority: P2 > Labels: beam-fixit > > We can set os variant to choose windows for Jenkins test, which can be > combined with JDK version test. So that we can have cross OS / cross JDK > version test. > This discussion came from > https://github.com/apache/beam/pull/3184#pullrequestreview-39303400 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-7317) TFRecordIOTest breaks on Spark runner
[ https://issues.apache.org/jira/browse/BEAM-7317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-7317: -- Labels: beam-fixit (was: ) > TFRecordIOTest breaks on Spark runner > - > > Key: BEAM-7317 > URL: https://issues.apache.org/jira/browse/BEAM-7317 > Project: Beam > Issue Type: Sub-task > Components: runner-spark >Reporter: Ismaël Mejía >Priority: P3 > Labels: beam-fixit > > org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidRecord FAILED > java.lang.AssertionError > org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidDataMask FAILED > java.lang.AssertionError > org.apache.beam.sdk.io.TFRecordIOTest > testReadInvalidLengthMask FAILED > java.lang.AssertionError -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-8665) Add infrastructure + test suites to run Beam tests on Windows/Mac platforms.
[ https://issues.apache.org/jira/browse/BEAM-8665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-8665: -- Labels: beam-fixit (was: ) > Add infrastructure + test suites to run Beam tests on Windows/Mac platforms. > - > > Key: BEAM-8665 > URL: https://issues.apache.org/jira/browse/BEAM-8665 > Project: Beam > Issue Type: Test > Components: testing >Reporter: Valentyn Tymofieiev >Priority: P2 > Labels: beam-fixit > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (BEAM-5302) Improve performance test documentation
[ https://issues.apache.org/jira/browse/BEAM-5302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5302: -- Labels: beam-fixit stale-P2 (was: stale-P2) > Improve performance test documentation > -- > > Key: BEAM-5302 > URL: https://issues.apache.org/jira/browse/BEAM-5302 > Project: Beam > Issue Type: Improvement > Components: testing >Reporter: Mark Liu >Priority: P2 > Labels: beam-fixit, stale-P2 > > Current documentation for performance testing and benchmarks missing > following areas: > How to write / use benchmark on Perfkit > How to run benchmark locally or on Jenkins > Benchmark summary / definition > How to use performance metrics data and where to find > Those documents can help new contributors to start and people who interested > in performance result to understand framework and look up results. -- This message was sent by Atlassian Jira (v8.3.4#803005)
