[jira] [Updated] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5779:
--
Description:
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
Path to failure:
* (Jenkins) beam_PostCommit_Python_Verify
* (gradle) :beam-sdks-python:postCommitITTests
* (py module) test_streaming_data_only
* (test class) apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
{code}
AssertionError:
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
messages.)
but: Expected 2 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
Unexpected: []
{code}
Looks like a legitimate failure. It is flakey as it went green later.
was:
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
Path to failure:
* beam_PostCommit_Python_Verify
* test_streaming_data_only
* apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
{code}
AssertionError:
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
messages.)
but: Expected 2 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
Unexpected: []
{code}
Looks like a legitimate failure. It is flakey as it went green later.
> Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
> --
>
> Key: BEAM-5779
> URL: https://issues.apache.org/jira/browse/BEAM-5779
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Ahmet Altay
>Priority: Major
> Labels: flake
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
> Path to failure:
> * (Jenkins) beam_PostCommit_Python_Verify
> * (gradle) :beam-sdks-python:postCommitITTests
> * (py module) test_streaming_data_only
> * (test class)
> apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
> {code}
> AssertionError:
> Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
> messages.)
> but: Expected 2 messages. Got 0 messages. Diffs (item, count):
> Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
> Unexpected: []
> {code}
> Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
Kenneth Knowles created BEAM-5779:
-
Summary: Failure in beam_PostCommit_Python_Verify
PubSubIntegrationTest on Dataflow
Key: BEAM-5779
URL: https://issues.apache.org/jira/browse/BEAM-5779
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Kenneth Knowles
Assignee: Ahmet Altay
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
Path to failure:
* beam_PostCommit_Python_Verify
* test_streaming_data_only
* apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
{code}
AssertionError:
Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
messages.)
but: Expected 2 messages. Got 0 messages. Diffs (item, count):
Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
Unexpected: []
{code}
Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-4663) Implement Cost calculations for Cost-Based Optimization (CBO)
[ https://issues.apache.org/jira/browse/BEAM-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4663: - Assignee: Kai Jiang > Implement Cost calculations for Cost-Based Optimization (CBO) > -- > > Key: BEAM-4663 > URL: https://issues.apache.org/jira/browse/BEAM-4663 > Project: Beam > Issue Type: Sub-task > Components: dsl-sql >Reporter: Kai Jiang >Assignee: Kai Jiang >Priority: Major > Time Spent: 1h 40m > Remaining Estimate: 0h > > To support CBO, we should implement methods in each Beam*Rel.java. > computeSelfCost(...) as our first step. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5773) Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for the Java Runtime Environment to continue."
[
https://issues.apache.org/jira/browse/BEAM-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652617#comment-16652617
]
Kenneth Knowles commented on BEAM-5773:
---
Looks like the same thing as happed on
[https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1401/] only in
this case Gradle could start threads but the Python test framework could not.
{code}
OpenBLAS blas_thread_init: RLIMIT_NPROC 10240 current, 10240 max
Process SyncManager-1:
Traceback (most recent call last):
File "/usr/lib/python2.7/multiprocessing/process.py", line 258, in _bootstrap
self.run()
File "/usr/lib/python2.7/multiprocessing/process.py", line 114, in run
self._target(*self._args, **self._kwargs)
File "/usr/lib/python2.7/multiprocessing/managers.py", line 558, in
_run_server
server.serve_forever()
File "/usr/lib/python2.7/multiprocessing/managers.py", line 184, in
serve_forever
t.start()
File "/usr/lib/python2.7/threading.py", line 736, in start
_start_new_thread(self.__bootstrap, ())
error: can't start new thread
interrupted
./scripts/run_postcommit.sh: line 124: 32380 Segmentation fault (core
dumped) python setup.py nosetests --attr $1 --nologcapture --processes=8
--process-timeout=3000 --test-pipeline-options="$JOINED_OPTS" $TESTS
{code}
> Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for
> the Java Runtime Environment to continue."
> --
>
> Key: BEAM-5773
> URL: https://issues.apache.org/jira/browse/BEAM-5773
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> Jenkins failed on the Python Dataflow ValidatesRunner postcommit because it
> Gradle allocate a thread.
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1402/console]
> Likely transient, but filing this to track if that is the case.
> {code}
> 15:07:52 [src] $
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/gradlew
> --info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
> -Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests
> :beam-sdks-python:validatesRunnerStreamingTests
> 15:07:52 #
> 15:07:52 # There is insufficient memory for the Java Runtime Environment to
> continue.
> 15:07:52 # Cannot create GC thread. Out of system resources.
> 15:07:52 # An error report file with more information is saved as:
> 15:07:52 #
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/hs_err_pid31336.log
> 15:07:53 Build step 'Invoke Gradle script' changed build result to FAILURE
> 15:07:53 Build step 'Invoke Gradle script' marked build as failure
> 15:07:56 Sending e-mails to: comm...@beam.apache.org
> 15:07:57 No emails were triggered.
> 15:07:57 Finished: FAILURE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5774) beam_Release_Gradle_NightlySnapshot timed out
[ https://issues.apache.org/jira/browse/BEAM-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652623#comment-16652623 ] Kenneth Knowles commented on BEAM-5774: --- No, it appears to be a plain-and-simple timeout. > beam_Release_Gradle_NightlySnapshot timed out > - > > Key: BEAM-5774 > URL: https://issues.apache.org/jira/browse/BEAM-5774 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Critical > > https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/209/ > Looking at the trend, this is not surprising: > https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5774) beam_Release_Gradle_NightlySnapshot timed out
[ https://issues.apache.org/jira/browse/BEAM-5774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652622#comment-16652622 ] Kenneth Knowles commented on BEAM-5774: --- TBD whether this is BEAM-5249 > beam_Release_Gradle_NightlySnapshot timed out > - > > Key: BEAM-5774 > URL: https://issues.apache.org/jira/browse/BEAM-5774 > Project: Beam > Issue Type: Bug > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Critical > > https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/209/ > Looking at the trend, this is not surprising: > https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5176) FailOnWarnings behave differently between CLI and Intellij build
[
https://issues.apache.org/jira/browse/BEAM-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652517#comment-16652517
]
Kenneth Knowles commented on BEAM-5176:
---
Actually when I do {{./gradlew --debug}} I see the following passed to the
failing javac command:
{code:java}
-Xlint:all -Werror -XepDisableWarningsInGeneratedCode
-XepExcludedPaths:(.*/)?(build/generated
.*avro-java|build/generated)/.* -Xep:MutableConstantField:OFF -Xlint:-options
-Xlint:-cast -Xlint:-deprecation -Xlint:-processing -Xlint:-rawtypes -Xlint:
-serial -Xlint:-try -Xlint:-unchecked -Xlint:-varargs{code}
> FailOnWarnings behave differently between CLI and Intellij build
> -
>
> Key: BEAM-5176
> URL: https://issues.apache.org/jira/browse/BEAM-5176
> Project: Beam
> Issue Type: Sub-task
> Components: build-system
>Reporter: Etienne Chauchot
>Assignee: Kenneth Knowles
>Priority: Major
>
> In command line the build passes but fails on the IDE because of warnings.
> To make it pass I had to put false in failOnWarnings in ApplyJavaNature
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5176) FailOnWarnings behave differently between CLI and Intellij build
[ https://issues.apache.org/jira/browse/BEAM-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652520#comment-16652520 ] Kenneth Knowles commented on BEAM-5176: --- Incidentally this repros on the command line for me suddenly. > FailOnWarnings behave differently between CLI and Intellij build > - > > Key: BEAM-5176 > URL: https://issues.apache.org/jira/browse/BEAM-5176 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Etienne Chauchot >Assignee: Kenneth Knowles >Priority: Major > > In command line the build passes but fails on the IDE because of warnings. > To make it pass I had to put false in failOnWarnings in ApplyJavaNature -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5773) Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for the Java Runtime Environment to continue."
[
https://issues.apache.org/jira/browse/BEAM-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652615#comment-16652615
]
Kenneth Knowles commented on BEAM-5773:
---
Removed auto-assignee.
> Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for
> the Java Runtime Environment to continue."
> --
>
> Key: BEAM-5773
> URL: https://issues.apache.org/jira/browse/BEAM-5773
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Priority: Major
>
> Jenkins failed on the Python Dataflow ValidatesRunner postcommit because it
> Gradle allocate a thread.
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1402/console]
> Likely transient, but filing this to track if that is the case.
> {code}
> 15:07:52 [src] $
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/gradlew
> --info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
> -Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests
> :beam-sdks-python:validatesRunnerStreamingTests
> 15:07:52 #
> 15:07:52 # There is insufficient memory for the Java Runtime Environment to
> continue.
> 15:07:52 # Cannot create GC thread. Out of system resources.
> 15:07:52 # An error report file with more information is saved as:
> 15:07:52 #
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/hs_err_pid31336.log
> 15:07:53 Build step 'Invoke Gradle script' changed build result to FAILURE
> 15:07:53 Build step 'Invoke Gradle script' marked build as failure
> 15:07:56 Sending e-mails to: comm...@beam.apache.org
> 15:07:57 No emails were triggered.
> 15:07:57 Finished: FAILURE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5773) Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for the Java Runtime Environment to continue."
[
https://issues.apache.org/jira/browse/BEAM-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5773:
-
Assignee: Kenneth Knowles
> Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for
> the Java Runtime Environment to continue."
> --
>
> Key: BEAM-5773
> URL: https://issues.apache.org/jira/browse/BEAM-5773
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Major
>
> Jenkins failed on the Python Dataflow ValidatesRunner postcommit because it
> Gradle allocate a thread.
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1402/console]
> Likely transient, but filing this to track if that is the case.
> {code}
> 15:07:52 [src] $
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/gradlew
> --info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
> -Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests
> :beam-sdks-python:validatesRunnerStreamingTests
> 15:07:52 #
> 15:07:52 # There is insufficient memory for the Java Runtime Environment to
> continue.
> 15:07:52 # Cannot create GC thread. Out of system resources.
> 15:07:52 # An error report file with more information is saved as:
> 15:07:52 #
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/hs_err_pid31336.log
> 15:07:53 Build step 'Invoke Gradle script' changed build result to FAILURE
> 15:07:53 Build step 'Invoke Gradle script' marked build as failure
> 15:07:56 Sending e-mails to: comm...@beam.apache.org
> 15:07:57 No emails were triggered.
> 15:07:57 Finished: FAILURE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5773) Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for the Java Runtime Environment to continue."
Kenneth Knowles created BEAM-5773:
-
Summary: Failure in beam_PostCommit_Py_VR_Dataflow "There is
insufficient memory for the Java Runtime Environment to continue."
Key: BEAM-5773
URL: https://issues.apache.org/jira/browse/BEAM-5773
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Kenneth Knowles
Assignee: Luke Cwik
Jenkins failed on the Python Dataflow ValidatesRunner postcommit because it
Gradle allocate a thread.
[https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1402/console]
Likely transient, but filing this to track if that is the case.
{code}
15:07:52 [src] $
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/gradlew
--info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
-Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests
:beam-sdks-python:validatesRunnerStreamingTests
15:07:52 #
15:07:52 # There is insufficient memory for the Java Runtime Environment to
continue.
15:07:52 # Cannot create GC thread. Out of system resources.
15:07:52 # An error report file with more information is saved as:
15:07:52 #
/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/hs_err_pid31336.log
15:07:53 Build step 'Invoke Gradle script' changed build result to FAILURE
15:07:53 Build step 'Invoke Gradle script' marked build as failure
15:07:56 Sending e-mails to: comm...@beam.apache.org
15:07:57 No emails were triggered.
15:07:57 Finished: FAILURE
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5773) Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for the Java Runtime Environment to continue."
[
https://issues.apache.org/jira/browse/BEAM-5773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5773:
-
Assignee: (was: Luke Cwik)
> Failure in beam_PostCommit_Py_VR_Dataflow "There is insufficient memory for
> the Java Runtime Environment to continue."
> --
>
> Key: BEAM-5773
> URL: https://issues.apache.org/jira/browse/BEAM-5773
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Priority: Major
>
> Jenkins failed on the Python Dataflow ValidatesRunner postcommit because it
> Gradle allocate a thread.
> [https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/1402/console]
> Likely transient, but filing this to track if that is the case.
> {code}
> 15:07:52 [src] $
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/gradlew
> --info --continue --max-workers=12 -Dorg.gradle.jvmargs=-Xms2g
> -Dorg.gradle.jvmargs=-Xmx4g :beam-sdks-python:validatesRunnerBatchTests
> :beam-sdks-python:validatesRunnerStreamingTests
> 15:07:52 #
> 15:07:52 # There is insufficient memory for the Java Runtime Environment to
> continue.
> 15:07:52 # Cannot create GC thread. Out of system resources.
> 15:07:52 # An error report file with more information is saved as:
> 15:07:52 #
> /home/jenkins/jenkins-slave/workspace/beam_PostCommit_Py_VR_Dataflow/src/hs_err_pid31336.log
> 15:07:53 Build step 'Invoke Gradle script' changed build result to FAILURE
> 15:07:53 Build step 'Invoke Gradle script' marked build as failure
> 15:07:56 Sending e-mails to: comm...@beam.apache.org
> 15:07:57 No emails were triggered.
> 15:07:57 Finished: FAILURE
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5812) Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT
[ https://issues.apache.org/jira/browse/BEAM-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659037#comment-16659037 ] Kenneth Knowles commented on BEAM-5812: --- The next build failed only on FastavroIT [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6342/] [https://scans.gradle.com/s/cmrthrxxlzy44/console-log?task=:beam-sdks-python:postCommitITTests] > Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, > FastavroIT > - > > Key: BEAM-5812 > URL: https://issues.apache.org/jira/browse/BEAM-5812 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: Valentyn Tymofieiev >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6341/] > [https://scans.gradle.com/s/ivjuxhni54azk/console-log?task=:beam-sdks-python:postCommitITTests] > I don't see anything about these tests that would say much. It could be > environmental but it might just imply that all the timeouts should be higher > if they are sensitive, or their was (and continues to be) an outage of some > sort. > Assignee chosen because I see you as reviewer of avro changes and author of > other changes to the py sdk in the last few days. If you have no idea, that's > fine, I just didn't want to leave it unassigned. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5813) Failure in ElasticSearchIOTest.testSplit
Kenneth Knowles created BEAM-5813:
-
Summary: Failure in ElasticSearchIOTest.testSplit
Key: BEAM-5813
URL: https://issues.apache.org/jira/browse/BEAM-5813
Project: Beam
Issue Type: Bug
Components: io-java-elasticsearch
Reporter: Kenneth Knowles
Assignee: Etienne Chauchot
[https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/]
[https://scans.gradle.com/s/xpsswtkdf62so/tests/y3up3nrnbr2ns-opjrlgkblx2ku]
{code:java}
Wrong number of splits expected:<25> but was:<22>
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5812) Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT
Kenneth Knowles created BEAM-5812: - Summary: Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT Key: BEAM-5812 URL: https://issues.apache.org/jira/browse/BEAM-5812 Project: Beam Issue Type: Bug Components: sdk-py-core Reporter: Kenneth Knowles Assignee: Valentyn Tymofieiev [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6341/] [https://scans.gradle.com/s/ivjuxhni54azk/console-log?task=:beam-sdks-python:postCommitITTests] I don't see anything about these tests that would say much. It could be environmental but it might just imply that all the timeouts should be higher if they are sensitive, or their was (and continues to be) an outage of some sort. Assignee chosen because I see you as reviewer of avro changes and author of other changes to the py sdk in the last few days. If you have no idea, that's fine, I just didn't want to leave it unassigned. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5814) Failure in RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
Kenneth Knowles created BEAM-5814:
-
Summary: Failure in
RegisterAndProcessBundleOperationTest.testProcessingBundleHandlesMultimapSideInputRequests
Key: BEAM-5814
URL: https://issues.apache.org/jira/browse/BEAM-5814
Project: Beam
Issue Type: Bug
Components: runner-dataflow
Reporter: Kenneth Knowles
Assignee: Ankur Goenka
https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1724/
https://scans.gradle.com/s/xpsswtkdf62so/tests/jpaz6qjpaw4ma-czu63fxt273w6
{code}
java.util.concurrent.ExecutionException:
org.mockito.exceptions.base.MockitoException:
No argument value was captured!
You might have forgotten to use argument.capture() in verify()...
...or you used capture() in stubbing but stubbed method was not called.
Be aware that it is recommended to use capture() only with verify()
...
{code}
Assigning to someone I think has some context as we have no code history or
blame for the Dataflow Worker. Do with this bug as you see fit. (filing all
flakes as critical for project health reasons)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
[
https://issues.apache.org/jira/browse/BEAM-5811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659032#comment-16659032
]
Kenneth Knowles commented on BEAM-5811:
---
Assignee just because code history looks like you might have most recent memory
of the test. What do you think?
> Timeout in Python datastore_write_it_test.DatastoreWriteIT
> --
>
> Key: BEAM-5811
> URL: https://issues.apache.org/jira/browse/BEAM-5811
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: yifan zou
>Priority: Critical
> Labels: flake
>
> [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
> [https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
> {code:java}
> TimedOutException: 'test_datastore_write_limit
> (apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5812) Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT
[ https://issues.apache.org/jira/browse/BEAM-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659040#comment-16659040 ] Kenneth Knowles commented on BEAM-5812: --- Longer ago we have one that fails UserScoreIT [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6326/] [https://scans.gradle.com/s/vqbs5bgqrhw3y/console-log?task=:beam-sdks-python:postCommitITTests] > Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, > FastavroIT > - > > Key: BEAM-5812 > URL: https://issues.apache.org/jira/browse/BEAM-5812 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: Valentyn Tymofieiev >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6341/] > [https://scans.gradle.com/s/ivjuxhni54azk/console-log?task=:beam-sdks-python:postCommitITTests] > I don't see anything about these tests that would say much. It could be > environmental but it might just imply that all the timeouts should be higher > if they are sensitive, or their was (and continues to be) an outage of some > sort. > Assignee chosen because I see you as reviewer of avro changes and author of > other changes to the py sdk in the last few days. If you have no idea, that's > fine, I just didn't want to leave it unassigned. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5811) Timeout in Python datastore_write_it_test.DatastoreWriteIT
Kenneth Knowles created BEAM-5811:
-
Summary: Timeout in Python datastore_write_it_test.DatastoreWriteIT
Key: BEAM-5811
URL: https://issues.apache.org/jira/browse/BEAM-5811
Project: Beam
Issue Type: Bug
Components: sdk-py-core
Reporter: Kenneth Knowles
Assignee: yifan zou
[https://builds.apache.org/job/beam_PostCommit_Python_Verify/6340/]
[https://scans.gradle.com/s/74drwrmqtaory/console-log?task=:beam-sdks-python:postCommitITTests]
{code:java}
TimedOutException: 'test_datastore_write_limit
(apache_beam.io.gcp.datastore_write_it_test.DatastoreWriteIT)'{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5828) Investigate publishing shaded source jars
Kenneth Knowles created BEAM-5828: - Summary: Investigate publishing shaded source jars Key: BEAM-5828 URL: https://issues.apache.org/jira/browse/BEAM-5828 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5810) Move Jenkins notifications to bui...@beam.apache.org
Kenneth Knowles created BEAM-5810: - Summary: Move Jenkins notifications to bui...@beam.apache.org Key: BEAM-5810 URL: https://issues.apache.org/jira/browse/BEAM-5810 Project: Beam Issue Type: Improvement Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles On list there is consensus to move them off commits@ and dev@ to a new list, builds@. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3138: - Assignee: Kenneth Knowles > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Assignee: Kenneth Knowles >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3138: - Assignee: (was: Kenneth Knowles) > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5817) Nexmark test of joining stream to files
Kenneth Knowles created BEAM-5817: - Summary: Nexmark test of joining stream to files Key: BEAM-5817 URL: https://issues.apache.org/jira/browse/BEAM-5817 Project: Beam Issue Type: New Feature Components: examples-nexmark Reporter: Kenneth Knowles Assignee: Kenneth Knowles Nexmark is a convenient framework for testing the use case of large scale stream enrichment. One way is joining a stream to files, and it can be tested via any source that Nexmark supports. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659679#comment-16659679 ] Kenneth Knowles commented on BEAM-3138: --- I've noticed this is causing needless overhead in the compilation of Nexmark - it depends on the shadowTest configuration of GCP IO, which then pulls in shadowTest of the SDK core. So there's a real cost to this misconfiguration. > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3573) Test jars should export only tests
[ https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3573: - Assignee: Kenneth Knowles > Test jars should export only tests > -- > > Key: BEAM-3573 > URL: https://issues.apache.org/jira/browse/BEAM-3573 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Today, we have test-jars that are used as libraries for testing. That is not > what "test jar" means, and dependency management actually does not work > correctly for this. It is OK to depend on a test jar in order to run the > tests therein, and not really OK to depend on one for another reason. > This ticket is a bucket ticket for fixes to this situation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-3138) Stop depending on Test JARs
[ https://issues.apache.org/jira/browse/BEAM-3138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3138: - Assignee: Kenneth Knowles > Stop depending on Test JARs > --- > > Key: BEAM-3138 > URL: https://issues.apache.org/jira/browse/BEAM-3138 > Project: Beam > Issue Type: Bug > Components: io-java-gcp, runner-core, sdk-java-core, sdk-java-harness >Reporter: Thomas Groh >Assignee: Kenneth Knowles >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Testing components can be in a testing or otherwise signaled package, but > shouldn't really be depended on by depending on a test jar in the test scope. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5812) Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT
[ https://issues.apache.org/jira/browse/BEAM-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16659668#comment-16659668 ] Kenneth Knowles commented on BEAM-5812: --- UserScoreIT here. Looking a little closer, it looks like it might be the dependency download that is failing. https://builds.apache.org/job/beam_PreCommit_Python_Cron/497/ https://scans.gradle.com/s/qcmrte5qmp26y/console-log?task=:beam-sdks-python:postCommitITTests > Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, > FastavroIT > - > > Key: BEAM-5812 > URL: https://issues.apache.org/jira/browse/BEAM-5812 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: Valentyn Tymofieiev >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6341/] > [https://scans.gradle.com/s/ivjuxhni54azk/console-log?task=:beam-sdks-python:postCommitITTests] > I don't see anything about these tests that would say much. It could be > environmental but it might just imply that all the timeouts should be higher > if they are sensitive, or their was (and continues to be) an outage of some > sort. > Assignee chosen because I see you as reviewer of avro changes and author of > other changes to the py sdk in the last few days. If you have no idea, that's > fine, I just didn't want to leave it unassigned. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-4539) TestBigQuery and TestBigQueryOptions are in the wrong path
[
https://issues.apache.org/jira/browse/BEAM-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-4539.
---
Resolution: Invalid
Fix Version/s: Not applicable
Actually src/test/java is a path for _test suites_ not _test utilities_. These
are test utilities that are meant to be used by other modules. One workaround
is to make them optional dependencies so they are not pulled in, assuming the
user will depend on them in their own test suite.
But since hamcrest and junit are both trivially tiny and self-contained there's
no point.
> TestBigQuery and TestBigQueryOptions are in the wrong path
> --
>
> Key: BEAM-4539
> URL: https://issues.apache.org/jira/browse/BEAM-4539
> Project: Beam
> Issue Type: Bug
> Components: io-java-gcp
>Reporter: Ismaël Mejía
>Assignee: Anton Kedin
>Priority: Major
> Fix For: Not applicable
>
>
> {{TestBigQuery}} and {{TestBigQueryOptions}} are currently in the
> src/main/java dir but are used only for tests so they should be part of the
> test artifact in src/test/java. As a side effect this makes the main GCPIO
> jar depend on both JUnit and Hamcrest which should be avoided..
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3608) Pre-shade Guava for things we want to keep using
[
https://issues.apache.org/jira/browse/BEAM-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3608:
--
Parent Issue: BEAM-5819 (was: BEAM-3606)
> Pre-shade Guava for things we want to keep using
>
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Instead of shading as part of our build, we can shade before build so that it
> is apparent when reading code, and in IDEs, that a particular class resides
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs
> unless they are annotated {{@Internal}}, and it makes sharing between our own
> modules never get broken by shading again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5819) Vendor dependencies that are currently shaded
Kenneth Knowles created BEAM-5819: - Summary: Vendor dependencies that are currently shaded Key: BEAM-5819 URL: https://issues.apache.org/jira/browse/BEAM-5819 Project: Beam Issue Type: New Feature Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles This is an umbrella ticket for the tasks that will lead to removing the shadow plugin from our main build, speeding our build, simplifying our IDE integration, and improving the readability of our code. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5821) Remove shadow plugin from SDK Java core
Kenneth Knowles created BEAM-5821: - Summary: Remove shadow plugin from SDK Java core Key: BEAM-5821 URL: https://issues.apache.org/jira/browse/BEAM-5821 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5820) Vendor Calcite
Kenneth Knowles created BEAM-5820: - Summary: Vendor Calcite Key: BEAM-5820 URL: https://issues.apache.org/jira/browse/BEAM-5820 Project: Beam Issue Type: Sub-task Components: dsl-sql Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5823) Vendor commons
Kenneth Knowles created BEAM-5823: - Summary: Vendor commons Key: BEAM-5823 URL: https://issues.apache.org/jira/browse/BEAM-5823 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5822) Vendor bytebuddy
Kenneth Knowles created BEAM-5822: - Summary: Vendor bytebuddy Key: BEAM-5822 URL: https://issues.apache.org/jira/browse/BEAM-5822 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5824) Vendor protobuf-java
Kenneth Knowles created BEAM-5824: - Summary: Vendor protobuf-java Key: BEAM-5824 URL: https://issues.apache.org/jira/browse/BEAM-5824 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5825) Vendor kryo
Kenneth Knowles created BEAM-5825: - Summary: Vendor kryo Key: BEAM-5825 URL: https://issues.apache.org/jira/browse/BEAM-5825 Project: Beam Issue Type: Sub-task Components: sdk-java-core Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5826) Vendor slf4j API
Kenneth Knowles created BEAM-5826: - Summary: Vendor slf4j API Key: BEAM-5826 URL: https://issues.apache.org/jira/browse/BEAM-5826 Project: Beam Issue Type: Sub-task Components: sdk-java-harness Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-3608) Vendor Guava
[
https://issues.apache.org/jira/browse/BEAM-3608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-3608:
--
Summary: Vendor Guava (was: Pre-shade Guava for things we want to keep
using)
> Vendor Guava
>
>
> Key: BEAM-3608
> URL: https://issues.apache.org/jira/browse/BEAM-3608
> Project: Beam
> Issue Type: Sub-task
> Components: runner-core, sdk-java-core
>Reporter: Kenneth Knowles
>Priority: Major
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> Instead of shading as part of our build, we can shade before build so that it
> is apparent when reading code, and in IDEs, that a particular class resides
> in a hidden namespace.
> {{import com.google.common.reflect.TypeToken}}
> becomes something like
> {{import org.apache.beam.private.guava21.com.google.common.reflect.TypeToken}}
> So we can very trivially ban `org.apache.beam.private` from public APIs
> unless they are annotated {{@Internal}}, and it makes sharing between our own
> modules never get broken by shading again.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5827) Vendor joda time
Kenneth Knowles created BEAM-5827: - Summary: Vendor joda time Key: BEAM-5827 URL: https://issues.apache.org/jira/browse/BEAM-5827 Project: Beam Issue Type: Sub-task Components: sdk-java-harness Reporter: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16656971#comment-16656971
]
Kenneth Knowles commented on BEAM-5779:
---
This now occurred in the Cron of the Python precommit. Checking the job, no
messages were read from pubsub. A flaky initialization maybe?
> Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
> --
>
> Key: BEAM-5779
> URL: https://issues.apache.org/jira/browse/BEAM-5779
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Ahmet Altay
>Priority: Major
> Labels: flake
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
> Path to failure:
> * (Jenkins) beam_PostCommit_Python_Verify
> * (gradle) :beam-sdks-python:postCommitITTests
> * (py module) test_streaming_data_only
> * (test class)
> apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
> {code}
> AssertionError:
> Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
> messages.)
> but: Expected 2 messages. Got 0 messages. Diffs (item, count):
> Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
> Unexpected: []
> {code}
> Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5779:
--
Comment: was deleted
(was: This now occurred in the Cron of the Python precommit. Checking the job,
no messages were read from pubsub. A flaky initialization maybe?)
> Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
> --
>
> Key: BEAM-5779
> URL: https://issues.apache.org/jira/browse/BEAM-5779
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Udi Meiri
>Priority: Major
> Labels: flake
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
> Path to failure:
> * (Jenkins) beam_PostCommit_Python_Verify
> * (gradle) :beam-sdks-python:postCommitITTests
> * (py module) test_streaming_data_only
> * (test class)
> apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
> {code}
> AssertionError:
> Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
> messages.)
> but: Expected 2 messages. Got 0 messages. Diffs (item, count):
> Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
> Unexpected: []
> {code}
> Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5779:
-
Assignee: Udi Meiri (was: Ahmet Altay)
> Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
> --
>
> Key: BEAM-5779
> URL: https://issues.apache.org/jira/browse/BEAM-5779
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Udi Meiri
>Priority: Major
> Labels: flake
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
> Path to failure:
> * (Jenkins) beam_PostCommit_Python_Verify
> * (gradle) :beam-sdks-python:postCommitITTests
> * (py module) test_streaming_data_only
> * (test class)
> apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
> {code}
> AssertionError:
> Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
> messages.)
> but: Expected 2 messages. Got 0 messages. Diffs (item, count):
> Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
> Unexpected: []
> {code}
> Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (BEAM-5286) [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] .sh script: text file busy.
[ https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5286: -- Comment: was deleted (was: https://builds.apache.org/job/beam_PreCommit_Java_Cron/462/) > [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] > .sh script: text file busy. > -- > > Key: BEAM-5286 > URL: https://issues.apache.org/jira/browse/BEAM-5286 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Alan Myrvold >Priority: Major > Labels: currently-failing > > Sample failure: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/] > Sample relevant log: > org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: > java.io.IOException: Cannot run program > "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": > error=26, Text file busy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5286) [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] .sh script: text file busy.
[ https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657060#comment-16657060 ] Kenneth Knowles edited comment on BEAM-5286 at 10/19/18 4:43 PM: - https://builds.apache.org/job/beam_PreCommit_Java_Cron/462/ was (Author: kenn): https://builds.apache.org/job/beam_PreCommit_Java_Cron/462/ > [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] > .sh script: text file busy. > -- > > Key: BEAM-5286 > URL: https://issues.apache.org/jira/browse/BEAM-5286 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Alan Myrvold >Priority: Major > Labels: currently-failing > > Sample failure: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/] > Sample relevant log: > org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: > java.io.IOException: Cannot run program > "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": > error=26, Text file busy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5286) [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] .sh script: text file busy.
[ https://issues.apache.org/jira/browse/BEAM-5286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657060#comment-16657060 ] Kenneth Knowles commented on BEAM-5286: --- https://builds.apache.org/job/beam_PreCommit_Java_Cron/462/ > [beam_PostCommit_Java_GradleBuild][org.apache.beam.examples.subprocess.ExampleEchoPipelineTest.testExampleEchoPipeline][Flake] > .sh script: text file busy. > -- > > Key: BEAM-5286 > URL: https://issues.apache.org/jira/browse/BEAM-5286 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Boyuan Zhang >Assignee: Alan Myrvold >Priority: Major > Labels: currently-failing > > Sample failure: > [https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1375/testReport/junit/org.apache.beam.examples.subprocess/ExampleEchoPipelineTest/testExampleEchoPipeline/] > Sample relevant log: > org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.lang.Exception: > java.io.IOException: Cannot run program > "/tmp/test-Echoo1519764280436328522/test-EchoAgain3143210610074994370.sh": > error=26, Text file busy -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5779) Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
[
https://issues.apache.org/jira/browse/BEAM-5779?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16657063#comment-16657063
]
Kenneth Knowles commented on BEAM-5779:
---
https://builds.apache.org/job/beam_PostCommit_Python_Verify/6321/
> Failure in beam_PostCommit_Python_Verify PubSubIntegrationTest on Dataflow
> --
>
> Key: BEAM-5779
> URL: https://issues.apache.org/jira/browse/BEAM-5779
> Project: Beam
> Issue Type: Bug
> Components: sdk-py-core
>Reporter: Kenneth Knowles
>Assignee: Udi Meiri
>Priority: Major
> Labels: flake
> Time Spent: 20m
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PostCommit_Python_Verify/6296/
> Path to failure:
> * (Jenkins) beam_PostCommit_Python_Verify
> * (gradle) :beam-sdks-python:postCommitITTests
> * (py module) test_streaming_data_only
> * (test class)
> apache_beam.io.gcp.pubsub_integration_test.PubSubIntegrationTest
> {code}
> AssertionError:
> Expected: (Test pipeline expected terminated in state: RUNNING and Expected 2
> messages.)
> but: Expected 2 messages. Got 0 messages. Diffs (item, count):
> Expected but not in actual: [('data002-seen', 1), ('data001-seen', 1)]
> Unexpected: []
> {code}
> Looks like a legitimate failure. It is flakey as it went green later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5795) Can SQL Query 5 be simplified?
Kenneth Knowles created BEAM-5795: - Summary: Can SQL Query 5 be simplified? Key: BEAM-5795 URL: https://issues.apache.org/jira/browse/BEAM-5795 Project: Beam Issue Type: Improvement Components: dsl-sql Reporter: Kenneth Knowles Assignee: Kenneth Knowles The original CQL query uses the ALL operator over the set of rows that are within a certain period from the watermark. We instead have a fancy join, due to windowing. Nonetheless, can this be simplified? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5176) FailOnWarnings behave differently between CLI and Intellij build
[ https://issues.apache.org/jira/browse/BEAM-5176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652531#comment-16652531 ] Kenneth Knowles commented on BEAM-5176: --- My mistake; I was on a funky branch. > FailOnWarnings behave differently between CLI and Intellij build > - > > Key: BEAM-5176 > URL: https://issues.apache.org/jira/browse/BEAM-5176 > Project: Beam > Issue Type: Sub-task > Components: build-system >Reporter: Etienne Chauchot >Assignee: Kenneth Knowles >Priority: Major > > In command line the build passes but fails on the IDE because of warnings. > To make it pass I had to put false in failOnWarnings in ApplyJavaNature -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5057) beam_Release_Gradle_NightlySnapshot failing due to a Javadoc error
[ https://issues.apache.org/jira/browse/BEAM-5057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16652620#comment-16652620 ] Kenneth Knowles commented on BEAM-5057: --- Is this now obsolete? > beam_Release_Gradle_NightlySnapshot failing due to a Javadoc error > -- > > Key: BEAM-5057 > URL: https://issues.apache.org/jira/browse/BEAM-5057 > Project: Beam > Issue Type: Bug > Components: testing >Reporter: Chamikara Jayalath >Assignee: Chamikara Jayalath >Priority: Major > Time Spent: 2h 50m > Remaining Estimate: 0h > > [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/127/console] > [https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/125/console] > > * What went wrong: > Execution failed for task ':beam-sdks-java-core:javadoc'. > > Javadoc generation failed. Generated Javadoc options file (useful for > > troubleshooting): > > '/home/jenkins/jenkins-slave/workspace/beam_Release_Gradle_NightlySnapshot/src/sdks/java/core/build/tmp/javadoc/javadoc.options' > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5774) beam_Release_Gradle_NightlySnapshot timed out
Kenneth Knowles created BEAM-5774: - Summary: beam_Release_Gradle_NightlySnapshot timed out Key: BEAM-5774 URL: https://issues.apache.org/jira/browse/BEAM-5774 Project: Beam Issue Type: Bug Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/209/ Looking at the trend, this is not surprising: https://builds.apache.org/job/beam_Release_Gradle_NightlySnapshot/buildTimeTrend -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5812) Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, FastavroIT
[ https://issues.apache.org/jira/browse/BEAM-5812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660952#comment-16660952 ] Kenneth Knowles commented on BEAM-5812: --- FastavroIT https://builds.apache.org/job/beam_PostCommit_Python_Verify/6353/ https://builds.apache.org/job/beam_PostCommit_Python_Verify/6353/console > Timeout in Python ITs: WordCountIT (fn api and legacy), HourlyTeamScoreIT, > FastavroIT > - > > Key: BEAM-5812 > URL: https://issues.apache.org/jira/browse/BEAM-5812 > Project: Beam > Issue Type: Bug > Components: sdk-py-core >Reporter: Kenneth Knowles >Assignee: Valentyn Tymofieiev >Priority: Critical > Labels: flake > > [https://builds.apache.org/job/beam_PostCommit_Python_Verify/6341/] > [https://scans.gradle.com/s/ivjuxhni54azk/console-log?task=:beam-sdks-python:postCommitITTests] > I don't see anything about these tests that would say much. It could be > environmental but it might just imply that all the timeouts should be higher > if they are sensitive, or their was (and continues to be) an outage of some > sort. > Assignee chosen because I see you as reviewer of avro changes and author of > other changes to the py sdk in the last few days. If you have no idea, that's > fine, I just didn't want to leave it unassigned. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5631) updateOfflineRepository fails with DefaultModuleComponentIdentifier error
[ https://issues.apache.org/jira/browse/BEAM-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661098#comment-16661098 ] Kenneth Knowles commented on BEAM-5631: --- Does Gradle 4.9 bring anything that we need? How about we permanently downgrade and go from 4.8 to 5? > updateOfflineRepository fails with DefaultModuleComponentIdentifier error > - > > Key: BEAM-5631 > URL: https://issues.apache.org/jira/browse/BEAM-5631 > Project: Beam > Issue Type: Bug > Components: build-system >Affects Versions: 2.8.0 >Reporter: Alan Myrvold >Assignee: Scott Wegner >Priority: Major > > ./gradlew :beam-examples:updateOfflineRepository > > Task :beam-examples-java:updateOfflineRepository FAILED > FAILURE: Build failed with an exception. > * What went wrong: > Execution failed for task ':beam-examples-java:updateOfflineRepository'. > > Could not find matching constructor for: > > org.gradle.internal.component.external.model.DefaultModuleComponentIdentifier(java.lang.String, > > java.lang.String, java.lang.String) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5631) updateOfflineRepository fails with DefaultModuleComponentIdentifier error
[ https://issues.apache.org/jira/browse/BEAM-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661176#comment-16661176 ] Kenneth Knowles commented on BEAM-5631: --- Gotcha. FWIW the latest is 4.10.2 and 4.9 is quite old by "close to HEAD" and open source standards. > updateOfflineRepository fails with DefaultModuleComponentIdentifier error > - > > Key: BEAM-5631 > URL: https://issues.apache.org/jira/browse/BEAM-5631 > Project: Beam > Issue Type: Bug > Components: build-system >Affects Versions: 2.8.0 >Reporter: Alan Myrvold >Assignee: Scott Wegner >Priority: Major > > ./gradlew :beam-examples:updateOfflineRepository > > Task :beam-examples-java:updateOfflineRepository FAILED > FAILURE: Build failed with an exception. > * What went wrong: > Execution failed for task ':beam-examples-java:updateOfflineRepository'. > > Could not find matching constructor for: > > org.gradle.internal.component.external.model.DefaultModuleComponentIdentifier(java.lang.String, > > java.lang.String, java.lang.String) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5833) Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless failures
Kenneth Knowles created BEAM-5833: - Summary: Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless failures Key: BEAM-5833 URL: https://issues.apache.org/jira/browse/BEAM-5833 Project: Beam Issue Type: Bug Components: sdk-java-harness Reporter: Kenneth Knowles Assignee: Kenneth Knowles https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5833) Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless failures
[ https://issues.apache.org/jira/browse/BEAM-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5833: -- Description: https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ https://scans.gradle.com/s/hqv45s2vaajvk was:https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ > Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless > failures > --- > > Key: BEAM-5833 > URL: https://issues.apache.org/jira/browse/BEAM-5833 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Blocker > > https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ > https://scans.gradle.com/s/hqv45s2vaajvk -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5833) Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless failures
[ https://issues.apache.org/jira/browse/BEAM-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5833: -- Description: https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ https://scans.gradle.com/s/big7mxohgz2ry was: https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ https://scans.gradle.com/s/hqv45s2vaajvk > Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless > failures > --- > > Key: BEAM-5833 > URL: https://issues.apache.org/jira/browse/BEAM-5833 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Blocker > > https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ > https://scans.gradle.com/s/big7mxohgz2ry -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5793) Python sdk docs target fails in flink_streaming_impulse_source
[ https://issues.apache.org/jira/browse/BEAM-5793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5793: -- Component/s: (was: sdk-py-core) test-failures > Python sdk docs target fails in flink_streaming_impulse_source > -- > > Key: BEAM-5793 > URL: https://issues.apache.org/jira/browse/BEAM-5793 > Project: Beam > Issue Type: Improvement > Components: test-failures >Reporter: Ruoyun Huang >Assignee: Maximilian Michels >Priority: Minor > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > gradle scan result: [https://gradle.com/s/w4icffibxs72a] > Error message: > projects/beam2/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py:docstring > of > apache_beam.io.flink.flink_streaming_impulse_source.FlinkStreamingImpulseSource.from_runner_api_parameter:11:Unexpected > indentation. > > Root cause seems to be that there are two annotations for a single function. > And sphinx does not like that. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5833) Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless failures
[ https://issues.apache.org/jira/browse/BEAM-5833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5833: -- Component/s: test-failures > Java precommit broken by CloseableFnDataReceiver#flush as well as Spotless > failures > --- > > Key: BEAM-5833 > URL: https://issues.apache.org/jira/browse/BEAM-5833 > Project: Beam > Issue Type: Bug > Components: sdk-java-harness, test-failures >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Blocker > > https://builds.apache.org/job/beam_PreCommit_Java_Cron/498/ > https://scans.gradle.com/s/big7mxohgz2ry -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5834) Document lack of support for GROUP BY and set operations on floating points in SQL
Kenneth Knowles created BEAM-5834: - Summary: Document lack of support for GROUP BY and set operations on floating points in SQL Key: BEAM-5834 URL: https://issues.apache.org/jira/browse/BEAM-5834 Project: Beam Issue Type: Sub-task Components: dsl-sql, website Reporter: Kenneth Knowles Assignee: Kenneth Knowles -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5829) SQL should probably not support GROUP BY or set operations on floating point numbers
[ https://issues.apache.org/jira/browse/BEAM-5829?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5829: -- Priority: Major (was: Critical) > SQL should probably not support GROUP BY or set operations on floating point > numbers > > > Key: BEAM-5829 > URL: https://issues.apache.org/jira/browse/BEAM-5829 > Project: Beam > Issue Type: Bug > Components: dsl-sql, test-failures >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > These are known to be super unreliable on most SQL engines, and generally > indicate a programming error. Floating points numbers are intended as > stand-ins for real numbers, for which equality (hence grouping and set > operations) are undecidable. And our build is broken because of it. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5830) What is an SDK? Users landing on the website have no reason to know
[ https://issues.apache.org/jira/browse/BEAM-5830?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-5830. --- Resolution: Fixed Fix Version/s: Not applicable > What is an SDK? Users landing on the website have no reason to know > --- > > Key: BEAM-5830 > URL: https://issues.apache.org/jira/browse/BEAM-5830 > Project: Beam > Issue Type: Bug > Components: website >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Fix For: Not applicable > > Time Spent: 20m > Remaining Estimate: 0h > > "SDK" is an abstract term that only a fraction of developers will have an > intuition for. Perhaps far fewer users. The Beam vision is "any language on > any runner" so why not use that language? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5307) Allow injection of state in DoFn.FinishBundle
[
https://issues.apache.org/jira/browse/BEAM-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-5307.
---
Resolution: Duplicate
Fix Version/s: Not applicable
> Allow injection of state in DoFn.FinishBundle
> -
>
> Key: BEAM-5307
> URL: https://issues.apache.org/jira/browse/BEAM-5307
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Sander Ploegsma
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> Example use case: a stateful {{DoFn}} that requires persisting its state to
> an external database. Instead of writing to the external database for each
> element, it is much more efficient to flush the state every once in a while,
> or when cleaning up.
> Currently, this is not possible because the {{@StateId}} injection is only
> available in processing functions and timers.
> [This|https://stackoverflow.com/questions/51789776/calculating-deltas-in-apache-beam-using-stateful-processing]
> might be a workaround, but I'm not even sure if it works.
> Instead, by allowing the use of {{@StateId}} inside a {{@FinishBundle}}
> function, we can make sure the internal state is persisted in all scenarios:
> {code:java}
> @FinishBundle
> public void flush(FinishBundleContext context, @StateId("myState")
> ValueState state) {
> repository.save(state.read());
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5307) Allow injection of state in DoFn.FinishBundle
[
https://issues.apache.org/jira/browse/BEAM-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661260#comment-16661260
]
Kenneth Knowles commented on BEAM-5307:
---
Hi [~sploegsma], thanks for the report. The trouble is that {{@FinishBundle}}
is not partitioned by key or window, but is a way to flush outgoing buffers in
a fairly naive way. What you want in the stateful case is to set a timer for
the window expiration and flush there.
We have some work in progress to make this a convenient {{@OnWindowExpiration}}
method - you can follow BEAM-1589 if you like.
> Allow injection of state in DoFn.FinishBundle
> -
>
> Key: BEAM-5307
> URL: https://issues.apache.org/jira/browse/BEAM-5307
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Sander Ploegsma
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> Example use case: a stateful {{DoFn}} that requires persisting its state to
> an external database. Instead of writing to the external database for each
> element, it is much more efficient to flush the state every once in a while,
> or when cleaning up.
> Currently, this is not possible because the {{@StateId}} injection is only
> available in processing functions and timers.
> [This|https://stackoverflow.com/questions/51789776/calculating-deltas-in-apache-beam-using-stateful-processing]
> might be a workaround, but I'm not even sure if it works.
> Instead, by allowing the use of {{@StateId}} inside a {{@FinishBundle}}
> function, we can make sure the internal state is persisted in all scenarios:
> {code:java}
> @FinishBundle
> public void flush(FinishBundleContext context, @StateId("myState")
> ValueState state) {
> repository.save(state.read());
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5221) Complile error: invalid LOC header (bad signature)
[
https://issues.apache.org/jira/browse/BEAM-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661264#comment-16661264
]
Kenneth Knowles commented on BEAM-5221:
---
This is the published {{beam-model-pipeline-2.6.0.jar}}?
> Complile error: invalid LOC header (bad signature)
> ---
>
> Key: BEAM-5221
> URL: https://issues.apache.org/jira/browse/BEAM-5221
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 2.6.0
> Environment: pring Tool Suite
> Version: 3.9.2.RELEASE
> Build Id: 201712210947
> Platform: Eclipse Oxygen.2 (4.7.2)
> Maven 3.3.9
>Reporter: Shantanu Kirkire
>Assignee: Kenneth Knowles
>Priority: Major
> Labels: build, compile-error, maven
>
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile)
> on project linking: Compilation failure: Compilation failure:
> [ERROR] error reading
> C:\Users\SKirkire\.m2\repository\org\apache\beam\beam-model-pipeline\2.6.0\beam-model-pipeline-2.6.0.jar;
> invalid LOC header (bad signature)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-4763) Add postCommit scripts and perfkit dashboards for nexmark on Samza runner
[ https://issues.apache.org/jira/browse/BEAM-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4763: - Assignee: Xinyu Liu (was: Kenneth Knowles) > Add postCommit scripts and perfkit dashboards for nexmark on Samza runner > - > > Key: BEAM-4763 > URL: https://issues.apache.org/jira/browse/BEAM-4763 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Xinyu Liu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5221) Complile error: invalid LOC header (bad signature)
[ https://issues.apache.org/jira/browse/BEAM-5221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661265#comment-16661265 ] Kenneth Knowles commented on BEAM-5221: --- I am handing this to someone who might have better context. > Complile error: invalid LOC header (bad signature) > --- > > Key: BEAM-5221 > URL: https://issues.apache.org/jira/browse/BEAM-5221 > Project: Beam > Issue Type: Bug > Components: beam-model >Affects Versions: 2.6.0 > Environment: pring Tool Suite > Version: 3.9.2.RELEASE > Build Id: 201712210947 > Platform: Eclipse Oxygen.2 (4.7.2) > Maven 3.3.9 >Reporter: Shantanu Kirkire >Assignee: Luke Cwik >Priority: Major > Labels: build, compile-error, maven > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) > on project linking: Compilation failure: Compilation failure: > [ERROR] error reading > C:\Users\SKirkire\.m2\repository\org\apache\beam\beam-model-pipeline\2.6.0\beam-model-pipeline-2.6.0.jar; > invalid LOC header (bad signature) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4760) Add postCommit scripts and perfkit dashboards for nexmark on Apex runner
[ https://issues.apache.org/jira/browse/BEAM-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661271#comment-16661271 ] Kenneth Knowles commented on BEAM-4760: --- Any interest in this [~thw]? > Add postCommit scripts and perfkit dashboards for nexmark on Apex runner > > > Key: BEAM-4760 > URL: https://issues.apache.org/jira/browse/BEAM-4760 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Thomas Weise >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4762) Add postCommit scripts and perfkit dashboards for nexmark on Gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4762: - Assignee: Manu Zhang (was: Kenneth Knowles) > Add postCommit scripts and perfkit dashboards for nexmark on Gearpump runner > > > Key: BEAM-4762 > URL: https://issues.apache.org/jira/browse/BEAM-4762 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Manu Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-4760) Add postCommit scripts and perfkit dashboards for nexmark on Apex runner
[ https://issues.apache.org/jira/browse/BEAM-4760?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-4760: - Assignee: Thomas Weise (was: Kenneth Knowles) > Add postCommit scripts and perfkit dashboards for nexmark on Apex runner > > > Key: BEAM-4760 > URL: https://issues.apache.org/jira/browse/BEAM-4760 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Thomas Weise >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4763) Add postCommit scripts and perfkit dashboards for nexmark on Samza runner
[ https://issues.apache.org/jira/browse/BEAM-4763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661268#comment-16661268 ] Kenneth Knowles commented on BEAM-4763: --- Any interest in this [~xinyu]? > Add postCommit scripts and perfkit dashboards for nexmark on Samza runner > - > > Key: BEAM-4763 > URL: https://issues.apache.org/jira/browse/BEAM-4763 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Xinyu Liu >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-4762) Add postCommit scripts and perfkit dashboards for nexmark on Gearpump runner
[ https://issues.apache.org/jira/browse/BEAM-4762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661270#comment-16661270 ] Kenneth Knowles commented on BEAM-4762: --- Any interest in this [~mauzhang]? > Add postCommit scripts and perfkit dashboards for nexmark on Gearpump runner > > > Key: BEAM-4762 > URL: https://issues.apache.org/jira/browse/BEAM-4762 > Project: Beam > Issue Type: Test > Components: examples-nexmark >Reporter: Etienne Chauchot >Assignee: Manu Zhang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5061) Invisible parameter type exception in JDK 10
[
https://issues.apache.org/jira/browse/BEAM-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles updated BEAM-5061:
--
Issue Type: Sub-task (was: Bug)
Parent: BEAM-2530
> Invisible parameter type exception in JDK 10
>
>
> Key: BEAM-5061
> URL: https://issues.apache.org/jira/browse/BEAM-5061
> Project: Beam
> Issue Type: Sub-task
> Components: beam-model
>Affects Versions: 2.5.0
>Reporter: Mike Pedersen
>Assignee: Kenneth Knowles
>Priority: Major
>
> When using JDK 10, using a ParDo after a CoGroupByKey seems to create the
> following exception when executed on local runner:
> {noformat}
> Exception in thread "main"
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: Invisible parameter type of Main$1 arg0 for
> public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
> ...
> Caused by: java.lang.IllegalStateException: Invisible parameter type of
> Main$1 arg0 for public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.InstrumentedType$Default.validated(InstrumentedType.java:925)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:465)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:170)
> ...
> {noformat}
> This error disappears completely when using JDK 8. Here is a minimal example
> to reproduce it:
> {code:java}
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.transforms.join.CoGbkResult;
> import org.apache.beam.sdk.transforms.join.CoGroupByKey;
> import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.apache.beam.sdk.values.TupleTag;
> import java.util.Arrays;
> import java.util.List;
> public class Main {
> public static void main(String[] args) {
> PipelineOptions options = PipelineOptionsFactory.create();
> Pipeline p = Pipeline.create(options);
> final TupleTag emailsTag = new TupleTag<>();
> final TupleTag phonesTag = new TupleTag<>();
> final List> emailsList =
> Arrays.asList(
> KV.of("amy", "a...@example.com"),
> KV.of("carl", "c...@example.com"),
> KV.of("julia", "ju...@example.com"),
> KV.of("carl", "c...@email.com"));
> final List> phonesList =
> Arrays.asList(
> KV.of("amy", "111-222-"),
> KV.of("james", "222-333-"),
> KV.of("amy", "333-444-"),
> KV.of("carl", "444-555-"));
> PCollection> emails = p.apply("CreateEmails",
> Create.of(emailsList));
> PCollection> phones = p.apply("CreatePhones",
> Create.of(phonesList));
> PCollection> results =
> KeyedPCollectionTuple.of(emailsTag, emails)
> .and(phonesTag, phones)
> .apply(CoGroupByKey.create());
> PCollection contactLines =
> results.apply(
> ParDo.of(
> new DoFn, String>() {
> @ProcessElement
> public void processElement(ProcessContext
> c) {
> KV e =
> c.element();
> String name = e.getKey();
> Iterable emailsIter =
> e.getValue().getAll(emailsTag);
> Iterable phonesIter =
> e.getValue().getAll(phonesTag);
> String formattedResult = "";
> c.output(formattedResult);
> }
>
[jira] [Commented] (BEAM-5061) Invisible parameter type exception in JDK 10
[
https://issues.apache.org/jira/browse/BEAM-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16661273#comment-16661273
]
Kenneth Knowles commented on BEAM-5061:
---
I will make this a subtask of BEAM-2530 as some folks are starting to pick
those up.
> Invisible parameter type exception in JDK 10
>
>
> Key: BEAM-5061
> URL: https://issues.apache.org/jira/browse/BEAM-5061
> Project: Beam
> Issue Type: Bug
> Components: beam-model
>Affects Versions: 2.5.0
>Reporter: Mike Pedersen
>Assignee: Kenneth Knowles
>Priority: Major
>
> When using JDK 10, using a ParDo after a CoGroupByKey seems to create the
> following exception when executed on local runner:
> {noformat}
> Exception in thread "main"
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: Invisible parameter type of Main$1 arg0 for
> public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
> ...
> Caused by: java.lang.IllegalStateException: Invisible parameter type of
> Main$1 arg0 for public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.InstrumentedType$Default.validated(InstrumentedType.java:925)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:465)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:170)
> ...
> {noformat}
> This error disappears completely when using JDK 8. Here is a minimal example
> to reproduce it:
> {code:java}
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.transforms.join.CoGbkResult;
> import org.apache.beam.sdk.transforms.join.CoGroupByKey;
> import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.apache.beam.sdk.values.TupleTag;
> import java.util.Arrays;
> import java.util.List;
> public class Main {
> public static void main(String[] args) {
> PipelineOptions options = PipelineOptionsFactory.create();
> Pipeline p = Pipeline.create(options);
> final TupleTag emailsTag = new TupleTag<>();
> final TupleTag phonesTag = new TupleTag<>();
> final List> emailsList =
> Arrays.asList(
> KV.of("amy", "a...@example.com"),
> KV.of("carl", "c...@example.com"),
> KV.of("julia", "ju...@example.com"),
> KV.of("carl", "c...@email.com"));
> final List> phonesList =
> Arrays.asList(
> KV.of("amy", "111-222-"),
> KV.of("james", "222-333-"),
> KV.of("amy", "333-444-"),
> KV.of("carl", "444-555-"));
> PCollection> emails = p.apply("CreateEmails",
> Create.of(emailsList));
> PCollection> phones = p.apply("CreatePhones",
> Create.of(phonesList));
> PCollection> results =
> KeyedPCollectionTuple.of(emailsTag, emails)
> .and(phonesTag, phones)
> .apply(CoGroupByKey.create());
> PCollection contactLines =
> results.apply(
> ParDo.of(
> new DoFn, String>() {
> @ProcessElement
> public void processElement(ProcessContext
> c) {
> KV e =
> c.element();
> String name = e.getKey();
> Iterable emailsIter =
> e.getValue().getAll(emailsTag);
> Iterable phonesIter =
> e.getValue().getAll(phonesTag);
> String formattedResult = "";
>
[jira] [Assigned] (BEAM-5645) RowCoder not equal after serialization
[
https://issues.apache.org/jira/browse/BEAM-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5645:
-
Assignee: Reuven Lax (was: Kenneth Knowles)
> RowCoder not equal after serialization
> --
>
> Key: BEAM-5645
> URL: https://issues.apache.org/jira/browse/BEAM-5645
> Project: Beam
> Issue Type: Bug
> Components: runner-core
>Affects Versions: 2.6.0
>Reporter: Julien Tournay
>Assignee: Reuven Lax
>Priority: Major
>
> The following code throws an exception:
> {code:scala}
> import org.apache.beam.sdk.coders._
> import org.apache.beam.sdk.schemas.Schema
> val schema =
> Schema
> .builder()
> .addInt32Field("c1")
> .addStringField("c2")
> .addDoubleField("c3")
> .build()
> val coder = RowCoder.of(schema)
> org.apache.beam.sdk.util.SerializableUtils.ensureSerializable(coder)
> {code}
> {code}
> java.lang.IllegalStateException: Coder not equal to original after
> serialization, indicating that the Coder may not implement serialization
> correctly. Before: org.apache.beam.sdk.coders.RowCoder@1323050e, after:
> org.apache.beam.sdk.coders.RowCoder@4c03150a
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (BEAM-3573) Test jars should export only tests, and only be exported for select modules
[ https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-3573: -- Summary: Test jars should export only tests, and only be exported for select modules (was: Test jars should export only tests) > Test jars should export only tests, and only be exported for select modules > --- > > Key: BEAM-3573 > URL: https://issues.apache.org/jira/browse/BEAM-3573 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 40m > Remaining Estimate: 0h > > Today, we have test-jars that are used as libraries for testing. That is not > what "test jar" means, and dependency management actually does not work > correctly for this. It is OK to depend on a test jar in order to run the > tests therein, and not really OK to depend on one for another reason. > This ticket is a bucket ticket for fixes to this situation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5887) packageTests and shadowTestJar write the same file
Kenneth Knowles created BEAM-5887:
-
Summary: packageTests and shadowTestJar write the same file
Key: BEAM-5887
URL: https://issues.apache.org/jira/browse/BEAM-5887
Project: Beam
Issue Type: Bug
Components: build-system
Reporter: Kenneth Knowles
Assignee: Kenneth Knowles
Pointed out by [~michel], we should be packaging unshaded tests to e.g.
{{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035,
BEAM-5116, BEAM-5207.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5907) Dataflow legacy worker test suite runs portability tests
[ https://issues.apache.org/jira/browse/BEAM-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667693#comment-16667693 ] Kenneth Knowles commented on BEAM-5907: --- I'll take a look. > Dataflow legacy worker test suite runs portability tests > > > Key: BEAM-5907 > URL: https://issues.apache.org/jira/browse/BEAM-5907 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Some tests of portability bits failed in a test suite for the legacy worker. > Could be a naming problem or a configuration problem. Notably, they failed > due to changes in unshaded test jars, which no one should be using. > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5907) Dataflow legacy worker test suite runs portability tests
[ https://issues.apache.org/jira/browse/BEAM-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5907: - Assignee: Kenneth Knowles (was: Boyuan Zhang) > Dataflow legacy worker test suite runs portability tests > > > Key: BEAM-5907 > URL: https://issues.apache.org/jira/browse/BEAM-5907 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > > Some tests of portability bits failed in a test suite for the legacy worker. > Could be a naming problem or a configuration problem. Notably, they failed > due to changes in unshaded test jars, which no one should be using. > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (BEAM-5913) Re-enable parallel builds on Jenkins
Kenneth Knowles created BEAM-5913: - Summary: Re-enable parallel builds on Jenkins Key: BEAM-5913 URL: https://issues.apache.org/jira/browse/BEAM-5913 Project: Beam Issue Type: Improvement Components: build-system Reporter: Kenneth Knowles Assignee: Kenneth Knowles Both Java precommit and multiple Java postcommit suites have parallel builds disabled. This has an intensely bad effect on runtimes, to the point where it can even change our workflows. We need to get parallel builds back ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5907) Dataflow worker unit test suite has thread-unsafe use of mockito
[ https://issues.apache.org/jira/browse/BEAM-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5907: -- Priority: Minor (was: Major) > Dataflow worker unit test suite has thread-unsafe use of mockito > > > Key: BEAM-5907 > URL: https://issues.apache.org/jira/browse/BEAM-5907 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Some tests of portability bits failed in a test suite for the legacy worker. > Could be a naming problem or a configuration problem. Notably, they failed > due to changes in unshaded test jars, which no one should be using. > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5907) Dataflow worker unit test suite has thread-unsafe use of mockito
[ https://issues.apache.org/jira/browse/BEAM-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668024#comment-16668024 ] Kenneth Knowles commented on BEAM-5907: --- Downgrading to minor as the basic race condition is solved and our current use doesn't seem to actually break Mockito. The tests are pretty laborious and probably test our mocking as much as the code under test, so still worth a look at some point. > Dataflow worker unit test suite has thread-unsafe use of mockito > > > Key: BEAM-5907 > URL: https://issues.apache.org/jira/browse/BEAM-5907 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Major > Time Spent: 50m > Remaining Estimate: 0h > > Some tests of portability bits failed in a test suite for the legacy worker. > Could be a naming problem or a configuration problem. Notably, they failed > due to changes in unshaded test jars, which no one should be using. > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5907) Dataflow worker unit test suite has thread-unsafe use of mockito
[ https://issues.apache.org/jira/browse/BEAM-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5907: - Assignee: (was: Kenneth Knowles) > Dataflow worker unit test suite has thread-unsafe use of mockito > > > Key: BEAM-5907 > URL: https://issues.apache.org/jira/browse/BEAM-5907 > Project: Beam > Issue Type: Bug > Components: runner-dataflow >Reporter: Kenneth Knowles >Priority: Minor > Time Spent: 50m > Remaining Estimate: 0h > > Some tests of portability bits failed in a test suite for the legacy worker. > Could be a naming problem or a configuration problem. Notably, they failed > due to changes in unshaded test jars, which no one should be using. > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (BEAM-5908) Postcommit failure after re-enabling parallel builds
[ https://issues.apache.org/jira/browse/BEAM-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles updated BEAM-5908: -- Summary: Postcommit failure after re-enabling parallel builds (was: Postcommit failure after fixing unshaded test jar classifier) > Postcommit failure after re-enabling parallel builds > > > Key: BEAM-5908 > URL: https://issues.apache.org/jira/browse/BEAM-5908 > Project: Beam > Issue Type: Bug > Components: test-failures >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Critical > Fix For: Not applicable > > Time Spent: 1h > Remaining Estimate: 0h > > https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/1778/#showFailuresLink -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5673) View.asMap on non-KV PCollection fails at runtime, not construction/submission time
[
https://issues.apache.org/jira/browse/BEAM-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles resolved BEAM-5673.
---
Resolution: Not A Problem
Assignee: Kenneth Knowles
Fix Version/s: Not applicable
> View.asMap on non-KV PCollection fails at runtime, not
> construction/submission time
> ---
>
> Key: BEAM-5673
> URL: https://issues.apache.org/jira/browse/BEAM-5673
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Przemyslaw Pastuszka
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> I'm trying to write a ParDo, which will use both Timer and Side Input, but it
> crashes when I try to run it with {{beam-runners-direct-java}} with
> {{IllegalArgumentException}} on a line
> [https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java#L167],
> because there are actually two inputs to ParDo (main PCollection and side
> input), while only one is expected. It looks like a bug in an implementation.
>
> Here's the code that reproduces the issue:
> {code:java}
> public class TestCrashesForTimerAndSideInput {
> @Rule
> public final transient TestPipeline p = TestPipeline.create();
> private static class DoFnWithTimer extends DoFn,
> String> {
> @TimerId("t")
> private final TimerSpec tSpec =
> TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
> private final PCollectionView> sideInput;
> private DoFnWithTimer(PCollectionView> sideInput)
> {
> this.sideInput = sideInput;
> }
> @ProcessElement
> public void processElement(ProcessContext c, @TimerId("t") Timer t) {
> KV element = c.element();
> c.output(element.getKey() + c.sideInput(sideInput).get(element));
> t.offset(Duration.standardSeconds(1)).setRelative();
> }
> @OnTimer("t")
> public void onTimerFire(OnTimerContext x) {
> x.output("Timer fired");
> }
> }
> @Test
> public void testCrashesForTimerAndSideInput() {
> ImmutableMap sideData = ImmutableMap. String>builder().
> put("x", "X").
> put("y", "Y").
> build();
> PCollectionView> sideInput =
> p.apply(Create.of(sideData)).apply(View.asMap());
> TestStream testStream =
> TestStream.create(StringUtf8Coder.of()).
> addElements("x").
> advanceProcessingTime(Duration.standardSeconds(1)).
> addElements("y").
> advanceProcessingTime(Duration.standardSeconds(1)).
> advanceWatermarkToInfinity();
> PCollection result = p.
> apply(testStream).
> apply(MapElements.into(kvs(strings(), strings())).via(v ->
> KV.of(v, v))).
> apply(ParDo.of(new
> DoFnWithTimer(sideInput)).withSideInputs(sideInput));
> PAssert.that(result).containsInAnyOrder("xX", "yY", "Timer fired");
> p.run();
> }
> }
> {code}
>
> and the error is:
> {code}
> java.lang.IllegalArgumentException: expected one element but was:
> KeyedWorkItem/ParMultiDo(ToKeyedWorkItem).output [PCollection],
> View.AsMap/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization).output
> [PCollection]>
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:322)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
> at
> org.apache.beam.runners.direct.QuiescenceDriver.fireTimers(QuiescenceDriver.java:167)
> at
> org.apache.beam.runners.direct.QuiescenceDriver.drive(QuiescenceDriver.java:110)
> at
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor$2.run(ExecutorServiceParallelExecutor.java:170)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5673) View.asMap on non-KV PCollection fails at runtime, not construction/submission time
[
https://issues.apache.org/jira/browse/BEAM-5673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16671737#comment-16671737
]
Kenneth Knowles commented on BEAM-5673:
---
Great!
> View.asMap on non-KV PCollection fails at runtime, not
> construction/submission time
> ---
>
> Key: BEAM-5673
> URL: https://issues.apache.org/jira/browse/BEAM-5673
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Affects Versions: 2.6.0, 2.7.0
>Reporter: Przemyslaw Pastuszka
>Priority: Major
> Fix For: Not applicable
>
>
> I'm trying to write a ParDo, which will use both Timer and Side Input, but it
> crashes when I try to run it with {{beam-runners-direct-java}} with
> {{IllegalArgumentException}} on a line
> [https://github.com/apache/beam/blob/master/runners/direct-java/src/main/java/org/apache/beam/runners/direct/QuiescenceDriver.java#L167],
> because there are actually two inputs to ParDo (main PCollection and side
> input), while only one is expected. It looks like a bug in an implementation.
>
> Here's the code that reproduces the issue:
> {code:java}
> public class TestCrashesForTimerAndSideInput {
> @Rule
> public final transient TestPipeline p = TestPipeline.create();
> private static class DoFnWithTimer extends DoFn,
> String> {
> @TimerId("t")
> private final TimerSpec tSpec =
> TimerSpecs.timer(TimeDomain.PROCESSING_TIME);
> private final PCollectionView> sideInput;
> private DoFnWithTimer(PCollectionView> sideInput)
> {
> this.sideInput = sideInput;
> }
> @ProcessElement
> public void processElement(ProcessContext c, @TimerId("t") Timer t) {
> KV element = c.element();
> c.output(element.getKey() + c.sideInput(sideInput).get(element));
> t.offset(Duration.standardSeconds(1)).setRelative();
> }
> @OnTimer("t")
> public void onTimerFire(OnTimerContext x) {
> x.output("Timer fired");
> }
> }
> @Test
> public void testCrashesForTimerAndSideInput() {
> ImmutableMap sideData = ImmutableMap. String>builder().
> put("x", "X").
> put("y", "Y").
> build();
> PCollectionView> sideInput =
> p.apply(Create.of(sideData)).apply(View.asMap());
> TestStream testStream =
> TestStream.create(StringUtf8Coder.of()).
> addElements("x").
> advanceProcessingTime(Duration.standardSeconds(1)).
> addElements("y").
> advanceProcessingTime(Duration.standardSeconds(1)).
> advanceWatermarkToInfinity();
> PCollection result = p.
> apply(testStream).
> apply(MapElements.into(kvs(strings(), strings())).via(v ->
> KV.of(v, v))).
> apply(ParDo.of(new
> DoFnWithTimer(sideInput)).withSideInputs(sideInput));
> PAssert.that(result).containsInAnyOrder("xX", "yY", "Timer fired");
> p.run();
> }
> }
> {code}
>
> and the error is:
> {code}
> java.lang.IllegalArgumentException: expected one element but was:
> KeyedWorkItem/ParMultiDo(ToKeyedWorkItem).output [PCollection],
> View.AsMap/View.VoidKeyToMultimapMaterialization/ParDo(VoidKeyToMultimapMaterialization)/ParMultiDo(VoidKeyToMultimapMaterialization).output
> [PCollection]>
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterators.getOnlyElement(Iterators.java:322)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.collect.Iterables.getOnlyElement(Iterables.java:294)
> at
> org.apache.beam.runners.direct.QuiescenceDriver.fireTimers(QuiescenceDriver.java:167)
> at
> org.apache.beam.runners.direct.QuiescenceDriver.drive(QuiescenceDriver.java:110)
> at
> org.apache.beam.runners.direct.ExecutorServiceParallelExecutor$2.run(ExecutorServiceParallelExecutor.java:170)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (BEAM-5928) ConcurrentModificationException from RowCoderGenerator lazy caching
[
https://issues.apache.org/jira/browse/BEAM-5928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5928:
-
Assignee: Reuven Lax (was: Kenneth Knowles)
> ConcurrentModificationException from RowCoderGenerator lazy caching
> ---
>
> Key: BEAM-5928
> URL: https://issues.apache.org/jira/browse/BEAM-5928
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core
>Reporter: Benson Tucker
>Assignee: Reuven Lax
>Priority: Major
>
> h3. Summary:
> RowCoderGenerator caches a delegate Coder once encode or decode is
> exercised, but there's not an API for caching this delegate eagerly.
> h3. Use Case:
> When creating several PCollections to perform distinct reads with the same
> schema, you might create one RowCoder.of(schema) before creating the list of
> PCollections / PCollectionsList. However, once the pipeline begins and rows
> arrive for encoding, these pipelines will simultaneously try to cache a
> delegate coder for the row's schema.
> h3. Workaround:
> You can force the eager caching of the code by exercising encode in the main
> application before creating PCollections using the RowCoder:
> {code:java}
> try {
> myRowCoder.encode(null, null);
> } catch (IOException | NullPointerException e) {
> // do nothing
> }
> {code}
> h3. Context:
> I've only encountered this during development with the direct runner.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5307) Allow injection of state in DoFn.FinishBundle
[
https://issues.apache.org/jira/browse/BEAM-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668683#comment-16668683
]
Kenneth Knowles commented on BEAM-5307:
---
Every element always has a window, so if you aren't assigning windows then they
are all in the global window. So in that case there is no window expiration
since the global window never ends. Using timers is the right idea. You'll want
to set the timer for a finite time in the future, not the end of the window.
StackOverflow is a better place for digging in to this, so I'll answer there.
> Allow injection of state in DoFn.FinishBundle
> -
>
> Key: BEAM-5307
> URL: https://issues.apache.org/jira/browse/BEAM-5307
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Sander Ploegsma
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> Example use case: a stateful {{DoFn}} that requires persisting its state to
> an external database. Instead of writing to the external database for each
> element, it is much more efficient to flush the state every once in a while,
> or when cleaning up.
> Currently, this is not possible because the {{@StateId}} injection is only
> available in processing functions and timers.
> [This|https://stackoverflow.com/questions/51789776/calculating-deltas-in-apache-beam-using-stateful-processing]
> might be a workaround, but I'm not even sure if it works.
> Instead, by allowing the use of {{@StateId}} inside a {{@FinishBundle}}
> function, we can make sure the internal state is persisted in all scenarios:
> {code:java}
> @FinishBundle
> public void flush(FinishBundleContext context, @StateId("myState")
> ValueState state) {
> repository.save(state.read());
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (BEAM-5307) Allow injection of state in DoFn.FinishBundle
[
https://issues.apache.org/jira/browse/BEAM-5307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668683#comment-16668683
]
Kenneth Knowles edited comment on BEAM-5307 at 10/30/18 1:22 PM:
-
Every element always has a window, so if you aren't assigning windows then they
are all in the global window. So in that case there is no window expiration
since the global window never ends. Using timers is the right idea.
StackOverflow is a better place for digging in to this, so I'll answer there.
was (Author: kenn):
Every element always has a window, so if you aren't assigning windows then they
are all in the global window. So in that case there is no window expiration
since the global window never ends. Using timers is the right idea. You'll want
to set the timer for a finite time in the future, not the end of the window.
StackOverflow is a better place for digging in to this, so I'll answer there.
> Allow injection of state in DoFn.FinishBundle
> -
>
> Key: BEAM-5307
> URL: https://issues.apache.org/jira/browse/BEAM-5307
> Project: Beam
> Issue Type: Improvement
> Components: sdk-java-core
>Reporter: Sander Ploegsma
>Assignee: Kenneth Knowles
>Priority: Major
> Fix For: Not applicable
>
>
> Example use case: a stateful {{DoFn}} that requires persisting its state to
> an external database. Instead of writing to the external database for each
> element, it is much more efficient to flush the state every once in a while,
> or when cleaning up.
> Currently, this is not possible because the {{@StateId}} injection is only
> available in processing functions and timers.
> [This|https://stackoverflow.com/questions/51789776/calculating-deltas-in-apache-beam-using-stateful-processing]
> might be a workaround, but I'm not even sure if it works.
> Instead, by allowing the use of {{@StateId}} inside a {{@FinishBundle}}
> function, we can make sure the internal state is persisted in all scenarios:
> {code:java}
> @FinishBundle
> public void flush(FinishBundleContext context, @StateId("myState")
> ValueState state) {
> repository.save(state.read());
> }
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
Kenneth Knowles created BEAM-5925:
-
Summary: Test flake in ElasticsearchIOTest.testWriteFullAddressing
Key: BEAM-5925
URL: https://issues.apache.org/jira/browse/BEAM-5925
Project: Beam
Issue Type: Bug
Components: io-java-elasticsearch
Reporter: Kenneth Knowles
Assignee: Etienne Chauchot
https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
https://scans.gradle.com/s/j42mwdsn5svcs
{code}
org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
listener timeout after waiting for [3] ms
{code}
Log looks like this:
{code}
[2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
[testWriteFullAddressing]: before test
[2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
[ElasticsearchIOTest#testWriteFullAddressing]: setting up test
[2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
[node_s0] adding template [random_index_template] for index patterns [*]
[2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
[ElasticsearchIOTest#testWriteFullAddressing]: all set up test
[2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[galilei] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[curie] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[darwin] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[copernicus] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[faraday] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[bohr] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[pasteur] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[einstein] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[maxwell] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService] [node_s0]
[newton] creating index, cause [auto(bulk api)], templates
[random_index_template], shards [6]/[0], mappings []
[2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0] Cluster
health status changed from [YELLOW] to [GREEN] (reason: [shards started
[[galilei][4], [galilei][5]] ...]).
[2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
[2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
[2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
[2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
[2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
[2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
[2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
[2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
[2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[darwin/g_sIfS5aQwi6BAXw_--vgw] create_mapping [TYPE_1]
[2018-10-31T04:06:39,201][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
[curie/PDuZqTZQROytGLowXGMxhA] create_mapping [TYPE_0]
[2018-10-31T04:06:40,030][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
[ElasticsearchIOTest#testWriteFullAddressing]: cleaning up after test
[2018-10-31T04:06:40,185][INFO ][o.e.c.m.MetaDataDeleteIndexService] [node_s0]
[bohr/9YTwB1yvTYKf9YjYCmHjwg] deleting index
[2018-10-31T04:06:40,185][INFO
[jira] [Assigned] (BEAM-5918) Add Cast transform for Rows
[ https://issues.apache.org/jira/browse/BEAM-5918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-5918: - Assignee: Gleb Kanterov (was: Kenneth Knowles) > Add Cast transform for Rows > --- > > Key: BEAM-5918 > URL: https://issues.apache.org/jira/browse/BEAM-5918 > Project: Beam > Issue Type: Improvement > Components: sdk-java-core >Reporter: Gleb Kanterov >Assignee: Gleb Kanterov >Priority: Major > Time Spent: 3h 10m > Remaining Estimate: 0h > > There is a need for a generic transform that given two Row schemas will > convert rows between them. There must be a possibility to opt-out from > certain kind of conversions, for instance, converting ints to shorts can > cause overflow. Another example, a schema could have a nullable field, but > never have NULL value in practice, because it was filtered out. > What is needed: > - widening values (e.g., int -> long) > - narrowwing (e.g., int -> short) > - runtime check for overflow while narrowing > - ignoring nullability (nullable=true -> nullable=false) > - weakening nullability (nullable=false -> nullable=true) > - projection (Schema(a: Int32, b: Int32) -> Schema(a: Int32)) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5925:
-
Assignee: Etienne Chauchot (was: Kenneth Knowles)
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [darwin/g_sIfS5aQwi6BAXw_--vgw] create_mapping [TYPE_1]
> [2018-10-31T04:06:39,201][INFO
[jira] [Commented] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669345#comment-16669345
]
Kenneth Knowles commented on BEAM-5925:
---
To be clear, this is v6: {{:beam-sdks-java-io-elasticsearch-tests-6:test}}
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [darwin/g_sIfS5aQwi6BAXw_--vgw]
[jira] [Assigned] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kenneth Knowles reassigned BEAM-5925:
-
Assignee: Kenneth Knowles (was: Etienne Chauchot)
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Kenneth Knowles
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [darwin/g_sIfS5aQwi6BAXw_--vgw] create_mapping [TYPE_1]
> [2018-10-31T04:06:39,201][INFO
[jira] [Commented] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669346#comment-16669346
]
Kenneth Knowles commented on BEAM-5925:
---
I'm going to take a minute to see if I can just increase the timeout.
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [faraday/vIMYG8vpTQKqNkyajcFOxw] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,576][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [copernicus/NzCZssInSiOdZKTmLCoXRw] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,890][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [darwin/g_sIfS5aQwi6BAXw_--vgw]
[jira] [Assigned] (BEAM-3573) Test jars should export only tests, and only be exported for select modules
[ https://issues.apache.org/jira/browse/BEAM-3573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles reassigned BEAM-3573: - Assignee: (was: Kenneth Knowles) > Test jars should export only tests, and only be exported for select modules > --- > > Key: BEAM-3573 > URL: https://issues.apache.org/jira/browse/BEAM-3573 > Project: Beam > Issue Type: Bug > Components: sdk-java-core >Reporter: Kenneth Knowles >Priority: Major > Time Spent: 1h 20m > Remaining Estimate: 0h > > Today, we have test-jars that are used as libraries for testing. That is not > what "test jar" means, and dependency management actually does not work > correctly for this. It is OK to depend on a test jar in order to run the > tests therein, and not really OK to depend on one for another reason. > This ticket is a bucket ticket for fixes to this situation. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (BEAM-5913) Re-enable parallel builds on Jenkins
[ https://issues.apache.org/jira/browse/BEAM-5913?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kenneth Knowles resolved BEAM-5913. --- Resolution: Fixed Fix Version/s: Not applicable > Re-enable parallel builds on Jenkins > > > Key: BEAM-5913 > URL: https://issues.apache.org/jira/browse/BEAM-5913 > Project: Beam > Issue Type: Improvement > Components: build-system >Reporter: Kenneth Knowles >Assignee: Kenneth Knowles >Priority: Critical > Fix For: Not applicable > > Time Spent: 3.5h > Remaining Estimate: 0h > > Both Java precommit and multiple Java postcommit suites have parallel builds > disabled. This has an intensely bad effect on runtimes, to the point where it > can even change our workflows. We need to get parallel builds back ASAP. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (BEAM-5887) packageTests and shadowTestJar write the same file
[
https://issues.apache.org/jira/browse/BEAM-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669341#comment-16669341
]
Kenneth Knowles commented on BEAM-5887:
---
Put this on 2.9.0 because it is possible we published a test jar that is
broken. But users shouldn't depend on them anyhow, with the possible exception
of third-party runners scraping for ValidatesRunner tests.
> packageTests and shadowTestJar write the same file
> --
>
> Key: BEAM-5887
> URL: https://issues.apache.org/jira/browse/BEAM-5887
> Project: Beam
> Issue Type: Bug
> Components: build-system
>Reporter: Kenneth Knowles
>Assignee: Michael Luckey
>Priority: Major
> Fix For: 2.9.0
>
> Time Spent: 4.5h
> Remaining Estimate: 0h
>
> Pointed out by [~michel], we should be packaging unshaded tests to e.g.
> {{tests-unshaded}} classifier. This is likely the root cause of BEAM-5035,
> BEAM-5116, BEAM-5207.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (BEAM-5925) Test flake in ElasticsearchIOTest.testWriteFullAddressing
[
https://issues.apache.org/jira/browse/BEAM-5925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669429#comment-16669429
]
Kenneth Knowles commented on BEAM-5925:
---
Looked into it, and I couldn't quite figure out what is going on, but I don't
think adding a timeout (which would be to the ES RetryConfiguration I think) is
the issue. Do {{:beam-sdks-java-io-elasticsearch-tests-2:test}},
{{:beam-sdks-java-io-elasticsearch-tests-5:test}}, and
{{:beam-sdks-java-io-elasticsearch-tests-6:test}} have some accidental conflict
in their clusters or teardown? I see they use different indices...
> Test flake in ElasticsearchIOTest.testWriteFullAddressing
> -
>
> Key: BEAM-5925
> URL: https://issues.apache.org/jira/browse/BEAM-5925
> Project: Beam
> Issue Type: Bug
> Components: io-java-elasticsearch
>Reporter: Kenneth Knowles
>Assignee: Etienne Chauchot
>Priority: Critical
>
> https://builds.apache.org/view/A-D/view/Beam/job/beam_PostCommit_Java_GradleBuild/1789/
> https://scans.gradle.com/s/j42mwdsn5svcs
> {code}
> org.apache.beam.sdk.Pipeline$PipelineExecutionException: java.io.IOException:
> listener timeout after waiting for [3] ms
> {code}
> Log looks like this:
> {code}
> [2018-10-31T04:06:07,571][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [testWriteFullAddressing]: before test
> [2018-10-31T04:06:07,572][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: setting up test
> [2018-10-31T04:06:07,589][INFO ][o.e.c.m.MetaDataIndexTemplateService]
> [node_s0] adding template [random_index_template] for index patterns [*]
> [2018-10-31T04:06:07,645][INFO ][o.a.b.s.i.e.ElasticsearchIOTest]
> [ElasticsearchIOTest#testWriteFullAddressing]: all set up test
> [2018-10-31T04:06:10,536][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [galilei] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:33,963][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [curie] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,034][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [darwin] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,050][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [copernicus] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,075][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [faraday] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,095][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [bohr] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,113][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [pasteur] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,142][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [einstein] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,205][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [maxwell] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:34,226][INFO ][o.e.c.m.MetaDataCreateIndexService]
> [node_s0] [newton] creating index, cause [auto(bulk api)], templates
> [random_index_template], shards [6]/[0], mappings []
> [2018-10-31T04:06:36,914][INFO ][o.e.c.r.a.AllocationService] [node_s0]
> Cluster health status changed from [YELLOW] to [GREEN] (reason: [shards
> started [[galilei][4], [galilei][5]] ...]).
> [2018-10-31T04:06:36,970][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [galilei/Vn1b8XXVSAmrTb5BVe2IJQ] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,137][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [newton/bjnImLt_QguBGEFH9lBJ6Q] create_mapping [TYPE_-1]
> [2018-10-31T04:06:37,385][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [maxwell/-RZ32NbRRZWaGaVfaptFIA] create_mapping [TYPE_0]
> [2018-10-31T04:06:37,636][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [einstein/2lgF5Vj6Ti2KTS-pYSzv3Q] create_mapping [TYPE_1]
> [2018-10-31T04:06:37,806][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [pasteur/832OwzleRSOHsWx85vOH-w] create_mapping [TYPE_0]
> [2018-10-31T04:06:38,103][INFO ][o.e.c.m.MetaDataMappingService] [node_s0]
> [bohr/9YTwB1yvTYKf9YjYCmHjwg] create_mapping [TYPE_1]
> [2018-10-31T04:06:38,229][INFO
[jira] [Commented] (BEAM-5061) Invisible parameter type exception in JDK 10
[
https://issues.apache.org/jira/browse/BEAM-5061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16669552#comment-16669552
]
Kenneth Knowles commented on BEAM-5061:
---
My (shallow) understanding is that a bytebuddy upgrade is needed, and it has
some new conventions to support new and tighter aspects of Java's access model.
> Invisible parameter type exception in JDK 10
>
>
> Key: BEAM-5061
> URL: https://issues.apache.org/jira/browse/BEAM-5061
> Project: Beam
> Issue Type: Sub-task
> Components: sdk-java-core
>Affects Versions: 2.5.0
>Reporter: Mike Pedersen
>Priority: Major
>
> When using JDK 10, using a ParDo after a CoGroupByKey seems to create the
> following exception when executed on local runner:
> {noformat}
> Exception in thread "main"
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.util.concurrent.UncheckedExecutionException:
> java.lang.IllegalStateException: Invisible parameter type of Main$1 arg0 for
> public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2214)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.get(LocalCache.java:4053)
> at
> org.apache.beam.repackaged.beam_runners_direct_java.com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4057)
> ...
> Caused by: java.lang.IllegalStateException: Invisible parameter type of
> Main$1 arg0 for public Main$1$DoFnInvoker(Main$1)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.InstrumentedType$Default.validated(InstrumentedType.java:925)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.MethodRegistry$Default.prepare(MethodRegistry.java:465)
> at
> org.apache.beam.repackaged.beam_sdks_java_core.net.bytebuddy.dynamic.scaffold.subclass.SubclassDynamicTypeBuilder.make(SubclassDynamicTypeBuilder.java:170)
> ...
> {noformat}
> This error disappears completely when using JDK 8. Here is a minimal example
> to reproduce it:
> {code:java}
> import org.apache.beam.sdk.Pipeline;
> import org.apache.beam.sdk.options.PipelineOptions;
> import org.apache.beam.sdk.options.PipelineOptionsFactory;
> import org.apache.beam.sdk.transforms.Create;
> import org.apache.beam.sdk.transforms.DoFn;
> import org.apache.beam.sdk.transforms.ParDo;
> import org.apache.beam.sdk.transforms.join.CoGbkResult;
> import org.apache.beam.sdk.transforms.join.CoGroupByKey;
> import org.apache.beam.sdk.transforms.join.KeyedPCollectionTuple;
> import org.apache.beam.sdk.values.KV;
> import org.apache.beam.sdk.values.PCollection;
> import org.apache.beam.sdk.values.TupleTag;
> import java.util.Arrays;
> import java.util.List;
> public class Main {
> public static void main(String[] args) {
> PipelineOptions options = PipelineOptionsFactory.create();
> Pipeline p = Pipeline.create(options);
> final TupleTag emailsTag = new TupleTag<>();
> final TupleTag phonesTag = new TupleTag<>();
> final List> emailsList =
> Arrays.asList(
> KV.of("amy", "a...@example.com"),
> KV.of("carl", "c...@example.com"),
> KV.of("julia", "ju...@example.com"),
> KV.of("carl", "c...@email.com"));
> final List> phonesList =
> Arrays.asList(
> KV.of("amy", "111-222-"),
> KV.of("james", "222-333-"),
> KV.of("amy", "333-444-"),
> KV.of("carl", "444-555-"));
> PCollection> emails = p.apply("CreateEmails",
> Create.of(emailsList));
> PCollection> phones = p.apply("CreatePhones",
> Create.of(phonesList));
> PCollection> results =
> KeyedPCollectionTuple.of(emailsTag, emails)
> .and(phonesTag, phones)
> .apply(CoGroupByKey.create());
> PCollection contactLines =
> results.apply(
> ParDo.of(
> new DoFn, String>() {
> @ProcessElement
> public void processElement(ProcessContext
> c) {
> KV e =
> c.element();
> String name = e.getKey();
> Iterable emailsIter =
> e.getValue().getAll(emailsTag);
> Iterable phonesIter =
> e.getValue().getAll(phonesTag);
> String formattedResult = "";
>
