suztomo edited a comment on pull request #14474: URL: https://github.com/apache/beam/pull/14474#issuecomment-816372956
https://ci-beam.apache.org/job/beam_PreCommit_Java_Phrase/3389/ got stuck at 22:53:00. ``` suztomo@apache-ci-beam-jenkins-11:~$ ps -ef |grep 'Gradle Test Executor' jenkins 4993 11152 3 02:41 ? 00:00:48 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.Bootstra pSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.8.6.jar_a26f6511a7813217be4cd6439d66563b/jacocoagent.jar=destfile=build /jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false -Xmx2g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.va riant -ea -cp /home/jenkins/.gradle/caches/6.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 320' jenkins 5751 5472 0 00:33 ? 00:01:05 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -DbeamTestPipelineOptions=["--project=apache-beam-testing","--tempLocation=gs:// temp-storage-for-perf-tests/loadtests","--tempRoot=gs://temp-storage-for-perf-tests/loadtests","--writeMethod=STREAMING_INSERTS","--writeFormat=JSON","--testBigQueryDatase t=beam_performance","--testBigQueryTable=bqio_write_10GB_java_stream_0409002212","--metricsBigQueryDataset=beam_performance","--metricsBigQueryTable=bqio_10GB_results_java _stream","--influxMeasurement=bqio_10GB_results_java_stream","--sourceOptions={\"numRecords\":\"10485760\",\"keySizeBytes\":\"1\",\"valueSizeBytes\":\"1024\"}","--runner=D ataflowRunner","--maxNumWorkers=5","--numWorkers=5","--autoscalingAlgorithm=NONE","--influxDatabase=beam_test_metrics","--influxHost=http://10.128.0.96:8086","--workerHarn essContainerImage=","--dataflowWorkerJar=/home/jenkins/jenkins-slave/workspace/beam_BiqQueryIO_Streaming_Performance_Test_Java/src/runners/google-cloud-dataflow-java/worke r/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.30.0-SNAPSHOT.jar","--region=us-central1"] -Djava.security.manager=worker.org.gradle.pro cess.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -Xmx2g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /hom e/jenkins/.gradle/caches/6.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 1' suztomo 20574 20550 0 03:07 pts/0 00:00:00 grep --color=auto Gradle Test Executor jenkins 21632 11377 3 00:03 ? 00:07:03 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam- testing","--region=us-central1","--tempRoot=gs://temp-storage-for-validates-runner-tests","--dataflowWorkerJar=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_V alidatesRunner_Dataflow/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.30.0-SNAPSHOT.jar"," --workerHarnessContainerImage="] -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -Xmx2g -Dfile.e ncoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/jenkins/.gradle/caches/6.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal. worker.GradleWorkerMain 'Gradle Test Executor 6' jenkins 21652 11377 3 00:03 ? 00:07:09 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -DbeamTestPipelineOptions=["--runner=TestDataflowRunner","--project=apache-beam- testing","--region=us-central1","--tempRoot=gs://temp-storage-for-validates-runner-tests","--dataflowWorkerJar=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Java_V alidatesRunner_Dataflow/src/runners/google-cloud-dataflow-java/worker/legacy-worker/build/libs/beam-runners-google-cloud-dataflow-java-legacy-worker-2.30.0-SNAPSHOT.jar"," --workerHarnessContainerImage="] -Djava.security.manager=worker.org.gradle.process.internal.worker.child.BootstrapSecurityManager -Dorg.gradle.native=false -Xmx2g -Dfile.e ncoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.variant -ea -cp /home/jenkins/.gradle/caches/6.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal. worker.GradleWorkerMain 'Gradle Test Executor 2' jenkins 25731 11152 3 02:49 ? 00:00:42 /usr/lib/jvm/java-8-openjdk-amd64/bin/java -Djava.security.manager=worker.org.gradle.process.internal.worker.child.Bootstra pSecurityManager -Dorg.gradle.native=false -javaagent:build/tmp/expandedArchives/org.jacoco.agent-0.8.6.jar_a26f6511a7813217be4cd6439d66563b/jacocoagent.jar=destfile=build /jacoco/test.exec,append=true,inclnolocationclasses=false,dumponexit=true,output=file,jmx=false -Xmx2g -Dfile.encoding=UTF-8 -Duser.country=US -Duser.language=en -Duser.va riant -ea -cp /home/jenkins/.gradle/caches/6.8/workerMain/gradle-worker.jar worker.org.gradle.process.internal.worker.GradleWorkerMain 'Gradle Test Executor 361' ``` With `sudo jstack 4993` (for "Gradle Test Executor 320"), I got https://gist.github.com/suztomo/6b7678f1c36cc67bf4e84df55c4a9611 in Jenkins's console. ### The test that got stuck This time, it was `BeamFnLoggingServiceTest`. ``` ... 23:10:59 at java.util.concurrent.AbstractExecutorService.invokeAll(AbstractExecutorService.java:244) 23:10:59 at org.apache.beam.runners.dataflow.worker.fn.logging.BeamFnLoggingServiceTest.testMultipleClientsFailingIsHandledGracefullyByServer(BeamFnLoggingServiceTest.java:150) ... ``` ### gRPC classes in the stack traces Interestingly the output includes gRPC-related classes: ``` 23:10:59 "grpc-nio-worker-ELG-3-8" #106 daemon prio=5 os_prio=0 tid=0x00007f7bc403d800 nid=0x2028 runnable [0x00007f7c94cc5000] 23:10:59 java.lang.Thread.State: RUNNABLE 23:10:59 at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) 23:10:59 at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) 23:10:59 at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93) 23:10:59 at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86) 23:10:59 - locked <0x00000000f967b450> (a org.apache.beam.vendor.grpc.v1p36p0.io.netty.channel.nio.SelectedSelectionKeySet) 23:10:59 - locked <0x00000000f967c508> (a java.util.Collections$UnmodifiableSet) 23:10:59 - locked <0x00000000f967b358> (a sun.nio.ch.EPollSelectorImpl) 23:10:59 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97) 23:10:59 at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:101) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.channel.nio.SelectedSelectionKeySetSelector.select(SelectedSelectionKeySetSelector.java:68) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.channel.nio.NioEventLoop.select(NioEventLoop.java:805) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:457) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) 23:10:59 at org.apache.beam.vendor.grpc.v1p36p0.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ``` This seems more about Netty used by gRPC, rather than gRPC itself. Trying to get another instance of timeout. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
