[
https://issues.apache.org/jira/browse/FLINK-16313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17052340#comment-17052340
]
Robert Metzger commented on FLINK-16313:
----------------------------------------
More information:
>From the coredump and gdb, I get the following:
{code}
root@ed674fdc4d9b:/home/test/flink/flink-libraries/flink-state-processing-api/target#
gdb $JAVA_HOME/bin/java core.11410
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
<http://www.gnu.org/software/gdb/bugs/>.
Find the GDB manual and other documentation resources online at:
<http://www.gnu.org/software/gdb/documentation/>.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from /usr/lib/jvm/java-8-openjdk-amd64/bin/java...(no debugging
symbols found)...done.
[New LWP 11468]
[New LWP 11470]
[New LWP 11475]
[New LWP 11476]
[New LWP 11481]
[New LWP 11480]
[New LWP 11486]
[New LWP 11469]
[New LWP 11471]
[New LWP 11472]
[New LWP 11473]
[New LWP 11474]
[New LWP 11478]
[New LWP 11477]
[New LWP 11479]
[New LWP 11482]
[New LWP 11463]
[New LWP 11823]
[New LWP 11460]
[New LWP 11465]
[New LWP 11421]
[New LWP 11483]
[New LWP 11492]
[New LWP 11975]
[New LWP 11821]
[New LWP 11426]
[New LWP 11416]
[New LWP 11417]
[New LWP 11466]
[New LWP 11484]
[New LWP 11418]
[New LWP 11414]
[New LWP 11433]
[New LWP 11424]
[New LWP 11412]
[New LWP 11415]
[New LWP 11462]
[New LWP 11411]
[New LWP 11430]
[New LWP 11429]
[New LWP 11422]
[New LWP 11461]
[New LWP 11978]
[New LWP 11977]
[New LWP 11427]
[New LWP 11488]
[New LWP 11434]
[New LWP 11419]
[New LWP 11428]
[New LWP 11431]
[New LWP 11425]
[New LWP 12072]
[New LWP 11464]
[New LWP 11485]
[New LWP 12136]
[New LWP 11410]
[New LWP 11423]
[New LWP 11420]
[New LWP 11432]
[New LWP 11467]
[New LWP 11413]
warning: Could not load shared library symbols for
/tmp/junit1547658367137473582/junit5791642854447555054/rocksdb-lib-b2b47d85aecef16c175b116af00b6d57/librocksdbjni-linux64.so.
Do you need "set solib-search-path" or "set sysroot"?
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
Core was generated by `/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m
-Xmx2048m -Dmvn.forkNum'.
Program terminated with signal SIGABRT, Aborted.
#0 0x00007f69aaec6428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
54 ../sysdeps/unix/sysv/linux/raise.c: No such file or directory.
[Current thread is 1 (Thread 0x7f68ade2c700 (LWP 11468))]
Installing openjdk unwinder
Traceback (most recent call last):
File
"/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py",
line 52, in <module>
class Types(object):
File
"/usr/share/gdb/auto-load/usr/lib/jvm/java-8-openjdk-amd64/jre/lib/amd64/server/libjvm.so-gdb.py",
line 66, in Types
nmethodp_t = gdb.lookup_type('nmethod').pointer()
gdb.error: No type named nmethod.
(gdb) where
#0 0x00007f69aaec6428 in __GI_raise (sig=sig@entry=6) at
../sysdeps/unix/sysv/linux/raise.c:54
#1 0x00007f69aaec802a in __GI_abort () at abort.c:89
#2 0x00007f69a960e84d in __gnu_cxx::__verbose_terminate_handler() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#3 0x00007f69a960c6b6 in ?? () from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#4 0x00007f69a960c701 in std::terminate() () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#5 0x00007f69a960d23f in __cxa_pure_virtual () from
/usr/lib/x86_64-linux-gnu/libstdc++.so.6
#6 0x00007f678e9110d5 in ?? ()
#7 0x00007f676c055b08 in ?? ()
#8 0x00007f676c053660 in ?? ()
#9 0x00007f68ade2b4af in ?? ()
#10 0x00007f68ade2b4b0 in ?? ()
#11 0x00007f68ade2b4c0 in ?? ()
#12 0x00007f68ade2b730 in ?? ()
#13 0x00007f676c055d38 in ?? ()
#14 0x00007f676c055b58 in ?? ()
#15 0x00007f69aaa6e270 in stack_used () from
/lib/x86_64-linux-gnu/libpthread.so.0
#16 0x0000000000000001 in ?? ()
#17 0x00007f68ade2c700 in ?? ()
#18 0x00007f69aa85d5c9 in __free_stacks (limit=41943040) at allocatestack.c:288
#19 queue_stack (stack=<optimized out>) at allocatestack.c:312
#20 __deallocate_stack (pd=<optimized out>) at allocatestack.c:774
#21 __free_tcb (pd=<optimized out>) at pthread_create.c:243
#22 0x0000000000000000 in ?? ()
{code}
Previous runs have also resulted in such files:
{code}
java.io.IOException: Stream closed
at
java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:283)
at java.io.BufferedInputStream.read(BufferedInputStream.java:345)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:284)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:326)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:178)
at java.io.InputStreamReader.read(InputStreamReader.java:184)
at java.io.Reader.read(Reader.java:100)
at java.util.Scanner.readInput(Scanner.java:804)
at java.util.Scanner.findWithinHorizon(Scanner.java:1685)
at java.util.Scanner.hasNextLine(Scanner.java:1500)
at
org.apache.maven.surefire.booter.PpidChecker$ProcessInfoConsumer.execute(PpidChecker.java:354)
at
org.apache.maven.surefire.booter.PpidChecker.unix(PpidChecker.java:190)
at
org.apache.maven.surefire.booter.PpidChecker.isProcessAlive(PpidChecker.java:123)
at
org.apache.maven.surefire.booter.ForkedBooter$2.run(ForkedBooter.java:214)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
# Created at 2020-03-05T15:49:56.955
System.exit() or native command error interrupted process checker.
java.lang.IllegalStateException: error [STOPPED] to read process 15274
at
org.apache.maven.surefire.booter.PpidChecker.checkProcessInfo(PpidChecker.java:145)
at
org.apache.maven.surefire.booter.PpidChecker.isProcessAlive(PpidChecker.java:124)
at
org.apache.maven.surefire.booter.ForkedBooter$2.run(ForkedBooter.java:214)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
I believe I am only able to reproduce this issue within this docker container
{{rmetzger/flink-ci:ubuntu-amd64-3528acd}}, and NOT on the azure hosted ubuntu
16.04 machines.
But on a {{CentOS Linux release 7.6.1810}} host machine.
> flink-state-processor-api: surefire execution unstable on Azure
> ---------------------------------------------------------------
>
> Key: FLINK-16313
> URL: https://issues.apache.org/jira/browse/FLINK-16313
> Project: Flink
> Issue Type: Bug
> Components: API / State Processor, Tests
> Reporter: Robert Metzger
> Assignee: Robert Metzger
> Priority: Critical
> Labels: test-stability
>
> Log file:
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=5686&view=logs&j=41cba0bb-1271-5adb-01cc-4768f26a8311&t=44574c85-1cd0-5978-cccf-f0cf7e87a36a
> {code}
> 2020-02-27T12:36:35.2860111Z [INFO] flink-table-planner
> ................................ SUCCESS [01:47 min]
> 2020-02-27T12:36:35.2860966Z [INFO] flink-cep-scala
> .................................... SUCCESS [ 5.041 s]
> 2020-02-27T12:36:35.2861740Z [INFO] flink-sql-client
> ................................... SUCCESS [03:00 min]
> 2020-02-27T12:36:35.2862503Z [INFO] flink-state-processor-api
> .......................... FAILURE [ 15.394 s]
> 2020-02-27T12:36:35.2863237Z [INFO]
> ------------------------------------------------------------------------
> 2020-02-27T12:36:35.2863587Z [INFO] BUILD FAILURE
> 2020-02-27T12:36:35.2864071Z [INFO]
> ------------------------------------------------------------------------
> 2020-02-27T12:36:35.2864428Z [INFO] Total time: 05:38 min
> 2020-02-27T12:36:35.2866349Z [INFO] Finished at: 2020-02-27T12:36:35+00:00
> 2020-02-27T12:36:35.9345815Z [INFO] Final Memory: 147M/2914M
> 2020-02-27T12:36:35.9347238Z [INFO]
> ------------------------------------------------------------------------
> 2020-02-27T12:36:35.9355362Z [WARNING] The requested profile
> "skip-webui-build" could not be activated because it does not exist.
> 2020-02-27T12:36:35.9367919Z [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test
> (integration-tests) on project flink-state-processor-api_2.11: There are test
> failures.
> 2020-02-27T12:36:35.9368804Z [ERROR]
> 2020-02-27T12:36:35.9369489Z [ERROR] Please refer to
> /__w/2/s/flink-libraries/flink-state-processing-api/target/surefire-reports
> for the individual test results.
> 2020-02-27T12:36:35.9370249Z [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> 2020-02-27T12:36:35.9370713Z [ERROR] ExecutionException Error occurred in
> starting fork, check output in log
> 2020-02-27T12:36:35.9371279Z [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException Error occurred in starting fork, check output in log
> 2020-02-27T12:36:35.9372275Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:510)
> 2020-02-27T12:36:35.9372917Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:457)
> 2020-02-27T12:36:35.9373498Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:298)
> 2020-02-27T12:36:35.9374064Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:246)
> 2020-02-27T12:36:35.9374636Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1183)
> 2020-02-27T12:36:35.9375344Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1011)
> 2020-02-27T12:36:35.9376194Z [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:857)
> 2020-02-27T12:36:35.9376791Z [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> 2020-02-27T12:36:35.9377375Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> 2020-02-27T12:36:35.9377898Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> 2020-02-27T12:36:35.9378435Z [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> 2020-02-27T12:36:35.9379063Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> 2020-02-27T12:36:35.9379709Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> 2020-02-27T12:36:35.9380367Z [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> 2020-02-27T12:36:35.9381007Z [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
> 2020-02-27T12:36:35.9381510Z [ERROR] at
> org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
> 2020-02-27T12:36:35.9381973Z [ERROR] at
> org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
> 2020-02-27T12:36:35.9382404Z [ERROR] at
> org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
> 2020-02-27T12:36:35.9382839Z [ERROR] at
> org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
> 2020-02-27T12:36:35.9383248Z [ERROR] at
> org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
> 2020-02-27T12:36:35.9383661Z [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-02-27T12:36:35.9384126Z [ERROR] at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-02-27T12:36:35.9384659Z [ERROR] at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-02-27T12:36:35.9385145Z [ERROR] at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-02-27T12:36:35.9385606Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
> 2020-02-27T12:36:35.9386293Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
> 2020-02-27T12:36:35.9386930Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
> 2020-02-27T12:36:35.9387471Z [ERROR] at
> org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
> 2020-02-27T12:36:35.9388056Z [ERROR] Caused by:
> org.apache.maven.surefire.booter.SurefireBooterForkException: Error occurred
> in starting fork, check output in log
> 2020-02-27T12:36:35.9388731Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.fork(ForkStarter.java:622)
> 2020-02-27T12:36:35.9389289Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.access$600(ForkStarter.java:115)
> 2020-02-27T12:36:35.9389864Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:444)
> 2020-02-27T12:36:35.9390411Z [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter$2.call(ForkStarter.java:420)
> 2020-02-27T12:36:35.9390986Z [ERROR] at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-02-27T12:36:35.9391458Z [ERROR] at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> 2020-02-27T12:36:35.9391991Z [ERROR] at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> 2020-02-27T12:36:35.9392419Z [ERROR] at java.lang.Thread.run(Thread.java:748)
> 2020-02-27T12:36:35.9392894Z [ERROR] -> [Help 1]
> 2020-02-27T12:36:35.9393077Z [ERROR]
> 2020-02-27T12:36:35.9393553Z [ERROR] To see the full stack trace of the
> errors, re-run Maven with the -e switch.
> 2020-02-27T12:36:35.9394108Z [ERROR] Re-run Maven using the -X switch to
> enable full debug logging.
> 2020-02-27T12:36:35.9394392Z [ERROR]
> 2020-02-27T12:36:35.9394713Z [ERROR] For more information about the errors
> and possible solutions, please read the following articles:
> 2020-02-27T12:36:35.9395211Z [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> 2020-02-27T12:36:35.9395525Z [ERROR]
> 2020-02-27T12:36:35.9395889Z [ERROR] After correcting the problems, you can
> resume the build with the command
> 2020-02-27T12:36:35.9396511Z [ERROR] mvn <goals> -rf
> :flink-state-processor-api_2.11
> 2020-02-27T12:36:36.2427441Z MVN exited with EXIT CODE: 1.
> 2020-02-27T12:36:36.2427867Z Trying to KILL watchdog (1633).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
