[
https://issues.apache.org/jira/browse/HDDS-2308?focusedWorklogId=339799&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-339799
]
ASF GitHub Bot logged work on HDDS-2308:
----------------------------------------
Author: ASF GitHub Bot
Created on: 07/Nov/19 08:19
Start Date: 07/Nov/19 08:19
Worklog Time Spent: 10m
Work Description: elek commented on pull request #9: HDDS-2308. Switch to
centos with the apache/ozone-build docker image
URL: https://github.com/apache/hadoop-docker-ozone/pull/9
## What changes were proposed in this pull request?
I realized multiple JVM crashes in the daily builds:
```
ERROR] ExecutionException The forked VM terminated without properly saying
goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7569723928289175829tmp surefire_947955725320624341206tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
[ERROR] ExecutionException The forked VM terminated without properly
saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7227403571189445391tmp surefire_1011197392458143645283tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] Crashed tests:
[ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
[ERROR]
org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire3938612864214747736tmp surefire_933162535733309260236tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
[ERROR] ExecutionException The forked VM terminated without properly
saying goodbye. VM crash or System.exit called?
[ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
/usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
-XX:+HeapDumpOnOutOfMemoryError -jar
/workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
/workdir/hadoop-ozone/ozonefs/target/surefire 2019-10-06T14-52-40_697-jvmRun1
surefire7569723928289175829tmp surefire_947955725320624341206tmp
[ERROR] Error occurred in starting fork, check output in log
[ERROR] Process Exit Code: 139
```
Based on the crash log (uploaded) it's related to the rocksdb JNI interface.
In the current ozone-build docker image (which provides the environment for
build) we use alpine where musl libc is used instead of the main glibc. I think
it would be more safe to use the same glibc what is used in production.
I tested with centos based docker image and it seems to be more stable.
Didn't see any more JVM crashes.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-2308
## How this patch can be tested?
Built and pushed to my own dockerhub account:
```
docker build -t elek/ozone-build:20191106-1 .
docker puseh elek/ozone-build:20191106-1
```
And used in the PR builds.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 339799)
Remaining Estimate: 0h
Time Spent: 10m
> Switch to centos with the apache/ozone-build docker image
> ---------------------------------------------------------
>
> Key: HDDS-2308
> URL: https://issues.apache.org/jira/browse/HDDS-2308
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Reporter: Marton Elek
> Assignee: Marton Elek
> Priority: Major
> Labels: pull-request-available
> Attachments: hs_err_pid16346.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> I realized multiple JVM crashes in the daily builds:
>
> {code:java}
> ERROR] ExecutionException The forked VM terminated without properly saying
> goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp
> surefire_947955725320624341206tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] Crashed tests:
>
>
> [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractRename
>
>
> [ERROR] ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter5429192218879128313.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7227403571189445391tmp
> surefire_1011197392458143645283tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] Crashed tests:
>
>
> [ERROR] org.apache.hadoop.fs.ozone.contract.ITestOzoneContractDistCp
>
>
> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter1355604543311368443.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire3938612864214747736tmp
> surefire_933162535733309260236tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139
>
>
> [ERROR] ExecutionException The forked VM terminated without properly
> saying goodbye. VM crash or System.exit called?
>
>
> [ERROR] Command was /bin/sh -c cd /workdir/hadoop-ozone/ozonefs &&
> /usr/lib/jvm/java-1.8-openjdk/jre/bin/java -Xmx2048m
> -XX:+HeapDumpOnOutOfMemoryError -jar
> /workdir/hadoop-ozone/ozonefs/target/surefire/surefirebooter9018689154779946208.jar
> /workdir/hadoop-ozone/ozonefs/target/surefire
> 2019-10-06T14-52-40_697-jvmRun1 surefire7569723928289175829tmp
> surefire_947955725320624341206tmp
>
>
> [ERROR] Error occurred in starting fork, check output in log
>
>
> [ERROR] Process Exit Code: 139 {code}
>
> Based on the crash log (uploaded) it's related to the rocksdb JNI interface.
> In the current ozone-build docker image (which provides the environment for
> build) we use alpine where musl libc is used instead of the main glibc. I
> think it would be more safe to use the same glibc what is used in production.
> I tested with centos based docker image and it seems to be more stable.
> Didn't see any more JVM crashes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
