[
https://issues.apache.org/jira/browse/HDFS-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18115317#comment-18115317
]
ASF GitHub Bot commented on HDFS-16021:
---------------------------------------
hadoop-yetus commented on PR #8739:
URL: https://github.com/apache/hadoop/pull/8739#issuecomment-5670600902
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|:----:|----------:|--------:|:--------:|:-------:|
| +0 :ok: | reexec | 1m 1s | | Docker mode activated. |
|||| _ Prechecks _ |
| +1 :green_heart: | dupname | 0m 1s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| -1 :x: | test4tests | 0m 0s | | The patch doesn't appear to include
any new or modified tests. Please justify why no new tests are needed for this
patch. Also please list what manual steps were performed to verify this patch.
|
|||| _ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 36m 20s | | trunk passed |
| +1 :green_heart: | compile | 4m 2s | | trunk passed with JDK
Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | compile | 4m 16s | | trunk passed with JDK
Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | mvnsite | 0m 56s | | trunk passed |
| +1 :green_heart: | shadedclient | 81m 20s | | branch has no errors
when building and testing our client artifacts. |
|||| _ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 3m 12s | | the patch passed |
| +1 :green_heart: | compile | 3m 11s | | the patch passed with JDK
Ubuntu-21.0.12+8-1-24.04-Ubuntu |
| +1 :green_heart: | cc | 3m 11s | | the patch passed |
| +1 :green_heart: | golang | 3m 11s | | the patch passed |
| +1 :green_heart: | javac | 3m 11s | | the patch passed |
| +1 :green_heart: | compile | 3m 11s | | the patch passed with JDK
Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| +1 :green_heart: | cc | 3m 11s | | the patch passed |
| +1 :green_heart: | golang | 3m 11s | | the patch passed |
| +1 :green_heart: | javac | 3m 11s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | mvnsite | 0m 20s | | the patch passed |
| +1 :green_heart: | shadedclient | 27m 27s | | patch has no errors
when building and testing our client artifacts. |
|||| _ Other Tests _ |
| +1 :green_heart: | unit | 9m 52s | | hadoop-hdfs-native-client in
the patch passed. |
| +1 :green_heart: | asflicense | 0m 34s | | The patch does not
generate ASF License warnings. |
| | | 133m 11s | | |
| Subsystem | Report/Notes |
|----------:|:-------------|
| Docker | ClientAPI=1.56 ServerAPI=1.56 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8739/2/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/8739 |
| Optional Tests | dupname asflicense compile cc mvnsite javac unit
codespell detsecrets golang |
| uname | Linux 50bd8f98415b 5.15.0-190-generic #200-Ubuntu SMP Fri Aug 7
15:06:04 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / 65a4e7b2dfe47ac93878904d1388290274823906 |
| Default Java | Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Multi-JDK versions |
/usr/lib/jvm/java-21-openjdk-amd64:Ubuntu-21.0.12+8-1-24.04-Ubuntu
/usr/lib/jvm/java-17-openjdk-amd64:Ubuntu-17.0.20+8-1-24.04-Ubuntu |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8739/2/testReport/ |
| Max. process+thread count | 636 (vs. ulimit of 10000) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-8739/2/console |
| versions | git=2.43.0 maven=3.9.15 |
| Powered by | Apache Yetus 0.14.1 https://yetus.apache.org |
This message was automatically generated.
> heap-use-after-free in hdfsThreadDestructor
> -------------------------------------------
>
> Key: HDFS-16021
> URL: https://issues.apache.org/jira/browse/HDFS-16021
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.9.1, 2.9.2, 3.3.0
> Reporter: Jeremy Coulon
> Assignee: Akira Ajisaka
> Priority: Major
> Labels: pull-request-available
> Attachments: fix-hdfsThreadDestructor.patch, hdfs-asan.log
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Related to HDFS-12628 HDFS-13585 HDFS-14488 HDFS-15270
>
> We have experienced crashes located in libhdfs hdfsThreadDestructor() for a
> long time. Crash is almost systematic with OpenJ9 and more sporadic with
> Hotspot JVM.
>
> I finally went to the root cause of this bug thanks to AddressSanitizer. This
> is quite difficult to setup because you need to rebuild both the test-case,
> hadoop and openjdk-hotspot >= 13 with specific compiler options.
>
> See hdfs-asan.log for details.
>
> *Analysis:*
> In hdfsThreadDestructor(), you are making several JNI calls in order to
> detach the thread from the JVM:
>
> {code:java}
> /* Detach the current thread from the JVM */
> if (env) {
> ret = (*env)->GetJavaVM(env, &vm);
> /*
> * More code here...
> */
> }{code}
> This is fine if the thread was created in the C/C++ world.
>
> However if the thread was created in the Java world, this is absolutely
> wrong. When a Java thread terminates, the JVM deallocates some memory which
> contains (among other things) the thread specific JNIEnv. Then
> hdfsThreadDestructor() is called. The *env* variable is not NULL but points
> to memory which was just released. This is heap-use-after-free detected by
> ASan.
>
> I have been working on a patch that fixes the issue (see attachment).
>
> Here is the idea:
> * In hdfsThreadDestructor(), we need to know if the thread was created by
> Java or C/C++ . If it was created by C/C++ we should make JNI calls in order
> to detach the current thread. If it was created by Java, we don't need to
> make any JNI call: thread is already detached.
> * In getGlobalJNIEnv(), we can detect if the thread was created by Java or
> C/C++. It can be done by calling *vm->GetEnv()*. Then we store this
> information inside ThreadLocalState.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
