[
https://issues.apache.org/jira/browse/HDFS-16021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18114725#comment-18114725
]
ASF GitHub Bot commented on HDFS-16021:
---------------------------------------
aajisaka opened a new pull request, #8739:
URL: https://github.com/apache/hadoop/pull/8739
### Description of PR
libhdfs registers a pthread TLS destructor, hdfsThreadDestructor, that
detaches the current thread from the JVM whenever it finds a cached JNIEnv. It
does so regardless of who attached the thread. When the JVM (or an embedding
application) attached the thread, the JNIEnv it still holds may already have
been freed by the time the destructor runs, so dereferencing it reads freed
memory (heap-use-after-free, SIGSEGV).
Track in ThreadLocalState whether libhdfs attached the thread itself,
determined in getGlobalJNIEnv via GetEnv before attaching, and skip the JNI
detach in hdfsThreadDestructor for threads libhdfs did not attach.
The original patch is authored by Jeremy Coulon.
Assisted-by: OpenCode (GLM-5.3-Flash)
JIRA: HDFS-16021
### How was this patch tested?
The same fix is reviewed in Apache DataFusion Comet side:
https://github.com/apache/datafusion-comet/pull/5890
### For code changes:
- [x] Does the title of this PR start with the corresponding JIRA issue id
(e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: Have the integration tests been executed and the
endpoint
declared according to the connector-specific documentation? *Note:
Automated CI
testing doesn't cover all cases so manual testing with cloud storage
is still
required.*
- [ ] If adding new dependencies to the code, are these dependencies
licensed in a way that is compatible for inclusion under [ASF
2.0](http://www.apache.org/legal/resolved.html#category-a)?
- [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`,
`NOTICE-binary` files?
### AI Tooling
If an AI tool was used:
- [x] The PR includes the phrase "Contains content generated by <tool>"
where <tool> is the name of the AI tool used.
- [x] My use of AI contributions follows the ASF legal policy
https://www.apache.org/legal/generative-tooling.html
> heap-use-after-free in hdfsThreadDestructor
> -------------------------------------------
>
> Key: HDFS-16021
> URL: https://issues.apache.org/jira/browse/HDFS-16021
> Project: Hadoop HDFS
> Issue Type: Bug
> Affects Versions: 2.9.1, 2.9.2, 3.3.0
> Reporter: Jeremy Coulon
> Assignee: Akira Ajisaka
> Priority: Major
> Labels: pull-request-available
> Attachments: fix-hdfsThreadDestructor.patch, hdfs-asan.log
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Related to HDFS-12628 HDFS-13585 HDFS-14488 HDFS-15270
>
> We have experienced crashes located in libhdfs hdfsThreadDestructor() for a
> long time. Crash is almost systematic with OpenJ9 and more sporadic with
> Hotspot JVM.
>
> I finally went to the root cause of this bug thanks to AddressSanitizer. This
> is quite difficult to setup because you need to rebuild both the test-case,
> hadoop and openjdk-hotspot >= 13 with specific compiler options.
>
> See hdfs-asan.log for details.
>
> *Analysis:*
> In hdfsThreadDestructor(), you are making several JNI calls in order to
> detach the thread from the JVM:
>
> {code:java}
> /* Detach the current thread from the JVM */
> if (env) {
> ret = (*env)->GetJavaVM(env, &vm);
> /*
> * More code here...
> */
> }{code}
> This is fine if the thread was created in the C/C++ world.
>
> However if the thread was created in the Java world, this is absolutely
> wrong. When a Java thread terminates, the JVM deallocates some memory which
> contains (among other things) the thread specific JNIEnv. Then
> hdfsThreadDestructor() is called. The *env* variable is not NULL but points
> to memory which was just released. This is heap-use-after-free detected by
> ASan.
>
> I have been working on a patch that fixes the issue (see attachment).
>
> Here is the idea:
> * In hdfsThreadDestructor(), we need to know if the thread was created by
> Java or C/C++ . If it was created by C/C++ we should make JNI calls in order
> to detach the current thread. If it was created by Java, we don't need to
> make any JNI call: thread is already detached.
> * In getGlobalJNIEnv(), we can detect if the thread was created by Java or
> C/C++. It can be done by calling *vm->GetEnv()*. Then we store this
> information inside ThreadLocalState.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
