[
https://issues.apache.org/jira/browse/HADOOP-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738263#comment-17738263
]
ASF GitHub Bot commented on HADOOP-18781:
-----------------------------------------
steveloughran commented on PR #5780:
URL: https://github.com/apache/hadoop/pull/5780#issuecomment-1611994029
@saxenapranav having the close() in finalize helps avoid leakage of threads
and things if caching for an fs schema is disabled and clients call
FileSystem.get(path), the way they should: normally apps shouldn't close it.
In #4966 we had to add finalizers to the FileContext bindings as they were
leaking.
One thing to consider is this problem surfaced as reports of the output
stream hanging in close(): it was waiting for the async uploads to finish but
because all was broken, that was not to be.
Mukund, as well as avoiding premature close(), can this PR have output
stream close() fail noisily if the thread pool was shut down? this is at least
testable, to an extent
> ABFS Output stream thread pools getting shutdown during GC.
> -----------------------------------------------------------
>
> Key: HADOOP-18781
> URL: https://issues.apache.org/jira/browse/HADOOP-18781
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
>
> Applications using AzureBlobFileSystem to create the AbfsOutputStream can use
> the AbfsOutputStream for the purpose of writing, however, the OutputStream
> doesn't hold any reference to the fs instance that created it, which can make
> the FS instance eligible for GC, when this occurs, AzureblobFileSystem's
> `finalize()` method gets called which in turn closes the FS, and in turn call
> the close for AzureBlobFileSystemStore, which uses the same Threadpool that
> is used by the AbfsOutputStream. This leads to the closing of the thread pool
> while the writing is happening in the background and leads to hanging while
> writing.
>
> *Solution:*
> Pass a backreference of AzureBlobFileSystem into AzureBlobFileSystemStore and
> AbfsOutputStream as well.
>
> Same should be done for AbfsInputStream as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
