[
https://issues.apache.org/jira/browse/HADOOP-18781?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739437#comment-17739437
]
ASF GitHub Bot commented on HADOOP-18781:
-----------------------------------------
mehakmeet commented on PR #5780:
URL: https://github.com/apache/hadoop/pull/5780#issuecomment-1617203652
> No similar test for InputStream?
So, InputStream doesn't actually use any threadpool passed from the FS
instance so would be tricky to test that. But, Let me just add a test for
checking back reference notNull, that should be enough I think.
> Can we also handle filesystem.close() in the output stream by failing
cleanly if the fs was explicitly closed()?
@steveloughran Was thinking about this, so essentially whenever an
Abfs.close() is closed we need to call the stream's close() as well, so that we
can flush any remaining writes etc? Was thinking how this should be handled say
mid way through a write, currently, I believe since the executor itself is shut
down it just simply rejects any request incoming like this
```
Could not submit task to executor
java.util.concurrent.ThreadPoolExecutor@5c5e7584[Terminated, pool size = 0,
active threads = 0, queued tasks = 0, completed tasks = 0]
```
> ABFS Output stream thread pools getting shutdown during GC.
> -----------------------------------------------------------
>
> Key: HADOOP-18781
> URL: https://issues.apache.org/jira/browse/HADOOP-18781
> Project: Hadoop Common
> Issue Type: Bug
> Components: fs/azure
> Reporter: Mehakmeet Singh
> Assignee: Mehakmeet Singh
> Priority: Major
> Labels: pull-request-available
>
> Applications using AzureBlobFileSystem to create the AbfsOutputStream can use
> the AbfsOutputStream for the purpose of writing, however, the OutputStream
> doesn't hold any reference to the fs instance that created it, which can make
> the FS instance eligible for GC, when this occurs, AzureblobFileSystem's
> `finalize()` method gets called which in turn closes the FS, and in turn call
> the close for AzureBlobFileSystemStore, which uses the same Threadpool that
> is used by the AbfsOutputStream. This leads to the closing of the thread pool
> while the writing is happening in the background and leads to hanging while
> writing.
>
> *Solution:*
> Pass a backreference of AzureBlobFileSystem into AzureBlobFileSystemStore and
> AbfsOutputStream as well.
>
> Same should be done for AbfsInputStream as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
