[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16583368#comment-16583368
]
Steve Loughran commented on HADOOP-15679:
-----------------------------------------
Thanks for the review.
I like your idea of debug level measuring of duration; will do. It'd work best
if the supplied hook instances have a to String value which is useful, but at
least we can log entry ID and priority.
bq. Have you considering per file system (like s3, wasb, etc.) shutdown timeout
(passed in when calling ShutdownHookManager#addShutdownHook in
FileSystem#getInternal() ) as needed while keep others with a small default
value?
I think the FS shutdown is set up before any are created, so it's not in a
position to ask...it'd get really complex to have it dynamically add it as new
entries were added.
The FS shutdown duration could be made another config point, independent of all
other shutdown hooks, I suppose. Making the base timeout extensible seems like
the simplest first step (and least to test/configure/document).
> ShutdownHookManager shutdown time needs to be configurable & extended
> ---------------------------------------------------------------------
>
> Key: HADOOP-15679
> URL: https://issues.apache.org/jira/browse/HADOOP-15679
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 2.8.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch,
> HADOOP-15679-002.patch
>
>
> HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging
> shutdowns. But the timeout is too short for applications where a large flush
> of data is needed on shutdown.
> A key example of this is Spark apps which save their history to object
> stores, where the file close() call triggers an upload of the final local
> cached block of data (could be 32+MB), and then execute the final mutipart
> commit.
> Proposed
> # make the default sleep time 30s, not 10s
> # make it configurable with a time duration property (with minimum time of
> 1s.?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-issues-h...@hadoop.apache.org
