[
https://issues.apache.org/jira/browse/HADOOP-15679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16584508#comment-16584508
]
Steve Loughran commented on HADOOP-15679:
-----------------------------------------
[~tmarquardt], thanks for the comments
h3. imports
Nominally alphabetical within groups, the groups being
java.*
java.*
(others)
org.apache.*
static import*
It's a bit lax and generally nobody tries to reorder stuff once it's in as it
only makes cherrypicking and merging harder. Imports are always a
false-positive merge conflict point. As an example, here all the java.* are
lower down, but I'm not going to move them.
h3. reentrant shutdown
I triggered this when writing my tests. as the atomic boolean was already being
set, I decided to follow through with some actions. Removing the logging of the
stack, given your premise that it'll be fairly meaningless.
h3. FileSystem.closeAll()
a faster shutdown would be good, especially when there's one FS instance per
container, but when it's triggered during JVM shutdown there's a risk that you
can't create new threads. So the shutdown executor pool needs to be created in
advance.
I think we do need to do more here...there's a collection of JIRAs related to
handling problems in shutdown. The timeout one handled one issue
(deadlock/retry problems in a shutdown hook breaking failover), but turns out
to be over-aggressive for some apps, especially when the final committing of
work is slow.
> ShutdownHookManager shutdown time needs to be configurable & extended
> ---------------------------------------------------------------------
>
> Key: HADOOP-15679
> URL: https://issues.apache.org/jira/browse/HADOOP-15679
> Project: Hadoop Common
> Issue Type: Bug
> Components: util
> Affects Versions: 2.8.0, 3.0.0
> Reporter: Steve Loughran
> Assignee: Steve Loughran
> Priority: Major
> Attachments: HADOOP-15679-001.patch, HADOOP-15679-002.patch,
> HADOOP-15679-002.patch
>
>
> HADOOP-12950 added a timeout on shutdowns to avoid problems with hanging
> shutdowns. But the timeout is too short for applications where a large flush
> of data is needed on shutdown.
> A key example of this is Spark apps which save their history to object
> stores, where the file close() call triggers an upload of the final local
> cached block of data (could be 32+MB), and then execute the final mutipart
> commit.
> Proposed
> # make the default sleep time 30s, not 10s
> # make it configurable with a time duration property (with minimum time of
> 1s.?)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
