[
https://issues.apache.org/jira/browse/HDFS-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13662441#comment-13662441
]
Andrew Wang commented on HDFS-4816:
-----------------------------------
I gathered some jstacks. It looks like there's a SBN servlet upload thread
blocked on a DataTransferThrottler, a NN servlet thread doing the image
download, and the main thread is hung in MiniDFSCluster#shutdown, specifically
HttpServer#stop. The javadoc for this says:
{quote}
Stops the component. The component may wait for current activities to complete
normally, but it can be interrupted.
{quote}
Apparently, we're dealing with a non-interrupting stop. Further googling
indicates that shutdown blocks until active requests are finished [1]. I tried
setting {{setGracefulShutdown}} to no avail. Dunno how to get around this
without patching Jetty.
Are we okay with the 100B/s set in the current patch? It easily hits the timing
window for me.
Setting 1B/s also does succeed, it just takes 9 minutes to complete the test.
[1]: http://docs.codehaus.org/display/JETTY/How+to+gracefully+shutdown
> transitionToActive blocks if the SBN is doing checkpoint image transfer
> -----------------------------------------------------------------------
>
> Key: HDFS-4816
> URL: https://issues.apache.org/jira/browse/HDFS-4816
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 3.0.0, 2.0.4-alpha
> Reporter: Andrew Wang
> Assignee: Andrew Wang
> Attachments: hdfs-4816-1.patch, hdfs-4816-2.patch, hdfs-4816-3.patch,
> hdfs-4816-slow-shutdown.txt
>
>
> The NN and SBN do this dance during checkpoint image transfer with nested
> HTTP GETs via {{HttpURLConnection}}. When an admin does a
> {{-transitionToActive}} during this transfer, part of that is interrupting an
> ongoing checkpoint so we can transition immediately.
> However, the {{thread.interrupt()}} in {{StandbyCheckpointer#stop}} gets
> swallowed by {{connection.getResponseCode()}} in
> {{TransferFsImage#doGetUrl}}. None of the methods in HttpURLConnection throw
> InterruptedException, so we need to do something else (perhaps HttpClient
> [1]):
> [1]: http://hc.apache.org/httpclient-3.x/
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
