[
https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396447#comment-13396447
]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
Hey Brandon,
I just got my implementation "working" (ie I can run a NN against remote
loggers and then successfully restart reading the edits remotely). It's nowhere
near finished/tested/pretty, but I pushed to github so I could share the
approach:
This commit
https://github.com/toddlipcon/hadoop-common/commit/c7aeeb83a653935463d0899968828abad3597ee8
refactors EditLogFileInputStream to deal with an interface instead of directly
a {{File}} reference. The interface just has to be able to provide an input
stream as well as its length and a name.
This commit
https://github.com/toddlipcon/hadoop-common/commit/2969ab41478c808ff26ef3f00dad2cd22a352e93
adds an implementation of the above interface which reads from a remote URL,
and changes the getEditLogManifest call to also expose the node's HTTP port (so
it can use ephemeral ports)
Then this commit
https://github.com/toddlipcon/hadoop-common/commit/477d937c258d4ffe67a5042258da7410df74c42e
implements {{selectInputStreams()}} to actually pull the edits off the remote
nodes while loading the namespace.
I'm not attacking HA yet in these patches, but the interfaces should work out
the same there, given the BK folks managed to get it working.
As I said above, the code's rough, so I don't expect anyone to code review the
github branch, but if you have questions on the approach, let me know. I'm
trying to reuse as much of your JournalService code as possible, but mostly
doing so by copy-pasting into these new files for now, to avoid conflicts, etc.
I anticipate running back over it to unify the code better at a later date.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get
> edit logs from journal service instead of from local or shared storage
> directories.
> As one of the first steps towards this goal, this JIRA is to enable standby
> namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
