[
https://issues.apache.org/jira/browse/HDFS-13150?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16377577#comment-16377577
]
Erik Krogen commented on HDFS-13150:
------------------------------------
Thanks for the read, Chao! Responses inline.
{quote}what is the relation of this with the in-progress edit log tailing?
also, is this a separate feature that can be turned on/off? or we target this
as a replacement for the current approach?{quote}
The addition of this feature does not necessarily remove the possibility of
using the traditional edit tail path to read in-progress edits, so we could
have it enabled via a separate flag. This would create a situation where, for
the feature to work as intended, you have to change 3 configurations: enable
in-progress edit log tailing, enable the fast-path, and turn the edit tail
period ({{dfs.ha.tail-edits.period}}) down to 0. We should probably leave the
edit tail period config as-is, but given that in-progress edit log tailing is a
new/experimental feature anyhow, perhaps we can just enable the fast path as
part of enabling in-progress edit log tailing, reducing it to two
configurations.
{quote}how much savings did you see by performing preloading edits outside the
lock? I did something similar in my experiment but it didn't show obvious
benefit.{quote}
I also didn't see a significant performance improvement, but I think in general
it is a good idea to avoid I/O inside of a lock if possible to protect against
temporary hiccups.
{quote}with the cache, does it mean we need to extra decode + encode on the
journal side? is there any perf impact on the journal?{quote}
Yes, unfortunately this adds an extra decode/encode cycle on the JN side. We
did not see any significant performance impact on the JNs in our experiments.
If this ends up being an issue, we can reduce this to a single decode (no
re-encode) by storing the cache as the original serialized form rather than the
deserialized form. This is a bit more complex so I would prefer not to unless
it turns out to be an issue in practice.
{quote}in the performance evaluation section, the max(ms) in scenario 5 is
extremely high comparing to others, is that a mistake?{quote}
No, this is what was measured. I haven't seen the same other times I've run the
experiment but didn't want to feel I was faking results by providing a cleaner
test run :) Given the very PoC nature of the code that this benchmark was on, I
hope to eliminate any such issues when we create a production-ready version. In
particular it may be helpful to log when latencies are higher than expected,
similar to the idea of HDFS-9145, so that we can track down any issues.
{quote}To achieve low latency the Observer NameNode also needs to pull from the
journal nodes in a very high frequency right? did you cover that in the
benchmark?{quote}
This is correct; in the benchmark the sleep period for edit tailing was turned
down to 0.
{quote}might be better to explain a little that on the high-level how this can
be used for Observer NameNode.{quote}
Sure, added a little bit of discussion in _Applicability to ObserverNode_ in v1.
> Create fast path for SbNN tailing edits from JNs
> ------------------------------------------------
>
> Key: HDFS-13150
> URL: https://issues.apache.org/jira/browse/HDFS-13150
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: hdfs, journal-node, namenode
> Reporter: Erik Krogen
> Assignee: Erik Krogen
> Priority: Major
> Attachments: edit-tailing-fast-path-design-v0.pdf
>
>
> In the interest of making coordinated/consistent reads easier to complete
> with low latency, it is advantageous to reduce the time between when a
> transaction is applied on the ANN and when it is applied on the SbNN. We
> propose adding a new "fast path" which can be used to tail edits when low
> latency is desired. We leave the existing tailing logic in place, and fall
> back to this path on startup, recovery, and when the fast path encounters
> unrecoverable errors.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
