[
https://issues.apache.org/jira/browse/HDFS-13767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16563928#comment-16563928
]
Erik Krogen commented on HDFS-13767:
------------------------------------
I agree with Konstantin's point that maintaining the ordering is not necessary.
I like the approach of sticking the call back into the queue rather than trying
to do some fancy deferral logic. This basically degrades to busy-waiting if
there's currently only requests which are not yet serviceable, but I don't
think this will be an issue in practice.
[Continuing
discussion|https://issues.apache.org/jira/browse/HDFS-13688?focusedCommentId=16562713&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16562713]
of whether to use {{FSImage#getLastAppliedOrWrittenTxId}} or
{{FSImage#getCorrectLastAppliedOrWrittenTxId}} within {{GlobalStateIdContext}}:
the "correct" version here only takes the lock on {{FSEditLog}}, not the global
{{FSNamesystemLock}}, so I think that the impact is not as bad as it could be -
for example reads from the namesystem can proceed as normal. If necessary, we
can even change {{FSEditLog}} to use a read-write lock, change this field to be
{{volatile}}, use an {{AtomicLong}}, etc to improve performance. However, I
have done some benchmarks in the past to estimate the scalability of the
read-from-standby architecture, in which I compared fetching the transaction ID
with vs without locking. My findings showed that the performance differential
between the two was not that large. For example, one datapoint was that when
doing 6k write QPS and 100k txn ID lookups per second, the client-side latency
was 166 ms when taking a lock, and 157 ms without the lock. The differential
increased a bit at even higher QPSes, but I think by using some of the
optimizations described above we could reduce it further.
As for why we need the lock, I do think it is required for correctness. Let's
say you {{mkdir /foo}} at txn ID 10, but {{getLastAppliedOrWrittenTxId}}
returns 9 because you took it without the lock. Now you do a {{getFileInfo
/foo}} which goes to the Observer, which is currently at txn ID 9, and get a
FNFE. We've broken the read-your-own-write semantics that we promised as part
of this feature.
> Add msync server implementation.
> --------------------------------
>
> Key: HDFS-13767
> URL: https://issues.apache.org/jira/browse/HDFS-13767
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
> Reporter: Chen Liang
> Assignee: Chen Liang
> Priority: Major
> Attachments: HDFS-13767.WIP.001.patch, HDFS-13767.WIP.002.patch,
> HDFS-13767.WIP.003.patch
>
>
> This is a followup on HDFS-13688, where msync API is introduced to
> {{ClientProtocol}} but the server side implementation is missing. This is
> Jira is to implement the server side logic.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
