[
https://issues.apache.org/jira/browse/HDFS-12943?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16298876#comment-16298876
]
Chao Sun commented on HDFS-12943:
---------------------------------
bq. Thanks Chao Sun, interesting results! You used only 1 SBN to server reads
right?
Yes I used 1 ANN + 1SBN + 1ONN (observer NN).
bq. In both configurations (with and without stale reads), I assume you were
saturating the system?
In the stale read case, the RPC queue time on the ANN was less than 5ms, while
on ONN it was between 0 to 30ms. In the non-stale read case, the RPC queue time
on ANN was around 130-140ms. So I guess the ANN was not saturated when stale
read is enabled?
bq. Did you use Namesystem unfair locking?
The ANN didn't use unfair locking. The ONN used unfair locking + async audit
logging (we have an internal patch to use log4j 2.x) + async edit logging. Do
you think it will make a difference if unfair locking is used on ANN?
bq. If I understand correctly, both your test and the Dynamometer test are more
like trace-driven micro benchmarks, where a container issues a certain type of
RPC at given timestamp. Chris was probably referring to a test job with "real
code" like if !file_exists(path) then create_file(path), where the blocking
relationship between calls are miniced.
Yes the test was pretty simple. It is basically:
{code}
loop {
x = randInt(0, 100)
if (x < 6) {
fs.createNewFile(..)
fs.rename(..)
fs.delete(..)
} else if (x < 10) {
fs.listStatus(..)
} else if (x < 40) {
fs.getFileBlockLocations(..)
} else {
fs.getFileStatus(..)
}
}
{code}
The file listing was done on a directory with 2K files.
Let me know if you have any suggestion on improving this. It's pretty easy to
change the code and re-run the benchmark.
> Consistent Reads from Standby Node
> ----------------------------------
>
> Key: HDFS-12943
> URL: https://issues.apache.org/jira/browse/HDFS-12943
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: hdfs
> Reporter: Konstantin Shvachko
> Attachments: ConsistentReadsFromStandbyNode.pdf
>
>
> StandbyNode in HDFS is a replica of the active NameNode. The states of the
> NameNodes are coordinated via the journal. It is natural to consider
> StandbyNode as a read-only replica. As with any replicated distributed system
> the problem of stale reads should be resolved. Our main goal is to provide
> reads from standby in a consistent way in order to enable a wide range of
> existing applications running on top of HDFS.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
