[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wei-Chiu Chuang updated HDFS-14272:
-----------------------------------
Description:
It is typical for integration tests to create some files and then check their
existence. For example, like the following simple bash script:
{code:java}
# hdfs dfs -touchz /tmp/abc
# hdfs dfs -ls /tmp/abc
{code}
The test executes HDFS bash command sequentially, but it may fail with
Consistent Standby Read because the -ls does not find the file.
Analysis: the second bash command, while launched sequentially after the first
one, is not aware of the state id returned from the first bash command. So
ObserverNode wouldn't wait for the the edits to get propagated, and thus fails.
I've got a cluster where the Observer has tens of seconds of RPC latency, and
this becomes very annoying. (I am still trying to figure out why this Observer
has such a long RPC latency. But that's another story.)
was:
It is typical for integration tests to create some files and then check its
existence. For example, like the following simple bash script:
{code:java}
# hdfs dfs -touchz /tmp/abc
# hdfs dfs -ls /tmp/abc
{code}
The test execute HDFS bash command sequentially, but it may fail with
Consistent Standby Read because the -ls may not find the file.
Analysis: the second bash command, while launched sequentially after the first
one, is not aware of the state id returned from the first bash command. So
ObserverNode wouldn't wait for the the edits to get propagated, and thus fails.
I've got a cluster where the Observer has tens of seconds of RPC latency, and
this becomes very annoying. (I am still trying to figure out why this Observer
has such a long RPC latency. But that's another story.)
> [SBN read] HDFS command line tools does not guarantee consistency
> -----------------------------------------------------------------
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
> Reporter: Wei-Chiu Chuang
> Priority: Major
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
