D M Murali Krishna Reddy created HDFS-18028:
-----------------------------------------------

             Summary: Fix Read Consistency issues from Observer Name Node
                 Key: HDFS-18028
                 URL: https://issues.apache.org/jira/browse/HDFS-18028
             Project: Hadoop HDFS
          Issue Type: Bug
            Reporter: D M Murali Krishna Reddy


With Observer reads enabled, data loss has been observed for a few Spark jobs 
on heavily loaded clusters.

The cause appears to be Observer lag in applying edits: from the audit logs, 
the gap between the write from a Spark executor (task commit rename on the 
Active) and the read from the Spark driver (listStatus served by the Observer) 
was ~5 ms, and the Observer had not yet applied the rename. The driver silently 
received a stale listing, the missing task output was dropped during job 
commit, and the job still succeeded. Details and reproduction: 
https://lists.apache.org/thread.html/wbrd50sr83fv6b52bzt51qwvwqlfnpkd (see also 
[SPARK-36810|https://issues.apache.org/jira/browse/SPARK-36810]).

One mitigation is to set dfs.client.failover.observer.auto-msync-period=0 for 
the driver, but this sends an msync before every read and would cut into the 
performance gains expected from Observer reads.

Another is to update FileOutputCommitter to msync() before the listStatus in 
commitJob() where this race was observed, but that fixes only one code path; 
any other application where one client reads what another client just wrote has 
the same exposure.

Raising this to discuss whether HDFS can provide a stronger default guarantee 
for cross-client read-after-write, rather than relying on every application to 
opt in.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to