Prashant Wason created HUDI-6084:
------------------------------------
Summary: Ensure write operations to MDT do not absorb failures
Key: HUDI-6084
URL: https://issues.apache.org/jira/browse/HUDI-6084
Project: Apache Hudi
Issue Type: Bug
Reporter: Prashant Wason
Assignee: Prashant Wason
Issue 1:
When we call compaction on MDT, we do not check the return value. Compaction
operation may have had errors reported in the WriteStatus. This will cause
missing data in MDT.
MDT operations should never succeed in case of errors.
Issue 2:
Once a deltacommit has completed, the WriteStatus has been used to finalize the
write and write the deltacommit action. The code was collecting the WriteStatus
on the driver side to check for any errors that occurred during the writing.
Since MDT write config has autoCommit, if there were any errors then there is
no value of checking them at this stage since the deltacommit has already
completed. Also, the write status RDD may have been unpersisted and if a cached
value is not available then it will lead to re-writing of the deltacommit.
Fix:
MDT uses FailOnFirstErrorWriteStatus which is designed to throw an exception
when the first write error is detected. Hence, we do not need to check for
write errors explicitly. If any write errors would have occurred then the write
itself would not have completed and thrown an exception.
Also, we do not need to check the WriteStatus after commit has completed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
