[
https://issues.apache.org/jira/browse/HUDI-5686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Purushotham Pushpavanthar updated HUDI-5686:
--------------------------------------------
Description:
See issue [https://github.com/apache/hudi/issues/7757] for more details.
Description of the issue:
If the HoodieDeltaStreamer is forcefully terminated before commit instant's
state is `COMPLETED`, it leaves the commit state in either `REQUESTED` or
`INFLIGHT`. When the HoodieDeltaStreamer is rerun, the first successful commit
writes first batch of records into Hudi Table. However, in the consecutive
commit, the changes committed by previous commit disappears. This causes *loss
of entire batch* of data written by the first commit after restart.
I observed this problem when HoodieDeltaStreamer is run in continuous mode and
when job gets resubmitted when AM container gets killed due to reasons like
loss of nodes or node going to unhealthy state. This issue is not limited to
continuous mode alone, this can happen anytime when Hudi write gets terminated
before instant is marked `COMPLETE`.
How to reproduce the issue:
# Run HoodieDeltaStreamer and yarn kill the job before commit instant reaches
`COMPLETE` state. Note the number of records after last successful commit (say
100)
# Upon re-submission of HoodieDeltaStreamer, there will be 2 new instants
created (1 Commit complete and 1 rollback complete). Note the number of delta
changes consumed(say 10 new records keys) in this run and total number of
records in hudi table( 110 unique records )
# On next run, wait till Hudi completes the commit assuming it received 5
records and check the count of unique records in hudi table (It was observed to
be 105). The delta records consumed in step 2 are entirely lost.
Reason:
To explain what happens internally lets take an example of a commit timeline
shown below
was:See issue [https://github.com/apache/hudi/issues/7757] for more details.
> Missing records when HoodieDeltaStreamer run in continuous
> ----------------------------------------------------------
>
> Key: HUDI-5686
> URL: https://issues.apache.org/jira/browse/HUDI-5686
> Project: Apache Hudi
> Issue Type: Bug
> Components: deltastreamer
> Reporter: Sagar Sumit
> Assignee: Purushotham Pushpavanthar
> Priority: Critical
> Fix For: 0.13.1
>
>
> See issue [https://github.com/apache/hudi/issues/7757] for more details.
> Description of the issue:
> If the HoodieDeltaStreamer is forcefully terminated before commit instant's
> state is `COMPLETED`, it leaves the commit state in either `REQUESTED` or
> `INFLIGHT`. When the HoodieDeltaStreamer is rerun, the first successful
> commit writes first batch of records into Hudi Table. However, in the
> consecutive commit, the changes committed by previous commit disappears. This
> causes *loss of entire batch* of data written by the first commit after
> restart.
> I observed this problem when HoodieDeltaStreamer is run in continuous mode
> and when job gets resubmitted when AM container gets killed due to reasons
> like loss of nodes or node going to unhealthy state. This issue is not
> limited to continuous mode alone, this can happen anytime when Hudi write
> gets terminated before instant is marked `COMPLETE`.
> How to reproduce the issue:
> # Run HoodieDeltaStreamer and yarn kill the job before commit instant reaches
> `COMPLETE` state. Note the number of records after last successful commit
> (say 100)
> # Upon re-submission of HoodieDeltaStreamer, there will be 2 new instants
> created (1 Commit complete and 1 rollback complete). Note the number of delta
> changes consumed(say 10 new records keys) in this run and total number of
> records in hudi table( 110 unique records )
> # On next run, wait till Hudi completes the commit assuming it received 5
> records and check the count of unique records in hudi table (It was observed
> to be 105). The delta records consumed in step 2 are entirely lost.
> Reason:
> To explain what happens internally lets take an example of a commit timeline
> shown below
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
