sivabalan narayanan created HUDI-4615:
-----------------------------------------
Summary: Fix empty commits being made by deltastreamer with
S3EventsSource when there is no data in SQS on starting a new pipeline
Key: HUDI-4615
URL: https://issues.apache.org/jira/browse/HUDI-4615
Project: Apache Hudi
Issue Type: Bug
Components: deltastreamer
Reporter: sivabalan narayanan
When we start a new deltastreamer with S3EventsSource, checkpoint is
Option.empty(). After consumption from source, if there is no data, the source
returns "val=0" as the checkpoint. So, deltastreamer assumes checkpoint has
changed and makes an empty commit. This needs fixing.
[https://github.com/apache/hudi/blob/0d0a4152cfd362185066519ae926ac4513c7a152/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/S3EventsMetaSelector.java#L151]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
