wwanix opened a new issue, #19693: URL: https://github.com/apache/hudi/issues/19693
### Describe the problem you faced **Issue**: Reading from upstream hudi lake with incremental query type in prod, got "HoodieException: Found hollow commit" error. Reason: Upstream hudi lake job failed or still running when access to it. **Environment**: EMR 7.1.0, hudi version 0.14.1, Spark version 3.5 **Work flow**: 1. In the first time run, use `snapshot` to read upstream hudi lake, generate a timestamp as extractId so next we use it to fetch the new data 2. Then twice a week to read the new/updated data with `incremental` type(pass extractId to "BEGIN_INSTANTTIME", generate new extractId) Hudi suggested to set up `hoodie.read.timeline.holes.resolution.policy` to handle the `hollow commit` error, but for those policy options 1. `Fail`: this the default behaviour, so won't fix the issue 2. `BLOCK`: The upstream lake generating job normally run over hours, so that means the hollow commit will stay there for hours and use this option, we need to wait until the job is done. We cannot wait that longer because we have downstream job need to run as well. 3. `USE_TRANSITION_TIME`: This option will allow us to read the completed commit instead of inflight commit. But as we discussed in the hudi weekly meeting, we were told this option is a new introduced feature, so don't know if it can fix the issue we have. We have been told bring this question up so hudi team can dig into it to find out the solution. We would like to have a solution that can fix the hollow commit without any data lose, and would like to know if we can switch to the solution without regenerating our whole lake. Thanks, Cheers, ### To Reproduce 1. Run a job(producer) to generate a new hudi lake without any failure(No any inflight commit) 2. Run another job(consumer) to read the hudi lake by using `snapshot` query type. Marked the timestamp. 3. Kick off a new producer job to keep it running or make it failed to generate`deltacommit.requested` file under the `.hoodie` folder 4. Run another consumer job to read that hudi lake with `incremental` query type, and pass the marked timestamp in step 2 to `BEGIN_INSTANTTIME` 5.To see if the consumer job failed with "hollow commit" error. ### Expected behavior After setting up `hoodie.read.timeline.holes.resolution.policy` to `USE_TRANSITION_TIME` 1. Make sure there are no error when reading the hudi lake with `incremental` query type 2. Make sure there are no data lose between different consequential runs ### Environment Description * Hudi version: 0.14.1 * Spark version: 3.5 * Flink version: * Hive version: * Hadoop version: * Storage (HDFS/S3/GCS..): * Running on Docker? (yes/no): ### Additional context _No response_ ### Stacktrace ```shell ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
