hudi-bot opened a new issue, #16174: URL: https://github.com/apache/hudi/issues/16174
1. Current batched commit string is of the type "commit#key", given this if we consider the following sample commit c1->k1 c1->k2 c1->k3 Lets say if the fetch next end exactly at c1#k3, then every fetch following that we would read entire commit c1 and then ignore. To solve this we would need another flag inside commit string like "commit#key#commit_complete_boolean_flag", this "commit_complete_boolean_flag" will help us avoid is sub optimal fetch in case we end up in the above scenario. 2. [https://github.com/apache/hudi/blob/05ac011316564f97de178b023e8e93ff768c37a4/hudi-utilities/src/main/java/org/apache/hudi/utilities/sources/helpers/IncrSourceHelper.java#L183] - QueryRunner.applyOrdering api call might not be needed as after filtering we are anyways order it again. Filtering does not need this ordering, we need to test and remove this ordering call (https://github.com/apache/hudi/pull/9433#discussion_r1291828096) 3. [https://github.com/apache/hudi/pull/9433#discussion_r1291826591] ## JIRA info - Link: https://issues.apache.org/jira/browse/HUDI-6687 - Type: Improvement -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
