alexguo8 opened a new pull request, #8269: URL: https://github.com/apache/hudi/pull/8269
### Change Logs _Describe context and summary for this change. Highlight if any code was copied._ - More details in the ticket https://issues.apache.org/jira/browse/HUDI-5967 - Currently, when performing a full table scan during a streaming read (when the start commit is EARLIEST), the partitions are collected into a HashSet which randomizes the order - For time partitioned tables, this means that Flink pipeline could read a partition for hour 11, advance the watermarks, then read the partitions for hour 8, and consider every record to be late - So, we use a TreeSet to force partition level ordering for full table scans - Test requires Hadoop to be running locally, similar to TestFileIndex tests ### Impact _Describe any public API or user-facing feature change or any performance impact._ - Since the full table scan only occurs once at the beginning of the streaming read, the partitions only need to be sorted once so there shouldn't be any performance impact - Reading from time partitioned tables should now work properly with time-sensitive Flink pipelines - Reading from non-time partitioned tables should have no functional impact ### Risk level (write none, low medium or high below) _If medium or high, explain what verification was done to mitigate the risks._ - None ### Documentation Update _Describe any necessary documentation update if there is any new feature, config, or user-facing change_ - _The config description must be updated if new configs are added or the default value of the configs are changed_ - _Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the [instruction](https://hudi.apache.org/contribute/developer-setup#website) to make changes to the website._ ### Contributor's checklist - [X] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [X] Change Logs and Impact were stated clearly - [X] Adequate tests were added if applicable - [ ] CI passed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
