xiarixiaoyao opened a new pull request #2858: URL: https://github.com/apache/hudi/pull/2858
…e result is wrong ## *Tips* - *Thank you very much for contributing to Apache Hudi.* - *Please review https://hudi.apache.org/contributing.html before opening a pull request.* ## What is the purpose of the pull request fix the bug of HoodieInputFormatUtils.getAffectedPartitions HoodieInputFormatUtils.getAffectedPartitions use String.contains method to filter the affected partition, this logical can't distinguish those partitions which have same prefix correctly： for example： partition1 fullPath is “/tmp/tableName/p=1” partition2 fullPath is "/tmp/tableName/p=11" partition3 fullPath is "/tmp/tableName/p=111 assume that we have a cow hudi table which has three partitions: partition1 fullPath is “/tmp/tableName/p=1” and this partition has only one file : file1 partition2 fullPath is "/tmp/tableName/p=11" and this partition has only one file : file2 partition3 fullPath is "/tmp/tableName/p=111" and this partition has only one file : file3 now we want to query the incr view of this hudi table by sparksql （the incr view involve partition1，partition2，partition3） sparksql will create hadoopRdds for each partition and union those rdds, since hudi table has three partitions, spark will create three hadoopRdd( rdd1, rdd2, rdd3) rdd1: holds the inputpath of “/tmp/tableName/p=1” rdd2: holds the inputpath of “/tmp/tableName/p=11” rdd2: holds the inputpath of “/tmp/tableName/p=111” when spark get the incremental files from rdd1, hoodie return file1. The return result is ok when spark try to get the incremental files from rdd2, since the bug of HoodieInputFormatUtils.getAffectedPartitions hudi will return file1 and file2. The return result is wrong, file1 should not be returned when spark try to get the incremental files from rdd3, since the bug of HoodieInputFormatUtils.getAffectedPartitions file1 and file2 and file 3 will be returned. This result is wrong, file1 and file2 should not be returned ## Brief change log *(for example:)* - *Modify AnnotationLocation checkstyle rule in checkstyle.xml* ## Verify this pull request add new UT, to cover this change ## Committer checklist - [ ] Has a corresponding JIRA in PR title & commit - [ ] Commit message is descriptive of the change - [ ] CI is green - [ ] Necessary doc changes done or have another open PR - [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
