codope commented on issue #9858: URL: https://github.com/apache/hudi/issues/9858#issuecomment-1762518643
@xuzifu666 Thanks for highlighting the issue. This is indeed a regression. The code block https://github.com/apache/hudi/blob/e6be39661a0aaed4227f8bcbec54e0abaf97fb22/hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/HoodieFileIndex.scala#L471-L474 is the main cause. It should be further special cased and not simply depend on number of partition columns. I am going to revert the eager listing first and then put in a proper fix, which should also take care of HUDI-6320. Additionally, we need some tests to assert spark DAGs for reads much like `TestSparkDataSourceDAGExecution`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
