bvaradar commented on issue #1798: URL: https://github.com/apache/hudi/issues/1798#issuecomment-658244220
The related code (HoodieROTablePathFilter) does not seem to have any relevant recent changes. @zherenyu831 From the control flow, since Spark deciphers the glob-path, it is first performing the listing of all matching entities and this is where I think it is slower try to list files under .aux. One option to try (for experimentation) is to skip ".hoodie" folder in glob pattern and see if it is faster. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
