pengzhiwei2018 commented on pull request #2651: URL: https://github.com/apache/hudi/pull/2651#issuecomment-796392798
> and > Thanks @pengzhiwei2018 for your effort and contributions around Spark SQL. I took a quick glance, and in general I agree with the idea of have our in-house Spark index implementation that opens the doors for other features we want to build. However, I see a number of high level concerns right now. > > This is affecting a critical code path, and it might be better to take this step by step. Can we first try to support `HoodieFileIndex` with the existing behavior Hudi has of supporting globbed paths ?. Once we have that, as next step we can build on it to also support just `basePath` based listing and partition pruning which is also needed for your Spark SQL Support. Sounds good! I am agree with you that first support both behavior. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
