nsivabalan commented on issue #4846: URL: https://github.com/apache/hudi/issues/4846#issuecomment-1046224183
this is my understanding by looking at the code of interest. will wait to hear from Balaji though. Its more of an optimization. If we don't override, we call into fetchLatestBaseFiles(partitionPath) which in turn calls into fetchAllStoredFileGroups(partitionPath). so this is essentially fetching all file groups for the partition of interest and then deducing from it. where as in our overridden method, we filter directly w/ rocksDB for the partition and fileId of interest and then fetching just the latest from it, which I feel might be faster. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
