umehrot2 commented on issue #1837: URL: https://github.com/apache/hudi/issues/1837#issuecomment-660392013
@bvaradar this was on our radar. I would ideally like to keep the whole **marker optimizations** separate. Also I received certain comments in my PR where folks do not want `hudi-common` to depend on `spark`. This means that we need to move out these listing logic our of `hudi-common` if we want to parallelize it with `spark context`. So lets target this in a separate PR. One of my teammates was interested in taking this up. Let me check with him if he can pick this and if not I will. For the meanwhile, I have assigned it to myself. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
