umehrot2 commented on issue #1829: URL: https://github.com/apache/hudi/issues/1829#issuecomment-659005038
I think the finding by @zuyanton seems correct. Increasing the `num-threads` will not help because we just set the `basepath` of the table in the `inputpath` of `jobconf`. I believe we will have a good speed up, if instead of `basePath` we set `all the partition paths` in the `inputpath` of `jobconf`, and then increase the `num-threads`. Another thing we can potentially explore is using Spark to perform this listing parallely on the cluster. But this seems like something we should target for `0.6.0` release with `Blocker` priority. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
