Github user ScrapCodes commented on the issue:
https://github.com/apache/spark/pull/22339
For numbers, while testing with object store having 50 files/dirs, without
this patch it took 130 REST requests for 2 batches to complete and with this
patch it took 56 rest requests. So number of rest calls are reduced, and this
translates to speedup. How much speed up is dependent on number of files, but
for the particular test, I have run, it was 2x.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
