[GitHub] [hudi] vinothchandar edited a comment on issue #1829: [SUPPORT] S3 slow file listing causes Hudi read performance.

[GitBox]

vinothchandar edited a comment on issue #1829:
URL: https://github.com/apache/hudi/issues/1829#issuecomment-658821557


   @zuyanton this seems like a general issue with `FileInputFormat` 
   
   ```
    int numThreads = job
           .getInt(
               
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.LIST_STATUS_NUM_THREADS,
               
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.DEFAULT_LIST_STATUS_NUM_THREADS);
   ```
   
   can you try adding 
`spark.hadoop.mapreduce.input.fileinputformat.list-status.num-threads=8` or 
something to sparkConf and see if helps? (default inside Hadoop is 1) 
   
   cc @n3nash IIRC you mentioned a similar approach done at uber?
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org