steveloughran commented on issue #25899: [SPARK-29089][SQL] Parallelize blocking FileSystem calls in DataSource#checkAndGlobPathIfNecessary URL: https://github.com/apache/spark/pull/25899#issuecomment-536993443 > Update: I tried increasing fs.s3a.connection.maximum and it did improve performance of the filesystem calls. 👍 > I still need to set up a benchmark that runs on EC2 instead of remote dev laptop, will update in a couple days. log the toString value of the FS instance at the end to see what the counters say
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
