steveloughran commented on pull request #2971: URL: https://github.com/apache/hadoop/pull/2971#issuecomment-918305235
rebase & setting up for GCS testing. the latest PR uses openFile(path).withFileStatus(st), so can open directly from a list() call to opening the files; saves one HEAD per manifest load. For a job with 1000 tasks, each generating a single file, that would reduce IO from 1 LIST, 1K HEAD, 1K GET, 1K Rename to the LIST, GET and rename, (+cleanup): shaving off a lot of load -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
