steveloughran commented on pull request #3289: URL: https://github.com/apache/hadoop/pull/3289#issuecomment-908567988
This is at a point where it's ready for some review and any benchmarking people can do. I've cut out a lot of HTTP IO per file create and load @mukund-thakur @mehakmeet @dongjoon-hyun @bogthe The only other big thing to consider is could we route the parallel POST calls in job commit through a fork/join thread pool and does that deliver better throughput due to the fact there's no need to yield to the OS scheduler to pick up the next bit of work. I am happy to do a live shared screen review of this PR next week, if people want do discuss things that way -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
