snvijaya opened a new pull request #2509: URL: https://github.com/apache/hadoop/pull/2509
When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended. If the data size written by Hadoop app is small, i.e. data size : before any of HFlush/HSync call is made or between 2 HFlush/Hsync API calls is less than write buffer size, 2 separate calls, one for append and another for flush is made, Apps that do such small writes eventually end up with almost similar number of calls for flush and append. When Hflush or Hsync APIs are called, a call is made to store backend to commit the data that was appended. If the data size written by Hadoop app is small, i.e. data size : before any of HFlush/HSync call is made or between 2 HFlush/Hsync API calls is less than write buffer size, 2 separate calls, one for append and another for flush is made, Apps that do such small writes eventually end up with almost similar number of calls for flush and append. This commit enables Flush to be piggybacked onto append call for such short write scenarios. This is guarded with config "fs.azure.write.enableappendwithflush" which is set to off by default as it needs a relevant change in backend to propogate. Tests asserting number of requests made, request data sizes, file sizes post append+flush and file content checks for various combinations of append/flush/close sets with and without the small write optimization is added. Existing tests in ITestAbfsNetworkStatistics asserting Http stats were rewritten for easy readability. (Test results published in end of PR conversation tab.) ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
