jeeteshm commented on a change in pull request #1611: HADOOP-16612 Track Azure Blob File System client-perceived latency URL: https://github.com/apache/hadoop/pull/1611#discussion_r339288731
########## File path: hadoop-tools/hadoop-azure/src/site/markdown/abfs.md ########## @@ -661,6 +661,47 @@ Hflush() being the only documented API that can provide persistent data transfer, Flush() also attempting to persist buffered data will lead to performance issues. +### <a name="perfoptions"></a> Perf Options + +#### <a name="abfstracklatencyoptions"></a> 1. HTTP Request Tracking Options +If you set `fs.azure.abfs.latency.track` to `true`, the module starts tracking the performance metrics of ABFS HTTP +traffic. To obtain these numbers on your machine or cluster, you will also need to enable +debug logging for the `AbfsPerfTracker` class in your `log4j` config. A typical perf log line appears like: + +``` +h=KARMA t=2019-10-25T20:21:14.518Z a=abfstest01.dfs.core.windows.net +c=abfs-testcontainer-84828169-6488-4a62-a875-1e674275a29f cr=delete ce=deletePath r=Succeeded l=32 ls=32 lc=1 s=200 +e= ci=95121dae-70a8-4187-b067-614091034558 ri=97effdcf-201f-0097-2d71-8bae00000000 ct=0 st=0 rt=0 bs=0 br=0 m=DELETE +u=https%3A%2F%2Fabfstest01.dfs.core.windows.net%2Fabfs-testcontainer%2Ftest%3Ftimeout%3D90%26recursive%3Dtrue +``` + +The fields have the following definitions: + +`h`: host name +`t`: time when this request was logged +`a`: Azure storage account name +`c`: container name +`cr`: name of the caller method +`ce`: name of the callee method +`r`: result (Succeeded/Failed) +`l`: latency (time spend in callee) Review comment: For l and lc, will fix the grammar in the next update. For ls and lc, will add "logged with the last callee" in the next update. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
