[ https://issues.apache.org/jira/browse/HADOOP-18146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Steve Loughran resolved HADOOP-18146. ------------------------------------- Fix Version/s: 3.3.9 Resolution: Fixed > ABFS: Add changes for expect hundred continue header with append requests > ------------------------------------------------------------------------- > > Key: HADOOP-18146 > URL: https://issues.apache.org/jira/browse/HADOOP-18146 > Project: Hadoop Common > Issue Type: Sub-task > Components: fs/azure > Affects Versions: 3.3.5 > Reporter: Anmol Asrani > Assignee: Anmol Asrani > Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.9 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Heavy load from a Hadoop cluster lead to high resource utilization at FE > nodes. Investigations from the server side indicate payload buffering at > Http.Sys as the cause. Payload of requests that eventually fail due to > throttling limits are also getting buffered, as its triggered before FE could > start request processing. > Approach: Client sends Append Http request with Expect header, but holds back > on payload transmission until server replies back with HTTP 100. We add this > header for all append requests so as to reduce. > We made several workload runs with and without hundred continue enabled and > the overall observation is that :- > # The ratio of TCP SYN packet count with and without expect hundred continue > enabled is 0.32 : 3 on average. > # The ingress into the machine at TCP level is almost 3 times lesser with > hundred continue enabled which implies a lot of bandwidth save. -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org