[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17350370#comment-17350370 ] Hadoop QA commented on YARN-10324: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 47s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 50s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 18s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 59s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 16s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 5s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 28s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 42s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 20m 53s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are enabled, using SpotBugs. {color} | | {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 2m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s{color} | {color:blue}{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 53s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 37s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.11+9-Ubuntu-0ubuntu2.20.04 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 37s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 4s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_292-8u292-b10-0ubuntu1~20.04-b10 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 4s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 55s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1014/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client.txt{color} | {color:orange} hadoop-mapreduce-project/hadoop-mapreduce-client: The patch generated 40 new + 79 unchanged - 0 fixed = 119 total (was 79) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 55s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349124#comment-17349124 ] Qi Zhu commented on YARN-10324: --- [~yaoguangdong] I'm not sure if you removed the original 003, and resubmitted it ? Waiting for the jenkins now, if it not triggered some hours later, you should attached it again to trigger. > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Assignee: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch, > YARN-10324.003.patch, image-2021-05-21-17-48-03-476.png > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349115#comment-17349115 ] Yao Guangdong commented on YARN-10324: -- [~zhuqi] Thanks for your help. I had submited it. Does it work? > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Assignee: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch, > YARN-10324.003.patch, image-2021-05-21-17-48-03-476.png > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349111#comment-17349111 ] Qi Zhu commented on YARN-10324: --- [~yaoguangdong] You should submitted it, make it patch available, then the jenkins will trigger. With the button: !image-2021-05-21-17-48-03-476.png|width=88,height=45! > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Assignee: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch, > YARN-10324.003.patch, image-2021-05-21-17-48-03-476.png > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17349110#comment-17349110 ] Yao Guangdong commented on YARN-10324: -- [~zhuqi] Thanks for your help.I have added a new patch [^YARN-10324.003.patch]. PTAL > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Assignee: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch, > YARN-10324.003.patch > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17344478#comment-17344478 ] Qi Zhu commented on YARN-10324: --- Hi [~yaoguangdong] Thanks for this work. I have added you to the contributor list. You can submit latest patch to trigger the jenkins. > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154217#comment-17154217 ] Yao Guangdong commented on YARN-10324: -- Fixed cached memory calculate incorrect bug. Add new patch YARN-10324.002.patch > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch, YARN-10324.002.patch > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10324) Fetch data from NodeManager may case read timeout when disk is busy
[ https://issues.apache.org/jira/browse/YARN-10324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17154155#comment-17154155 ] Brahma Reddy Battula commented on YARN-10324: - Updated the target version to 3.3.1 as 3.3.0 is going to release. > Fetch data from NodeManager may case read timeout when disk is busy > --- > > Key: YARN-10324 > URL: https://issues.apache.org/jira/browse/YARN-10324 > Project: Hadoop YARN > Issue Type: Improvement > Components: auxservices >Affects Versions: 2.7.0, 3.2.1 >Reporter: Yao Guangdong >Priority: Minor > Labels: patch > Attachments: YARN-10324.001.patch > > > With the cluster size become more and more big.The cost time on Reduce > fetch Map's result from NodeManager become more and more long.We often see > the WARN logs in the reduce's logs as follow. > {quote}2020-06-19 15:43:15,522 WARN [fetcher#8] > org.apache.hadoop.mapreduce.task.reduce.Fetcher: Failed to connect to > TX-196-168-211.com:13562 with 5 map outputs > java.net.SocketTimeoutException: Read timed out > at java.net.SocketInputStream.socketRead0(Native Method) > at java.net.SocketInputStream.socketRead(SocketInputStream.java:116) > at java.net.SocketInputStream.read(SocketInputStream.java:171) > at java.net.SocketInputStream.read(SocketInputStream.java:141) > at java.io.BufferedInputStream.fill(BufferedInputStream.java:246) > at java.io.BufferedInputStream.read1(BufferedInputStream.java:286) > at java.io.BufferedInputStream.read(BufferedInputStream.java:345) > at sun.net.www.http.HttpClient.parseHTTPHeader(HttpClient.java:735) > at sun.net.www.http.HttpClient.parseHTTP(HttpClient.java:678) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1587) > at > sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1492) > at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:480) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.verifyConnection(Fetcher.java:434) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.setupConnectionsWithRetry(Fetcher.java:400) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.openShuffleUrl(Fetcher.java:271) > at > org.apache.hadoop.mapreduce.task.reduce.Fetcher.copyFromHost(Fetcher.java:330) > at org.apache.hadoop.mapreduce.task.reduce.Fetcher.run(Fetcher.java:198) > {quote} > We check the NodeManager server find that the disk IO util and connections > became very high when the read timeout happened.We analyze that if we have > 20,000 maps and 1,000 reduces which will make NodeManager generate 20 million > times IO stream operate in the shuffle phase.If the reduce fetch data size is > very small from map output files.Which make the disk IO util become very high > in big cluster.Then read timeout happened frequently.The application finished > time become longer. > We find ShuffleHandler have IndexCache for cache file.out.index file.Then we > want to change the small IO to big IO which can reduce the small disk IO > times. So we try to cache all the small file data(file.out) in memory when > the first fetch request come.Then the others fetch request only need read > data from memory avoid disk IO operation.After we cache data to memory we > find the read timeout disappeared. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org