[jira] [Commented] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config
[
https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237037#comment-17237037
]
Hadoop QA commented on MAPREDUCE-7309:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
33s{color} | | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | | {color:green} No case conflicting files found. {color} |
| {color:blue}0{color} | {color:blue} codespell {color} | {color:blue} 0m
1s{color} | | {color:blue} codespell was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | | {color:green} The patch does not contain any @author tags.
{color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | | {color:green} The patch appears to include 1 new or modified
test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
14s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
39s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
33s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
27s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
37s{color} | | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 14s{color} | | {color:green} branch has no errors when building and
testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | | {color:green} trunk passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
28s{color} | | {color:green} trunk passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m
59s{color} | | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
29s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | | {color:green} the patch passed with JDK Private
Build-1.8.0_272-8u272-b10-0ubuntu1~18.04-b10 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} blanks {color} | {color:green} 0m
0s{color} | | {color:green} The patch has no blanks issues. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 20s{color} |
[/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt|https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/37/artifact/out/results-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt]
| {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 2 new + 244 unchanged - 0 fixed = 246 total (was 244)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 51s{color} | | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | | {color:green} the patch passed with JDK
Ubuntu-11.0.9+11-Ubuntu-0ubuntu1.18.04.1 {color} |
| {color:green}+1{color} | {color:green}
[jira] [Commented] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config
[ https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17237031#comment-17237031 ] Peter Bacsko commented on MAPREDUCE-7309: - Thanks for the patch [~wangda], but I feel it's maybe too complicated. For the sake of simplicity and easier backport, I suggest a lighter approach in patch v3. > Improve performance of reading resource request for mapper/reducers from > config > --- > > Key: MAPREDUCE-7309 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0 >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309.001.patch, > MAPREDUCE-7309.002.patch > > > This is an issue could affect all the releases which includes YARN-6927. > Basically, we use regex match repeatly when we read mapper/reducer resource > request from config files. When we have large config file, and large number > of splits, it could take a long time. > We saw AM could take hours to parse config when we have 200k+ splits, with a > large config file (hundreds of kbs). > We should do proper caching for pre-configured resource requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7309) Improve performance of reading resource request for mapper/reducers from config
[ https://issues.apache.org/jira/browse/MAPREDUCE-7309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Bacsko updated MAPREDUCE-7309: Attachment: MAPREDUCE-7309-003.patch > Improve performance of reading resource request for mapper/reducers from > config > --- > > Key: MAPREDUCE-7309 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7309 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: applicationmaster >Affects Versions: 3.0.0, 3.1.0, 3.2.0, 3.3.0 >Reporter: Wangda Tan >Assignee: Wangda Tan >Priority: Major > Attachments: MAPREDUCE-7309-003.patch, MAPREDUCE-7309.001.patch, > MAPREDUCE-7309.002.patch > > > This is an issue could affect all the releases which includes YARN-6927. > Basically, we use regex match repeatly when we read mapper/reducer resource > request from config files. When we have large config file, and large number > of splits, it could take a long time. > We saw AM could take hours to parse config when we have 200k+ splits, with a > large config file (hundreds of kbs). > We should do proper caching for pre-configured resource requests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
