[jira] [Commented] (YARN-11094) Follow up changes for YARN-10547
[
https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510927#comment-17510927
]
Hadoop QA commented on YARN-11094:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
22s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 54s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 19m
26s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 0m
46s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
22s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 21s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1282/artifact/out/diff-compile-javac-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red}
hadoop-tools_hadoop-sls-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 0 unchanged - 1 fixed
= 1 total (was 1) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1282/artifact/out/diff-compile-javac-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt{color}
| {color:red}
hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new +
0 unchanged - 1 fixed = 1 total (was 1) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1282/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt{color}
| {color:orange} hadoop-tools/hadoop-sls: The patch generated 41 new + 43
unchanged - 4 fixed = 84 total (was 47) {color} |
| {color:green}+1{co
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[
https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11094:
--
Attachment: YARN-11094.001.patch
> Follow up changes of YARN-10547
> ---
>
> Key: YARN-11094
> URL: https://issues.apache.org/jira/browse/YARN-11094
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler-load-simulator
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-11094.001.patch
>
>
> As a follow up of YARN-10547, we need to fix the raw use of Map across the
> following classes:
> - AMDefinitionSLS
> - SLSRunner
> See [this
> comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
> for details.
> The following methods are affected:
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
> - Methods of TaskContainerDefinition.Builder
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
> - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
> Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
> 1. Redundant @SuppressWarnings("unchecked") annotations
> 2. Explicit type arguments for some collections
> 3. Remove exceptions from methods that never throwing them
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes for YARN-10547
[
https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11094:
--
Summary: Follow up changes for YARN-10547 (was: Follow up changes of
YARN-10547)
> Follow up changes for YARN-10547
>
>
> Key: YARN-11094
> URL: https://issues.apache.org/jira/browse/YARN-11094
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler-load-simulator
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-11094.001.patch
>
>
> As a follow up of YARN-10547, we need to fix the raw use of Map across the
> following classes:
> - AMDefinitionSLS
> - SLSRunner
> See [this
> comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
> for details.
> The following methods are affected:
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
> - Methods of TaskContainerDefinition.Builder
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
> - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
> Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
> 1. Redundant @SuppressWarnings("unchecked") annotations
> 2. Explicit type arguments for some collections
> 3. Remove exceptions from methods that never throwing them
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[
https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11094:
--
Description:
As a follow up of YARN-10547, we need to fix the raw use of Map across the
following classes:
- AMDefinitionSLS
- SLSRunner
See [this
comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
for details.
The following methods are affected:
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
- org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
- Methods of TaskContainerDefinition.Builder
- org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
- org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
1. Redundant @SuppressWarnings("unchecked") annotations
2. Explicit type arguments for some collections
3. Remove exceptions from methods that never throwing them
was:
As a follow up of YARN-10547, we need to fix the raw use of Map across the
following classes:
- AMDefinitionSLS
- SLSRunner
See [this
comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
for details.
The following methods are affected:
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
- org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
- Methods of TaskContainerDefinition.Builder
- org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
- org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
1. Redundant @SuppressWarnings("unchecked") annotations
2. Explicit type arguments for some collections
> Follow up changes of YARN-10547
> ---
>
> Key: YARN-11094
> URL: https://issues.apache.org/jira/browse/YARN-11094
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler-load-simulator
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
>
> As a follow up of YARN-10547, we need to fix the raw use of Map across the
> following classes:
> - AMDefinitionSLS
> - SLSRunner
> See [this
> comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
> for details.
> The following methods are affected:
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
> - Methods of TaskContainerDefinition.Builder
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
> - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
> Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
> 1. Redundant @SuppressWarnings("unchecked") annotations
> 2. Explicit type arguments for some collections
> 3. Remove exceptions from methods that never throwing them
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[
https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Szilard Nemeth updated YARN-11094:
--
Description:
As a follow up of YARN-10547, we need to fix the raw use of Map across the
following classes:
- AMDefinitionSLS
- SLSRunner
See [this
comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
for details.
The following methods are affected:
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
- org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
- Methods of TaskContainerDefinition.Builder
- org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
- org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
1. Redundant @SuppressWarnings("unchecked") annotations
2. Explicit type arguments for some collections
was:
As a follow up of YARN-10547, we need to fix the raw use of Map across the
following classes:
- AMDefinitionSLS
- SLSRunner
See [this
comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
for details.
The following methods are affected:
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
- org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
- org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
- Methods of TaskContainerDefinition.Builder
- org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
- org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
> Follow up changes of YARN-10547
> ---
>
> Key: YARN-11094
> URL: https://issues.apache.org/jira/browse/YARN-11094
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: scheduler-load-simulator
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
>
> As a follow up of YARN-10547, we need to fix the raw use of Map across the
> following classes:
> - AMDefinitionSLS
> - SLSRunner
> See [this
> comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839]
> for details.
> The following methods are affected:
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace
> - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers
> - Methods of TaskContainerDefinition.Builder
> - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create
> - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace
> Moreover, some generic cleanup in SLSRunner could be performed to eliminate:
> 1. Redundant @SuppressWarnings("unchecked") annotations
> 2. Explicit type arguments for some collections
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[ https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-11094: -- Description: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. The following methods are affected: - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers - Methods of TaskContainerDefinition.Builder - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace was: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. The following methods are affected: - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS - Methods of TaskContainerDefinition.Builder - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace > Follow up changes of YARN-10547 > --- > > Key: YARN-11094 > URL: https://issues.apache.org/jira/browse/YARN-11094 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > > As a follow up of YARN-10547, we need to fix the raw use of Map across the > following classes: > - AMDefinitionSLS > - SLSRunner > See [this > comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] > for details. > The following methods are affected: > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS > - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers > - Methods of TaskContainerDefinition.Builder > - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create > - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[ https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-11094: -- Description: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. The following methods are affected: - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS - Methods of TaskContainerDefinition.Builder - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace was: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. The following methods are affected: - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace > Follow up changes of YARN-10547 > --- > > Key: YARN-11094 > URL: https://issues.apache.org/jira/browse/YARN-11094 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > > As a follow up of YARN-10547, we need to fix the raw use of Map across the > following classes: > - AMDefinitionSLS > - SLSRunner > See [this > comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] > for details. > The following methods are affected: > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS > - Methods of TaskContainerDefinition.Builder > - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers > - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create > - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10547) Decouple job parsing logic from SLSRunner
[
https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510903#comment-17510903
]
Hadoop QA commented on YARN-10547:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Logfile || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
32s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} || ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} No case conflicting files
found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green}{color} | {color:green} The patch does not contain any
@author tags. {color} |
| {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color}
| {color:green}test4tests{color} | {color:green} The patch appears to include 2
new or modified test files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 36m
18s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
31s{color} | {color:green}{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 16s{color} | {color:green}{color} | {color:green} branch has no errors when
building and testing our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green}{color} | {color:green} trunk passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
23s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private
Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 19m
52s{color} | {color:blue}{color} | {color:blue} Both FindBugs and SpotBugs are
enabled, using SpotBugs. {color} |
| {color:green}+1{color} | {color:green} spotbugs {color} | {color:green} 0m
47s{color} | {color:green}{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} || ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
21s{color} | {color:green}{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 22s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1281/artifact/out/diff-compile-javac-hadoop-tools_hadoop-sls-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04.txt{color}
| {color:red}
hadoop-tools_hadoop-sls-jdkUbuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 with JDK
Ubuntu-11.0.14+9-Ubuntu-0ubuntu2.20.04 generated 1 new + 0 unchanged - 1 fixed
= 1 total (was 1) {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
19s{color} | {color:green}{color} | {color:green} the patch passed with JDK
Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 19s{color}
|
{color:red}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1281/artifact/out/diff-compile-javac-hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07.txt{color}
| {color:red}
hadoop-tools_hadoop-sls-jdkPrivateBuild-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07
with JDK Private Build-1.8.0_312-8u312-b07-0ubuntu1~20.04-b07 generated 1 new +
0 unchanged - 1 fixed = 1 total (was 1) {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 15s{color} |
{color:orange}https://ci-hadoop.apache.org/job/PreCommit-YARN-Build/1281/artifact/out/diff-checkstyle-hadoop-tools_hadoop-sls.txt{color}
| {color:orange} hadoop-tools/hadoop-sls: The patch generated 40 new + 45
unchanged - 2 fixed = 85 total (was 47) {color} |
| {color:green}+1{co
[jira] [Comment Edited] (YARN-10547) Decouple job parsing logic from SLSRunner
[
https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510839#comment-17510839
]
Szilard Nemeth edited comment on YARN-10547 at 3/22/22, 9:14 PM:
-
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
I do suggest a follow-up jira for this cleanup.
3. This falls into the same category as 2., so the same follow-up jira could
clean this whole "typelessness" thingy up.
was (Author: snemeth):
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
I do suggest a follow-up jira for this cleanup.
3. This falls into the same category as 2., so the same follow-up jira could
clean this whole "typelessness" thingy up.
> Decouple job parsing logic from SLSRunner
> -
>
> Key: YARN-10547
> URL: https://issues.apache.org/jira/browse/YARN-10547
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-10547.001.patch, YARN-10547.002.patch,
> YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch,
> YARN-10547.006.patch
>
>
> SLSRunner has too many responsibilities.
> One of them is to parse the job details from the SLS input formats and launch
> the AMs and task containers.
> As a first step, the job parser logic could be decoupled from this class.
> There are 3 types of inputs:
> - SLS trace
> - Synth
> - Rumen
> Their job parsing method are:
> - SLS trace:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526
> - Synth:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790
> - Rumen:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11094) Follow up changes of YARN-10547
[ https://issues.apache.org/jira/browse/YARN-11094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-11094: -- Description: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. The following methods are affected: - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace was: As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. > Follow up changes of YARN-10547 > --- > > Key: YARN-11094 > URL: https://issues.apache.org/jira/browse/YARN-11094 > Project: Hadoop YARN > Issue Type: Improvement > Components: scheduler-load-simulator >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > > As a follow up of YARN-10547, we need to fix the raw use of Map across the > following classes: > - AMDefinitionSLS > - SLSRunner > See [this > comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] > for details. > The following methods are affected: > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#createFromSlsTrace > - org.apache.hadoop.yarn.sls.AMDefinitionFactory#getAMContainerResourceSLS > - org.apache.hadoop.yarn.sls.AMDefinitionSLS#getTaskContainers > - org.apache.hadoop.yarn.sls.AMDefinitionSLS.Builder#create > - org.apache.hadoop.yarn.sls.SLSRunner#startAMFromSLSTrace -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11094) Follow up changes of YARN-10547
Szilard Nemeth created YARN-11094: - Summary: Follow up changes of YARN-10547 Key: YARN-11094 URL: https://issues.apache.org/jira/browse/YARN-11094 Project: Hadoop YARN Issue Type: Improvement Components: scheduler-load-simulator Reporter: Szilard Nemeth Assignee: Szilard Nemeth As a follow up of YARN-10547, we need to fix the raw use of Map across the following classes: - AMDefinitionSLS - SLSRunner See [this comment|https://issues.apache.org/jira/browse/YARN-10547?focusedCommentId=17510839&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17510839] for details. -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10547) Decouple job parsing logic from SLSRunner
[ https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Szilard Nemeth updated YARN-10547: -- Attachment: YARN-10547.006.patch > Decouple job parsing logic from SLSRunner > - > > Key: YARN-10547 > URL: https://issues.apache.org/jira/browse/YARN-10547 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-10547.001.patch, YARN-10547.002.patch, > YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch, > YARN-10547.006.patch > > > SLSRunner has too many responsibilities. > One of them is to parse the job details from the SLS input formats and launch > the AMs and task containers. > As a first step, the job parser logic could be decoupled from this class. > There are 3 types of inputs: > - SLS trace > - Synth > - Rumen > Their job parsing method are: > - SLS trace: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526 > - Synth: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790 > - Rumen: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10547) Decouple job parsing logic from SLSRunner
[
https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510839#comment-17510839
]
Szilard Nemeth edited comment on YARN-10547 at 3/22/22, 7:16 PM:
-
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
I do suggest a follow-up jira for this cleanup.
3. This falls into the same category as 2., so the same follow-up jira could
clean this whole "typelessness" thingy up.
was (Author: snemeth):
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
I do suggest a follow-up jira for this cleanup.
> Decouple job parsing logic from SLSRunner
> -
>
> Key: YARN-10547
> URL: https://issues.apache.org/jira/browse/YARN-10547
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-10547.001.patch, YARN-10547.002.patch,
> YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch
>
>
> SLSRunner has too many responsibilities.
> One of them is to parse the job details from the SLS input formats and launch
> the AMs and task containers.
> As a first step, the job parser logic could be decoupled from this class.
> There are 3 types of inputs:
> - SLS trace
> - Synth
> - Rumen
> Their job parsing method are:
> - SLS trace:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526
> - Synth:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790
> - Rumen:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10547) Decouple job parsing logic from SLSRunner
[
https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510839#comment-17510839
]
Szilard Nemeth edited comment on YARN-10547 at 3/22/22, 7:15 PM:
-
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
I do suggest a follow-up jira for this cleanup.
was (Author: snemeth):
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
> Decouple job parsing logic from SLSRunner
> -
>
> Key: YARN-10547
> URL: https://issues.apache.org/jira/browse/YARN-10547
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-10547.001.patch, YARN-10547.002.patch,
> YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch
>
>
> SLSRunner has too many responsibilities.
> One of them is to parse the job details from the SLS input formats and launch
> the AMs and task containers.
> As a first step, the job parser logic could be decoupled from this class.
> There are 3 types of inputs:
> - SLS trace
> - Synth
> - Rumen
> Their job parsing method are:
> - SLS trace:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526
> - Synth:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790
> - Rumen:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10547) Decouple job parsing logic from SLSRunner
[
https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510839#comment-17510839
]
Szilard Nemeth commented on YARN-10547:
---
Hi [~quapaw],
Thanks for your review.
Fixed the folllowing:
1. SLSRunner#L488: Checked exception is not thrown.
2. As I only moved this part of the code, I don't think it's worth to do a
refactor on top of moving the code here. The raw type of map could be fixed
easily with Jackson type references:
https://stackoverflow.com/questions/6846244/jackson-and-generic-type-reference
However, as the unparameterized Map is passed along to many places, the changes
would cascade too deeply, so the fix won't be that simple.
See the method definition and all the called methods that receive the Map:
{code}
public static AMDefinitionSLS createFromSlsTrace(Map jsonJob,
SLSRunner slsRunner) throws YarnException {
{code}
> Decouple job parsing logic from SLSRunner
> -
>
> Key: YARN-10547
> URL: https://issues.apache.org/jira/browse/YARN-10547
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Szilard Nemeth
>Assignee: Szilard Nemeth
>Priority: Minor
> Attachments: YARN-10547.001.patch, YARN-10547.002.patch,
> YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch
>
>
> SLSRunner has too many responsibilities.
> One of them is to parse the job details from the SLS input formats and launch
> the AMs and task containers.
> As a first step, the job parser logic could be decoupled from this class.
> There are 3 types of inputs:
> - SLS trace
> - Synth
> - Rumen
> Their job parsing method are:
> - SLS trace:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526
> - Synth:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790
> - Rumen:
> https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11093) In fs-support-append false case, timelineserver reads event files in arbitrary order
[ https://issues.apache.org/jira/browse/YARN-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11093: -- Labels: pull-request-available (was: ) > In fs-support-append false case, timelineserver reads event files in > arbitrary order > > > Key: YARN-11093 > URL: https://issues.apache.org/jira/browse/YARN-11093 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.2 >Reporter: shailesh gupta >Priority: Critical > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > In our setup, we are using Hive and Tez and using offline tez-ui mode, where > we copy the ATS events files to other place and start tez-ui. To keep the > event files small, we are using > fs-support-append = false, and creating a new file for writing the events. We > can see that, in this mode, file name contains > [suffix|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382] > as timestamp. > But at the time of > [read|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844], > we are relying on the FS to provide correct file order. If that fails, we > are adding event out of order, leading to event being discarded, or incorrect > information. > Fix could be sorting of the file names, based on suffix if append mode is not > used. > sample file names: > * summarylog-appattempt_1647348120288_0001_01_460237 > * > entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147 > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11093) In fs-support-append false case, timelineserver reads event files in arbitrary order
[ https://issues.apache.org/jira/browse/YARN-11093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] shailesh gupta updated YARN-11093: -- Summary: In fs-support-append false case, timelineserver reads event files in arbitrary order (was: In fs-support-append false case, server reads event files in arbitrary order) > In fs-support-append false case, timelineserver reads event files in > arbitrary order > > > Key: YARN-11093 > URL: https://issues.apache.org/jira/browse/YARN-11093 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.3.2 >Reporter: shailesh gupta >Priority: Critical > > In our setup, we are using Hive and Tez and using offline tez-ui mode, where > we copy the ATS events files to other place and start tez-ui. To keep the > event files small, we are using > fs-support-append = false, and creating a new file for writing the events. We > can see that, in this mode, file name contains > [suffix|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382] > as timestamp. > But at the time of > [read|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844], > we are relying on the FS to provide correct file order. If that fails, we > are adding event out of order, leading to event being discarded, or incorrect > information. > Fix could be sorting of the file names, based on suffix if append mode is not > used. > sample file names: > * summarylog-appattempt_1647348120288_0001_01_460237 > * > entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147 > -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11093) In fs-support-append false case, server reads event files in arbitrary order
shailesh gupta created YARN-11093: - Summary: In fs-support-append false case, server reads event files in arbitrary order Key: YARN-11093 URL: https://issues.apache.org/jira/browse/YARN-11093 Project: Hadoop YARN Issue Type: Bug Components: timelineserver Affects Versions: 3.3.2 Reporter: shailesh gupta In our setup, we are using Hive and Tez and using offline tez-ui mode, where we copy the ATS events files to other place and start tez-ui. To keep the event files small, we are using fs-support-append = false, and creating a new file for writing the events. We can see that, in this mode, file name contains [suffix|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L382] as timestamp. But at the time of [read|https://github.com/apache/hadoop/blob/1d5650c4d0acf33f141d593762682b3603523104/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-timeline-pluginstorage/src/main/java/org/apache/hadoop/yarn/server/timeline/EntityGroupFSTimelineStore.java#L844], we are relying on the FS to provide correct file order. If that fails, we are adding event out of order, leading to event being discarded, or incorrect information. Fix could be sorting of the file names, based on suffix if append mode is not used. sample file names: * summarylog-appattempt_1647348120288_0001_01_460237 * entitylog-timelineEntityGroupId_1647348120288_1_dag_1647348120288_0001_1_673147 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10547) Decouple job parsing logic from SLSRunner
[ https://issues.apache.org/jira/browse/YARN-10547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17510442#comment-17510442 ] András Győri commented on YARN-10547: - Only had a few comments on this one, I will commit it if you fix them: * SLSRunner#L488: Checked exception is not thrown * SLSRunner#L464: Raw use of Map * AMDefinitionSLS#L41: Unchecked conversion, this could easily fail. In case of conversion exception, I would default it to an empty list, because it is already checked in the next line properly > Decouple job parsing logic from SLSRunner > - > > Key: YARN-10547 > URL: https://issues.apache.org/jira/browse/YARN-10547 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Szilard Nemeth >Priority: Minor > Attachments: YARN-10547.001.patch, YARN-10547.002.patch, > YARN-10547.003.patch, YARN-10547.004.patch, YARN-10547.005.patch > > > SLSRunner has too many responsibilities. > One of them is to parse the job details from the SLS input formats and launch > the AMs and task containers. > As a first step, the job parser logic could be decoupled from this class. > There are 3 types of inputs: > - SLS trace > - Synth > - Rumen > Their job parsing method are: > - SLS trace: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L479-L526 > - Synth: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L722-L790 > - Rumen: > https://github.com/apache/hadoop/blob/005b854f6bad66defafae0abf95dabc6c36ca8b1/hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/SLSRunner.java#L651-L716 -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
