[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16452110#comment-16452110
]
Sunil G commented on MAPREDUCE-7042:
+1 on the path. If no objections , i ll commit it later today
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch,
> MAPREDUCE-7042.3.patch, MAPREDUCE-7042.4.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451944#comment-16451944
]
Hadoop QA commented on MAPREDUCE-7042:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
36s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
41s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 22s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 3 new + 356 unchanged - 0 fixed = 359 total (was 356)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 9m
9s{color} | {color:green} hadoop-mapreduce-client-app in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
24s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 13s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | MAPREDUCE-7042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12920597/MAPREDUCE-7042.4.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 4c1fba138aa8 3.13.0-139-generic #188-Ubuntu SMP Tue Jan 9
14:43:09 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 69e1e6a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_162 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7400/artifact/out/diff-checkstyle-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/7400/testReport/ |
| Max. process+thread count | 584 (vs. ulimit of 1) |
| modules | C:
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U:
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451845#comment-16451845
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
Updated patch fixing test failure.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch,
> MAPREDUCE-7042.3.patch, MAPREDUCE-7042.4.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16451700#comment-16451700
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
[~xgong] shall I take over this?
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch,
> MAPREDUCE-7042.3.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16445409#comment-16445409
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
Many tests are failing, looks we also need to change tests since many tests
verifies ats data published.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch,
> MAPREDUCE-7042.3.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443867#comment-16443867
]
Hadoop QA commented on MAPREDUCE-7042:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 10m
58s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
24s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
38s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 19s{color} | {color:orange}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app:
The patch generated 1 new + 301 unchanged - 0 fixed = 302 total (was 301)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 17s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
18s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 15s{color}
| {color:red} hadoop-mapreduce-client-app in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 68m 39s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.mapreduce.v2.app.TestMRAppComponentDependencies |
| | hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler |
| | hadoop.mapreduce.v2.app.TestMRAppMaster |
| | hadoop.mapreduce.v2.app.TestFetchFailure |
| | hadoop.mapreduce.v2.app.TestStagingCleanup |
| | hadoop.mapreduce.v2.app.TestRecovery |
| | hadoop.mapreduce.v2.app.TestAMInfos |
| | hadoop.mapreduce.v2.app.TestMRApp |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8620d2b |
| JIRA Issue | MAPREDUCE-7042 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12919755/MAPREDUCE-7042.3.patch
|
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 052ca8024cbd 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16443598#comment-16443598
]
Xuan Gong commented on MAPREDUCE-7042:
--
[~rohithsharma]
Address the latest comment
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch,
> MAPREDUCE-7042.3.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442587#comment-16442587
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
Patch looks great.. one nit
# Method handleTimelineEvent has two if check. First if check {{if
(timelineV2Client !=null || timelineClient != null)}} could be removed.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441275#comment-16441275
]
Xuan Gong commented on MAPREDUCE-7042:
--
bq. Could we use AsyncDispatcher instead of implementing own dispatcher?
Make sense.
Uploaded a new patch which use AsyncDispatcher
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch, MAPREDUCE-7042.2.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439087#comment-16439087
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
bq. When we got the JHS event, we push the event to JHS event queue.
Could we use AsyncDispatcher instead of implementing own dispatcher?
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16438029#comment-16438029
]
Xuan Gong commented on MAPREDUCE-7042:
--
[~rohithsharma]
The basic workflow will be:
* When we got the JHS event, we push the event to JHS event queue.
* We have separate JHS event handler thread periodically pick and handle the
event from JHS event queue.
* When we start to handle the event, if ATS is enabled, we could push the
event immediately into ATS event queue.
* We have separate ATS event handler thread periodically pick and handle the
event from ATS event queue.
To separate JHS event and ATS event here, I think that I need two different
queues and handler threads. So, when we handle the event, we can simply push
the event into ATS event queue and continue to handle this as JHS event. So,
even there is anything wrong with ATS, it would not block the process of
handling JHS event.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435387#comment-16435387
]
Rohith Sharma K S commented on MAPREDUCE-7042:
--
Thanks [~xgong] for the patch. Definitely I agree that we should dispatch it
separate thread to minimize the impact.
I see that patch is creating a new dispatcher. How about using AsyncDispatcher
in JHS and registering JobHistoryEvent type. With this, we can have simple
event handler code as current patch is doing. Using AsyncDispatcher minimize
the creating of new thread with runnable implementation. Almost very similar to
RMStateStore implementation.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16434657#comment-16434657
]
Xuan Gong commented on MAPREDUCE-7042:
--
The goal is that we handle the JHS event and ATS event separately. So if there
is anything wrong while we handle the ats event, it should not block the
process of handling JHS event.
The solution we used here is:
* create an additional queue for the timeline event
* In JHSEventHandlingThread, we simply put the timeline event to this
additional timeline event queue.
* Create a separate eventhandlingThread to handle the timeline event
[~rohithsharma] Could you review this, please?
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
> Attachments: MAPREDUCE-7042.1.patch
>
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430022#comment-16430022
]
Xuan Gong commented on MAPREDUCE-7042:
--
Here is what really happens:
1) We send a kill signal to AM, and AM starts to stop gracefully. The first
service which AM starts to stop is JobHistoryEventHandler.
2) In JobHistoryEventHandler, we handle the ATS events and JHS events together.
Right now, the event handler grabs the lock and tries to publish ATS entity to
ATS v2. But due to the Connection issue (whatever issue which causes the
SocketTimeoutException), the event handler gets blocked. So, the whole process
which stops JobHistoryEventHandler gets blocked as well.
3) When we send a kill signal to AM, we set a
"yarn.app.mapreduce.am.hard-kill-timeout-ms" (by default 10s). After 10s
waiting, it would send a kill signal to RM, and do the forced kill for this
application.
4) The event handler still gets blocked, so it does not get a chance to handle
the rest of the JHS events before it gets forced killed.
So, we can not find anything for this application in JHS.
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7042) Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when ATS v2 is enabled
[
https://issues.apache.org/jira/browse/MAPREDUCE-7042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16430023#comment-16430023
]
Xuan Gong commented on MAPREDUCE-7042:
--
To solve this issue, we should separate JobHistory Handling from ATS
> Killed mapreduce job data does not move to mapreduce.jobhistory.done-dir when
> ATS v2 is enabled
> ---
>
> Key: MAPREDUCE-7042
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7042
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Yesha Vora
>Assignee: Xuan Gong
>Priority: Major
>
> Steps:
> 1) Start a mapreduce job
> {code}
> hadoop jar
> /usr/hdp/current/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient-3.0.0.3.0.0.0-751-tests.jar
> sleep "-Dmapreduce.job.user.name=hrt_qa" -m 10 -r 1 -mt 1000 -rt
> 1000{code}
> 2) kill job
> 3) Validate job is present at mapreduce.jobhistory.done-dir
> mapreduce.jobhistory.done-dir dir does not have job_1516776025831_0048 entry.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
