[jira] [Commented] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16089458#comment-16089458 ] Sunil G commented on MAPREDUCE-6889: Committing shortly. > Provide an API Job#close to shutdown MR client services. > > > Key: MAPREDUCE-6889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: MAPREDUCE-6889.0001.patch, MAPREDUCE-6889.0002.patch > > > ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer > initialization. If writer is not closed, then there is possibility of OOM see > YARN-5438 fixes closing FS object. > TimelineClient is used by YarnClient. So all the user who uses YarnClient > with ATS1.5 need to stop service properly. Otherwise there is big chance of > FS object leak. > Of course MR uses YARN client submit job. If MR do not stop YarnClient then > there is FS object leak. > JobClient provides a API to stop all these service using *JobClient#close*. > But many MR clients uses *Job* object to submit a job. But do not stop > started services by default. > So, provide a API *Job#close* to shutdown MR client services. This API can be > utilized by caller explicitly to stop service which avoids FS leak. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020998#comment-16020998 ] Sunil G commented on MAPREDUCE-6889: +1 for the patch. Thanks for correct the jira title as well. I could commit tomorrow if there are no objections. > Provide an API Job#close to shutdown MR client services. > > > Key: MAPREDUCE-6889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: MAPREDUCE-6889.0001.patch, MAPREDUCE-6889.0002.patch > > > ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer > initialization. If writer is not closed, then there is possibility of OOM see > YARN-5438 fixes closing FS object. > TimelineClient is used by YarnClient. So all the user who uses YarnClient > with ATS1.5 need to stop service properly. Otherwise there is big chance of > FS object leak. > Of course MR uses YARN client submit job. If MR do not stop YarnClient then > there is FS object leak. > JobClient provides a API to stop all these service using *JobClient#close*. > But many MR clients uses *Job* object to submit a job. But do not stop > started services by default. > So, provide a API *Job#close* to shutdown MR client services. This API can be > utilized by caller explicitly to stop service which avoids FS leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
[
https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017722#comment-16017722
]
Hadoop QA commented on MAPREDUCE-6889:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 14m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 7s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
37s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
27s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 7s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
28s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 1s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 53s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 114m 49s
{color} | {color:red} hadoop-mapreduce-client-jobclient in the patch failed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 147m 38s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:14b5c93 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12868968/MAPREDUCE-6889.0002.patch
|
| JIRA Issue | MAPREDUCE-6889 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 5641169a87da 3.13.0-107-generic #154-Ubuntu SMP Tue Dec 20
09:57:27 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 009b9f3 |
| Default Java | 1.8.0_131 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/6982/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt
|
[jira] [Commented] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012632#comment-16012632 ] Jason Lowe commented on MAPREDUCE-6889: --- I don't know the full history behind the newInstance call. Some software archaeology shows the newInstance change was added during the review rework process for YARN-4696, but even before that the code was explicitly setting the conf key to disable caching for the filesystem. The disabling of the filesystem cache was done partway through the review on YARN-4234 but I could not find a discussion that explained why that addition to the patch was necessary. Pinging [~xgong] and [~gtCarrera9] to see if they know the logic behind avoiding the fs cache in FileSystemTimelineWriter. > Provide an API Job#close to shutdown MR client services. > > > Key: MAPREDUCE-6889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: MAPREDUCE-6889.0001.patch > > > ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer > initialization. If writer is not closed, then there is possibility of OOM see > YARN-5438 fixes closing FS object. > TimelineClient is used by YarnClient. So all the user who uses YarnClient > with ATS1.5 need to stop service properly. Otherwise there is big chance of > FS object leak. > Of course MR uses YARN client submit job. If MR do not stop YarnClient then > there is FS object leak. > JobClient provides a API to stop all these service using *JobClient#close*. > But many MR clients uses *Job* object to submit a job. But do not stop > started services by default. > So, provide a API *Job#close* to shutdown MR client services. This API can be > utilized by caller explicitly to stop service which avoids FS leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6889) Provide an API Job#close to shutdown MR client services.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16012589#comment-16012589 ] Rohith Sharma K S commented on MAPREDUCE-6889: -- Just for curiosity, many folks including me get general doubt that, why FileSystemTimelineWriter did not implement *FileSystem#get()* that uses cache rather than FileSystem#newInstance() which creates new object? One reason I would think Is for making independent TimelineClient so that closing FS object in JVM should not impact TimelineClient. Is my thought right? > Provide an API Job#close to shutdown MR client services. > > > Key: MAPREDUCE-6889 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6889 > Project: Hadoop Map/Reduce > Issue Type: Bug >Reporter: Rohith Sharma K S >Assignee: Rohith Sharma K S > Attachments: MAPREDUCE-6889.0001.patch > > > ATS1.5 uses FileSystemTimelineWriter which creates FS object on every writer > initialization. If writer is not closed, then there is possibility of OOM see > YARN-5438 fixes closing FS object. > TimelineClient is used by YarnClient. So all the user who uses YarnClient > with ATS1.5 need to stop service properly. Otherwise there is big chance of > FS object leak. > Of course MR uses YARN client submit job. If MR do not stop YarnClient then > there is FS object leak. > JobClient provides a API to stop all these service using *JobClient#close*. > But many MR clients uses *Job* object to submit a job. But do not stop > started services by default. > So, provide a API *Job#close* to shutdown MR client services. This API can be > utilized by caller explicitly to stop service which avoids FS leak. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
