[jira] [Commented] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202893#comment-17202893 ] Hadoop QA commented on MAPREDUCE-7298: -- | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 49s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} {color} | {color:green} 0m 0s{color} | {color:green}test4tests{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 55s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 33s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 21s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 30s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 17m 18s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 0m 49s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 29s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 26s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 26s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 21s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 15s{color} | {color:orange}https://ci-hadoop.apache.org/job/PreCommit-MAPREDUCE-Build/16/artifact/out/diff-checkstyle-hadoop-tools_hadoop-distcp.txt{color} | {color:orange} hadoop-tools/hadoop-distcp: The patch generated 3 new + 30 unchanged - 0 fixed = 33 total (was 30) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 24s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 15m 46s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 22s{color} |
[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated MAPREDUCE-7298: --- Attachment: MAPREDUCE-7298.02.patch Status: Patch Available (was: Open) > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch, MAPREDUCE-7298.02.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated MAPREDUCE-7298: --- Status: Open (was: Patch Available) > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch, MAPREDUCE-7298.02.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202859#comment-17202859 ] Aasha Medhi commented on MAPREDUCE-7298: 1. UT added 2. Description added to Jira [~shashikant] Job will get cleaned only if its a blocking call to run. If its a non blocking call, I don't see a way to get the job reference. How will that get cleaned up? > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202846#comment-17202846 ] Bilwa S T commented on MAPREDUCE-6786: -- Hi [~devaraj] [~Naganarasimha] i would like to work on this. So assigning it . > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated MAPREDUCE-7298: --- Description: Distcp doesn't close the job after the job is completed. This leads to leaked Truststore Reloader Threads. The fix is to close the job once it is complete. job.close internally calls yarnClient.close(), which then calls timelineConnector.serviceStop() . This destroys the sslFactory cleaning up the ReloadingX509TrustManager. Without the patch for each distcp job, a new ReloadingX509TrustManager is created which creates a new thread. These threads are never killed and they remain like that till HS2 is restarted. With the close, the thread will be cleaned up once the job is completed. was:Distcp doesn't close the job after the job is completed. This leads to leaked Truststore Reloader Threads > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202820#comment-17202820 ] Shashikant Banerjee commented on MAPREDUCE-7298: Thanks [~aasha], for working on this. 1) Can you please add a unit test for the case ? 2) Can you also please add some description how did you figure out the problem? > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202819#comment-17202819 ] Hadoop QA commented on MAPREDUCE-7298: -- | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Logfile || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 0s{color} | {color:blue}{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || || | {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} No case conflicting files found. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s{color} | {color:red}{color} | {color:red} The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color} | || || || || {color:brown} trunk Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 36m 34s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 30s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 39s{color} | {color:green}{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 48s{color} | {color:green}{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green}{color} | {color:green} trunk passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s{color} | {color:green}{color} | {color:green} trunk passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m 3s{color} | {color:blue}{color} | {color:blue} Used deprecated FindBugs config; considering switching to SpotBugs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 0s{color} | {color:green}{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || || | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 31s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 27s{color} | {color:green}{color} | {color:green} the patch passed with JDK Ubuntu-11.0.8+10-post-Ubuntu-0ubuntu118.04.1 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 27s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 25s{color} | {color:green}{color} | {color:green} the patch passed with JDK Private Build-1.8.0_265-8u265-b01-0ubuntu2~18.04-b01 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 25s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 29s{color} | {color:green}{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green}{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 18m 29s{color} | {color:green}{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 23s{color} | {color:green}{color} | {color:green} the patch passed with JDK
[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aasha Medhi updated MAPREDUCE-7298: --- Attachment: MAPREDUCE-7298.01.patch Status: Patch Available (was: Open) > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Created] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
Aasha Medhi created MAPREDUCE-7298: -- Summary: Distcp doesn't close the job after the job is completed Key: MAPREDUCE-7298 URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 Project: Hadoop Map/Reduce Issue Type: Task Reporter: Aasha Medhi Distcp doesn't close the job after the job is completed. This leads to leaked Truststore Reloader Threads -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-6786) TaskAttemptImpl state changes for containers reuse
[ https://issues.apache.org/jira/browse/MAPREDUCE-6786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T reassigned MAPREDUCE-6786: Assignee: Bilwa S T (was: Naganarasimha G R) > TaskAttemptImpl state changes for containers reuse > -- > > Key: MAPREDUCE-6786 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6786 > Project: Hadoop Map/Reduce > Issue Type: Sub-task > Components: applicationmaster, mrv2 >Reporter: Devaraj Kavali >Assignee: Bilwa S T >Priority: Major > Attachments: MAPREDUCE-6786-v0.patch > > > Update TaskAttemptImpl to support the reuse of containers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org