[jira] [Commented] (YARN-9438) launchTime not written to state store for running applications
[
https://issues.apache.org/jira/browse/YARN-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899223#comment-16899223
]
Jonathan Hung commented on YARN-9438:
-
002 changes:
* Use the same ApplicationStateData constructor in RMAppImpl as in
RMStateStore#storeNewApplication
* Fix TestRMRestart#testRMRestartWaitForPreviousSucceededAttempt: first call
to updateApplicationStateInternal should not be skipped (first one is saving
launchTime), second one should be skipped
* Fix TestRMRestart#testClientRetryOnKillingApplication: similar to above.
TestMemoryRMStateStore's {{count}} is incremented when storing launchTime, so
increment the counts in this test accordingly
* Fix TestRMRestart#testRMRestartOnMissingAttempts: storing launchTime creates
a new ApplicationStateData in memory state store, hence move the app state
getter after the attempts have been launched
> launchTime not written to state store for running applications
> --
>
> Key: YARN-9438
> URL: https://issues.apache.org/jira/browse/YARN-9438
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.10.0
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9438.001.patch, YARN-9438.002.patch
>
>
> launchTime is only saved to state store after application finishes, so if
> restart happens, any running applications will have launchTime set as -1
> (since this is the default timestamp of the recovery event).
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9438) launchTime not written to state store for running applications
[ https://issues.apache.org/jira/browse/YARN-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9438: Attachment: YARN-9438.002.patch > launchTime not written to state store for running applications > -- > > Key: YARN-9438 > URL: https://issues.apache.org/jira/browse/YARN-9438 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9438.001.patch, YARN-9438.002.patch > > > launchTime is only saved to state store after application finishes, so if > restart happens, any running applications will have launchTime set as -1 > (since this is the default timestamp of the recovery event). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16891378#comment-16891378 ] Jonathan Hung commented on YARN-9668: - Darn it. Attached an addendum for branch-3.0. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.0.001.addendum, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-3.0.001.addendum > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.0.001.addendum, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reopened YARN-9668: - > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.0.001.addendum, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16890350#comment-16890350 ] Jonathan Hung commented on YARN-9668: - Thanks Haibo! Committed to branch-3.2, branch-3.1, branch-3.0 as well. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.2.001.patch, > YARN-9668.001.patch, YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Fix Version/s: 3.2.2 3.1.3 3.0.4 > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.2.001.patch, > YARN-9668.001.patch, YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9438) launchTime not written to state store for running applications
[ https://issues.apache.org/jira/browse/YARN-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9438: Affects Version/s: 2.10.0 > launchTime not written to state store for running applications > -- > > Key: YARN-9438 > URL: https://issues.apache.org/jira/browse/YARN-9438 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9438.001.patch > > > launchTime is only saved to state store after application finishes, so if > restart happens, any running applications will have launchTime set as -1 > (since this is the default timestamp of the recovery event). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9438) launchTime not written to state store for running applications
[ https://issues.apache.org/jira/browse/YARN-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886574#comment-16886574 ] Jonathan Hung commented on YARN-9438: - 001 saves app's launchTime to state store when attempt is launched. > launchTime not written to state store for running applications > -- > > Key: YARN-9438 > URL: https://issues.apache.org/jira/browse/YARN-9438 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9438.001.patch > > > launchTime is only saved to state store after application finishes, so if > restart happens, any running applications will have launchTime set as -1 > (since this is the default timestamp of the recovery event). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9438) launchTime not written to state store for running applications
[ https://issues.apache.org/jira/browse/YARN-9438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9438: Attachment: YARN-9438.001.patch > launchTime not written to state store for running applications > -- > > Key: YARN-9438 > URL: https://issues.apache.org/jira/browse/YARN-9438 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9438.001.patch > > > launchTime is only saved to state store after application finishes, so if > restart happens, any running applications will have launchTime set as -1 > (since this is the default timestamp of the recovery event). -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885443#comment-16885443 ] Jonathan Hung commented on YARN-9668: - TestCapacityOverTimePolicy test failure looks related to YARN-9450 > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.2.001.patch, > YARN-9668.001.patch, YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-2.002.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-2.002.patch, YARN-9668-branch-3.2.001.patch, > YARN-9668.001.patch, YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883389#comment-16883389 ] Jonathan Hung commented on YARN-9668: - Yes. Uploaded branch-2 patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-2.001.patch, > YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883358#comment-16883358 ] Jonathan Hung commented on YARN-9668: - Uploaded branch-3.2 version. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9688-branch-3.2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: (was: YARN-9688-branch-3.2.001.patch) > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668-branch-3.2.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668-branch-3.2.001.patch, YARN-9668.001.patch, > YARN-9668.002.patch, YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16881202#comment-16881202 ] Jonathan Hung commented on YARN-9668: - TestCombinedSystemMetricsPublisher: YARN-9541 TestRMAppTransitions: YARN-9540 TestFairSchedulerPreemption: YARN-9333 Ran *TestRMHAForAsyncScheduler* test locally and it succeeded. [~haibochen] mind taking another look? Thanks! > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880892#comment-16880892 ] Jonathan Hung commented on YARN-9668: - Only [TestRMRestart.testSynchronouslyRenewDTOnRecovery|https://builds.apache.org/job/PreCommit-YARN-Build/24349/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestRMRestart/testSynchronouslyRenewDTOnRecovery_CAPACITY_/] appears related. testSynchronouslyRenewDTOnRecovery specifies kerberos auth in the conf, but it doesn't get propagated to UGI, so it actually uses simple. After applying this patch this test will use kerberos which causes the test failures. 003 fixes this test to use TestSecurityMockRM so the test doesn't try to login. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668.003.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch, > YARN-9668.003.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16880643#comment-16880643 ] Jonathan Hung commented on YARN-9668: - Thanks [~haibochen] makes sense, attached 002 to address this. > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668.002.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch, YARN-9668.002.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Affects Version/s: 2.10.0 > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 2.10.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
[ https://issues.apache.org/jira/browse/YARN-9668?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9668: Attachment: YARN-9668.001.patch > UGI conf doesn't read user overridden configurations on RM and NM startup > - > > Key: YARN-9668 > URL: https://issues.apache.org/jira/browse/YARN-9668 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9668.001.patch > > > Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not > passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9668) UGI conf doesn't read user overridden configurations on RM and NM startup
Jonathan Hung created YARN-9668: --- Summary: UGI conf doesn't read user overridden configurations on RM and NM startup Key: YARN-9668 URL: https://issues.apache.org/jira/browse/YARN-9668 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Hung Assignee: Jonathan Hung Similar to HADOOP-15150. Configs overridden thru e.g. -D or -conf are not passed to the UGI conf on RM or NM startup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9615) Add dispatcher metrics to RM
[ https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16868001#comment-16868001 ] Jonathan Hung commented on YARN-9615: - Thanks [~bibinchundatt], I just attached YARN-9615.poc.patch, it would be great if you could take a look. It adds a couple metrics for RM async dispatcher + scheduler dispatcher events. If your approach is similar, perhaps we can use this patch as a basis. If not , I'd be interested to see what your implementation looks like. > Add dispatcher metrics to RM > > > Key: YARN-9615 > URL: https://issues.apache.org/jira/browse/YARN-9615 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9615.poc.patch, screenshot-1.png > > > It'd be good to have counts/processing times for each event type in RM async > dispatcher and scheduler async dispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9615) Add dispatcher metrics to RM
[ https://issues.apache.org/jira/browse/YARN-9615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9615: Attachment: YARN-9615.poc.patch > Add dispatcher metrics to RM > > > Key: YARN-9615 > URL: https://issues.apache.org/jira/browse/YARN-9615 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9615.poc.patch, screenshot-1.png > > > It'd be good to have counts/processing times for each event type in RM async > dispatcher and scheduler async dispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16867094#comment-16867094 ] Jonathan Hung commented on YARN-9559: - Hi [~wangda], it's related to YARN-8849. Basically we want to remove the hardcoded logic in ContainersLauncher that actually launches the user's specified container, and plug in our own logic for container launch. > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch, > YARN-9559.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16866983#comment-16866983 ] Jonathan Hung commented on YARN-9559: - Hi [~giovanni.fumarola], mind taking a look at the latest patch? (003) It should address all of your comments. Thanks! > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch, > YARN-9559.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9615) Add dispatcher metrics to RM
Jonathan Hung created YARN-9615: --- Summary: Add dispatcher metrics to RM Key: YARN-9615 URL: https://issues.apache.org/jira/browse/YARN-9615 Project: Hadoop YARN Issue Type: Task Reporter: Jonathan Hung Assignee: Jonathan Hung It'd be good to have counts/processing times for each event type in RM async dispatcher and scheduler async dispatcher. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9593) Updating scheduler conf with comma in config value fails
[
https://issues.apache.org/jira/browse/YARN-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16860209#comment-16860209
]
Jonathan Hung commented on YARN-9593:
-
Yeah [~erwaman], that seems reasonable. Are you interested in taking this up?
> Updating scheduler conf with comma in config value fails
>
>
> Key: YARN-9593
> URL: https://issues.apache.org/jira/browse/YARN-9593
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0, 3.2.0, 3.1.2
>Reporter: Anthony Hsu
>Priority: Major
>
> For example:
> {code:java}
> $ yarn schedulerconf -update "root.gridops:acl_administer_queue=user1,user2
> group1,group2"
> Specify configuration key value as confKey=confVal.{code}
> This fails because there is a comma in the config value and the SchedConfCLI
> splits on comma first, expecting each split to a k=v pair.
> {noformat}
> void globalUpdates(String args, SchedConfUpdateInfo updateInfo) {
> if (args == null) {
> return;
> }
> HashMap globalUpdates = new HashMap<>();
> for (String globalUpdate : args.split(",")) {
> putKeyValuePair(globalUpdates, globalUpdate);
> }
> updateInfo.setGlobalParams(globalUpdates);
> }{noformat}
> Cc: [~jhung]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9593) Updating scheduler conf with comma in config value fails
[
https://issues.apache.org/jira/browse/YARN-9593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16853525#comment-16853525
]
Jonathan Hung commented on YARN-9593:
-
Thanks [~erwaman] for the report, this is a known issue, but confs of this
variety can be changed via the REST API as well:
[https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Scheduler_Configuration_Mutation_API].
I suppose it's possible to change the delimiter in the CLI. But {{,}} seems
quite natural to me. Most confs don't require this either. What do you think?
> Updating scheduler conf with comma in config value fails
>
>
> Key: YARN-9593
> URL: https://issues.apache.org/jira/browse/YARN-9593
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0, 3.2.0, 3.1.2
>Reporter: Anthony Hsu
>Priority: Major
>
> For example:
> {code:java}
> $ yarn schedulerconf -update "root.gridops:acl_administer_queue=user1,user2
> group1,group2"
> Specify configuration key value as confKey=confVal.{code}
> This fails because there is a comma in the config value and the SchedConfCLI
> splits on comma first, expecting each split to a k=v pair.
> {noformat}
> void globalUpdates(String args, SchedConfUpdateInfo updateInfo) {
> if (args == null) {
> return;
> }
> HashMap globalUpdates = new HashMap<>();
> for (String globalUpdate : args.split(",")) {
> putKeyValuePair(globalUpdates, globalUpdate);
> }
> updateInfo.setGlobalParams(globalUpdates);
> }{noformat}
> Cc: [~jhung]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841792#comment-16841792 ] Jonathan Hung commented on YARN-9559: - Thanks [~giovanni.fumarola] , TestFederationInterceptor passes locally. TestContainerSchedulerQueuing all fail but they also fail prepatch locally. It shouldn’t be related (since 001 does not have this failure and 001/003 are functionally the same). Let’s see the yetus output for 003. 003 is same as 002 but removes a couple new lines. > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch, > YARN-9559.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9559: Attachment: YARN-9559.003.patch > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch, > YARN-9559.003.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9559: Attachment: YARN-9559.002.patch > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16841685#comment-16841685 ] Jonathan Hung commented on YARN-9559: - Thanks [~giovanni.fumarola], TestYarnConfigurationFields should be related, it's fixed (along with checkstyles) in 002. Other two unit tests seem to be known issues: YARN-7597 and YARN-9558 > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch, YARN-9559.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16840877#comment-16840877 ] Jonathan Hung commented on YARN-9559: - attached 001 patch which creates AbstractContainersLauncher class > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
[ https://issues.apache.org/jira/browse/YARN-9559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9559: Attachment: YARN-9559.001.patch > Create AbstractContainersLauncher for pluggable ContainersLauncher logic > > > Key: YARN-9559 > URL: https://issues.apache.org/jira/browse/YARN-9559 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9559.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9559) Create AbstractContainersLauncher for pluggable ContainersLauncher logic
Jonathan Hung created YARN-9559: --- Summary: Create AbstractContainersLauncher for pluggable ContainersLauncher logic Key: YARN-9559 URL: https://issues.apache.org/jira/browse/YARN-9559 Project: Hadoop YARN Issue Type: Task Reporter: Jonathan Hung Assignee: Jonathan Hung -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839699#comment-16839699
]
Jonathan Hung commented on YARN-9518:
-
Yes, makes sense [~Jim_Brennan]. Attached a patch to YARN-2194. Closing this
ticket.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.7
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518-branch-2.7.002.patch,
> YARN-9518-branch-2.7.7.001.patch, YARN-9518-trunk.001.patch, YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is compatible with cgroup path of history os version such as
> centos6, centos7 , and universally applicable to cgroup subsystem paths such
> as cgroup network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>
[jira] [Updated] (YARN-2194) Cgroups cease to work in RHEL7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-2194: Attachment: YARN-2194-branch-2.7.001.patch > Cgroups cease to work in RHEL7 > -- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-2194-1.patch, YARN-2194-2.patch, YARN-2194-3.patch, > YARN-2194-4.patch, YARN-2194-5.patch, YARN-2194-6.patch, YARN-2194-7.patch, > YARN-2194-branch-2.7.001.patch > > > In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the > controller name leads to container launch failure. > RHEL7 deprecates libcgroup and recommends the user of systemd. However, > systemd has certain shortcomings as identified in this JIRA (see comments). > This JIRA only fixes the failure, and doesn't try to use systemd. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-2194) Cgroups cease to work in RHEL7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839698#comment-16839698 ] Jonathan Hung commented on YARN-2194: - Attaching a branch-2.7 patch. (Ref: YARN-9518) > Cgroups cease to work in RHEL7 > -- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-2194-1.patch, YARN-2194-2.patch, YARN-2194-3.patch, > YARN-2194-4.patch, YARN-2194-5.patch, YARN-2194-6.patch, YARN-2194-7.patch, > YARN-2194-branch-2.7.001.patch > > > In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the > controller name leads to container launch failure. > RHEL7 deprecates libcgroup and recommends the user of systemd. However, > systemd has certain shortcomings as identified in this JIRA (see comments). > This JIRA only fixes the failure, and doesn't try to use systemd. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2194) Cgroups cease to work in RHEL7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-2194: Attachment: YARN-9518-branch-2.7.002.patch > Cgroups cease to work in RHEL7 > -- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-2194-1.patch, YARN-2194-2.patch, YARN-2194-3.patch, > YARN-2194-4.patch, YARN-2194-5.patch, YARN-2194-6.patch, YARN-2194-7.patch > > > In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the > controller name leads to container launch failure. > RHEL7 deprecates libcgroup and recommends the user of systemd. However, > systemd has certain shortcomings as identified in this JIRA (see comments). > This JIRA only fixes the failure, and doesn't try to use systemd. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-2194) Cgroups cease to work in RHEL7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-2194: Attachment: (was: YARN-9518-branch-2.7.002.patch) > Cgroups cease to work in RHEL7 > -- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-2194-1.patch, YARN-2194-2.patch, YARN-2194-3.patch, > YARN-2194-4.patch, YARN-2194-5.patch, YARN-2194-6.patch, YARN-2194-7.patch > > > In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the > controller name leads to container launch failure. > RHEL7 deprecates libcgroup and recommends the user of systemd. However, > systemd has certain shortcomings as identified in this JIRA (see comments). > This JIRA only fixes the failure, and doesn't try to use systemd. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Reopened] (YARN-2194) Cgroups cease to work in RHEL7
[ https://issues.apache.org/jira/browse/YARN-2194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reopened YARN-2194: - > Cgroups cease to work in RHEL7 > -- > > Key: YARN-2194 > URL: https://issues.apache.org/jira/browse/YARN-2194 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager >Affects Versions: 2.7.0 >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Critical > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: YARN-2194-1.patch, YARN-2194-2.patch, YARN-2194-3.patch, > YARN-2194-4.patch, YARN-2194-5.patch, YARN-2194-6.patch, YARN-2194-7.patch > > > In RHEL7, the CPU controller is named "cpu,cpuacct". The comma in the > controller name leads to container launch failure. > RHEL7 deprecates libcgroup and recommends the user of systemd. However, > systemd has certain shortcomings as identified in this JIRA (see comments). > This JIRA only fixes the failure, and doesn't try to use systemd. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16839643#comment-16839643
]
Jonathan Hung commented on YARN-9518:
-
Yes, I agree with [~Jim_Brennan]. Actually we had this patch backported
internally, I uploaded it for convenience, hope you don't mind [~shurong.mai].
[~shurong.mai] can you check that this patch (YARN-9518-branch-2.7.002.patch)
works for you? Also if you two could review it that would be great.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.7
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518-branch-2.7.002.patch,
> YARN-9518-branch-2.7.7.001.patch, YARN-9518-trunk.001.patch, YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is compatible with cgroup path of history os version such as
> centos6, centos7 , and universally applicable to cgroup subsystem paths such
> as cgroup network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
>
[jira] [Updated] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hung updated YARN-9518:
Attachment: YARN-9518-branch-2.7.002.patch
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 2.7.7
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518-branch-2.7.002.patch,
> YARN-9518-branch-2.7.7.001.patch, YARN-9518-trunk.001.patch, YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is compatible with cgroup path of history os version such as
> centos6, centos7 , and universally applicable to cgroup subsystem paths such
> as cgroup network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error while initializing CGroups.
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16834123#comment-16834123 ] Jonathan Hung commented on YARN-9529: - Thanks [~haibochen]!! > Log correct cpu controller path on error while initializing CGroups. > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement > Components: nodemanager >Affects Versions: 2.10.0, 3.2.0 >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Labels: 2.10.0, 3.3.0 > Fix For: 2.10.0, 3.0.4, 3.3.0, 3.1.3, 3.2.2 > > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9529) Log correct cpu controller path on error
[
https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832960#comment-16832960
]
Jonathan Hung commented on YARN-9529:
-
Thanks! This was when reconfiguring
{{yarn.nodemanager.linux-container-executor.cgroups.hierarchy}}. The previous
log prints this dir's parent dir which was never writable, so the NM should
never have been able to start up. But it's checking the writability of this
configured dir (not its parent), hence the logging change.
> Log correct cpu controller path on error
>
>
> Key: YARN-9529
> URL: https://issues.apache.org/jira/browse/YARN-9529
> Project: Hadoop YARN
> Issue Type: Improvement
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9529.001.patch
>
>
> The base cpu controller path is logged instead of the hadoop cgroup path.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9529) Log correct cpu controller path on error
[ https://issues.apache.org/jira/browse/YARN-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9529: Attachment: YARN-9529.001.patch > Log correct cpu controller path on error > > > Key: YARN-9529 > URL: https://issues.apache.org/jira/browse/YARN-9529 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9529.001.patch > > > The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9529) Log correct cpu controller path on error
Jonathan Hung created YARN-9529: --- Summary: Log correct cpu controller path on error Key: YARN-9529 URL: https://issues.apache.org/jira/browse/YARN-9529 Project: Hadoop YARN Issue Type: Improvement Reporter: Jonathan Hung Assignee: Jonathan Hung Attachments: YARN-9529.001.patch The base cpu controller path is logged instead of the hadoop cgroup path. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832878#comment-16832878
]
Jonathan Hung edited comment on YARN-9518 at 5/3/19 10:23 PM:
--
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us. (we're running 2.7.4)
was (Author: jhung):
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is universally applicable to cgroup subsystem paths, such as cgroup
> network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at
[jira] [Commented] (YARN-9518) can't use CGroups with YARN in centos7
[
https://issues.apache.org/jira/browse/YARN-9518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16832878#comment-16832878
]
Jonathan Hung commented on YARN-9518:
-
[~shurong.mai] have you seen YARN-2194? We had the same issue and this fixed it
for us.
> can't use CGroups with YARN in centos7
> ---
>
> Key: YARN-9518
> URL: https://issues.apache.org/jira/browse/YARN-9518
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.2.0, 2.9.2, 2.8.5, 2.7.7, 3.1.2
>Reporter: Shurong Mai
>Priority: Major
> Labels: cgroup, patch
> Attachments: YARN-9518.patch
>
>
> The os version is centos7.
> {code:java}
> cat /etc/redhat-release
> CentOS Linux release 7.3.1611 (Core)
> {code}
> When I had set configuration variables for cgroup with yarn, nodemanager
> could be start without any matter. But when I ran a job, the job failed with
> these exceptional nodemanager logs in the end.
> In these logs, the important logs is " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> After I analysed, I found the reason. In centos6, the cgroup "cpu" and
> "cpuacct" subsystem are as follows:
> {code:java}
> /sys/fs/cgroup/cpu
> /sys/fs/cgroup/cpuacct
> {code}
> But in centos7, as follows:
> {code:java}
> /sys/fs/cgroup/cpu -> cpu,cpuacct
> /sys/fs/cgroup/cpuacct -> cpu,cpuacct
> /sys/fs/cgroup/cpu,cpuacct{code}
> "cpu" and "cpuacct" have merge as "cpu,cpuacct". "cpu" and "cpuacct" are
> symbol links.
> As I look at source code, nodemamager get the cgroup subsystem info by
> reading /proc/mounts. So It get the cpu and cpuacct subsystem path are also
> "/sys/fs/cgroup/cpu,cpuacct".
> The resource description arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> There is a comma in the cgroup path, but the comma is separator of multi
> resource. Therefore, the cgroup path is truncated by container-executor as
> "/sys/fs/cgroup/cpu" rather than correct cgroup path "
> /sys/fs/cgroup/cpu,cpuacct/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> " and report the error in the log " Can't open file /sys/fs/cgroup/cpu as
> node manager - Is a directory "
> Hence I modify the source code and submit a patch. The idea of patch is that
> nodemanager get the cgroup cpu path as "/sys/fs/cgroup/cpu" rather than
> "/sys/fs/cgroup/cpu,cpuacct". As a result, the resource description
> arguments of container-executor is such as follows:
> {code:java}
> cgroups=/sys/fs/cgroup/cpu/hadoop-yarn/container_1554210318404_0057_02_01/tasks
> {code}
> Note that there is no comma in the path, and is a valid path because
> "/sys/fs/cgroup/cpu" is symbol link to "/sys/fs/cgroup/cpu,cpuacct".
> After applied the patch, the problem is resolved and the job can run
> successfully.
> The patch is universally applicable to cgroup subsystem paths, such as cgroup
> network subsystem as follows:
> {code:java}
> /sys/fs/cgroup/net_cls -> net_cls,net_prio
> /sys/fs/cgroup/net_prio -> net_cls,net_prio
> /sys/fs/cgroup/net_cls,net_prio{code}
>
>
> ##
> {panel:title=exceptional nodemanager logs:}
> 2019-04-19 20:17:20,095 INFO
> org.apache.hadoop.yarn.server.nodemanager.containermanager.container.ContainerImpl:
> Container container_1554210318404_0042_01_01 transitioned from LOCALIZED
> to RUNNING
> 2019-04-19 20:17:20,101 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exit code
> from container container_1554210318404_0042_01_01 is : 27
> 2019-04-19 20:17:20,103 WARN
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor: Exception
> from container-launch with container ID: container_155421031840
> 4_0042_01_01 and exit code: 27
> ExitCodeException exitCode=27:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:585)
> at org.apache.hadoop.util.Shell.run(Shell.java:482)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:776)
> at
> org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:299)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Created] (YARN-9438) launchTime not written to state store for running applications
Jonathan Hung created YARN-9438: --- Summary: launchTime not written to state store for running applications Key: YARN-9438 URL: https://issues.apache.org/jira/browse/YARN-9438 Project: Hadoop YARN Issue Type: Bug Reporter: Jonathan Hung Assignee: Jonathan Hung launchTime is only saved to state store after application finishes, so if restart happens, any running applications will have launchTime set as -1 (since this is the default timestamp of the recovery event). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[
https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16805334#comment-16805334
]
Jonathan Hung commented on YARN-8200:
-
* javac warning:
{noformat}
[WARNING]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/LightWeightResource.java:[114,14]
[deprecation] setMemory(int) in Resource has been deprecated
[WARNING]
/testptch/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/LightWeightResource.java:[108,13]
[deprecation] getMemory() in Resource has been deprecated{noformat}
due to [https://bugs.openjdk.java.net/browse/JDK-8033421] which does not seem
to be fixed in openjdk7
* TestSecureLogins failure related to HADOOP-16031
* TestTimelineAuthFilterForV2 failure seems unrelated (also fails in branch-2
nightly)
* ASF license warning seems related to MAPREDUCE-7036
* Not sure what's going on with hadoop-mapreduce-client-jobclient, but tests
pass and the timeout seems to exist in both branch-2 and trunk nightlies
> Backport resource types/GPU features to branch-3.0/branch-2
> ---
>
> Key: YARN-8200
> URL: https://issues.apache.org/jira/browse/YARN-8200
> Project: Hadoop YARN
> Issue Type: Task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-8200-branch-2.001.patch,
> YARN-8200-branch-2.002.patch, YARN-8200-branch-3.0.001.patch,
> counter.scheduler.operation.allocate.csv.defaultResources,
> counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json
>
>
> Currently we have a need for GPU scheduling on our YARN clusters to support
> deep learning workloads. However, our main production clusters are running
> older versions of branch-2 (2.7 in our case). To prevent supporting too many
> very different hadoop versions across multiple clusters, we would like to
> backport the resource types/resource profiles feature to branch-2, as well as
> the GPU specific support.
>
> We have done a trial backport of YARN-3926 and some miscellaneous patches in
> YARN-7069 based on issues we uncovered, and the backport was fairly smooth.
> We also did a trial backport of most of YARN-6223 (sans docker support).
>
> Regarding the backports, perhaps we can do the development in a feature
> branch and then merge to branch-2 when ready.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803546#comment-16803546 ] Jonathan Hung commented on YARN-8200: - * TestSecureLogins failure related to HADOOP-16031 * [TestOpportunisticContainerAllocatorAMService.testContainerPromoteAndDemoteBeforeContainerStart|https://builds.apache.org/job/PreCommit-YARN-Build/23820/testReport/org.apache.hadoop.yarn.server.resourcemanager/TestOpportunisticContainerAllocatorAMService/testContainerPromoteAndDemoteBeforeContainerStart/] related to YARN-8011 * TestOpportunisticContainerAllocatorAMService.testAppAttemptRemovalAfterNodeRemoval is already failing in branch-3.0 * TestNodeLabelContainerAllocation related to YARN-9006 > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-2.002.patch, YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803519#comment-16803519 ] Jonathan Hung commented on YARN-8200: - Attached YARN-8200-branch-2.002 containing all the commits targeted for branch-2. (The commit list is at YARN-8200 branch which has been rebased on latest branch-2). > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-2.002.patch, YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-8200: Attachment: YARN-8200-branch-2.002.patch > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-2.002.patch, YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803251#comment-16803251 ] Jonathan Hung commented on YARN-8200: - Attached YARN-8200-branch-3.0.001 containing all the commits targeted for branch-3.0. (The commit list is at YARN-8200.branch3 branch which has been rebased on latest branch-3.0). > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-8200: Attachment: YARN-8200-branch-3.0.001.patch > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803250#comment-16803250 ] Jonathan Hung commented on YARN-8200: - Yes [~Jim_Brennan], we plan to do a 2.10 release with this feature. > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > YARN-8200-branch-3.0.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-9412) Backport YARN-6909 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung resolved YARN-9412. - Resolution: Fixed This ended up being a clean port. Closing. > Backport YARN-6909 to branch-2 > -- > > Key: YARN-9412 > URL: https://issues.apache.org/jira/browse/YARN-9412 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16803163#comment-16803163 ] Jonathan Hung edited comment on YARN-9409 at 3/27/19 6:11 PM: -- Thanks Zhe, committed to YARN-8200 and YARN-8200.branch3 was (Author: jhung): Thanks Zhe, committed to YARN-8200 > Port resource type changes from YARN-7237 to branch-3.0/branch-2 > > > Key: YARN-9409 > URL: https://issues.apache.org/jira/browse/YARN-9409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9409-YARN-8200.branch3.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802203#comment-16802203 ] Jonathan Hung commented on YARN-9409: - Thanks Anthony. Yup, was gonna do that after this patch is in :) created YARN-9412 for this. > Port resource type changes from YARN-7237 to branch-3.0/branch-2 > > > Key: YARN-9409 > URL: https://issues.apache.org/jira/browse/YARN-9409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9409-YARN-8200.branch3.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9412) Backport YARN-6909 to branch-2
Jonathan Hung created YARN-9412: --- Summary: Backport YARN-6909 to branch-2 Key: YARN-9412 URL: https://issues.apache.org/jira/browse/YARN-9412 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Hung Assignee: Jonathan Hung -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16802072#comment-16802072 ] Jonathan Hung commented on YARN-9272: - kicking jenkins again to fix the findbugs warning > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.001.patch, > YARN-9272-YARN-8200.branch3.001.patch, YARN-9272-YARN-8200.branch3.002.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16801179#comment-16801179 ] Jonathan Hung commented on YARN-9409: - Unit test failures should be unrelated. > Port resource type changes from YARN-7237 to branch-3.0/branch-2 > > > Key: YARN-9409 > URL: https://issues.apache.org/jira/browse/YARN-9409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9409-YARN-8200.branch3.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16800996#comment-16800996 ] Jonathan Hung commented on YARN-9272: - Attached branch-2 version which is the same as the branch-3 one but with initMockRMWithOldConf argument declared as final (requirement in java 7) > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.001.patch, > YARN-9272-YARN-8200.branch3.001.patch, YARN-9272-YARN-8200.branch3.002.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9272: Attachment: YARN-9272-YARN-8200.001.patch > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.001.patch, > YARN-9272-YARN-8200.branch3.001.patch, YARN-9272-YARN-8200.branch3.002.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-9409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9409: Attachment: YARN-9409-YARN-8200.branch3.001.patch > Port resource type changes from YARN-7237 to branch-3.0/branch-2 > > > Key: YARN-9409 > URL: https://issues.apache.org/jira/browse/YARN-9409 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9409-YARN-8200.branch3.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9409) Port resource type changes from YARN-7237 to branch-3.0/branch-2
Jonathan Hung created YARN-9409: --- Summary: Port resource type changes from YARN-7237 to branch-3.0/branch-2 Key: YARN-9409 URL: https://issues.apache.org/jira/browse/YARN-9409 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Hung Assignee: Jonathan Hung -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16799473#comment-16799473
]
Jonathan Hung commented on YARN-9272:
-
Issue with TestReservationSystemWithRMHA was that it sets scheduler conf and
passes it to RM, which previously would get passed to scheduler on
reinitialize, but {{loadNewConfiguration}} bypasses this.
This was fixed as part of YARN-6124, so 002 adds the changes in RMHATestBase
from YARN-6124.
> Backport YARN-7738 for refreshing max allocation for multiple resource types
>
>
> Key: YARN-9272
> URL: https://issues.apache.org/jira/browse/YARN-9272
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9272-YARN-8200.branch3.001.patch,
> YARN-9272-YARN-8200.branch3.002.patch
>
>
> Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for
> branch-2)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9272: Attachment: YARN-9272-YARN-8200.branch3.002.patch > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.branch3.001.patch, > YARN-9272-YARN-8200.branch3.002.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798597#comment-16798597 ] Jonathan Hung edited comment on YARN-9272 at 3/22/19 1:02 AM: -- * TestNodeLabelContainerAllocation also fails locally pre-patch. * TestCapacityOverTimePolicy passes locally. * TestOpportunisticContainerAllocatorAMService seems flaky, it passes and fails intermittently locally. * TestReservationSystemWithRMHA failures look related. was (Author: jhung): * TestNodeLabelContainerAllocation also fails locally pre-patch. * TestCapacityOverTimePolicy passes locally. * TestOpportunisticContainerAllocatorAMService and TestReservationSystemWithRMHA failures look related. > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.branch3.001.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798597#comment-16798597 ] Jonathan Hung commented on YARN-9272: - * TestNodeLabelContainerAllocation also fails locally pre-patch. * TestCapacityOverTimePolicy passes locally. * TestOpportunisticContainerAllocatorAMService and TestReservationSystemWithRMHA failures look related. > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.branch3.001.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798423#comment-16798423
]
Jonathan Hung edited comment on YARN-9272 at 3/21/19 9:09 PM:
--
Attached branch-3.0 version. A few diffs from YARN-7738:
* Omit the vcores > Integer.MAX_VALUE check in Resource class (this is already
handled by castToIntSafely in branch-3.0)
* Add an extra constructor in ResourceInformation class (this was added as
part of YARN-7254)
* Add the {{loadNewConfiguration}} method in AdminService where
resource-types.xml file is added to the conf which is called when refreshing
scheduler. (This method was added as part of YARN-6124)
* Add TestCapacitySchedulerWithMultiResourceTypes test class (this was added
as part of YARN-7237).
The branch-3.0 patch also applies cleanly on YARN-8200.
was (Author: jhung):
Attached branch-3.0 version. A few diffs from YARN-7738:
* Omit the vcores > Integer.MAX_VALUE check in Resource class (this is already
handled by castToIntSafely in branch-3.0)
* Add an extra constructor in ResourceInformation class (this was added as
part of YARN-7254)
* Add the {{loadNewConfiguration}} method in AdminService where
resource-types.xml file is added to the conf which is called when refreshing
scheduler. (This method was added as part of YARN-6124)
* Add TestCapacitySchedulerWithMultiResourceTypes test class (this was added
as part of YARN-7237).
> Backport YARN-7738 for refreshing max allocation for multiple resource types
>
>
> Key: YARN-9272
> URL: https://issues.apache.org/jira/browse/YARN-9272
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9272-YARN-8200.branch3.001.patch
>
>
> Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for
> branch-2)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[
https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16798423#comment-16798423
]
Jonathan Hung commented on YARN-9272:
-
Attached branch-3.0 version. A few diffs from YARN-7738:
* Omit the vcores > Integer.MAX_VALUE check in Resource class (this is already
handled by castToIntSafely in branch-3.0)
* Add an extra constructor in ResourceInformation class (this was added as
part of YARN-7254)
* Add the {{loadNewConfiguration}} method in AdminService where
resource-types.xml file is added to the conf which is called when refreshing
scheduler. (This method was added as part of YARN-6124)
* Add TestCapacitySchedulerWithMultiResourceTypes test class (this was added
as part of YARN-7237).
> Backport YARN-7738 for refreshing max allocation for multiple resource types
>
>
> Key: YARN-9272
> URL: https://issues.apache.org/jira/browse/YARN-9272
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9272-YARN-8200.branch3.001.patch
>
>
> Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for
> branch-2)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9272) Backport YARN-7738 for refreshing max allocation for multiple resource types
[ https://issues.apache.org/jira/browse/YARN-9272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9272: Attachment: YARN-9272-YARN-8200.branch3.001.patch > Backport YARN-7738 for refreshing max allocation for multiple resource types > > > Key: YARN-9272 > URL: https://issues.apache.org/jira/browse/YARN-9272 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9272-YARN-8200.branch3.001.patch > > > Need to port to YARN-8200.branch3 (for branch-3.0) and YARN-8200 (for > branch-2) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796754#comment-16796754 ] Jonathan Hung commented on YARN-9271: - mapreduce jobclient unit tests seem to be failing in branch-2 as well: [https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/265/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient.txt] asflicense appears to be related to MAPREDUCE-7036. > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9271: Attachment: YARN-9271-YARN-8200.001.patch > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16796414#comment-16796414 ] Jonathan Hung commented on YARN-9271: - Attached 001 patch. It's identical to YARN-6927 but replaces some java 8 syntax when finding log lines in tests with java 7 compatible syntax: # TestTaskAttempt#testReducerMemoryRequestOverriding # TestTaskAttempt#testReducerCpuRequestOverriding # TestYARNRunner#testAMRMemoryRequestOverriding > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9271-YARN-8200.001.patch > > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-9271) Backport YARN-6927 for resource type support in MapReduce
[ https://issues.apache.org/jira/browse/YARN-9271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung reassigned YARN-9271: --- Assignee: Jonathan Hung > Backport YARN-6927 for resource type support in MapReduce > - > > Key: YARN-9271 > URL: https://issues.apache.org/jira/browse/YARN-9271 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > > This is already in branch-3.0. Need to port it to YARN-8200/branch-2. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9291) Backport YARN-7637 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795508#comment-16795508 ] Jonathan Hung edited comment on YARN-9291 at 3/19/19 12:56 AM: --- Test failure was fixed as part of YARN-9397. Checkstyle failure is part of original patch. was (Author: jhung): Test failure was fixed as part of YARN-9397. > Backport YARN-7637 to branch-2 > -- > > Key: YARN-9291 > URL: https://issues.apache.org/jira/browse/YARN-9291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9291-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9291) Backport YARN-7637 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16795508#comment-16795508 ] Jonathan Hung commented on YARN-9291: - Test failure was fixed as part of YARN-9397. > Backport YARN-7637 to branch-2 > -- > > Key: YARN-9291 > URL: https://issues.apache.org/jira/browse/YARN-9291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9291-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9397) Fix empty NMResourceInfo object test failures in branch-2
Jonathan Hung created YARN-9397: --- Summary: Fix empty NMResourceInfo object test failures in branch-2 Key: YARN-9397 URL: https://issues.apache.org/jira/browse/YARN-9397 Project: Hadoop YARN Issue Type: Sub-task Reporter: Jonathan Hung Assignee: Jonathan Hung Attachments: YARN-9397-YARN-8200.001.patch Appears the empty object handling behavior changed in jersey versions (branch-2 is on jersey 1.9, branch-3 on 1.19). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9397) Fix empty NMResourceInfo object test failures in branch-2
[ https://issues.apache.org/jira/browse/YARN-9397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9397: Attachment: YARN-9397-YARN-8200.001.patch > Fix empty NMResourceInfo object test failures in branch-2 > - > > Key: YARN-9397 > URL: https://issues.apache.org/jira/browse/YARN-9397 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9397-YARN-8200.001.patch > > > Appears the empty object handling behavior changed in jersey versions > (branch-2 is on jersey 1.9, branch-3 on 1.19). -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8200) Backport resource types/GPU features to branch-3.0/branch-2
[ https://issues.apache.org/jira/browse/YARN-8200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16793941#comment-16793941 ] Jonathan Hung commented on YARN-8200: - Hey [~Jim_Brennan], yeah, I plan to finish the ports/start a merge-vote this month. > Backport resource types/GPU features to branch-3.0/branch-2 > --- > > Key: YARN-8200 > URL: https://issues.apache.org/jira/browse/YARN-8200 > Project: Hadoop YARN > Issue Type: Task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-8200-branch-2.001.patch, > counter.scheduler.operation.allocate.csv.defaultResources, > counter.scheduler.operation.allocate.csv.gpuResources, synth_sls.json > > > Currently we have a need for GPU scheduling on our YARN clusters to support > deep learning workloads. However, our main production clusters are running > older versions of branch-2 (2.7 in our case). To prevent supporting too many > very different hadoop versions across multiple clusters, we would like to > backport the resource types/resource profiles feature to branch-2, as well as > the GPU specific support. > > We have done a trial backport of YARN-3926 and some miscellaneous patches in > YARN-7069 based on issues we uncovered, and the backport was fairly smooth. > We also did a trial backport of most of YARN-6223 (sans docker support). > > Regarding the backports, perhaps we can do the development in a feature > branch and then merge to branch-2 when ready. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9291) Backport YARN-7637 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763864#comment-16763864 ] Jonathan Hung commented on YARN-9291: - 001 is exactly the same as YARN-7637 but parameterizes a list in TestGpuResourceHandler > Backport YARN-7637 to branch-2 > -- > > Key: YARN-9291 > URL: https://issues.apache.org/jira/browse/YARN-9291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9291-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9291) Backport YARN-7637 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9291: Attachment: YARN-9291-YARN-8200.001.patch > Backport YARN-7637 to branch-2 > -- > > Key: YARN-9291 > URL: https://issues.apache.org/jira/browse/YARN-9291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9291-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9291) Backport YARN-7637 to branch-2
[ https://issues.apache.org/jira/browse/YARN-9291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9291: Issue Type: Sub-task (was: Task) Parent: YARN-8200 > Backport YARN-7637 to branch-2 > -- > > Key: YARN-9291 > URL: https://issues.apache.org/jira/browse/YARN-9291 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9291-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-9291) Backport YARN-7637 to branch-2
Jonathan Hung created YARN-9291: --- Summary: Backport YARN-7637 to branch-2 Key: YARN-9291 URL: https://issues.apache.org/jira/browse/YARN-9291 Project: Hadoop YARN Issue Type: Task Reporter: Jonathan Hung Assignee: Jonathan Hung -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763308#comment-16763308 ] Jonathan Hung commented on YARN-9289: - Test failure seems to be related to HADOOP-16031. > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9289-YARN-8200.001.patch, > YARN-9289-YARN-8200.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763259#comment-16763259 ] Jonathan Hung commented on YARN-9289: - Attached 002 patch: * fix whitespace * re-add the annotation in GpuDeviceInformation. Seems the behavior in the original patch is not consistent, it will deserialize with the tag but serialize with . So I just changed TestNMWebServices#testGetNMResourceInfo to check for driver_version instead of driverVersion. * Created a subclass of NMResourceInfo in TestNMWebServices instead of anonymous class since it doesn't seem like the annotations were inheriting correctly. (Also the json return types were strings and being compared against longs, so I parsed the json strings as longs before asserting) > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9289-YARN-8200.001.patch, > YARN-9289-YARN-8200.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9289: Attachment: YARN-9289-YARN-8200.002.patch > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9289-YARN-8200.001.patch, > YARN-9289-YARN-8200.002.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[
https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763183#comment-16763183
]
Jonathan Hung commented on YARN-9289:
-
001 patch diffs from branch-3:
* Adds annotations and default constructors to AssignedGpuDevice, GpuDevice,
NMGpuResourceInfo
* Some import changes
* Remove charset from REST return types (was added as part of HADOOP-10075)
* Removed an annotation in GpuDeviceInformation:
\{{@javax.xml.bind.annotation.XmlElement(name = "driver_version")}}. It seems
in the original patch the unit test expects driverVersion: {{
assertEquals("1.2.3",
json.getJSONObject("gpuDeviceInformation").get("driverVersion"));}}
so the annotation wasn't doing anything.
> Backport YARN-7330 for GPU in UI to branch-2
>
>
> Key: YARN-9289
> URL: https://issues.apache.org/jira/browse/YARN-9289
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9289-YARN-8200.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[
https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16763183#comment-16763183
]
Jonathan Hung edited comment on YARN-9289 at 2/8/19 12:08 AM:
--
001 patch diffs from branch-3:
* Adds annotations and default constructors to AssignedGpuDevice, GpuDevice,
NMGpuResourceInfo
* Some import changes
* Remove charset from REST return types (was added as part of HADOOP-10075)
* Removed an annotation in GpuDeviceInformation:
{{@javax.xml.bind.annotation.XmlElement(name = "driver_version")}}. It seems in
the original patch the unit test expects driverVersion:
{noformat}
assertEquals("1.2.3",
json.getJSONObject("gpuDeviceInformation").get("driverVersion"));{noformat}
so the annotation wasn't doing anything.
was (Author: jhung):
001 patch diffs from branch-3:
* Adds annotations and default constructors to AssignedGpuDevice, GpuDevice,
NMGpuResourceInfo
* Some import changes
* Remove charset from REST return types (was added as part of HADOOP-10075)
* Removed an annotation in GpuDeviceInformation:
\{{@javax.xml.bind.annotation.XmlElement(name = "driver_version")}}. It seems
in the original patch the unit test expects driverVersion: {{
assertEquals("1.2.3",
json.getJSONObject("gpuDeviceInformation").get("driverVersion"));}}
so the annotation wasn't doing anything.
> Backport YARN-7330 for GPU in UI to branch-2
>
>
> Key: YARN-9289
> URL: https://issues.apache.org/jira/browse/YARN-9289
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Jonathan Hung
>Assignee: Jonathan Hung
>Priority: Major
> Attachments: YARN-9289-YARN-8200.001.patch
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9289: Attachment: YARN-7330-YARN-8200.001.patch > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-7330-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9289: Attachment: YARN-9289-YARN-8200.001.patch > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9289-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9289) Backport YARN-7330 for GPU in UI to branch-2
[ https://issues.apache.org/jira/browse/YARN-9289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hung updated YARN-9289: Attachment: (was: YARN-9289-YARN-8200.001.patch) > Backport YARN-7330 for GPU in UI to branch-2 > > > Key: YARN-9289 > URL: https://issues.apache.org/jira/browse/YARN-9289 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Jonathan Hung >Assignee: Jonathan Hung >Priority: Major > Attachments: YARN-9289-YARN-8200.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
