[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[
https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811154#comment-17811154
]
ASF GitHub Bot commented on YARN-11653:
---
hadoop-yetus commented on PR #6501:
URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1911616762
:confetti_ball: **+1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 30s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 0s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 1 new or modified test files. |
_ trunk Compile Tests _ |
| +1 :green_heart: | mvninstall | 45m 34s | | trunk passed |
| +1 :green_heart: | compile | 0m 59s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 0m 51s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 0m 53s | | trunk passed |
| +1 :green_heart: | mvnsite | 0m 58s | | trunk passed |
| +1 :green_heart: | javadoc | 0m 54s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 47s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 4s | | trunk passed |
| +1 :green_heart: | shadedclient | 38m 21s | | branch has no errors
when building and testing our client artifacts. |
_ Patch Compile Tests _ |
| +1 :green_heart: | mvninstall | 0m 47s | | the patch passed |
| +1 :green_heart: | compile | 0m 53s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 0m 53s | | the patch passed |
| +1 :green_heart: | compile | 0m 45s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 0m 45s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| +1 :green_heart: | checkstyle | 0m 40s | | the patch passed |
| +1 :green_heart: | mvnsite | 0m 48s | | the patch passed |
| +1 :green_heart: | javadoc | 0m 43s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 0m 39s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 2m 1s | | the patch passed |
| +1 :green_heart: | shadedclient | 38m 16s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 100m 53s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | asflicense | 0m 35s | | The patch does not
generate ASF License warnings. |
| | | 241m 6s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6501 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux 79ff9e6564ad 5.15.0-88-generic #98-Ubuntu SMP Mon Oct 2
15:18:56 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | dev-support/bin/hadoop.sh |
| git revision | trunk / c77fc7353211cde90e4dfb42e4e2eadfb1b610c7 |
| Default Java | Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Multi-JDK versions |
/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04
/usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| Test Results |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/1/testReport/ |
| Max. process+thread count | 970 (vs. ulimit of 5500) |
| modules | C:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
|
| Console output |
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6501/1/console |
| versions | git=2.25.1 maven=3.6.3 spotbugs=4.2.2 |
| Powered by | Apache Yetus 0.14.0 https://yetus.apache.org |
This message was
[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9997: - Component/s: resourcemanager > Code cleanup in ZKConfigurationStore > > > Key: YARN-9997 > URL: https://issues.apache.org/jira/browse/YARN-9997 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9997.001.patch, YARN-9997.002.patch, > YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, > YARN-9997.006.patch, YARN-9997.branch-3.2.001.patch > > > Many thins can be improved: > * znodeParentPath could be a local variable > * zkManager could be private, VisibleForTesting annotation is not needed > anymore > * Do something with unchecked casts > * zkManager.safeSetData calls are almost having the same set of parameters: > Simplify this > * Extract zkManager calls to their own methods: They are repeated > * Remove TODOs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9997) Code cleanup in ZKConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-9997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9997: - Affects Version/s: 3.3.0 3.4.0 > Code cleanup in ZKConfigurationStore > > > Key: YARN-9997 > URL: https://issues.apache.org/jira/browse/YARN-9997 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9997.001.patch, YARN-9997.002.patch, > YARN-9997.003.patch, YARN-9997.004.patch, YARN-9997.005.patch, > YARN-9997.006.patch, YARN-9997.branch-3.2.001.patch > > > Many thins can be improved: > * znodeParentPath could be a local variable > * zkManager could be private, VisibleForTesting annotation is not needed > anymore > * Do something with unchecked casts > * zkManager.safeSetData calls are almost having the same set of parameters: > Simplify this > * Extract zkManager calls to their own methods: They are repeated > * Remove TODOs -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10001: -- Affects Version/s: 3.4.0 > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10001-branch-3.2.001.patch, YARN-10001.001.patch, > YARN-10001.002.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10001: -- Affects Version/s: 3.3.0 > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0 >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10001-branch-3.2.001.patch, YARN-10001.001.patch, > YARN-10001.002.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9998) Code cleanup in LeveldbConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-9998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9998: - Affects Version/s: 3.3.0 3.4.0 > Code cleanup in LeveldbConfigurationStore > - > > Key: YARN-9998 > URL: https://issues.apache.org/jira/browse/YARN-9998 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9998.001.patch, YARN-9998.002.patch, > YARN-9998.branch-3.2.001.patch > > > Many things can be improved: > * Field compactionTimer could be a local variable > * Field versiondb should be camelcase > * initDatabase is a very long method: Initialize db / versionDb should be in > separate methods, split this method into smaller chunks > * Remove TODOs > * Remove duplicated code block in > LeveldbConfigurationStore.CompactionTimerTask > * Any other cleanup -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9998) Code cleanup in LeveldbConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-9998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9998: - Component/s: resourcemanager > Code cleanup in LeveldbConfigurationStore > - > > Key: YARN-9998 > URL: https://issues.apache.org/jira/browse/YARN-9998 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9998.001.patch, YARN-9998.002.patch, > YARN-9998.branch-3.2.001.patch > > > Many things can be improved: > * Field compactionTimer could be a local variable > * Field versiondb should be camelcase > * initDatabase is a very long method: Initialize db / versionDb should be in > separate methods, split this method into smaller chunks > * Remove TODOs > * Remove duplicated code block in > LeveldbConfigurationStore.CompactionTimerTask > * Any other cleanup -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10001) Add explanation of unimplemented methods in InMemoryConfigurationStore
[ https://issues.apache.org/jira/browse/YARN-10001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10001: -- Component/s: capacity scheduler > Add explanation of unimplemented methods in InMemoryConfigurationStore > -- > > Key: YARN-10001 > URL: https://issues.apache.org/jira/browse/YARN-10001 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.3.0 >Reporter: Szilard Nemeth >Assignee: Siddharth Ahuja >Priority: Major > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10001-branch-3.2.001.patch, YARN-10001.001.patch, > YARN-10001.002.patch > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10002) Code cleanup and improvements in ConfigurationStoreBaseTest
[ https://issues.apache.org/jira/browse/YARN-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10002: -- Component/s: yarn > Code cleanup and improvements in ConfigurationStoreBaseTest > --- > > Key: YARN-10002 > URL: https://issues.apache.org/jira/browse/YARN-10002 > Project: Hadoop YARN > Issue Type: Improvement > Components: test, yarn >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10002.001.patch, YARN-10002.002.patch, > YARN-10002.003.patch, YARN-10002.004.patch, YARN-10002.005.patch, > YARN-10002.006.patch, YARN-10002.branch-3.2.001.patch > > > * Some protected fields could be package-private > * Could add a helper method that prepares a simple LogMutation with 1, 2 or 3 > updates (Key + value) as this pattern is used extensively in subclasses -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10002) Code cleanup and improvements in ConfigurationStoreBaseTest
[ https://issues.apache.org/jira/browse/YARN-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10002: -- Affects Version/s: 3.3.0 3.4.0 > Code cleanup and improvements in ConfigurationStoreBaseTest > --- > > Key: YARN-10002 > URL: https://issues.apache.org/jira/browse/YARN-10002 > Project: Hadoop YARN > Issue Type: Improvement > Components: test >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10002.001.patch, YARN-10002.002.patch, > YARN-10002.003.patch, YARN-10002.004.patch, YARN-10002.005.patch, > YARN-10002.006.patch, YARN-10002.branch-3.2.001.patch > > > * Some protected fields could be package-private > * Could add a helper method that prepares a simple LogMutation with 1, 2 or 3 > updates (Key + value) as this pattern is used extensively in subclasses -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10002) Code cleanup and improvements in ConfigurationStoreBaseTest
[ https://issues.apache.org/jira/browse/YARN-10002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10002: -- Component/s: test > Code cleanup and improvements in ConfigurationStoreBaseTest > --- > > Key: YARN-10002 > URL: https://issues.apache.org/jira/browse/YARN-10002 > Project: Hadoop YARN > Issue Type: Improvement > Components: test >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-10002.001.patch, YARN-10002.002.patch, > YARN-10002.003.patch, YARN-10002.004.patch, YARN-10002.005.patch, > YARN-10002.006.patch, YARN-10002.branch-3.2.001.patch > > > * Some protected fields could be package-private > * Could add a helper method that prepares a simple LogMutation with 1, 2 or 3 > updates (Key + value) as this pattern is used extensively in subclasses -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10106) Yarn logs CLI filtering by application attempt
[
https://issues.apache.org/jira/browse/YARN-10106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10106:
--
Affects Version/s: 3.4.0
> Yarn logs CLI filtering by application attempt
> --
>
> Key: YARN-10106
> URL: https://issues.apache.org/jira/browse/YARN-10106
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Adam Antal
>Assignee: Hudáky Márton Gyula
>Priority: Trivial
> Fix For: 3.4.0
>
> Attachments: YARN-10106.001.patch, YARN-10106.002.patch,
> YARN-10106.003.patch, YARN-10106.004.patch, YARN-10106.005.patch,
> YARN-10106.006.patch, YARN-10106.007.patch, YARN-10106.008.patch,
> YARN-10106.009.patch, YARN-10106.010.patch, YARN-10106.011.patch,
> YARN-10106.012.patch, YARN-10106.013.patch
>
>
> {{ContainerLogsRequest}} got a new parameter in YARN-10101, which is the
> {{applicationAttempt}} - we can use this new parameter in Yarn logs CLI as
> well to filter by application attempt.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10125) In Federation, kill application from client does not kill Unmanaged AM's and containers launched by Unmanaged AM
[ https://issues.apache.org/jira/browse/YARN-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10125: -- Affects Version/s: 3.4.0 > In Federation, kill application from client does not kill Unmanaged AM's and > containers launched by Unmanaged AM > > > Key: YARN-10125 > URL: https://issues.apache.org/jira/browse/YARN-10125 > Project: Hadoop YARN > Issue Type: Sub-task > Components: client, federation, router >Affects Versions: 3.4.0 >Reporter: D M Murali Krishna Reddy >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.5.0 > > Attachments: YARN-10125.001.patch, YARN-10125.002.patch > > > In Federation, killing an application from client using "bin/yarn application > -kill ", kills the containers only of the home subcluster, > the Unmanaged AM and the containers launched in other subcluster are not > being killed causing blocking of resources. > The containers get killed after the task gets completed and The unmanaged AM > gets killed after 10 minutes of killing the application, killing any > remaining running containers in that subcluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10154) CS Dynamic Queues cannot be configured with absolute resources
[ https://issues.apache.org/jira/browse/YARN-10154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10154: -- Component/s: capacity scheduler > CS Dynamic Queues cannot be configured with absolute resources > -- > > Key: YARN-10154 > URL: https://issues.apache.org/jira/browse/YARN-10154 > Project: Hadoop YARN > Issue Type: Bug > Components: capacity scheduler >Affects Versions: 3.1.3 >Reporter: Sunil G >Assignee: Manikandan R >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10154.001.patch, YARN-10154.002.patch, > YARN-10154.003.patch, YARN-10154.addendum-001.patch, > YARN-10154.addendum-002.patch, YARN-10154.addendum-003.patch, > YARN-10154.addendum-004.patch > > > In CS, ManagedParent Queue and its template cannot take absolute resource > value like > [memory=8192,vcores=8] > Thsi Jira is to track and improve the configuration reading module of > DynamicQueue to support absolute resource values. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10219) YARN service placement constraints is broken
[
https://issues.apache.org/jira/browse/YARN-10219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10219:
--
Component/s: yarn
> YARN service placement constraints is broken
>
>
> Key: YARN-10219
> URL: https://issues.apache.org/jira/browse/YARN-10219
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.1.0, 3.2.0, 3.1.1, 3.1.2, 3.3.0, 3.2.1, 3.1.3
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Blocker
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10219.001.patch, YARN-10219.002.patch,
> YARN-10219.003.patch, YARN-10219.004.patch, YARN-10219.005.patch
>
>
> YARN service placement constraint does not work with node label nor node
> attributes. Example of placement constraints:
> {code}
> "placement_policy": {
> "constraints": [
> {
> "type": "AFFINITY",
> "scope": "NODE",
> "node_attributes": {
> "label":["genfile"]
> },
> "target_tags": [
> "ping"
> ]
> }
> ]
> },
> {code}
> Node attribute added:
> {code} ./bin/yarn nodeattributes -add "host-3.example.com:label=genfile"
> {code}
> Scheduling activities shows:
> {code} Node does not match partition or placement constraints,
> unsatisfied PC expression="in,node,ping", target-type=ALLOCATION_TAG
>
> 1
> host-3.example.com:45454{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10201: -- Affects Version/s: 3.4.0 > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Affects Versions: 3.4.0 >Reporter: Young Chen >Assignee: Shilun Fan >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10201.v0.patch, YARN-10201.v1.patch, > YARN-10201.v10.patch, YARN-10201.v2.patch, YARN-10201.v3.patch, > YARN-10201.v4.patch, YARN-10201.v5.patch, YARN-10201.v6.patch, > YARN-10201.v7.patch, YARN-10201.v8.patch, YARN-10201.v9.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10228) Yarn Service fails if am java opts contains ZK authentication file path
[
https://issues.apache.org/jira/browse/YARN-10228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10228:
--
Component/s: yarn
> Yarn Service fails if am java opts contains ZK authentication file path
> ---
>
> Key: YARN-10228
> URL: https://issues.apache.org/jira/browse/YARN-10228
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.3.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.3.0, 3.4.0
>
> Attachments: YARN-10228.001.patch
>
>
> If i configure
> {code:java}
> yarn.service.am.java.opts=-Xmx768m
> -Djava.security.auth.login.config=/opt/hadoop/etc/jaas-zk.conf
> {code}
> Invalid character error is getting printed .
> This is due to jvm opts validation added in YARN-9718
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10286) PendingContainers bugs in the scheduler outputs
[
https://issues.apache.org/jira/browse/YARN-10286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10286:
--
Component/s: resourcemanager
> PendingContainers bugs in the scheduler outputs
> ---
>
> Key: YARN-10286
> URL: https://issues.apache.org/jira/browse/YARN-10286
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
>Affects Versions: 3.3.0
>Reporter: Adam Antal
>Assignee: Andras Gyori
>Priority: Critical
> Fix For: 3.2.2, 3.4.0, 3.3.1
>
> Attachments: YARN-10286.001.patch, YARN-10286.branch-3.2.001.patch,
> YARN-10286.branch-3.3.001.patch
>
>
> There are some problems with the {{ws/v1/cluster/scheduler}} output of the
> ResourceManager:
> Even though the pendingContainers field is
> [documented|https://hadoop.apache.org/docs/r3.2.0/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html#Cluster_Scheduler_API]
> in the FairScheduler output, it is
> [missing|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/FairSchedulerQueueInfo.java]
> from the actual output. In case of the CapacityScheduler this field
> [exists|https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/dao/CapacitySchedulerQueueInfo.java],
> but it is missing from the documentation. Let's fix both cases!
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10233) [YARN UI2] No Logs were found in "YARN Daemon Logs" page
[ https://issues.apache.org/jira/browse/YARN-10233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10233: -- Affects Version/s: 3.4.0 > [YARN UI2] No Logs were found in "YARN Daemon Logs" page > > > Key: YARN-10233 > URL: https://issues.apache.org/jira/browse/YARN-10233 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: Akhil PB >Assignee: Akhil PB >Priority: Blocker > Fix For: 3.4.0 > > Attachments: YARN-10233.001.patch, > YARN_UI2_Tools_Daemon_Logs_Page_Fixed.png > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query
[ https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10304: -- Component/s: yarn > Create an endpoint for remote application log directory path query > -- > > Key: YARN-10304 > URL: https://issues.apache.org/jira/browse/YARN-10304 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.4.0 >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10304.001.patch, YARN-10304.002.patch, > YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, > YARN-10304.006.patch, YARN-10304.007.patch, YARN-10304.008.patch, > YARN-10304.009.patch > > > The logic of the aggregated log directory path determination (currently based > on configuration) is scattered around the codebase and duplicated multiple > times. By providing a separate class for creating the path for a specific > user, it allows for an abstraction over this logic. This could be used in > place of the previously duplicated logic, moreover, we could provide an > endpoint to query this path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10304) Create an endpoint for remote application log directory path query
[ https://issues.apache.org/jira/browse/YARN-10304?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10304: -- Affects Version/s: 3.4.0 > Create an endpoint for remote application log directory path query > -- > > Key: YARN-10304 > URL: https://issues.apache.org/jira/browse/YARN-10304 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10304.001.patch, YARN-10304.002.patch, > YARN-10304.003.patch, YARN-10304.004.patch, YARN-10304.005.patch, > YARN-10304.006.patch, YARN-10304.007.patch, YARN-10304.008.patch, > YARN-10304.009.patch > > > The logic of the aggregated log directory path determination (currently based > on configuration) is scattered around the codebase and duplicated multiple > times. By providing a separate class for creating the path for a specific > user, it allows for an abstraction over this logic. This could be used in > place of the previously duplicated logic, moreover, we could provide an > endpoint to query this path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10308) Update javadoc and variable names for keytab in yarn services as it supports filesystems other than hdfs and local file system
[
https://issues.apache.org/jira/browse/YARN-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10308:
--
Component/s: documentation
> Update javadoc and variable names for keytab in yarn services as it supports
> filesystems other than hdfs and local file system
> --
>
> Key: YARN-10308
> URL: https://issues.apache.org/jira/browse/YARN-10308
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10308.001.patch, YARN-10308.002.patch
>
>
> 1. Below description should be updated
> {code:java}
> @ApiModelProperty(value = "The URI of the kerberos keytab. It supports two " +
> "schemes \"hdfs\" and \"file\". If the URI starts with \"hdfs://\" " +
> "scheme, it indicates the path on hdfs where the keytab is stored. The
> " +
> "keytab will be localized by YARN and made available to AM in its
> local" +
> " directory. If the URI starts with \"file://\" scheme, it indicates a
> " +
> "path on the local host where the keytab is presumbaly installed by " +
> "admins upfront. ")
> public String getKeytab() {
> return keytab;
> }
> {code}
> 2. Variables below are still named on hdfs which is confusing
> {code:java}
> if ("file".equals(keytabURI.getScheme())) {
> LOG.info("Using a keytab from localhost: " + keytabURI);
> } else {
> Path keytabOnhdfs = new Path(keytabURI);
> if (!fileSystem.getFileSystem().exists(keytabOnhdfs)) {
> LOG.warn(service.getName() + "'s keytab (principalName = "
> + principalName + ") doesn't exist at: " + keytabOnhdfs);
> return;
> }
> LocalResource keytabRes = fileSystem.createAmResource(keytabOnhdfs,
> LocalResourceType.FILE);
> localResource.put(String.format(YarnServiceConstants.KEYTAB_LOCATION,
> service.getName()), keytabRes);
> LOG.info("Adding " + service.getName() + "'s keytab for "
> + "localization, uri = " + keytabOnhdfs);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10325) Document max-parallel-apps for Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10325: -- Affects Version/s: 3.3.1 3.4.0 > Document max-parallel-apps for Capacity Scheduler > - > > Key: YARN-10325 > URL: https://issues.apache.org/jira/browse/YARN-10325 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler, capacityscheduler >Affects Versions: 3.4.0, 3.3.1 >Reporter: Peter Bacsko >Assignee: Peter Bacsko >Priority: Major > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10325-001.patch, YARN-10325-branch-3.3.001.patch > > > New feature introduced by YARN-9930 should be reflected in the upstream > documentation. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10407) Add phantomjsdriver.log to gitignore
[
https://issues.apache.org/jira/browse/YARN-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10407:
--
Component/s: yarn
> Add phantomjsdriver.log to gitignore
>
>
> Key: YARN-10407
> URL: https://issues.apache.org/jira/browse/YARN-10407
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0, 3.3.1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.4.0, 3.3.1
>
>
> After testing hadoop-yarn-applications-catalog-webapp, it genrates
> phantomjsdriver.log.
> {noformat}
> $ mvn test --projects
> org.apache.hadoop:hadoop-yarn-applications-catalog-webapp
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10308) Update javadoc and variable names for keytab in yarn services as it supports filesystems other than hdfs and local file system
[
https://issues.apache.org/jira/browse/YARN-10308?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10308:
--
Affects Version/s: 3.4.0
> Update javadoc and variable names for keytab in yarn services as it supports
> filesystems other than hdfs and local file system
> --
>
> Key: YARN-10308
> URL: https://issues.apache.org/jira/browse/YARN-10308
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Minor
> Fix For: 3.4.0
>
> Attachments: YARN-10308.001.patch, YARN-10308.002.patch
>
>
> 1. Below description should be updated
> {code:java}
> @ApiModelProperty(value = "The URI of the kerberos keytab. It supports two " +
> "schemes \"hdfs\" and \"file\". If the URI starts with \"hdfs://\" " +
> "scheme, it indicates the path on hdfs where the keytab is stored. The
> " +
> "keytab will be localized by YARN and made available to AM in its
> local" +
> " directory. If the URI starts with \"file://\" scheme, it indicates a
> " +
> "path on the local host where the keytab is presumbaly installed by " +
> "admins upfront. ")
> public String getKeytab() {
> return keytab;
> }
> {code}
> 2. Variables below are still named on hdfs which is confusing
> {code:java}
> if ("file".equals(keytabURI.getScheme())) {
> LOG.info("Using a keytab from localhost: " + keytabURI);
> } else {
> Path keytabOnhdfs = new Path(keytabURI);
> if (!fileSystem.getFileSystem().exists(keytabOnhdfs)) {
> LOG.warn(service.getName() + "'s keytab (principalName = "
> + principalName + ") doesn't exist at: " + keytabOnhdfs);
> return;
> }
> LocalResource keytabRes = fileSystem.createAmResource(keytabOnhdfs,
> LocalResourceType.FILE);
> localResource.put(String.format(YarnServiceConstants.KEYTAB_LOCATION,
> service.getName()), keytabRes);
> LOG.info("Adding " + service.getName() + "'s keytab for "
> + "localization, uri = " + keytabOnhdfs);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10328) Too many ZK Curator NodeExists exception logs in YARN Service AM logs
[
https://issues.apache.org/jira/browse/YARN-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10328:
--
Component/s: yarn
> Too many ZK Curator NodeExists exception logs in YARN Service AM logs
> -
>
> Key: YARN-10328
> URL: https://issues.apache.org/jira/browse/YARN-10328
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10328.001.patch
>
>
> Following debug logs are printed everytime when component is started.
> {code:java}
> [pool-6-thread-3] DEBUG zk.CuratorService - path already present:
> /registry/users/server/services/yarn-service/default-worker/components
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for
> /registry/users/hetuserver/services/yarn-service/default-worker/components
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1480)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.hadoop.registry.client.impl.zk.CuratorService.zkMkPath(CuratorService.java:587)
> at
> org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.mknode(RegistryOperationsService.java:99)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:146)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:128)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateServiceRecord(ComponentInstance.java:511)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateContainerStatus(ComponentInstance.java:449)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance$ContainerStatusRetriever.run(ComponentInstance.java:620)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10331) Upgrade node.js to 10.21.0
[ https://issues.apache.org/jira/browse/YARN-10331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10331: -- Affects Version/s: 3.3.1 3.4.0 > Upgrade node.js to 10.21.0 > -- > > Key: YARN-10331 > URL: https://issues.apache.org/jira/browse/YARN-10331 > Project: Hadoop YARN > Issue Type: Bug > Components: build, yarn-ui-v2 >Affects Versions: 3.4.0, 3.3.1 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Fix For: 3.4.0, 3.3.1, 3.2.3, 2.10.3 > > > YARN-10036 upgraded Node.js to 8.17.0, but Node.js 8.x is already EoL. > https://nodejs.org/en/about/releases/ -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10328) Too many ZK Curator NodeExists exception logs in YARN Service AM logs
[
https://issues.apache.org/jira/browse/YARN-10328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10328:
--
Affects Version/s: 3.4.0
> Too many ZK Curator NodeExists exception logs in YARN Service AM logs
> -
>
> Key: YARN-10328
> URL: https://issues.apache.org/jira/browse/YARN-10328
> Project: Hadoop YARN
> Issue Type: Bug
>Affects Versions: 3.4.0
>Reporter: Bilwa S T
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10328.001.patch
>
>
> Following debug logs are printed everytime when component is started.
> {code:java}
> [pool-6-thread-3] DEBUG zk.CuratorService - path already present:
> /registry/users/server/services/yarn-service/default-worker/components
> org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode =
> NodeExists for
> /registry/users/hetuserver/services/yarn-service/default-worker/components
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:128)
> at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
> at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1480)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:740)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl$11.call(CreateBuilderImpl.java:723)
> at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:109)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:720)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:484)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:474)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:454)
> at
> org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
> at
> org.apache.hadoop.registry.client.impl.zk.CuratorService.zkMkPath(CuratorService.java:587)
> at
> org.apache.hadoop.registry.client.impl.zk.RegistryOperationsService.mknode(RegistryOperationsService.java:99)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:146)
> at
> org.apache.hadoop.yarn.service.registry.YarnRegistryViewForProviders.putComponent(YarnRegistryViewForProviders.java:128)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateServiceRecord(ComponentInstance.java:511)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance.updateContainerStatus(ComponentInstance.java:449)
> at
> org.apache.hadoop.yarn.service.component.instance.ComponentInstance$ContainerStatusRetriever.run(ComponentInstance.java:620)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10355) Refactor NM ContainerLaunch.java#orderEnvByDependencies
[
https://issues.apache.org/jira/browse/YARN-10355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10355:
--
Affects Version/s: 3.4.0
> Refactor NM ContainerLaunch.java#orderEnvByDependencies
> ---
>
> Key: YARN-10355
> URL: https://issues.apache.org/jira/browse/YARN-10355
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Assignee: Tamas Domok
>Priority: Minor
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 6h
> Remaining Estimate: 0h
>
> The
> {{org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch#orderEnvByDependencies}}
> and it's helper method \{{getEnvDependencies }}(together with the overrides)
> is hard to read. Some improvements could be made:
> * use Pattern matching in the overrides of getEnvDependencies instead of
> iterating through the environmental variable strings char by char
> * the unit tests contains a lot of repeated code and generally the test
> methods are long - they could be separated into different setup/helper and
> assertion methods
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10350) TestUserGroupMappingPlacementRule fails
[
https://issues.apache.org/jira/browse/YARN-10350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10350:
--
Affects Version/s: 3.4.0
> TestUserGroupMappingPlacementRule fails
> ---
>
> Key: YARN-10350
> URL: https://issues.apache.org/jira/browse/YARN-10350
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Bilwa S T
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10350.001.patch, YARN-10350.002.patch
>
>
> TestUserGroupMappingPlacementRule fails on trunk:
> {noformat}
> [INFO] Running
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
> [ERROR] Tests run: 31, Failures: 1, Errors: 2, Skipped: 0, Time elapsed:
> 2.662 s <<< FAILURE! - in
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule
> [ERROR]
> testResolvedQueueIsNotManaged(org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule)
> Time elapsed: 0.03 s <<< ERROR!
> java.lang.Exception: Unexpected exception,
> expected but
> was
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:28)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.lang.AssertionError: Queue expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.verifyQueueMapping(TestUserGroupMappingPlacementRule.java:236)
> at
> org.apache.hadoop.yarn.server.resourcemanager.placement.TestUserGroupMappingPlacementRule.testResolvedQueueIsNotManaged(TestUserGroupMappingPlacementRule.java:516)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.ExpectException.evaluate(ExpectException.java:19)
> ... 18 more
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10367) Failed to get nodejs 10.21.0 when building docker image
[
https://issues.apache.org/jira/browse/YARN-10367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10367:
--
Affects Version/s: 3.3.1
3.4.0
> Failed to get nodejs 10.21.0 when building docker image
> ---
>
> Key: YARN-10367
> URL: https://issues.apache.org/jira/browse/YARN-10367
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build, webapp
>Affects Versions: 3.4.0, 3.3.1
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Blocker
> Fix For: 3.4.0, 3.3.1
>
>
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/213/console
> {noformat}
> Version '10.21.0-1nodesource1' for 'nodejs' was not found
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10499) TestRouterWebServicesREST fails
[
https://issues.apache.org/jira/browse/YARN-10499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10499:
--
Affects Version/s: 3.4.0
> TestRouterWebServicesREST fails
> ---
>
> Key: YARN-10499
> URL: https://issues.apache.org/jira/browse/YARN-10499
> Project: Hadoop YARN
> Issue Type: Bug
> Components: test
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Attachments:
> patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-router.txt
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> [https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-2488/1/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn.txt]
> {noformat}
> [ERROR] Failures:
> [ERROR]
> TestRouterWebServicesREST.testAppAttemptXML:720->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR]
> TestRouterWebServicesREST.testAppPriorityXML:796->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR] TestRouterWebServicesREST.testAppQueueXML:846->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR] TestRouterWebServicesREST.testAppStateXML:744->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR]
> TestRouterWebServicesREST.testAppTimeoutXML:920->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR]
> TestRouterWebServicesREST.testAppTimeoutsXML:896->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR] TestRouterWebServicesREST.testAppXML:696->performGetCalls:274
> expected:<200> but was:<204>
> [ERROR] TestRouterWebServicesREST.testUpdateAppPriorityXML:832
> expected:<200> but was:<500>
> [ERROR] TestRouterWebServicesREST.testUpdateAppQueueXML:882 expected:<200>
> but was:<500>
> [ERROR] TestRouterWebServicesREST.testUpdateAppStateXML:782 expected:<202>
> but was:<500>
> [ERROR] Errors:
> [ERROR]
> TestRouterWebServicesREST.testGetAppAttemptXML:1292->getAppAttempt:1464 »
> ClientHandler
> [ERROR]
> TestRouterWebServicesREST.testGetAppsMultiThread:1337->testGetContainersXML:1317->getAppAttempt:1464
> » ClientHandler
> [ERROR]
> TestRouterWebServicesREST.testGetContainersXML:1317->getAppAttempt:1464 »
> ClientHandler {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10407) Add phantomjsdriver.log to gitignore
[
https://issues.apache.org/jira/browse/YARN-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10407:
--
Affects Version/s: 3.3.1
3.4.0
> Add phantomjsdriver.log to gitignore
>
>
> Key: YARN-10407
> URL: https://issues.apache.org/jira/browse/YARN-10407
> Project: Hadoop YARN
> Issue Type: Improvement
>Affects Versions: 3.4.0, 3.3.1
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Minor
> Fix For: 3.4.0, 3.3.1
>
>
> After testing hadoop-yarn-applications-catalog-webapp, it genrates
> phantomjsdriver.log.
> {noformat}
> $ mvn test --projects
> org.apache.hadoop:hadoop-yarn-applications-catalog-webapp
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811130#comment-17811130 ] ASF GitHub Bot commented on YARN-11650: --- slfan1989 commented on PR #6490: URL: https://github.com/apache/hadoop/pull/6490#issuecomment-1911516346 @yangjiandan Thanks for the contribution! let’s take a look at other partners’ suggestions. If there are no comments, we will merge this PR in 1 - 2 days. > Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > --- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811129#comment-17811129 ] ASF GitHub Bot commented on YARN-11653: --- slfan1989 commented on PR #6501: URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1911512755 LGTM. > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1786#comment-1786 ] ASF GitHub Bot commented on YARN-11653: --- yangjiandan commented on PR #6501: URL: https://github.com/apache/hadoop/pull/6501#issuecomment-1911436943 In my test env, the nodes page is below:  > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated YARN-11653: -- Labels: pull-request-available (was: ) > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
[ https://issues.apache.org/jira/browse/YARN-11653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1783#comment-1783 ] ASF GitHub Bot commented on YARN-11653: --- yangjiandan opened a new pull request, #6501: URL: https://github.com/apache/hadoop/pull/6501 ### Description of PR see YARN-11653 ### How was this patch tested? ### For code changes: - [ ] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')? - [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation? - [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under [ASF 2.0](http://www.apache.org/legal/resolved.html#category-a)? - [ ] If applicable, have you updated the `LICENSE`, `LICENSE-binary`, `NOTICE-binary` files? > Add Totoal_Memory and Total_Vcores columns in Nodes page > > > Key: YARN-11653 > URL: https://issues.apache.org/jira/browse/YARN-11653 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > > Currently, the RM nodes page includes used and available memory/vcore , but > it lacks a total column, which is not intuitive enough for users. When the > resource capacities of nodes in the cluster vary widely, we may need to sort > the nodes to facilitate the comparison of metrics among same types of nodes. > Therefore, it is necessary to add columns for total CPU/memory on the nodes > page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11653) Add Totoal_Memory and Total_Vcores columns in Nodes page
Jiandan Yang created YARN-11653: Summary: Add Totoal_Memory and Total_Vcores columns in Nodes page Key: YARN-11653 URL: https://issues.apache.org/jira/browse/YARN-11653 Project: Hadoop YARN Issue Type: New Feature Reporter: Jiandan Yang Assignee: Jiandan Yang Currently, the RM nodes page includes used and available memory/vcore , but it lacks a total column, which is not intuitive enough for users. When the resource capacities of nodes in the cluster vary widely, we may need to sort the nodes to facilitate the comparison of metrics among same types of nodes. Therefore, it is necessary to add columns for total CPU/memory on the nodes page. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11650) Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, FiCaSchedulerApp and AbstractCSQueue
[ https://issues.apache.org/jira/browse/YARN-11650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17811096#comment-17811096 ] ASF GitHub Bot commented on YARN-11650: --- yangjiandan commented on PR #6490: URL: https://github.com/apache/hadoop/pull/6490#issuecomment-1911328177 @slfan1989 Please help me review this patch again. Thanks. > Refactoring variable names related multiNodePolicy in MultiNodePolicySpec, > FiCaSchedulerApp and AbstractCSQueue > --- > > Key: YARN-11650 > URL: https://issues.apache.org/jira/browse/YARN-11650 > Project: Hadoop YARN > Issue Type: Improvement > Components: RM >Reporter: Jiandan Yang >Assignee: Jiandan Yang >Priority: Major > Labels: pull-request-available > > In classes related to MultiNodePolicy support, some variable names do not > accurately reflect their true meanings. For instance, a variable named > *queue* is actually representing the class name of a policy, and a variable > named *policyName* denotes the class name of the policy. This may cause > confusion for the readers of the code. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-5597) YARN Federation improvements
[ https://issues.apache.org/jira/browse/YARN-5597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-5597: - Release Note: We have enhanced the YARN Federation functionality for improved usability. The enhanced features are as follows: 1. YARN Router now boasts a full implementation of all interfaces including the ApplicationClientProtocol, ResourceManagerAdministrationProtocol, and RMWebServiceProtocol. 2. YARN Router support for application cleanup and automatic offline mechanisms for subCluster. 3. Code improvements were undertaken for the Router and AMRMProxy, along with enhancements to previously pending functionalities. 4. Audit logs and Metrics for Router received upgrades. 5. A boost in cluster security features was achieved, with the inclusion of Kerberos support. 6. The page function of the router has been enhanced. 7. A set of commands has been added to the Router side for operating on SubClusters and Policies. > YARN Federation improvements > > > Key: YARN-5597 > URL: https://issues.apache.org/jira/browse/YARN-5597 > Project: Hadoop YARN > Issue Type: Improvement > Components: federation >Affects Versions: 3.4.0 >Reporter: Subramaniam Krishnan >Assignee: Subramaniam Krishnan >Priority: Major > Labels: federation > Fix For: 3.4.0 > > > This umbrella JIRA tracks set of improvements over the YARN Federation MVP > (YARN-2915) -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-11041) Replace all occurences of queuePath with the new QueuePath class - followup
[
https://issues.apache.org/jira/browse/YARN-11041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17810992#comment-17810992
]
ASF GitHub Bot commented on YARN-11041:
---
hadoop-yetus commented on PR #6499:
URL: https://github.com/apache/hadoop/pull/6499#issuecomment-1910729696
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Logfile | Comment |
|::|--:|:|::|:---:|
| +0 :ok: | reexec | 0m 23s | | Docker mode activated. |
_ Prechecks _ |
| +1 :green_heart: | dupname | 0m 2s | | No case conflicting files
found. |
| +0 :ok: | codespell | 0m 0s | | codespell was not available. |
| +0 :ok: | detsecrets | 0m 0s | | detect-secrets was not available.
|
| +1 :green_heart: | @author | 0m 0s | | The patch does not contain
any @author tags. |
| +1 :green_heart: | test4tests | 0m 0s | | The patch appears to
include 90 new or modified test files. |
_ trunk Compile Tests _ |
| +0 :ok: | mvndep | 14m 33s | | Maven dependency ordering for branch |
| -1 :x: | mvninstall | 24m 55s |
[/branch-mvninstall-root.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6499/1/artifact/out/branch-mvninstall-root.txt)
| root in trunk failed. |
| +1 :green_heart: | compile | 3m 56s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | compile | 3m 27s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | checkstyle | 1m 23s | | trunk passed |
| +1 :green_heart: | mvnsite | 1m 56s | | trunk passed |
| +1 :green_heart: | javadoc | 1m 42s | | trunk passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 49s | | trunk passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 1s | | trunk passed |
| +1 :green_heart: | shadedclient | 19m 32s | | branch has no errors
when building and testing our client artifacts. |
| -0 :warning: | patch | 19m 46s | | Used diff version of patch file.
Binary files and potentially other changes not applied. Please rebase and
squash commits if necessary. |
_ Patch Compile Tests _ |
| +0 :ok: | mvndep | 0m 20s | | Maven dependency ordering for patch |
| +1 :green_heart: | mvninstall | 1m 10s | | the patch passed |
| +1 :green_heart: | compile | 3m 12s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javac | 3m 12s | | the patch passed |
| +1 :green_heart: | compile | 3m 13s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | javac | 3m 13s | | the patch passed |
| +1 :green_heart: | blanks | 0m 0s | | The patch has no blanks
issues. |
| -0 :warning: | checkstyle | 0m 59s |
[/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt](https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6499/1/artifact/out/results-checkstyle-hadoop-yarn-project_hadoop-yarn.txt)
| hadoop-yarn-project/hadoop-yarn: The patch generated 7 new + 1792 unchanged
- 197 fixed = 1799 total (was 1989) |
| +1 :green_heart: | mvnsite | 1m 44s | | the patch passed |
| +1 :green_heart: | javadoc | 1m 32s | | the patch passed with JDK
Ubuntu-11.0.21+9-post-Ubuntu-0ubuntu120.04 |
| +1 :green_heart: | javadoc | 1m 40s | | the patch passed with JDK
Private Build-1.8.0_392-8u392-ga-1~20.04-b08 |
| +1 :green_heart: | spotbugs | 3m 22s | | the patch passed |
| +1 :green_heart: | shadedclient | 19m 18s | | patch has no errors
when building and testing our client artifacts. |
_ Other Tests _ |
| +1 :green_heart: | unit | 86m 9s | |
hadoop-yarn-server-resourcemanager in the patch passed. |
| +1 :green_heart: | unit | 25m 22s | | hadoop-yarn-client in the patch
passed. |
| +1 :green_heart: | unit | 0m 30s | | hadoop-yarn-server-router in
the patch passed. |
| +1 :green_heart: | unit | 0m 51s | |
hadoop-yarn-server-globalpolicygenerator in the patch passed. |
| +1 :green_heart: | asflicense | 0m 36s | | The patch does not
generate ASF License warnings. |
| | | 231m 11s | | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | ClientAPI=1.44 ServerAPI=1.44 base:
https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-6499/1/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/6499 |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets |
| uname | Linux b7941499822e 5.15.0-88-generic
[jira] [Updated] (YARN-10570) Remove "experimental" warning message from fs2cs
[
https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10570:
--
Component/s: scheduler
> Remove "experimental" warning message from fs2cs
>
>
> Key: YARN-10570
> URL: https://issues.apache.org/jira/browse/YARN-10570
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 3.4.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10570-001.patch
>
>
> Although {{fs2cs}} tool has been in constant development, it's been used and
> tested by a group of people, so let's remove the following message:
> {{WARNING: This feature is experimental and not intended for production use!}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10503) Support queue capacity in terms of absolute resources with custom resourceType.
[
https://issues.apache.org/jira/browse/YARN-10503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10503:
--
Component/s: gpu
> Support queue capacity in terms of absolute resources with custom
> resourceType.
> ---
>
> Key: YARN-10503
> URL: https://issues.apache.org/jira/browse/YARN-10503
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: gpu
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 3.4.0, 3.3.1, 3.2.4
>
> Attachments: YARN-10503-branch-3.3.010.patch, YARN-10503.001.patch,
> YARN-10503.002.patch, YARN-10503.003.patch, YARN-10503.004.patch,
> YARN-10503.005.patch, YARN-10503.006.patch, YARN-10503.007.patch,
> YARN-10503.008.patch, YARN-10503.009.patch, YARN-10503.010.patch
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> Now the absolute resources are memory and cores.
> {code:java}
> /**
> * Different resource types supported.
> */
> public enum AbsoluteResourceType {
> MEMORY, VCORES;
> }{code}
> But in our GPU production clusters, we need to support more resourceTypes.
> It's very import for cluster scaling when with different resourceType
> absolute demands.
>
> This Jira will handle GPU first.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10561) Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog webapp
[ https://issues.apache.org/jira/browse/YARN-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10561: -- Affects Version/s: 3.3.2 3.4.0 > Upgrade node.js to 12.22.1 and yarn to 1.22.5 in YARN application catalog > webapp > > > Key: YARN-10561 > URL: https://issues.apache.org/jira/browse/YARN-10561 > Project: Hadoop YARN > Issue Type: Bug > Components: webapp >Affects Versions: 3.4.0, 3.3.2 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Critical > Labels: pull-request-available > Fix For: 3.4.0, 3.3.2 > > Time Spent: 1h 20m > Remaining Estimate: 0h > > YARN application catalog webapp is using node.js 8.11.3, and 8.x are already > EoL. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10520) Deprecated the residual nested class for the LCEResourceHandler
[ https://issues.apache.org/jira/browse/YARN-10520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10520: -- Affects Version/s: 3.4.0 > Deprecated the residual nested class for the LCEResourceHandler > --- > > Key: YARN-10520 > URL: https://issues.apache.org/jira/browse/YARN-10520 > Project: Hadoop YARN > Issue Type: Test > Components: nodemanager >Affects Versions: 3.4.0 >Reporter: Wanqiang Ji >Assignee: Wanqiang Ji >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 50m > Remaining Estimate: 0h > > The old LCEResourceHandler interface hierarchy was deprecated, but some > nested class was left. Such as > CustomCgroupsLCEResourceHandler/MockLinuxContainerExecutor/TestResourceHandler > etc. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10609: -- Component/s: documentation > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task > Components: documentation >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10609.001.patch, YARN-10609.002.patch, > YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10889) [Umbrella] Queue Creation in Capacity Scheduler - Tech debts
[ https://issues.apache.org/jira/browse/YARN-10889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10889: Assignee: Benjamin Teke (was: Szilard Nemeth) > [Umbrella] Queue Creation in Capacity Scheduler - Tech debts > > > Key: YARN-10889 > URL: https://issues.apache.org/jira/browse/YARN-10889 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Major > > Follow-up of YARN-10496 -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Consider legacy auto queue creation absolute resource template to avoid rounding errors
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10590: -- Component/s: capacity scheduler > Consider legacy auto queue creation absolute resource template to avoid > rounding errors > --- > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Reporter: Qi Zhu >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > Time Spent: 40m > Remaining Estimate: 0h > > "Because as we discussed in YARN-10504 , the initialization of auto created > queues from template was changed (see comment and comment)." > 1. As the comment we discussed, we found the effective core is different(the > gap), because the update effective will override the absolute auto created > leaf queue. > 2. But actually, the new logic in YARN-10504 override is right, the > difference is caused by test case , don't consider the calculation loss of > multi resource type, the cap/absolute are all calculated by one type, > (memory) in DefaultResourceCalculator, (dominant type) in > DominantResourceCalculator. As we known in the comment, the absolute auto > created leaf queue will merge the effective resource by cap/absolute > calculated result, this caused the gap. > 2. In other case(not absolute case) in the auto created leaf queue, the merge > will not cause the gap, in update effective resource override will also use > the one type calculated result. > 3. So this jira just make the test right, the calculation result is already > right. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10590) Consider legacy auto queue creation absolute resource template to avoid rounding errors
[ https://issues.apache.org/jira/browse/YARN-10590?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10590: -- Affects Version/s: 3.4.0 > Consider legacy auto queue creation absolute resource template to avoid > rounding errors > --- > > Key: YARN-10590 > URL: https://issues.apache.org/jira/browse/YARN-10590 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Andras Gyori >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10590.001.patch, YARN-10590.002.patch > > Time Spent: 40m > Remaining Estimate: 0h > > "Because as we discussed in YARN-10504 , the initialization of auto created > queues from template was changed (see comment and comment)." > 1. As the comment we discussed, we found the effective core is different(the > gap), because the update effective will override the absolute auto created > leaf queue. > 2. But actually, the new logic in YARN-10504 override is right, the > difference is caused by test case , don't consider the calculation loss of > multi resource type, the cap/absolute are all calculated by one type, > (memory) in DefaultResourceCalculator, (dominant type) in > DominantResourceCalculator. As we known in the comment, the absolute auto > created leaf queue will merge the effective resource by cap/absolute > calculated result, this caused the gap. > 2. In other case(not absolute case) in the auto created leaf queue, the merge > will not cause the gap, in update effective resource override will also use > the one type calculated result. > 3. So this jira just make the test right, the calculation result is already > right. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10609) Update the document for YARN-10531(Be able to disable user limit factor for CapacityScheduler Leaf Queue)
[ https://issues.apache.org/jira/browse/YARN-10609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10609: -- Affects Version/s: 3.4.0 > Update the document for YARN-10531(Be able to disable user limit factor for > CapacityScheduler Leaf Queue) > - > > Key: YARN-10609 > URL: https://issues.apache.org/jira/browse/YARN-10609 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10609.001.patch, YARN-10609.002.patch, > YARN-10609.003.patch, YARN-10609.004.patch, YARN-10609.005.patch > > > Since we have finished YARN-10531. > We should update the corresponding document. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10570) Remove "experimental" warning message from fs2cs
[
https://issues.apache.org/jira/browse/YARN-10570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10570:
--
Affects Version/s: 3.4.0
> Remove "experimental" warning message from fs2cs
>
>
> Key: YARN-10570
> URL: https://issues.apache.org/jira/browse/YARN-10570
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Peter Bacsko
>Assignee: Peter Bacsko
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10570-001.patch
>
>
> Although {{fs2cs}} tool has been in constant development, it's been used and
> tested by a group of people, so let's remove the following message:
> {{WARNING: This feature is experimental and not intended for production use!}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10611) Fix that shaded should be used for google guava imports in YARN-10352.
[ https://issues.apache.org/jira/browse/YARN-10611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10611: -- Component/s: test > Fix that shaded should be used for google guava imports in YARN-10352. > -- > > Key: YARN-10611 > URL: https://issues.apache.org/jira/browse/YARN-10611 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10611.001.patch > > > Fix that shaded should be used for google guava imports in YARN-10352. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10610) Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath
[ https://issues.apache.org/jira/browse/YARN-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10610: -- Affects Version/s: 3.4.0 > Add queuePath to RESTful API for CapacityScheduler consistent with > FairScheduler queuePath > -- > > Key: YARN-10610 > URL: https://issues.apache.org/jira/browse/YARN-10610 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10610.001.patch, YARN-10610.002.patch, > YARN-10610.003.patch, image-2021-02-03-13-47-13-516.png, > image-2021-02-11-20-58-37-436.png > > > The cs only have queueName, but not full queuePath. > !image-2021-02-03-13-47-13-516.png|width=631,height=356! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10610) Add queuePath to RESTful API for CapacityScheduler consistent with FairScheduler queuePath
[ https://issues.apache.org/jira/browse/YARN-10610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10610: -- Component/s: capacity scheduler > Add queuePath to RESTful API for CapacityScheduler consistent with > FairScheduler queuePath > -- > > Key: YARN-10610 > URL: https://issues.apache.org/jira/browse/YARN-10610 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10610.001.patch, YARN-10610.002.patch, > YARN-10610.003.patch, image-2021-02-03-13-47-13-516.png, > image-2021-02-11-20-58-37-436.png > > > The cs only have queueName, but not full queuePath. > !image-2021-02-03-13-47-13-516.png|width=631,height=356! > > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10623) Capacity scheduler should support refresh queue automatically by a thread policy.
[ https://issues.apache.org/jira/browse/YARN-10623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10623: -- Affects Version/s: 3.4.0 > Capacity scheduler should support refresh queue automatically by a thread > policy. > - > > Key: YARN-10623 > URL: https://issues.apache.org/jira/browse/YARN-10623 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10623.001.patch, YARN-10623.002.patch, > YARN-10623.003.patch, YARN-10623.004.patch, YARN-10623.005.patch, > YARN-10623.006.patch, YARN-10623.007.patch, YARN-10623.008.patch, > YARN-10623.009.patch, YARN-10623.010.patch > > Time Spent: 20m > Remaining Estimate: 0h > > In fair scheduler, it is supported that refresh queue related conf > automatically by a thread to reload, but in capacity scheduler we only > support to refresh queue related changes by refreshQueues, it is needed for > our cluster to realize queue manage. > cc [~wangda] [~ztang] [~pbacsko] [~snemeth] [~gandras] [~bteke] [~shuzirra] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10921) AbstractCSQueue: Node Labels logic is scattered and iteration logic is repeated all over the place
[ https://issues.apache.org/jira/browse/YARN-10921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10921: - Parent Issue: YARN-11652 (was: YARN-10889) > AbstractCSQueue: Node Labels logic is scattered and iteration logic is > repeated all over the place > -- > > Key: YARN-10921 > URL: https://issues.apache.org/jira/browse/YARN-10921 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Peter Szucs >Priority: Minor > > TODO items: > - Check original Node labels epic / jiras? > - Think about ways to improve repetitive iteration on configuredNodeLabels > - Search for: "String label" in code > Code blocks to handle Node labels: > - AbstractCSQueue#setupQueueConfigs > - AbstractCSQueue#getQueueConfigurations > - AbstractCSQueue#accessibleToPartition > - AbstractCSQueue#getNodeLabelsForQueue > - AbstractCSQueue#updateAbsoluteCapacities > - AbstractCSQueue#updateConfigurableResourceRequirement > - CSQueueUtils#loadCapacitiesByLabelsFromConf > - AutoCreatedLeafQueue -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10615) Fix Auto Queue Creation hierarchy construction to use queue path instead of short queue name
[ https://issues.apache.org/jira/browse/YARN-10615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10615: -- Affects Version/s: 3.4.0 > Fix Auto Queue Creation hierarchy construction to use queue path instead of > short queue name > > > Key: YARN-10615 > URL: https://issues.apache.org/jira/browse/YARN-10615 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Affects Versions: 3.4.0 >Reporter: Andras Gyori >Assignee: Andras Gyori >Priority: Critical > Fix For: 3.4.0 > > Attachments: YARN-10615.001.patch > > > The CSAutoQueueHandler validates the parent hierarchy of a queue path on > creation. The queues are queried by their short name, which might cause > ambiguity (short name of root.a is the same as root.b.a). We need to query > the queues from QueueManager by full queue path. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10618) RM UI2 Application page shows the AM preempted containers instead of the nonAM ones
[ https://issues.apache.org/jira/browse/YARN-10618?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10618: -- Affects Version/s: 3.4.0 > RM UI2 Application page shows the AM preempted containers instead of the > nonAM ones > --- > > Key: YARN-10618 > URL: https://issues.apache.org/jira/browse/YARN-10618 > Project: Hadoop YARN > Issue Type: Bug > Components: yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: Benjamin Teke >Assignee: Benjamin Teke >Priority: Minor > Fix For: 3.4.0 > > Attachments: YARN-10618.001.patch > > > YARN RM UIv2 application page shows the AM preempted containers under both > the _Num Non-AM container preempted_ and _Num AM container preempted_. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10627) Extend logging to give more information about weight mode
[
https://issues.apache.org/jira/browse/YARN-10627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10627:
--
Affects Version/s: 3.4.0
> Extend logging to give more information about weight mode
> -
>
> Key: YARN-10627
> URL: https://issues.apache.org/jira/browse/YARN-10627
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: yarn
>Affects Versions: 3.4.0
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Fix For: 3.4.0
>
> Attachments: YARN-10627.001.patch, YARN-10627.002.patch,
> YARN-10627.003.patch, YARN-10627.004.patch, YARN-10627.005.patch,
> YARN-10627.006.patch, image-2021-02-20-00-07-09-875.png
>
>
> In YARN-10504 weight mode was added, however the logged information about the
> created queues or the toString methods weren't updated accordingly. Some
> examples:
> ParentQueue#setupQueueConfigs:
> {code:java}
> LOG.info(queueName + ", capacity=" + this.queueCapacities.getCapacity()
> + ", absoluteCapacity=" + this.queueCapacities.getAbsoluteCapacity()
> + ", maxCapacity=" + this.queueCapacities.getMaximumCapacity()
> + ", absoluteMaxCapacity=" + this.queueCapacities
> .getAbsoluteMaximumCapacity() + ", state=" + getState() + ", acls="
> + aclsString + ", labels=" + labelStrBuilder.toString() + "\n"
> + ", reservationsContinueLooking=" + reservationsContinueLooking
> + ", orderingPolicy=" + getQueueOrderingPolicyConfigName()
> + ", priority=" + priority
> + ", allowZeroCapacitySum=" + allowZeroCapacitySum);
> {code}
> ParentQueue#toString:
> {code:java}
> public String toString() {
> return queueName + ": " +
> "numChildQueue= " + childQueues.size() + ", " +
> "capacity=" + queueCapacities.getCapacity() + ", " +
> "absoluteCapacity=" + queueCapacities.getAbsoluteCapacity() + ", " +
> "usedResources=" + queueUsage.getUsed() +
> "usedCapacity=" + getUsedCapacity() + ", " +
> "numApps=" + getNumApplications() + ", " +
> "numContainers=" + getNumContainers();
> }
> {code}
> LeafQueue#setupQueueConfigs:
> {code:java}
> LOG.info(
> "Initializing " + getQueuePath() + "\n" + "capacity = "
> + queueCapacities.getCapacity()
> + " [= (float) configuredCapacity / 100 ]" + "\n"
> + "absoluteCapacity = " + queueCapacities.getAbsoluteCapacity()
> + " [= parentAbsoluteCapacity * capacity ]" + "\n"
> + "maxCapacity = " + queueCapacities.getMaximumCapacity()
> + " [= configuredMaxCapacity ]" + "\n" + "absoluteMaxCapacity =
> "
> + queueCapacities.getAbsoluteMaximumCapacity()
> + " [= 1.0 maximumCapacity undefined, "
> + "(parentAbsoluteMaxCapacity * maximumCapacity) / 100
> otherwise ]"
> + "\n" + "effectiveMinResource=" +
> getEffectiveCapacity(CommonNodeLabelsManager.NO_LABEL) + "\n"
> + " , effectiveMaxResource=" +
> getEffectiveMaxCapacity(CommonNodeLabelsManager.NO_LABEL)
> + "\n" + "userLimit = " + usersManager.getUserLimit()
> + " [= configuredUserLimit ]" + "\n" + "userLimitFactor = "
> + usersManager.getUserLimitFactor()
> + " [= configuredUserLimitFactor ]" + "\n" + "maxApplications =
> "
> + maxApplications
> + " [= configuredMaximumSystemApplicationsPerQueue or"
> + " (int)(configuredMaximumSystemApplications *
> absoluteCapacity)]"
> + "\n" + "maxApplicationsPerUser = " + maxApplicationsPerUser
> + " [= (int)(maxApplications * (userLimit / 100.0f) * "
> + "userLimitFactor) ]" + "\n"
> + "maxParallelApps = " + getMaxParallelApps() + "\n"
> + "usedCapacity = " +
> + queueCapacities.getUsedCapacity() + " [= usedResourcesMemory
> / "
> + "(clusterResourceMemory * absoluteCapacity)]" + "\n"
> + "absoluteUsedCapacity = " + absoluteUsedCapacity
> + " [= usedResourcesMemory / clusterResourceMemory]" + "\n"
> + "maxAMResourcePerQueuePercent = " +
> maxAMResourcePerQueuePercent
> + " [= configuredMaximumAMResourcePercent ]" + "\n"
> + "minimumAllocationFactor = " + minimumAllocationFactor
> + " [= (float)(maximumAllocationMemory -
> minimumAllocationMemory) / "
> + "maximumAllocationMemory ]" + "\n" + "maximumAllocation = "
> + maximumAllocation + " [= configuredMaxAllocation ]" + "\n"
> + "numContainers = " + numContainers
> + " [=
[jira] [Updated] (YARN-10646) TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario
[
https://issues.apache.org/jira/browse/YARN-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10646:
--
Component/s: capacity scheduler
> TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the
> correct scenario
> -
>
> Key: YARN-10646
> URL: https://issues.apache.org/jira/browse/YARN-10646
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: capacity scheduler
>Affects Versions: 3.4.0
> Environment:
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There is a mixup in TestCapacitySchedulerWeightMode in the configuration
> creator method comments and the test case descriptor comments. See the
> following code:
> {code:java}
> /*
>* Queue structure:
>* root (*)
>* ___
>* / \
>* a x(=100%), y(50%) b y(=50%), z(=100%)
>* __
>* / / \
>* a1 ([x,y]: w=100)b1(no) b2([y,z]: w=100)
>*
>* Parent uses weight, child uses percentage
>*/
> public static Configuration getCSConfWithLabelsParentUsePctChildUseWeight(
> Configuration config) {
> {code}
> While inside the method all the queues (including the second level ones) are
> configured with capacity, only some labels are configured with weights:
> {code:java}
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "x", 100);
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "y", 100);
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "z", 100);
> ...
> conf.setQueues(A, new String[] { "a1" });
> conf.setCapacityByLabel(A1, RMNodeLabelsManager.NO_LABEL, 100);
> conf.setMaximumCapacity(A1, 100);
> conf.setAccessibleNodeLabels(A1, toSet("x", "y"));
> conf.setDefaultNodeLabelExpression(A1, "x");
> conf.setCapacityByLabel(A1, "x", 100);
> conf.setCapacityByLabel(A1, "y", 100);
> conf.setQueues(B, new String[] { "b1", "b2" });
> conf.setCapacityByLabel(B1, RMNodeLabelsManager.NO_LABEL, 50);
> conf.setMaximumCapacity(B1, 50);
> conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET);
> conf.setCapacityByLabel(B2, RMNodeLabelsManager.NO_LABEL, 50);
> conf.setMaximumCapacity(B2, 50);
> conf.setAccessibleNodeLabels(B2, toSet("y", "z"));
> conf.setCapacityByLabel(B2, "y", 100);
> conf.setCapacityByLabel(B2, "z", 100);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10646) TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the correct scenario
[
https://issues.apache.org/jira/browse/YARN-10646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10646:
--
Affects Version/s: 3.4.0
> TestCapacitySchedulerWeightMode test descriptor comments doesn't reflect the
> correct scenario
> -
>
> Key: YARN-10646
> URL: https://issues.apache.org/jira/browse/YARN-10646
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.4.0
> Environment:
>Reporter: Benjamin Teke
>Assignee: Benjamin Teke
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h 10m
> Remaining Estimate: 0h
>
> There is a mixup in TestCapacitySchedulerWeightMode in the configuration
> creator method comments and the test case descriptor comments. See the
> following code:
> {code:java}
> /*
>* Queue structure:
>* root (*)
>* ___
>* / \
>* a x(=100%), y(50%) b y(=50%), z(=100%)
>* __
>* / / \
>* a1 ([x,y]: w=100)b1(no) b2([y,z]: w=100)
>*
>* Parent uses weight, child uses percentage
>*/
> public static Configuration getCSConfWithLabelsParentUsePctChildUseWeight(
> Configuration config) {
> {code}
> While inside the method all the queues (including the second level ones) are
> configured with capacity, only some labels are configured with weights:
> {code:java}
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "x", 100);
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "y", 100);
> conf.setLabeledQueueWeight(CapacitySchedulerConfiguration.ROOT, "z", 100);
> ...
> conf.setQueues(A, new String[] { "a1" });
> conf.setCapacityByLabel(A1, RMNodeLabelsManager.NO_LABEL, 100);
> conf.setMaximumCapacity(A1, 100);
> conf.setAccessibleNodeLabels(A1, toSet("x", "y"));
> conf.setDefaultNodeLabelExpression(A1, "x");
> conf.setCapacityByLabel(A1, "x", 100);
> conf.setCapacityByLabel(A1, "y", 100);
> conf.setQueues(B, new String[] { "b1", "b2" });
> conf.setCapacityByLabel(B1, RMNodeLabelsManager.NO_LABEL, 50);
> conf.setMaximumCapacity(B1, 50);
> conf.setAccessibleNodeLabels(B1, RMNodeLabelsManager.EMPTY_STRING_SET);
> conf.setCapacityByLabel(B2, RMNodeLabelsManager.NO_LABEL, 50);
> conf.setMaximumCapacity(B2, 50);
> conf.setAccessibleNodeLabels(B2, toSet("y", "z"));
> conf.setCapacityByLabel(B2, "y", 100);
> conf.setCapacityByLabel(B2, "z", 100);
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10647) Fix TestRMNodeLabelsManager failed after YARN-10501.
[ https://issues.apache.org/jira/browse/YARN-10647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10647: -- Component/s: test > Fix TestRMNodeLabelsManager failed after YARN-10501. > > > Key: YARN-10647 > URL: https://issues.apache.org/jira/browse/YARN-10647 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Attachments: YARN-10647.001.patch > > > YARN-10501 introduced the TestRMNodeLabelsManager test failed. > Because the deactive for label, should be null. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10647) Fix TestRMNodeLabelsManager failed after YARN-10501.
[ https://issues.apache.org/jira/browse/YARN-10647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10647: -- Affects Version/s: 3.3.1 3.4.0 > Fix TestRMNodeLabelsManager failed after YARN-10501. > > > Key: YARN-10647 > URL: https://issues.apache.org/jira/browse/YARN-10647 > Project: Hadoop YARN > Issue Type: Bug > Components: test >Affects Versions: 3.4.0, 3.3.1 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Attachments: YARN-10647.001.patch > > > YARN-10501 introduced the TestRMNodeLabelsManager test failed. > Because the deactive for label, should be null. > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10888) [Umbrella] New capacity modes for CS
[ https://issues.apache.org/jira/browse/YARN-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke resolved YARN-10888. -- Resolution: Fixed > [Umbrella] New capacity modes for CS > > > Key: YARN-10888 > URL: https://issues.apache.org/jira/browse/YARN-10888 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: capacity_scheduler_queue_capacity.pdf > > > *Investigate how resource allocation configuration could be more consistent > in CapacityScheduler* > It would be nice if everywhere where a capacity can be defined could be > defined the same way: > * With fixed amounts (e.g. 1 GB memory, 8 vcores, 3 GPU) > * With percentages > ** Percentage of all resources (eg 10% of all memory, vcore, GPU) > ** Percentage per resource type (eg 10% memory, 25% vcore, 50% GPU) > * Allow mixing different modes under one hierarchy but not under the same > parent queues. > We need to determine all configuration options where capacities can be > defined, and see if it is possible to extend the configuration, or if it > makes sense in that case. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10653) Fixed the findbugs issues introduced by YARN-10647.
[ https://issues.apache.org/jira/browse/YARN-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10653: -- Component/s: test > Fixed the findbugs issues introduced by YARN-10647. > --- > > Key: YARN-10653 > URL: https://issues.apache.org/jira/browse/YARN-10653 > Project: Hadoop YARN > Issue Type: Improvement > Components: test >Affects Versions: 3.4.0, 3.3.1 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Attachments: YARN-10653.001.patch, YARN-10653.002.patch, > image-2021-02-26-13-49-18-241.png > > > In YARN-10647 > I fixed TestRMNodeLabelsManager failed after YARN-10501. > But the finding bugs should be fixed also. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10656) Parsing error in CapacityScheduler.md
[
https://issues.apache.org/jira/browse/YARN-10656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10656:
--
Affects Version/s: 3.4.0
> Parsing error in CapacityScheduler.md
> -
>
> Key: YARN-10656
> URL: https://issues.apache.org/jira/browse/YARN-10656
> Project: Hadoop YARN
> Issue Type: Bug
> Components: documentation
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Akira Ajisaka
>Priority: Major
> Labels: pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 1h
> Remaining Estimate: 0h
>
> mvn site failed:
> https://ci-hadoop.apache.org/job/hadoop-qbt-trunk-java8-linux-x86_64/429/artifact/out/patch-mvnsite-root.txt
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-site-plugin:3.6:site (default-site) on project
> hadoop-yarn-site: Error parsing
> '/home/jenkins/jenkins-home/workspace/hadoop-qbt-trunk-java8-linux-x86_64/sourcedir/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md':
> line [-1] Error parsing the model: Unable to execute macro in the document:
> toc -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10657) We should make max application per queue to support node label.
[ https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10657: -- Component/s: capacity scheduler > We should make max application per queue to support node label. > --- > > Key: YARN-10657 > URL: https://issues.apache.org/jira/browse/YARN-10657 > Project: Hadoop YARN > Issue Type: Sub-task > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10657.001.patch, YARN-10657.002.patch, > YARN-10657.003.patch, YARN-10657.004.patch, YARN-10657.005.patch > > > https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708 > As we discussed in above comment: > We should deep into the label related max applications per queue. > I think when node label enabled in queue, max applications should consider > the max capacity of all labels. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10658) CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName.
[ https://issues.apache.org/jira/browse/YARN-10658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10658: -- Affects Version/s: 3.4.0 > CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. > -- > > Key: YARN-10658 > URL: https://issues.apache.org/jira/browse/YARN-10658 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10658.001.patch, YARN-10658.002.patch, > YARN-10658.003.patch > > > After the leaf queue can use same name, QueueInfo class getQueueName method > should have ambiguous QueueName. we should add queue path field to avoid > ambiguous QueueName, and make it consistent with fair scheduler. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10657) We should make max application per queue to support node label.
[ https://issues.apache.org/jira/browse/YARN-10657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10657: -- Affects Version/s: 3.4.0 > We should make max application per queue to support node label. > --- > > Key: YARN-10657 > URL: https://issues.apache.org/jira/browse/YARN-10657 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Andras Gyori >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10657.001.patch, YARN-10657.002.patch, > YARN-10657.003.patch, YARN-10657.004.patch, YARN-10657.005.patch > > > https://issues.apache.org/jira/browse/YARN-10641?focusedCommentId=17291708=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17291708 > As we discussed in above comment: > We should deep into the label related max applications per queue. > I think when node label enabled in queue, max applications should consider > the max capacity of all labels. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10674) fs2cs should generate auto-created queue deletion properties
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10674:
--
Component/s: scheduler
> fs2cs should generate auto-created queue deletion properties
>
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: scheduler
>Affects Versions: 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch, YARN-10674.016.patch, YARN-10674.017.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10658) CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName.
[ https://issues.apache.org/jira/browse/YARN-10658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10658: -- Component/s: capacity scheduler > CapacityScheduler QueueInfo add queue path field to avoid ambiguous QueueName. > -- > > Key: YARN-10658 > URL: https://issues.apache.org/jira/browse/YARN-10658 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-10658.001.patch, YARN-10658.002.patch, > YARN-10658.003.patch > > > After the leaf queue can use same name, QueueInfo class getQueueName method > should have ambiguous QueueName. we should add queue path field to avoid > ambiguous QueueName, and make it consistent with fair scheduler. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10653) Fixed the findbugs issues introduced by YARN-10647.
[ https://issues.apache.org/jira/browse/YARN-10653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10653: -- Affects Version/s: 3.3.1 3.4.0 > Fixed the findbugs issues introduced by YARN-10647. > --- > > Key: YARN-10653 > URL: https://issues.apache.org/jira/browse/YARN-10653 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0, 3.3.1 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Fix For: 3.4.0, 3.3.1, 3.2.3 > > Attachments: YARN-10653.001.patch, YARN-10653.002.patch, > image-2021-02-26-13-49-18-241.png > > > In YARN-10647 > I fixed TestRMNodeLabelsManager failed after YARN-10501. > But the finding bugs should be fixed also. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10693) Add documentation for YARN-10623 auto refresh queue conf in CS
[ https://issues.apache.org/jira/browse/YARN-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10693: -- Component/s: capacity scheduler > Add documentation for YARN-10623 auto refresh queue conf in CS > -- > > Key: YARN-10693 > URL: https://issues.apache.org/jira/browse/YARN-10693 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10693.001.patch, YARN-10693.002.patch, > YARN-10693.003.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-11652) [Umbrella] Follow-up after YARN-10888/YARN-10889
Benjamin Teke created YARN-11652: Summary: [Umbrella] Follow-up after YARN-10888/YARN-10889 Key: YARN-11652 URL: https://issues.apache.org/jira/browse/YARN-11652 Project: Hadoop YARN Issue Type: Improvement Affects Versions: 3.5.0 Reporter: Benjamin Teke Assignee: Benjamin Teke Follow-up improvements after the changes in YARN-10888/YARN-10889. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10692) Add Node GPU Utilization and apply to NodeMetrics.
[ https://issues.apache.org/jira/browse/YARN-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10692: -- Affects Version/s: 3.3.1 3.4.0 > Add Node GPU Utilization and apply to NodeMetrics. > -- > > Key: YARN-10692 > URL: https://issues.apache.org/jira/browse/YARN-10692 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0, 3.3.1 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10692.001.patch, YARN-10692.002.patch, > YARN-10692.003.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Now there are no node level GPU Utilization, this issue will add it, and add > it to NodeMetrics first. > cc [~pbacsko] [~Jim_Brennan] [~ebadger] [~gandras] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10674) fs2cs should generate auto-created queue deletion properties
[
https://issues.apache.org/jira/browse/YARN-10674?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10674:
--
Affects Version/s: 3.4.0
> fs2cs should generate auto-created queue deletion properties
>
>
> Key: YARN-10674
> URL: https://issues.apache.org/jira/browse/YARN-10674
> Project: Hadoop YARN
> Issue Type: Sub-task
>Affects Versions: 3.4.0
>Reporter: Qi Zhu
>Assignee: Qi Zhu
>Priority: Major
> Labels: fs2cs
> Fix For: 3.4.0
>
> Attachments: YARN-10674.001.patch, YARN-10674.002.patch,
> YARN-10674.003.patch, YARN-10674.004.patch, YARN-10674.005.patch,
> YARN-10674.006.patch, YARN-10674.007.patch, YARN-10674.008.patch,
> YARN-10674.009.patch, YARN-10674.010.patch, YARN-10674.011.patch,
> YARN-10674.012.patch, YARN-10674.013.patch, YARN-10674.014.patch,
> YARN-10674.015.patch, YARN-10674.016.patch, YARN-10674.017.patch
>
>
> In FS the auto deletion check interval is 10s.
> {code:java}
> @Override
> public void onCheck() {
> queueMgr.removeEmptyDynamicQueues();
> queueMgr.removePendingIncompatibleQueues();
> }
> while (running) {
> try {
> synchronized (this) {
> reloadListener.onCheck();
> }
> ...
> Thread.sleep(reloadIntervalMs);
> }
> /** Time to wait between checks of the allocation file */
> public static final long ALLOC_RELOAD_INTERVAL_MS = 10 * 1000;{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10692) Add Node GPU Utilization and apply to NodeMetrics.
[ https://issues.apache.org/jira/browse/YARN-10692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10692: -- Component/s: gpu > Add Node GPU Utilization and apply to NodeMetrics. > -- > > Key: YARN-10692 > URL: https://issues.apache.org/jira/browse/YARN-10692 > Project: Hadoop YARN > Issue Type: Sub-task > Components: gpu >Affects Versions: 3.4.0, 3.3.1 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.1 > > Attachments: YARN-10692.001.patch, YARN-10692.002.patch, > YARN-10692.003.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Now there are no node level GPU Utilization, this issue will add it, and add > it to NodeMetrics first. > cc [~pbacsko] [~Jim_Brennan] [~ebadger] [~gandras] -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10886) Cluster based and parent based max capacity in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke updated YARN-10886: - Parent Issue: YARN-11652 (was: YARN-10888) > Cluster based and parent based max capacity in Capacity Scheduler > - > > Key: YARN-10886 > URL: https://issues.apache.org/jira/browse/YARN-10886 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Priority: Major > > We want to introduce the percentage modes relative to the cluster, not the > parent, i.e > The property root.users.maximum-capacity will mean one of the following > things: > *Either Parent Percentage:* maximum capacity relative to its parent. If it’s > set to 50, then it means that the capacity is capped with respect to the > parent. This can be covered by the current format, no change there. > *Or Cluster Percentage:* maximum capacity expressed as a percentage of the > overall cluster capacity. This case is the new scenario, for example: > yarn.scheduler.capacity.root.users.max-capacity = c:50% > yarn.scheduler.capacity.root.users.max-capacity = c:50%, c:30% -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10693) Add documentation for YARN-10623 auto refresh queue conf in CS
[ https://issues.apache.org/jira/browse/YARN-10693?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10693: -- Affects Version/s: 3.4.0 > Add documentation for YARN-10623 auto refresh queue conf in CS > -- > > Key: YARN-10693 > URL: https://issues.apache.org/jira/browse/YARN-10693 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Qi Zhu >Assignee: Qi Zhu >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-10693.001.patch, YARN-10693.002.patch, > YARN-10693.003.patch > > Time Spent: 1h 40m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9809) NMs should supply a health status when registering with RM
[ https://issues.apache.org/jira/browse/YARN-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9809: - Component/s: nodemanager resourcemanager > NMs should supply a health status when registering with RM > -- > > Key: YARN-9809 > URL: https://issues.apache.org/jira/browse/YARN-9809 > Project: Hadoop YARN > Issue Type: Bug > Components: nodemanager, resourcemanager >Affects Versions: 3.4.0, 3.3.1 >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.3.1 > > Attachments: YARN-9809-branch-3.2.007.patch, > YARN-9809-branch-3.2.008.patch, YARN-9809-branch-3.2.009.patch, > YARN-9809.001.patch, YARN-9809.002.patch, YARN-9809.003.patch, > YARN-9809.004.patch, YARN-9809.005.patch, YARN-9809.006.patch, > YARN-9809.007.patch > > > Currently if the NM registers with the RM and it is unhealthy, it can be > scheduled many containers before the first heartbeat. After the first > heartbeat, the RM will mark the NM as unhealthy and kill all of the > containers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9279) Remove the old hamlet package
[ https://issues.apache.org/jira/browse/YARN-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9279: - Component/s: webapp > Remove the old hamlet package > - > > Key: YARN-9279 > URL: https://issues.apache.org/jira/browse/YARN-9279 > Project: Hadoop YARN > Issue Type: Improvement > Components: webapp >Affects Versions: 3.4.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-9279.01.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > The old hamlet package was deprecated in HADOOP-11875. Let's remove this to > improve the maintenability. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9809) NMs should supply a health status when registering with RM
[ https://issues.apache.org/jira/browse/YARN-9809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9809: - Affects Version/s: 3.3.1 3.4.0 > NMs should supply a health status when registering with RM > -- > > Key: YARN-9809 > URL: https://issues.apache.org/jira/browse/YARN-9809 > Project: Hadoop YARN > Issue Type: Bug >Affects Versions: 3.4.0, 3.3.1 >Reporter: Eric Badger >Assignee: Eric Badger >Priority: Major > Fix For: 3.2.2, 3.4.0, 3.3.1 > > Attachments: YARN-9809-branch-3.2.007.patch, > YARN-9809-branch-3.2.008.patch, YARN-9809-branch-3.2.009.patch, > YARN-9809.001.patch, YARN-9809.002.patch, YARN-9809.003.patch, > YARN-9809.004.patch, YARN-9809.005.patch, YARN-9809.006.patch, > YARN-9809.007.patch > > > Currently if the NM registers with the RM and it is unhealthy, it can be > scheduled many containers before the first heartbeat. After the first > heartbeat, the RM will mark the NM as unhealthy and kill all of the > containers. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10888) [Umbrella] New capacity modes for CS
[ https://issues.apache.org/jira/browse/YARN-10888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benjamin Teke reassigned YARN-10888: Assignee: Benjamin Teke (was: Szilard Nemeth) > [Umbrella] New capacity modes for CS > > > Key: YARN-10888 > URL: https://issues.apache.org/jira/browse/YARN-10888 > Project: Hadoop YARN > Issue Type: Improvement >Reporter: Szilard Nemeth >Assignee: Benjamin Teke >Priority: Major > Fix For: 3.4.0 > > Attachments: capacity_scheduler_queue_capacity.pdf > > > *Investigate how resource allocation configuration could be more consistent > in CapacityScheduler* > It would be nice if everywhere where a capacity can be defined could be > defined the same way: > * With fixed amounts (e.g. 1 GB memory, 8 vcores, 3 GPU) > * With percentages > ** Percentage of all resources (eg 10% of all memory, vcore, GPU) > ** Percentage per resource type (eg 10% memory, 25% vcore, 50% GPU) > * Allow mixing different modes under one hierarchy but not under the same > parent queues. > We need to determine all configuration options where capacities can be > defined, and see if it is possible to extend the configuration, or if it > makes sense in that case. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils
[ https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9354: - Affects Version/s: 3.3.0 3.4.0 > Resources should be created with ResourceTypesTestHelper instead of TestUtils > - > > Key: YARN-9354 > URL: https://issues.apache.org/jira/browse/YARN-9354 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 3.3.0, 3.4.0 >Reporter: Szilard Nemeth >Assignee: Andras Gyori >Priority: Trivial > Labels: newbie, newbie++ > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9354.001.patch, YARN-9354.002.patch, > YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, > YARN-9354.branch-3.2.002.patch, YARN-9354.branch-3.2.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource > has not identical, but very similar implementation to > org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. > Since these 2 methods are doing the same essentially and > ResourceTypesTestHelper is newer and used more, TestUtils#createResource > should be replaced with ResourceTypesTestHelper#newResource with all > occurrence. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9354) Resources should be created with ResourceTypesTestHelper instead of TestUtils
[ https://issues.apache.org/jira/browse/YARN-9354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9354: - Component/s: resourcemanager > Resources should be created with ResourceTypesTestHelper instead of TestUtils > - > > Key: YARN-9354 > URL: https://issues.apache.org/jira/browse/YARN-9354 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Szilard Nemeth >Assignee: Andras Gyori >Priority: Trivial > Labels: newbie, newbie++ > Fix For: 3.3.0, 3.2.2, 3.4.0 > > Attachments: YARN-9354.001.patch, YARN-9354.002.patch, > YARN-9354.003.patch, YARN-9354.004.patch, YARN-9354.branch-3.2.001.patch, > YARN-9354.branch-3.2.002.patch, YARN-9354.branch-3.2.003.patch > > > org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestUtils#createResource > has not identical, but very similar implementation to > org.apache.hadoop.yarn.resourcetypes.ResourceTypesTestHelper#newResource. > Since these 2 methods are doing the same essentially and > ResourceTypesTestHelper is newer and used more, TestUtils#createResource > should be replaced with ResourceTypesTestHelper#newResource with all > occurrence. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9279) Remove the old hamlet package
[ https://issues.apache.org/jira/browse/YARN-9279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9279: - Affects Version/s: 3.4.0 > Remove the old hamlet package > - > > Key: YARN-9279 > URL: https://issues.apache.org/jira/browse/YARN-9279 > Project: Hadoop YARN > Issue Type: Improvement >Affects Versions: 3.4.0 >Reporter: Akira Ajisaka >Assignee: Akira Ajisaka >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-9279.01.patch > > Time Spent: 1.5h > Remaining Estimate: 0h > > The old hamlet package was deprecated in HADOOP-11875. Let's remove this to > improve the maintenability. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6492) Generate queue metrics for each partition
[ https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6492: - Affects Version/s: 3.3.1 3.4.0 > Generate queue metrics for each partition > - > > Key: YARN-6492 > URL: https://issues.apache.org/jira/browse/YARN-6492 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler >Affects Versions: 3.4.0, 3.3.1 >Reporter: Jonathan Hung >Assignee: Manikandan R >Priority: Major > Fix For: 2.9.3, 3.2.2, 2.10.1, 3.4.0, 3.3.1 > > Attachments: PartitionQueueMetrics_default_partition.txt, > PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt, > YARN-6492-branch-2.10.016.patch, YARN-6492-branch-2.10.019.patch, > YARN-6492-branch-2.8.014.patch, YARN-6492-branch-2.9.015.patch, > YARN-6492-branch-3.1.018.patch, YARN-6492-branch-3.2.017.patch, > YARN-6492-junits.patch, YARN-6492.001.patch, YARN-6492.002.patch, > YARN-6492.003.patch, YARN-6492.004.patch, YARN-6492.005.WIP.patch, > YARN-6492.006.WIP.patch, YARN-6492.007.WIP.patch, YARN-6492.008.WIP.patch, > YARN-6492.009.WIP.patch, YARN-6492.010.WIP.patch, YARN-6492.011.WIP.patch, > YARN-6492.012.WIP.patch, YARN-6492.013.patch, partition_metrics.txt > > > We are interested in having queue metrics for all partitions. Right now each > queue has one QueueMetrics object which captures metrics either in default > partition or across all partitions. (After YARN-6467 it will be in default > partition) > But having the partition metrics would be very useful. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6476) Advanced Federation UI based on YARN UI v2
[ https://issues.apache.org/jira/browse/YARN-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6476: - Component/s: yarn yarn-ui-v2 > Advanced Federation UI based on YARN UI v2 > -- > > Key: YARN-6476 > URL: https://issues.apache.org/jira/browse/YARN-6476 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn, yarn-ui-v2 >Affects Versions: 3.4.0 >Reporter: Carlo Curino >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-6476-screenshots.zip, YARN-6476.001.patch > > > Extension of YARN-3661 based on new YARN UI YARN-3368. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6476) Advanced Federation UI based on YARN UI v2
[ https://issues.apache.org/jira/browse/YARN-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6476: - Affects Version/s: 3.4.0 > Advanced Federation UI based on YARN UI v2 > -- > > Key: YARN-6476 > URL: https://issues.apache.org/jira/browse/YARN-6476 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Carlo Curino >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Attachments: YARN-6476-screenshots.zip, YARN-6476.001.patch > > > Extension of YARN-3661 based on new YARN UI YARN-3368. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6572) Refactoring Router services to use common util classes for pipeline creations
[ https://issues.apache.org/jira/browse/YARN-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6572: - Component/s: federation > Refactoring Router services to use common util classes for pipeline creations > - > > Key: YARN-6572 > URL: https://issues.apache.org/jira/browse/YARN-6572 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6537) Running RM tests against the Router
[ https://issues.apache.org/jira/browse/YARN-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6537: - Fix Version/s: 3.4.0 > Running RM tests against the Router > --- > > Key: YARN-6537 > URL: https://issues.apache.org/jira/browse/YARN-6537 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, resourcemanager >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > We should run all the YARN RM tests that involve ApplicationClientProtocol, > AdminProtocol and RestAPI against the Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6537) Running RM tests against the Router
[ https://issues.apache.org/jira/browse/YARN-6537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6537: - Affects Version/s: 3.4.0 > Running RM tests against the Router > --- > > Key: YARN-6537 > URL: https://issues.apache.org/jira/browse/YARN-6537 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation, resourcemanager >Affects Versions: 3.4.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Minor > Labels: pull-request-available > Fix For: 3.4.0 > > > We should run all the YARN RM tests that involve ApplicationClientProtocol, > AdminProtocol and RestAPI against the Router. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6572) Refactoring Router services to use common util classes for pipeline creations
[ https://issues.apache.org/jira/browse/YARN-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-6572: - Affects Version/s: 3.4.0 > Refactoring Router services to use common util classes for pipeline creations > - > > Key: YARN-6572 > URL: https://issues.apache.org/jira/browse/YARN-6572 > Project: Hadoop YARN > Issue Type: Sub-task >Affects Versions: 3.4.0 >Reporter: Giovanni Matteo Fumarola >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 4h 50m > Remaining Estimate: 0h > -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-9586) [QA] Need more doc for yarn.federation.policy-manager-params when LoadBasedRouterPolicy is used
[ https://issues.apache.org/jira/browse/YARN-9586?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-9586: - Affects Version/s: 3.4.0 > [QA] Need more doc for yarn.federation.policy-manager-params when > LoadBasedRouterPolicy is used > --- > > Key: YARN-9586 > URL: https://issues.apache.org/jira/browse/YARN-9586 > Project: Hadoop YARN > Issue Type: Wish > Components: federation >Affects Versions: 3.4.0 >Reporter: Shen Yinjie >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > We picked LoadBasedRouterPolicy for YARN federation, but had no idea what to > set to yarn.federation.policy-manager-params. Is there a demo config or more > detailed description for this. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10218) [GPG] Support HTTPS in GPG
[ https://issues.apache.org/jira/browse/YARN-10218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10218: -- Affects Version/s: 3.4.0 > [GPG] Support HTTPS in GPG > -- > > Key: YARN-10218 > URL: https://issues.apache.org/jira/browse/YARN-10218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Affects Versions: 3.4.0 >Reporter: Bilwa S T >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HTTPS support in Router is handled as part of Jira YARN-10120. Https Rest > calls from GPG to Router must be supported -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10218) [GPG] Support HTTPS in GPG
[ https://issues.apache.org/jira/browse/YARN-10218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10218: -- Component/s: federation > [GPG] Support HTTPS in GPG > -- > > Key: YARN-10218 > URL: https://issues.apache.org/jira/browse/YARN-10218 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Bilwa S T >Assignee: Shilun Fan >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > > HTTPS support in Router is handled as part of Jira YARN-10120. Https Rest > calls from GPG to Router must be supported -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10744) add example for yarn.federation.policy-manager-params
[ https://issues.apache.org/jira/browse/YARN-10744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10744: -- Fix Version/s: (was: 3.4.0) > add example for yarn.federation.policy-manager-params > - > > Key: YARN-10744 > URL: https://issues.apache.org/jira/browse/YARN-10744 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: chaosju >Assignee: chaosju >Priority: Trivial > Labels: pull-request-available > Attachments: YARN-10744.001.patch > > Time Spent: 20m > Remaining Estimate: 0h > > add example for yarn.federation.policy-manager-params -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10747) Bump YARN CSI protobuf version to 3.7.1
[ https://issues.apache.org/jira/browse/YARN-10747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shilun Fan updated YARN-10747: -- Affects Version/s: 3.3.5 3.4.0 > Bump YARN CSI protobuf version to 3.7.1 > --- > > Key: YARN-10747 > URL: https://issues.apache.org/jira/browse/YARN-10747 > Project: Hadoop YARN > Issue Type: Task >Affects Versions: 3.4.0, 3.3.5 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0, 3.3.5 > > Time Spent: 2h 10m > Remaining Estimate: 0h > > Bumping YARN CSI protobuf version to 3.7.1 to keep it consistent with > hadoop's protobuf version. -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10757) jsonschema2pojo-maven-plugin version is not defined
[
https://issues.apache.org/jira/browse/YARN-10757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shilun Fan updated YARN-10757:
--
Affects Version/s: 3.4.0
> jsonschema2pojo-maven-plugin version is not defined
> ---
>
> Key: YARN-10757
> URL: https://issues.apache.org/jira/browse/YARN-10757
> Project: Hadoop YARN
> Issue Type: Bug
> Components: build
>Affects Versions: 3.4.0
>Reporter: Akira Ajisaka
>Assignee: Tamas Domok
>Priority: Major
> Labels: newbie, pull-request-available
> Fix For: 3.4.0
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> The below maven plugin version is not defined.
> {noformat}
> $ mvn install
> [INFO] Scanning for projects...
> [WARNING]
> [WARNING] Some problems were encountered while building the effective model
> for org.apache.hadoop:hadoop-yarn-server-resourcemanager:jar:3.4.0-SNAPSHOT
> [WARNING] 'build.plugins.plugin.version' for
> org.jsonschema2pojo:jsonschema2pojo-maven-plugin is missing. @ line 448,
> column 15
> [WARNING]
> [WARNING] It is highly recommended to fix these problems because they
> threaten the stability of your build.
> [WARNING]
> [WARNING] For this reason, future Maven versions might no longer support
> building such malformed projects.
> [WARNING]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
