[jira] [Updated] (YARN-10201) Make AMRMProxyPolicy aware of SC load
[ https://issues.apache.org/jira/browse/YARN-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Young Chen updated YARN-10201: -- Attachment: YARN-10201.v10.patch > Make AMRMProxyPolicy aware of SC load > - > > Key: YARN-10201 > URL: https://issues.apache.org/jira/browse/YARN-10201 > Project: Hadoop YARN > Issue Type: Sub-task > Components: amrmproxy >Reporter: Young Chen >Assignee: Young Chen >Priority: Major > Attachments: YARN-10201.v0.patch, YARN-10201.v1.patch, > YARN-10201.v10.patch, YARN-10201.v2.patch, YARN-10201.v3.patch, > YARN-10201.v4.patch, YARN-10201.v5.patch, YARN-10201.v6.patch, > YARN-10201.v7.patch, YARN-10201.v8.patch, YARN-10201.v9.patch > > > LocalityMulticastAMRMProxyPolicy is currently unaware of SC load when > splitting resource requests. We propose changes to the policy so that it > receives feedback from SCs and can load balance requests across the federated > cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-6526: Attachment: YARN-6526.004.patch > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6526.001.patch, YARN-6526.002.patch, > YARN-6526.003.patch, YARN-6526.004.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9898) Dependency netty-all-4.1.27.Final doesn't support ARM platform
[
https://issues.apache.org/jira/browse/YARN-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17103056#comment-17103056
]
liusheng commented on YARN-9898:
yes, thank you, I have also tested OK locally before uploading this patch, and
this patch have no side affect on x86 functionalities. not sure about the CI
results.
> Dependency netty-all-4.1.27.Final doesn't support ARM platform
> --
>
> Key: YARN-9898
> URL: https://issues.apache.org/jira/browse/YARN-9898
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: liusheng
>Assignee: liusheng
>Priority: Major
> Attachments: YARN-9898.001.patch, YARN-9898.002.patch
>
>
> Hadoop dependent the Netty package, but the *netty-all-4.1.27.Final* of
> io.netty maven repo, cannot support ARM platform.
> When run the test *TestCsiClient.testIdentityService* on ARM server, it will
> raise error like following:
> {code:java}
> Caused by: java.io.FileNotFoundException:
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> ... 46 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6492) Generate queue metrics for each partition
[
https://issues.apache.org/jira/browse/YARN-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102945#comment-17102945
]
Eric Payne commented on YARN-6492:
--
Thanks [~maniraj...@gmail.com] for working to move this issue forward, and
thanks [~jhung] for the great reviews.
Mani, I try to build patch 008 on trunk, and it gives me the following
compilation errors:
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:3.1:compile (default-compile) on
project hadoop-yarn-server-resourcemanager: Compilation failure
[ERROR]
/home/ericp/hadoop/source/YARN-6492/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java:[320,39]
cannot find symbol
[ERROR] symbol: class PartitionQueueMetrics
[ERROR] location: class
org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetrics
{noformat}
As I mentioned [in this comment,
above|https://issues.apache.org/jira/browse/YARN-6492?focusedCommentId=16903050=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16903050],
I am concerned about naming the default partition "default":
{code:java}
public static final String DEFAULT_PARTITION = "default";
{code}
Will the field names for the default partition come out as follows?
{panel}
"name": "Hadoop:service=ResourceManager,name=PartitionQueueMetrics,p0=default"
"tag.Partition": "default"
{panel}
In order to be consistent with other API responses like
{{/ws/v1/cluster/scheduler}}, I think this should just be an empty string. So,
I would expect the JMX response to look like the following for
DEFAULT_PARTITION:
{panel}
"name": "Hadoop:service=ResourceManager,name=PartitionQueueMetrics,p0="
"tag.Partition": ""
{panel}
{quote}
IMO the bottom line is that after this JIRA is committed, the existing
QueueMetrics should still only contain metrics for default partition, and
partitioned queue metrics should only be in the newly added metrics.
{quote}
I agree with Jonathan's statement.
> Generate queue metrics for each partition
> -
>
> Key: YARN-6492
> URL: https://issues.apache.org/jira/browse/YARN-6492
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Jonathan Hung
>Assignee: Manikandan R
>Priority: Major
> Attachments: PartitionQueueMetrics_default_partition.txt,
> PartitionQueueMetrics_x_partition.txt, PartitionQueueMetrics_y_partition.txt,
> YARN-6492.001.patch, YARN-6492.002.patch, YARN-6492.003.patch,
> YARN-6492.004.patch, YARN-6492.005.WIP.patch, YARN-6492.006.WIP.patch,
> YARN-6492.007.WIP.patch, YARN-6492.008.WIP.patch, partition_metrics.txt
>
>
> We are interested in having queue metrics for all partitions. Right now each
> queue has one QueueMetrics object which captures metrics either in default
> partition or across all partitions. (After YARN-6467 it will be in default
> partition)
> But having the partition metrics would be very useful.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[
https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102905#comment-17102905
]
Hadoop QA commented on YARN-6526:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
39s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 21s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
26s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
18s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common in
trunk has 1 extant findbugs warnings. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
33s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 16s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common:
The patch generated 1 new + 2 unchanged - 0 fixed = 3 total (was 2) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
31s{color} | {color:red}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
generated 1 new + 1 unchanged - 0 fixed = 2 total (was 1) {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m
1s{color} | {color:green} hadoop-yarn-server-common in the patch passed.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 70m 50s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs |
module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common
|
| |
org.apache.hadoop.yarn.server.federation.store.impl.SQLFederationStateStore.getPoliciesConfigurations(GetSubClusterPoliciesConfigurationsRequest)
may fail to clean up java.sql.ResultSet Obligation to clean up resource
created at SQLFederationStateStore.java:up java.sql.ResultSet Obligation to
clean up resource created at SQLFederationStateStore.java:[line 934] is not
discharged |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
[jira] [Commented] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102846#comment-17102846 ] Bilwa S T commented on YARN-6526: - Hi [~elgoiri] I have attached patch by adding metrics and related testcase too. Please check Thanks > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6526.001.patch, YARN-6526.002.patch, > YARN-6526.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6526) Refactoring SQLFederationStateStore by avoiding to recreate a connection at every call
[ https://issues.apache.org/jira/browse/YARN-6526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bilwa S T updated YARN-6526: Attachment: YARN-6526.003.patch > Refactoring SQLFederationStateStore by avoiding to recreate a connection at > every call > -- > > Key: YARN-6526 > URL: https://issues.apache.org/jira/browse/YARN-6526 > Project: Hadoop YARN > Issue Type: Sub-task > Components: federation >Reporter: Giovanni Matteo Fumarola >Assignee: Bilwa S T >Priority: Major > Fix For: 3.4.0 > > Attachments: YARN-6526.001.patch, YARN-6526.002.patch, > YARN-6526.003.patch > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[
https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102828#comment-17102828
]
Hadoop QA commented on YARN-10254:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
31s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
42s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
41s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
32s{color} | {color:green}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 0 new + 15 unchanged - 1 fixed = 15 total (was 16) {color}
|
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 20s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
44s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}103m 59s{color}
| {color:red} hadoop-yarn-server-resourcemanager in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}169m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoQueueCreation
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestQueueManagementDynamicEditPolicy
|
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacitySchedulerAutoCreatedQueuePreemption
|
| | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart |
| |
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestAbsoluteResourceWithAutoQueue
|
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-YARN-Build/26006/artifact/out/Dockerfile
|
| JIRA Issue | YARN-10254 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002425/YARN-10254.003.patch |
| Optional Tests |
[jira] [Commented] (YARN-9444) YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource
[
https://issues.apache.org/jira/browse/YARN-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102782#comment-17102782
]
Eric Payne commented on YARN-9444:
--
Thanks for the original patch [~shuzirra] and for the reviews [~snemeth] and
[~adam.antal] and [~shuzirra].
I will commit the branch-2.10 patch on Monday if there are no additional
comments at that time.
> YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize
> yarn.io/gpu as a valid resource
> --
>
> Key: YARN-9444
> URL: https://issues.apache.org/jira/browse/YARN-9444
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9444.001.patch, YARN-9444.002.patch,
> YARN-9444.003.patch, YARN-9444.branch-2.10.003.patch
>
>
> The original issue was the jobclient test did not send the requested resource
> type, when specified in the command line eg:
> {code:java}
> hadoop jar hadoop-mapreduce-client-jobclient-tests.jar sleep
> -Dmapreduce.reduce.resource.yarn.io/gpu=1 -m 10 -r 1 -mt 9
> {code}
> After some investigation, it turned out it only affects resource types with
> name containing '.' characters. And the root cause is regexp from the
> getRequestedResourcesFromConfig method.
> {code:java}
> "^" + Pattern.quote(prefix) + "[^.]+$"
> {code}
> This regexp explicitly forbids any dots in the resource type name, which is
> inconsistent with the default resource type for gpu and fpga, which are
> yarn.io/gpu and yarn.io/fpga respectively.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9898) Dependency netty-all-4.1.27.Final doesn't support ARM platform
[
https://issues.apache.org/jira/browse/YARN-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102750#comment-17102750
]
Ayush Saxena commented on YARN-9898:
Thanx [~seanlau] for the patch.
I have re triggered the build here, I don't think the failure is related to the
patch :
https://builds.apache.org/job/PreCommit-YARN-Build/26007/
Have verified the fix in AARCH-64 :
{noformat}
[INFO] ---
[INFO] T E S T S
[INFO] ---
[INFO] Running org.apache.hadoop.yarn.csi.client.TestCsiClient
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.835 s
- in org.apache.hadoop.yarn.csi.client.TestCsiClient
[INFO]
[INFO] Results:
[INFO]
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
{noformat}
{noformat}
[INFO]
[INFO] Detecting the operating system and CPU architecture
[INFO]
[INFO] os.detected.name: linux
[INFO] os.detected.arch: aarch_64
[INFO] os.detected.version: 4.15
[INFO] os.detected.version.major: 4
[INFO] os.detected.version.minor: 15
[INFO] os.detected.release: ubuntu
[INFO] os.detected.release.version: 16.04
[INFO] os.detected.release.like.ubuntu: true
[INFO] os.detected.release.like.debian: true
[INFO] os.detected.classifier: linux-aarch_64
[INFO]
{noformat}
[~weichiu] Let me know for any concerns, will hold this up for you a couple of
days to have a look.
> Dependency netty-all-4.1.27.Final doesn't support ARM platform
> --
>
> Key: YARN-9898
> URL: https://issues.apache.org/jira/browse/YARN-9898
> Project: Hadoop YARN
> Issue Type: Bug
>Reporter: liusheng
>Assignee: liusheng
>Priority: Major
> Attachments: YARN-9898.001.patch, YARN-9898.002.patch
>
>
> Hadoop dependent the Netty package, but the *netty-all-4.1.27.Final* of
> io.netty maven repo, cannot support ARM platform.
> When run the test *TestCsiClient.testIdentityService* on ARM server, it will
> raise error like following:
> {code:java}
> Caused by: java.io.FileNotFoundException:
> META-INF/native/libnetty_transport_native_epoll_aarch_64.so
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:161)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:243)
> at
> io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:124)
> ... 45 more
> Suppressed: java.lang.UnsatisfiedLinkError: no
> netty_transport_native_epoll_aarch_64 in java.library.path
> at
> java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
> at java.lang.Runtime.loadLibrary0(Runtime.java:870)
> at java.lang.System.loadLibrary(System.java:1122)
> at
> io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:263)
> at java.security.AccessController.doPrivileged(Native
> Method)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:255)
> at
> io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:233)
> ... 46 more
> {code}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gergely Pollak updated YARN-10254: -- Attachment: YARN-10254.003.patch > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch, > YARN-10254.003.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9898) Dependency netty-all-4.1.27.Final doesn't support ARM platform
[
https://issues.apache.org/jira/browse/YARN-9898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102632#comment-17102632
]
Hadoop QA commented on YARN-9898:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m
26s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
8s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
52s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 18m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 18m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
74m 58s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
36s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
33s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
11s{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} hadoop-yarn-csi in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 17m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 17m
32s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 17m
57s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
6s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 32s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
37s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}600m 58s{color}
| {color:red} root in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 1m
27s{color} | {color:red} The patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black}766m 37s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.TestDecommissionWithStriped |
| | hadoop.hdfs.server.namenode.ha.TestConfiguredFailoverProxyProvider |
| | hadoop.yarn.applications.distributedshell.TestDistributedShell |
| | hadoop.io.compress.snappy.TestSnappyCompressorDecompressor |
| | hadoop.io.compress.TestCompressorDecompressor |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-YARN-Build/26004/artifact/out/Dockerfile
|
| JIRA Issue | YARN-9898 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002354/YARN-9898.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 12d0786e2768 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-9444) YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource
[
https://issues.apache.org/jira/browse/YARN-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102622#comment-17102622
]
Adam Antal commented on YARN-9444:
--
+1 (non-binding) on the branch-2.10 patch. Thanks for the cherry-pick [~epayne].
> YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize
> yarn.io/gpu as a valid resource
> --
>
> Key: YARN-9444
> URL: https://issues.apache.org/jira/browse/YARN-9444
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9444.001.patch, YARN-9444.002.patch,
> YARN-9444.003.patch, YARN-9444.branch-2.10.003.patch
>
>
> The original issue was the jobclient test did not send the requested resource
> type, when specified in the command line eg:
> {code:java}
> hadoop jar hadoop-mapreduce-client-jobclient-tests.jar sleep
> -Dmapreduce.reduce.resource.yarn.io/gpu=1 -m 10 -r 1 -mt 9
> {code}
> After some investigation, it turned out it only affects resource types with
> name containing '.' characters. And the root cause is regexp from the
> getRequestedResourcesFromConfig method.
> {code:java}
> "^" + Pattern.quote(prefix) + "[^.]+$"
> {code}
> This regexp explicitly forbids any dots in the resource type name, which is
> inconsistent with the default resource type for gpu and fpga, which are
> yarn.io/gpu and yarn.io/fpga respectively.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102597#comment-17102597 ] Peter Bacsko edited comment on YARN-10254 at 5/8/20, 1:42 PM: -- Regarding #2 I'm voting for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. A counter-argument could be that a slower and completed application is still better than having no running application at all, especially if the jobs are scheduled. [~sunilg] thoughts? was (Author: pbacsko): Regarding #2 I'm voting for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102597#comment-17102597 ] Peter Bacsko edited comment on YARN-10254 at 5/8/20, 1:39 PM: -- Regarding #2 I'm voting for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. was (Author: pbacsko): Regarding #2 I'm voting for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect the SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102597#comment-17102597 ] Peter Bacsko commented on YARN-10254: - Regarding #2 I would vot for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect the SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102597#comment-17102597 ] Peter Bacsko edited comment on YARN-10254 at 5/8/20, 1:38 PM: -- Regarding #2 I'm voting for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect the SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. was (Author: pbacsko): Regarding #2 I would vot for throwing an exception. For example, I use the mapping "u:%user:root.grp.%primary_group", but the administrator changes the queue structure to "root.groups" from "root.grp". The mapping becomes invalid and all (or some) applications end up running in "root.default". In situatons like that, I prefer seeing a failure immediately so I can act ASAP. Having a running app might be preferable in some scenarios, but it will likely affect the SLAs and job running times. The users have to realize that their application have been placed into the wrong queue and it takes time. These kind of problems can be enormously frustrating. > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10254) CapacityScheduler incorrect User Group Mapping after leaf queue change
[ https://issues.apache.org/jira/browse/YARN-10254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102593#comment-17102593 ] Gergely Pollak commented on YARN-10254: --- [~pbacsko] thank you for your feedback, #1) Yes it got smaller and smaller each iteration I made changes to the class, and I just didn't want to remove it yet, but you are right, there is no point in keeping this method. Originally I wanted to have some validation here, but those were effectively the same as the validations in other methods. Will remove it in next patch #2) It is a good question, I honestly don't know which is the better approach, my thought process was if the mapping rule is invalid we still might want to place the applciation to default queue rather than bouncing it back, but this case is a kind of misconfiguration, where an exception could be in place. On the other hand configuration errors should fail at initialization time. So I'm not sure which is the best solution, thoghts anyone? #3) Will be fixed :) #4) Ditto #5) Yes you are right, we should query the queue by parent.getQueuePath() + "." + leafQueueName, because the get wont recoginze a short parent name + a short leaf name, only a full parent path + short leafName. #6) Alter mapping just alters a mapping, it depends on the user if it is a normalization, or just a simple overwrite, so that's why the method name isn't more expressive. comment will be added. > CapacityScheduler incorrect User Group Mapping after leaf queue change > -- > > Key: YARN-10254 > URL: https://issues.apache.org/jira/browse/YARN-10254 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Gergely Pollak >Assignee: Gergely Pollak >Priority: Major > Attachments: YARN-10254.001.patch, YARN-10254.002.patch > > > YARN-9879 and YARN-10198 introduced some major changes to user group mapping, > and some of them unfortunately had some negative impact on the way mapping > works. > In some cases incorrect PlacementContexts were created, where full queue path > was passed as leaf queue name. This affects how the yarn cli app list > displays the queues. > u:%user:%primary_group.%user mapping fails with an incorrect validation error > when the %primary_group parent queue was a managed parent. > Group based rules in certain cases are mapped to root.[primary_group] rules, > loosing the ability to create deeper structures. > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9444) YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize yarn.io/gpu as a valid resource
[
https://issues.apache.org/jira/browse/YARN-9444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102423#comment-17102423
]
Gergely Pollak commented on YARN-9444:
--
[~epayne] thank you for helping with the backport, LGTM + 1 (non-binding).
As I see the findbugs error is unrelated.
> YARN API ResourceUtils's getRequestedResourcesFromConfig doesn't recognize
> yarn.io/gpu as a valid resource
> --
>
> Key: YARN-9444
> URL: https://issues.apache.org/jira/browse/YARN-9444
> Project: Hadoop YARN
> Issue Type: Bug
> Components: api
>Reporter: Gergely Pollak
>Assignee: Gergely Pollak
>Priority: Minor
> Fix For: 3.3.0
>
> Attachments: YARN-9444.001.patch, YARN-9444.002.patch,
> YARN-9444.003.patch, YARN-9444.branch-2.10.003.patch
>
>
> The original issue was the jobclient test did not send the requested resource
> type, when specified in the command line eg:
> {code:java}
> hadoop jar hadoop-mapreduce-client-jobclient-tests.jar sleep
> -Dmapreduce.reduce.resource.yarn.io/gpu=1 -m 10 -r 1 -mt 9
> {code}
> After some investigation, it turned out it only affects resource types with
> name containing '.' characters. And the root cause is regexp from the
> getRequestedResourcesFromConfig method.
> {code:java}
> "^" + Pattern.quote(prefix) + "[^.]+$"
> {code}
> This regexp explicitly forbids any dots in the resource type name, which is
> inconsistent with the default resource type for gpu and fpga, which are
> yarn.io/gpu and yarn.io/fpga respectively.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-9460) QueueACLsManager and ReservationsACLManager should not use instanceof checks
[
https://issues.apache.org/jira/browse/YARN-9460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17102384#comment-17102384
]
Hadoop QA commented on YARN-9460:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 2m
11s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} dupname {color} | {color:green} 0m
0s{color} | {color:green} No case conflicting files found. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
17s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
20m 25s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:blue}0{color} | {color:blue} spotbugs {color} | {color:blue} 1m
58s{color} | {color:blue} Used deprecated FindBugs config; considering
switching to SpotBugs. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
55s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
47s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 34s{color} | {color:orange}
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:
The patch generated 6 new + 108 unchanged - 0 fixed = 114 total (was 108)
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
50s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
17m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
2s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 91m
56s{color} | {color:green} hadoop-yarn-server-resourcemanager in the patch
passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}170m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | ClientAPI=1.40 ServerAPI=1.40 base:
https://builds.apache.org/job/PreCommit-YARN-Build/26005/artifact/out/Dockerfile
|
| JIRA Issue | YARN-9460 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/13002368/YARN-9460.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux d1919662cd60 4.15.0-74-generic #84-Ubuntu SMP Thu Dec 19
08:06:28 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / e9e1ead089c |
| Default Java | Private Build-1.8.0_252-8u252-b09-1~18.04-b09 |
| checkstyle |
