[jira] [Created] (YARN-10940) Fix Documentation for AWS-Hadoop integration / yarn-site.xml
Girish Ganesan created YARN-10940: - Summary: Fix Documentation for AWS-Hadoop integration / yarn-site.xml Key: YARN-10940 URL: https://issues.apache.org/jira/browse/YARN-10940 Project: Hadoop YARN Issue Type: Task Reporter: Girish Ganesan The following document on AWS-Hadoop integration specified authenticating via AWS environment variables: [https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Authenticating_via_the_AWS_Environment_Variables] It provides a warning: _Important: These environment variables are generally not propagated from client to server when YARN applications are launched. That is: having the AWS environment variables set when an application is launched will not permit the launched application to access S3 resources. The environment variables must (somehow) be set on the hosts/processes where the work is executed._ This is somewhat cryptic. A few things need to be clarified in the doc: # This is true even when Yarn is running on a single node (pseudo distributed). # *This also affects authentication via named profile* credentials:[https://hadoop.apache.org/docs/stable/hadoop-aws/tools/hadoop-aws/index.html#Using_Named_Profile_Credentials_with_ProfileCredentialsProvider] __ This method depends on AWS_PROFILE variable. # Please give some pointers on how the variables can be propagated. One way is to whitelist the variable in yarn.nodemanager.env-whitelist (set in yarn-site.xml): [https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/ClusterSetup.html#Configuring_Environment_of_Hadoop_Daemons] I was trying to figure out why hive was failing on a query (using mapred) on an external table created from S3. After a while I realized it was not getting the AWS_PROFILE variable. Eventually I realized that adding the variable to the Yarn whitelist will do the trick. Hopefully this ticket will help someone else. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10902) Resources on application blacklisted node with reserved container can not allocate to other applications
[
https://issues.apache.org/jira/browse/YARN-10902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412926#comment-17412926
]
jackwangcs commented on YARN-10902:
---
Hi [~Tao Yang], [~bteke], [~snemeth], could you help to review this path?
Thanks.
> Resources on application blacklisted node with reserved container can not
> allocate to other applications
>
>
> Key: YARN-10902
> URL: https://issues.apache.org/jira/browse/YARN-10902
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: jackwangcs
>Assignee: jackwangcs
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> If a node has reserved container of an application and the application adds
> this node into its blacklist, resources on the node can not allocate to other
> applications in the current allocation process.
> In RegularContainerAllocator, if it finds the node is in blacklist, it will
> not allocate resources. Furthermore, this node has a reserved container,
> other queues or application will not have opportunity to allocate.
> {code:java}
> ContainerAllocation tryAllocateOnNode(Resource clusterResource,
> FiCaSchedulerNode node, SchedulingMode schedulingMode,
> ResourceLimits resourceLimits, SchedulerRequestKey schedulerKey,
> RMContainer reservedContainer) {
> ContainerAllocation result;
> // Sanity checks before assigning to this node
> result = checkIfNodeBlackListed(node, schedulerKey);
> if (null != result) {
> return result;
> }
> //
> }{code}
> In this case, the reserved container should be cancelled.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10903) Too many "Failed to accept allocation proposal" because of wrong Headroom check for DRF
[
https://issues.apache.org/jira/browse/YARN-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412922#comment-17412922
]
Tao Yang commented on YARN-10903:
-
+1 for the PR, will merge it after a few days if there are no objections.
> Too many "Failed to accept allocation proposal" because of wrong Headroom
> check for DRF
> ---
>
> Key: YARN-10903
> URL: https://issues.apache.org/jira/browse/YARN-10903
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: jackwangcs
>Assignee: jackwangcs
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The headroom check in `ParentQueue.canAssign` and
> `RegularContainerAllocator#checkHeadroom` does not consider the DRF cases.
> This will cause a lot of "Failed to accept allocation proposal" when a queue
> is near-fully used.
> In the log:
> Headroom: memory:256, vCores:729
> Request: memory:56320, vCores:5
> clusterResource: memory:673966080, vCores:110494
> If use the DRF, then
> {code:java}
> Resources.greaterThanOrEqual(rc, clusterResource, Resources.add(
> currentResourceLimits.getHeadroom(), resourceCouldBeUnReserved),
> required); {code}
> will be true but in fact we can not allocate resources to the request due to
> the max limit(no enough memory).
> {code:java}
> 2021-07-21 23:49:39,012 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1626747977559_95859
> headRoom= currentConsumption=0
> 2021-07-21 23:49:39,012 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.LocalityAppPlacementAllocator:
> Request={AllocationRequestId: -1, Priority: 1, Capability: vCores:5>, # Containers: 19, Location: *, Relax Locality: true, Execution
> Type Request: null, Node Label Expression: prod-best-effort-node}
> .
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Try to commit allocation proposal=New
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest:
> ALLOCATED=[(Application=appattempt_1626747977559_95859_01;
> Node=:8041; Resource=)]
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.UsersManager:
> userLimit is fetched. userLimit=,
> userSpecificUserLimit=,
> schedulingMode=RESPECT_PARTITION_EXCLUSIVITY, partition=prod-best-effort-node
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
> Headroom calculation for user x: userLimit=
> queueMaxAvailRes= consumed=
> partition=prod-best-effort-node
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue:
> Used resource= exceeded maxResourceLimit of the
> queue =
> 2021-07-21 23:49:39,013 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Failed to accept allocation proposal
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10928) Support default queue properties of capacity scheduler to simplify configuration management
[
https://issues.apache.org/jira/browse/YARN-10928?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412918#comment-17412918
]
Tao Yang commented on YARN-10928:
-
The PR LGTM now, +1 from my side. I will merge this PR after a few days if
there are no objections.
> Support default queue properties of capacity scheduler to simplify
> configuration management
> ---
>
> Key: YARN-10928
> URL: https://issues.apache.org/jira/browse/YARN-10928
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Weihao Zheng
>Assignee: Weihao Zheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> There are many user cases that one user owns many queues in his
> organization's cluster for different business usages in practice. These
> queues often share the same properties, such as minimum-user-limit-percent
> and user-limit-factor. Users have to write one property for every queue they
> use if they want to use customized these shared properties. Adding default
> queue properties for these cases will simplify capacity scheduler's
> configuration file and make it easy to adjust queue's common properties.
>
> CHANGES:
> Add two properties as queue's default value in capacity scheduler's
> configuration:
> * {{yarn.scheduler.capacity.minimum-user-limit-percent}}
> * {{yarn.scheduler.capacity.user-limit-factor}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412909#comment-17412909 ] hao commented on YARN-10939: [~bteke]Hello, sir. Can you help me with this question? Thank you！！！ > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > Attachments: 企业微信截图_16311769849737.png, 企业微信截图_1631177020581.png > > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Issue Comment Deleted] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated YARN-10939: --- Comment: was deleted (was: [~aajisaka] Hello, sir. Can you help me with this question? Thank you！！！) > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > Attachments: 企业微信截图_16311769849737.png, 企业微信截图_1631177020581.png > > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10903) Too many "Failed to accept allocation proposal" because of wrong Headroom check for DRF
[
https://issues.apache.org/jira/browse/YARN-10903?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412899#comment-17412899
]
jackwangcs commented on YARN-10903:
---
Hi [~Tao Yang], thanks for your review, I have fixed the check-style warnings.
> Too many "Failed to accept allocation proposal" because of wrong Headroom
> check for DRF
> ---
>
> Key: YARN-10903
> URL: https://issues.apache.org/jira/browse/YARN-10903
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacityscheduler
>Reporter: jackwangcs
>Assignee: jackwangcs
>Priority: Major
> Labels: pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> The headroom check in `ParentQueue.canAssign` and
> `RegularContainerAllocator#checkHeadroom` does not consider the DRF cases.
> This will cause a lot of "Failed to accept allocation proposal" when a queue
> is near-fully used.
> In the log:
> Headroom: memory:256, vCores:729
> Request: memory:56320, vCores:5
> clusterResource: memory:673966080, vCores:110494
> If use the DRF, then
> {code:java}
> Resources.greaterThanOrEqual(rc, clusterResource, Resources.add(
> currentResourceLimits.getHeadroom(), resourceCouldBeUnReserved),
> required); {code}
> will be true but in fact we can not allocate resources to the request due to
> the max limit(no enough memory).
> {code:java}
> 2021-07-21 23:49:39,012 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.SchedulerApplicationAttempt:
> showRequests: application=application_1626747977559_95859
> headRoom= currentConsumption=0
> 2021-07-21 23:49:39,012 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.LocalityAppPlacementAllocator:
> Request={AllocationRequestId: -1, Priority: 1, Capability: vCores:5>, # Containers: 19, Location: *, Relax Locality: true, Execution
> Type Request: null, Node Label Expression: prod-best-effort-node}
> .
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Try to commit allocation proposal=New
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.common.ResourceCommitRequest:
> ALLOCATED=[(Application=appattempt_1626747977559_95859_01;
> Node=:8041; Resource=)]
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.UsersManager:
> userLimit is fetched. userLimit=,
> userSpecificUserLimit=,
> schedulingMode=RESPECT_PARTITION_EXCLUSIVITY, partition=prod-best-effort-node
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue:
> Headroom calculation for user x: userLimit=
> queueMaxAvailRes= consumed=
> partition=prod-best-effort-node
> 2021-07-21 23:49:39,013 DEBUG
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.AbstractCSQueue:
> Used resource= exceeded maxResourceLimit of the
> queue =
> 2021-07-21 23:49:39,013 INFO
> org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler:
> Failed to accept allocation proposal
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10893) Add metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor
[ https://issues.apache.org/jira/browse/YARN-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412676#comment-17412676 ] Íñigo Goiri commented on YARN-10893: Thanks [~akbordia] for the patch. Merged PR 3325 to trunk. > Add metrics for getClusterMetrics and getApplications APIs in > FederationClientInterceptor > - > > Key: YARN-10893 > URL: https://issues.apache.org/jira/browse/YARN-10893 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akshat Bordia >Assignee: Akshat Bordia >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently getClusterMetrics and getApplications APIs in > FederationClientInterceptor do not have metrics being recorded. Need to add > the metrics for the latency, successful and failed attempt counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Resolved] (YARN-10893) Add metrics for getClusterMetrics and getApplications APIs in FederationClientInterceptor
[ https://issues.apache.org/jira/browse/YARN-10893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri resolved YARN-10893. Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed > Add metrics for getClusterMetrics and getApplications APIs in > FederationClientInterceptor > - > > Key: YARN-10893 > URL: https://issues.apache.org/jira/browse/YARN-10893 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Akshat Bordia >Assignee: Akshat Bordia >Priority: Major > Labels: pull-request-available > Fix For: 3.4.0 > > Time Spent: 2h 20m > Remaining Estimate: 0h > > Currently getClusterMetrics and getApplications APIs in > FederationClientInterceptor do not have metrics being recorded. Need to add > the metrics for the latency, successful and failed attempt counts. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-1115) Provide optional means for a scheduler to check real user ACLs
[ https://issues.apache.org/jira/browse/YARN-1115?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412659#comment-17412659 ] Andras Gyori commented on YARN-1115: Thank you [~epayne] for working on this one. I have checked you latest patch a couple times and I think I understand the flow of the logic. However, it might only be me, but I have found the following a slightly confusing at first sight: * Submitting an app without a proxy user: ** user is the real user ** realUser is null * Submitting an app with a proxy user: ** user is the proxy user ** realUser is the real user I might find it a bit more intuitive to define proxyUser instead, thereby reverting this part of the logic a bit. Again, this is only a subjective preference, so others could find it less intuitive. > Provide optional means for a scheduler to check real user ACLs > -- > > Key: YARN-1115 > URL: https://issues.apache.org/jira/browse/YARN-1115 > Project: Hadoop YARN > Issue Type: Improvement > Components: capacity scheduler, scheduler >Affects Versions: 2.8.5 >Reporter: Eric Payne >Priority: Major > Attachments: YARN-1115.001.patch > > > In the framework for secure implementation using UserGroupInformation.doAs > (https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/Superusers.html), > a trusted superuser can submit jobs on behalf of another user in a secure > way. In this framework, the superuser is referred to as the real user and the > proxied user is referred to as the effective user. > Currently when a job is submitted as an effective user, the ACLs for the > effective user are checked against the queue on which the job is to be run. > Depending on an optional configuration, the scheduler should also check the > ACLs of the real user if the configuration to do so is set. > For example, suppose my superuser name is super, and super is configured to > securely proxy as joe. Also suppose there is a Hadoop queue named ops which > only allows ACLs for super, not for joe. > When super proxies to joe in order to submit a job to the ops queue, it will > fail because joe, as the effective user, does not have ACLs on the ops queue. > In many cases this is what you want, in order to protect queues that joe > should not be using. > However, there are times when super may need to proxy to many users, and the > client running as super just wants to use the ops queue because the ops queue > is already dedicated to the client's purpose, and, to keep the ops queue > dedicated to that purpose, super doesn't want to open up ACLs to joe in > general on the ops queue. Without this functionality, in this case, the > client running as super needs to figure out which queue each user has ACLs > opened up for, and then coordinate with other tasks using those queues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10938) Support reservation scheduling enabled switch for capacity scheduler
[
https://issues.apache.org/jira/browse/YARN-10938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated YARN-10938:
--
Labels: pull-request-available (was: )
> Support reservation scheduling enabled switch for capacity scheduler
>
>
> Key: YARN-10938
> URL: https://issues.apache.org/jira/browse/YARN-10938
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: capacity scheduler
>Reporter: Weihao Zheng
>Assignee: Weihao Zheng
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently the CapacityScheduler uses reservations in order to handle requests
> for large containers and the fact there might not currently be enough space
> available on a single host. But this algorithm is not suitable for small
> cluster which only have very limited resources. So we can add a switch
> property in capacity scheduler's configuration to avoid reservation
> scheduling in these use cases.
> CHANGES:
> Add {{"yarn.scheduler.capacity.reservation.enabled"}} in capacity scheduler's
> configuration.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17412470#comment-17412470 ] hao commented on YARN-10939: [~aajisaka] Hello, sir. Can you help me with this question? Thank you！！！ > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > Attachments: 企业微信截图_16311769849737.png, 企业微信截图_1631177020581.png > > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated YARN-10939: --- Attachment: 企业微信截图_16311769849737.png > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > Attachments: 企业微信截图_16311769849737.png, 企业微信截图_1631177020581.png > > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated YARN-10939: --- Attachment: 企业微信截图_1631177020581.png > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > Attachments: 企业微信截图_16311769849737.png, 企业微信截图_1631177020581.png > > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-10939) The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI
[ https://issues.apache.org/jira/browse/YARN-10939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hao updated YARN-10939: --- Summary: The task status of the same application on the yarn jobhistory is inconsistent with that on the yarn Web UI (was: The task status of yarn on jobhistory is inconsistent with that on yarn UI) > The task status of the same application on the yarn jobhistory is > inconsistent with that on the yarn Web UI > --- > > Key: YARN-10939 > URL: https://issues.apache.org/jira/browse/YARN-10939 > Project: Hadoop YARN > Issue Type: Bug >Reporter: hao >Priority: Major > > The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10939) The task status of yarn on jobhistory is inconsistent with that on yarn UI
hao created YARN-10939: -- Summary: The task status of yarn on jobhistory is inconsistent with that on yarn UI Key: YARN-10939 URL: https://issues.apache.org/jira/browse/YARN-10939 Project: Hadoop YARN Issue Type: Bug Reporter: hao The task status of yarn on jobhistory is inconsistent with that on yarn UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10938) Support reservation scheduling enabled switch for capacity scheduler
Weihao Zheng created YARN-10938:
---
Summary: Support reservation scheduling enabled switch for
capacity scheduler
Key: YARN-10938
URL: https://issues.apache.org/jira/browse/YARN-10938
Project: Hadoop YARN
Issue Type: Improvement
Components: capacity scheduler
Reporter: Weihao Zheng
Assignee: Weihao Zheng
Currently the CapacityScheduler uses reservations in order to handle requests
for large containers and the fact there might not currently be enough space
available on a single host. But this algorithm is not suitable for small
cluster which only have very limited resources. So we can add a switch property
in capacity scheduler's configuration to avoid reservation scheduling in these
use cases.
CHANGES:
Add {{"yarn.scheduler.capacity.reservation.enabled"}} in capacity scheduler's
configuration.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Assigned] (YARN-10927) Explain assertion literals in testcases of CapacityScheduler and related test classes
[ https://issues.apache.org/jira/browse/YARN-10927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tamas Domok reassigned YARN-10927: -- Assignee: Tamas Domok > Explain assertion literals in testcases of CapacityScheduler and related test > classes > - > > Key: YARN-10927 > URL: https://issues.apache.org/jira/browse/YARN-10927 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Szilard Nemeth >Assignee: Tamas Domok >Priority: Minor > > In the existing tests the assertion literals could be explained for easier > understanding As there are too many test classes, we can tackle this more > easily in a feature by feature fashion -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10937) Fix log message arguments in LogAggregationFileController
Tamas Domok created YARN-10937:
--
Summary: Fix log message arguments in LogAggregationFileController
Key: YARN-10937
URL: https://issues.apache.org/jira/browse/YARN-10937
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.3.1
Reporter: Tamas Domok
Assignee: Tamas Domok
More arguments provided (1) than placeholders specified (0) in the following
log message:
{code:java}
LOG.info("Unable to set permissions for configured filesystem since"
+ " it does not support this", remoteFS.getScheme());{code}
This is logged two times, both of them is affected.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Created] (YARN-10936) Fix typo in LogAggregationFileController
Tamas Domok created YARN-10936:
--
Summary: Fix typo in LogAggregationFileController
Key: YARN-10936
URL: https://issues.apache.org/jira/browse/YARN-10936
Project: Hadoop YARN
Issue Type: Bug
Affects Versions: 3.3.1
Reporter: Tamas Domok
Assignee: Tamas Domok
{code:java}
-LOG.warn("Failed to check if FileSystem suppports permissions on "
+LOG.warn("Failed to check if FileSystem supports permissions on "{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
