+1 (binding)
- Built from source
- Brought up a non-secure virtual cluster w/ NN, 1 DN, RM, AHS, JHS, and 3 NMs
- Validated inter- and intra-queue preemption
- Validated exclusive node labels
Thanks a lot Chao for your diligence and hard work on this release.
Eric
On
[
https://issues.apache.org/jira/browse/YARN-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-10848.
---
Resolution: Not A Problem
I am closing this JIRA based on the above discussion.
> Vcore allocat
[
https://issues.apache.org/jira/browse/YARN-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9975.
--
Resolution: Duplicate
I'm closing this as a dup of YARN-1115. Please reopen if you disagree.
> Supp
Eric Payne created YARN-10935:
-
Summary: AM Total Queue Limit goes below per-uwer AM Limit if
parent is full.
Key: YARN-10935
URL: https://issues.apache.org/jira/browse/YARN-10935
Project: Hadoop YARN
Eric Payne created YARN-10834:
-
Summary: Intra-queue preemption: apps that don't use defined
custom resource won't be preempted.
Key: YARN-10834
URL: https://issues.apache.org/jira/browse/YARN-10834
Eric Payne created YARN-10613:
-
Summary: Config to allow Intra-queue preemption to enable/disable
conservativeDRF
Key: YARN-10613
URL: https://issues.apache.org/jira/browse/YARN-10613
Project: Hadoop
[
https://issues.apache.org/jira/browse/YARN-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-10164.
---
Resolution: Won't Do
> Allow NM to start even when custom resource type not defi
Eric Payne created YARN-10471:
-
Summary: Prevent logs for any container from becoming larger than
a configurable size.
Key: YARN-10471
URL: https://issues.apache.org/jira/browse/YARN-10471
Project
Eric Payne created YARN-10456:
-
Summary: RM PartitionQueueMetrics records are named QueueMetrics
in Simon metrics registry
Key: YARN-10456
URL: https://issues.apache.org/jira/browse/YARN-10456
Project
Eric Payne created YARN-10451:
-
Summary: RM (v1) UI NodesPage can NPE when yarn.io/gpu resource
type is defined.
Key: YARN-10451
URL: https://issues.apache.org/jira/browse/YARN-10451
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-1741.
--
Resolution: Won't Fix
bq. Since branch-2.8 is EOL, I propose that we close this as Won't Fix.
+1
Eric Payne created YARN-10343:
-
Summary: Legacy RM UI should include labeled metrics for
allocated, total, and reserved resources.
Key: YARN-10343
URL: https://issues.apache.org/jira/browse/YARN-10343
[
https://issues.apache.org/jira/browse/YARN-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9767.
--
Resolution: Duplicate
> PartitionQueueMetrics Iss
Eric Payne created YARN-10251:
-
Summary: Show extended resources on legacy RM UI.
Key: YARN-10251
URL: https://issues.apache.org/jira/browse/YARN-10251
Project: Hadoop YARN
Issue Type
Eric Payne created YARN-10164:
-
Summary: Allow NM to start even when custom resource type not
defined
Key: YARN-10164
URL: https://issues.apache.org/jira/browse/YARN-10164
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9790.
--
Fix Version/s: 2.10.1
3.1.4
3.2.2
Resolution: Fixed
Eric Payne created YARN-10084:
-
Summary: Allow inheritance of max app lifetime / default app
lifetime
Key: YARN-10084
URL: https://issues.apache.org/jira/browse/YARN-10084
Project: Hadoop YARN
Eric Payne created YARN-10033:
-
Summary: TestProportionalCapacityPreemptionPolicy not initializing
vcores for effective max resources
Key: YARN-10033
URL: https://issues.apache.org/jira/browse/YARN-10033
Eric Payne created YARN-10009:
-
Summary: DRF can treat minimum user limit percent as a max when
custom resource is defined
Key: YARN-10009
URL: https://issues.apache.org/jira/browse/YARN-10009
Project
[
https://issues.apache.org/jira/browse/YARN-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9773.
--
Fix Version/s: 3.1.4
3.2.2
3.3.0
Resolution: Fixed
Thanks
Eric Payne created YARN-9911:
Summary: Backport YARN-9773 (Add QueueMetrics for Custom
Resources) to branch-2 and branch-2.10
Key: YARN-9911
URL: https://issues.apache.org/jira/browse/YARN-9911
Project
Eric Payne created YARN-9894:
Summary: CapacitySchedulerPerf test for measuring hundreds of apps
in a large number of queues.
Key: YARN-9894
URL: https://issues.apache.org/jira/browse/YARN-9894
Project
Eric Payne created YARN-9756:
Summary: Create metric that sums total memory/vcores preempted per
round
Key: YARN-9756
URL: https://issues.apache.org/jira/browse/YARN-9756
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9685.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
3.2.1
It is my pleasure to announce that Eric Badger has accepted an invitation to
become a Hadoop Core committer.
Congratulations, Eric! This is well-deserved!
-Eric Payne
+1 (binding)
- RM refresh updates values as expected
- Streaming jobs complete successfully
- Moving apps between queues succeeds
- Inter-queue preemption works as expected
- Successfully ran selected yarn unit tests.
===
Eric Payne
===
On Tuesday, January 8, 2019, 5:42:46 AM CST
a blocker for
release. I'm on the fence.
Thanks,
-Eric
On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne
wrote:
Sunil,
So, the basic symptoms are that if preemption is enabled on any queue, the
preemption is disabled after a 'yarn rm -refreshQueues'. In addition, all of
the
CST, Eric Payne
wrote:
Sunil, thanks for all of the hard work on this release.
I have discovered that queue refresh doesn't work in some cases. For example,
when I change yarn.scheduler.capacity.root.default.disable_preemption, it
doesn't take effect unless I restart the RM.
I am still
+1 (binding)
-- Built from source-- Installed on 6-node pseudo cluster-- Tested intra-
inter-queue preemption, user weights-- Ran streaming jobs, word count, and tara
gen/sort tests
Thanks Akira for all of the hard work.-Eric Payne
On Tuesday, November 13, 2018, 7:02:51 PM CST, Akira
Thanks Wangda for creating this release.
+1 (binding)
Tested:
- Built from source
- Deployed to 6-node, multi-tennant, unsecured pseudo cluster with hierarchical
queue structure (CS)
- Refreshed queue (CS) properties
- Intra-queue preemption (CS)
- inter-queue preemption (CS)
- User weights (CS)
[
https://issues.apache.org/jira/browse/YARN-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-8425.
--
Resolution: Not A Bug
> Yarn container getting killed due to running beyond physical memory lim
Sorry, Yongjun. My +1 is also binding+1 (binding)-Eric Payne
On Friday, June 1, 2018, 12:25:36 PM CDT, Eric Payne
wrote:
Thanks a lot, Yongjun, for your hard work on this release.
+1
- Built from source
- Installed on 6 node pseudo cluster
Tested the following in the Capacity
, with and without size-based
weight
- tested user weights
Thanks!
Eric Payne
On Monday, April 16, 2018, 7:00:03 PM CDT, Lei Xu <l...@apache.org> wrote:
Hi, All
I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
address missing source jars in the maven repository in RC-0.
weights with FifoOrderingPolicy to ensure that weights were
assigned to users as expected.
Eric Payne
On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu <l...@apache.org> wrote:
Hi, All
I've created release candidate RC-0 for Apache Hadoop 3.0.2.
Please note: this is an ame
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan
<wheele...@gmail.
Thanks for working on this release!
+1 (binding)
I tested the following:
- yarn distributed shell job
- yarn streaming job
- inter-queue preemption
- compared behavior of fair and fifo ordering policy
- both userlimit_first mode and priority_first mode of intra-queue preemption
Eric Payne
Eric Payne created YARN-7947:
Summary: Capacity Scheduler intra-queue preemption can NPE for
non-schedulable apps
Key: YARN-7947
URL: https://issues.apache.org/jira/browse/YARN-7947
Project: Hadoop YARN
Eric Payne created YARN-7927:
Summary: YARN-7813 caused test failure in
TestRMWebServicesSchedulerActivities
Key: YARN-7927
URL: https://issues.apache.org/jira/browse/YARN-7927
Project: Hadoop YARN
Eric Payne created YARN-7813:
Summary: Capacity Scheduler Intra-queue Preemption should be
configurable for each queue
Key: YARN-7813
URL: https://issues.apache.org/jira/browse/YARN-7813
Project: Hadoop
[
https://issues.apache.org/jira/browse/YARN-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-7424.
--
Resolution: Invalid
bq. In order to create the "desired" behavior, we would have to fun
Eric Payne created YARN-7728:
Summary: Expose and expand container preemptions in Capacity
Scheduler queue metrics
Key: YARN-7728
URL: https://issues.apache.org/jira/browse/YARN-7728
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-7658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-7658.
--
Resolution: Duplicate
> Capacity scheduler UI hangs when rendering if labels are pres
Eric Payne created YARN-7658:
Summary: Capacity scheduler UI hangs when rendering if labels are
present
Key: YARN-7658
URL: https://issues.apache.org/jira/browse/YARN-7658
Project: Hadoop YARN
Thanks for the hard work on this release, Konstantin.
+1 (binding)
- Built from source
- Verified that refreshing of queues works as expected.
- Verified can run multiple users in a single queue
- Ran terasort test
- Verified that cross-queue preemption works as expected
Thanks. Eric Payne
Eric Payne created YARN-7619:
Summary: Max AM Resource value in CS UI is different for every user
Key: YARN-7619
URL: https://issues.apache.org/jira/browse/YARN-7619
Project: Hadoop YARN
Issue
this feature.
Huge thanks to everyone who helped with reviews, commits, guidance, and
technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
Rohith Sharma K S, Eric Payne .
[1] :
http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201711.mbox/%3CCACYiTuhKhF1JCtR7ZFuZSEKQ4s
Eric Payne created YARN-7575:
Summary: When using absolute capacity configuration with no max
capacity, scheduler UI NPEs and can't grow queue
Key: YARN-7575
URL: https://issues.apache.org/jira/browse/YARN-7575
Eric Payne created YARN-7501:
Summary: Capacity Scheduler Intra-queue preemption should have a
"dead zone" around user limit
Key: YARN-7501
URL: https://issues.apache.org/jira/browse/YARN-7501
Eric Payne created YARN-7496:
Summary: CS Intra-queue preemption user-limit calculations are not
in line with LeafQueue user-limit calculations
Key: YARN-7496
URL: https://issues.apache.org/jira/browse/YARN-7496
Eric Payne created YARN-7469:
Summary: Capacity Scheduler Intra-queue preemption: User can
starve if newest app is exactly at user limit
Key: YARN-7469
URL: https://issues.apache.org/jira/browse/YARN-7469
Eric Payne created YARN-7424:
Summary: Capacity Scheduler Intra-queue preemption: add property
to only preempt up to configured MULP
Key: YARN-7424
URL: https://issues.apache.org/jira/browse/YARN-7424
+1 (binding)
Thanks a lot, Junping!
I built and installed the source on a 6-node pseudo cluster. I simple sleep and
streaming jobs that exercised intra-queue and inter-queue preemption, and used
user weights.
-Eric
From: Junping Du
To:
Eric Payne created YARN-7370:
Summary: Intra-queue preemption properties should be refreshable
Key: YARN-7370
URL: https://issues.apache.org/jira/browse/YARN-7370
Project: Hadoop YARN
Issue Type
) preemption, both USERFIRST and PRIORITYFIRST
o User weights not equals to 1
o User weights in conjunction with in-queue preemption.
-Eric Payne
From: Andrew Wang <andrew.w...@cloudera.com>
To: "common-...@hadoop.apache.org" <common-...@hadoop.apache.org>;
"hdfs-.
Eric Payne created YARN-7245:
Summary: In Cap Sched UI, Max AM Resource column in Active Users
Info section should be per-user
Key: YARN-7245
URL: https://issues.apache.org/jira/browse/YARN-7245
Project
Eric Payne created YARN-7149:
Summary: Cross-queue preemption sometimes starves an underserved
queue
Key: YARN-7149
URL: https://issues.apache.org/jira/browse/YARN-7149
Project: Hadoop YARN
Eric Payne created YARN-7120:
Summary: CapacitySchedulerPage NPE in "Aggregate scheduler counts"
section
Key: YARN-7120
URL: https://issues.apache.org/jira/browse/YARN-7120
Project: H
Eric Payne created YARN-7052:
Summary: RM SchedulingMonitor should use HadoopExecutors when
creating ScheduledExecutorService
Key: YARN-7052
URL: https://issues.apache.org/jira/browse/YARN-7052
Project
Eric Payne created YARN-7051:
Summary: FifoIntraQueuePreemptionPlugin can get concurrent
modification exception/
Key: YARN-7051
URL: https://issues.apache.org/jira/browse/YARN-7051
Project: Hadoop YARN
+1 (binding)
Tested the following:
- Application History Server
-- Apps can be observed from UI
-- App and container metadata can be retrieved via REST APIs
- RM UI
-- Can kill an app from the RM UI
- Apps run in different frameworks. Frameworks tested: MR and yarn shell
-- In yarn
Thanks Andrew.
I downloaded the source, built it, and installed it onto a pseudo distributed
4-node cluster.
I ran mapred and streaming test cases, including sleep and wordcount.
+1 (non-binding)
-Eric
From: Andrew Wang
To: "common-...@hadoop.apache.org"
Hey does anyone know why the precommit builds seem broken? I try to kick one
and it hangs for a long time and then produces the following:
#16288 (pending—H14 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H15 doesn’t
have label Hadoop&&!H9&&!H5&&!H6; H16 doesn’t have label Hadoop&&!H9&&!H5&&!H6;
Eric Payne created YARN-6585:
Summary: RM fails to start when upgrading from 2.7 to 2.8 for
clusters with node labels.
Key: YARN-6585
URL: https://issues.apache.org/jira/browse/YARN-6585
Project: Hadoop
Eric Payne created YARN-6248:
Summary: Killing an app with pending container requests leaves the
user in UsersManager
Key: YARN-6248
URL: https://issues.apache.org/jira/browse/YARN-6248
Project: Hadoop
Eric Payne created YARN-6165:
Summary: Intra-queue preemption occurs even when preemption is
turned off for a specific queue.
Key: YARN-6165
URL: https://issues.apache.org/jira/browse/YARN-6165
Project
Eric Payne created YARN-5973:
Summary: TestCapacitySchedulerSurgicalPreemption sometimes fails
Key: YARN-5973
URL: https://issues.apache.org/jira/browse/YARN-5973
Project: Hadoop YARN
Issue Type
How do we come to a resolution regarding whether or not re-cut branch-2.8 or
release it as it is (after fixing blockers)?
There are some things in branch-2 that I would like to pull back into
branch-2.8, so a resolution to this question will affect how I proceed.
Thanks,-Eric
From: Karthik
jobs are running
in labelled queues (YARN-4751).
- Ensure that a yarn distributed shell application can be launched and complete
successfully.
Eric Payne
From: Vinod Kumar Vavilapalli <vino...@apache.org>
To: "common-...@hadoop.apache.org" <common-...@hadoop
Eric Payne created YARN-4751:
Summary: In 2.7, Labeled queue usage not shown properly in
capacity scheduler UI
Key: YARN-4751
URL: https://issues.apache.org/jira/browse/YARN-4751
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-4390.
--
Resolution: Duplicate
Closing this ticket in favor of YARN-4108
> Consider container request s
[
https://issues.apache.org/jira/browse/YARN-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-4226.
--
Resolution: Won't Fix
Since the code works and is only slightly confusing, I am closing this ticket
Eric Payne created YARN-4422:
Summary: Generic AHS sometimes doesn't show started, node, or logs
on App page
Key: YARN-4422
URL: https://issues.apache.org/jira/browse/YARN-4422
Project: Hadoop YARN
Eric Payne created YARN-4390:
Summary: Consider container request size during CS preemption
Key: YARN-4390
URL: https://issues.apache.org/jira/browse/YARN-4390
Project: Hadoop YARN
Issue Type
Eric Payne created YARN-4225:
Summary: Add preemption status to {{yarn queue -status}}
Key: YARN-4225
URL: https://issues.apache.org/jira/browse/YARN-4225
Project: Hadoop YARN
Issue Type: Bug
Eric Payne created YARN-4226:
Summary: Make capacity scheduler queue's preemption status REST
API consistent with GUI
Key: YARN-4226
URL: https://issues.apache.org/jira/browse/YARN-4226
Project: Hadoop
Eric Payne created YARN-3978:
Summary: Configurably turn off the saving of container info in
Generic AHS
Key: YARN-3978
URL: https://issues.apache.org/jira/browse/YARN-3978
Project: Hadoop YARN
Eric Payne created YARN-3905:
Summary: Application History Server UI NPEs when accessing apps
run after RM restart
Key: YARN-3905
URL: https://issues.apache.org/jira/browse/YARN-3905
Project: Hadoop YARN
Eric Payne created YARN-3769:
Summary: Preemption occurring unnecessarily because preemption
doesn't consider user limit
Key: YARN-3769
URL: https://issues.apache.org/jira/browse/YARN-3769
Project
Eric Payne created YARN-3540:
Summary: Fetcher#copyMapOutput is leaking usedMemory upon
IOException during InMemoryMapOutput shuffle handler
Key: YARN-3540
URL: https://issues.apache.org/jira/browse/YARN-3540
Eric Payne created YARN-3275:
Summary: Preemption happening on non-preemptable queues
Key: YARN-3275
URL: https://issues.apache.org/jira/browse/YARN-3275
Project: Hadoop YARN
Issue Type: Bug
[
https://issues.apache.org/jira/browse/YARN-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-2592.
--
Resolution: Invalid
Preemption can kill containers to fulfil need of already over-capacity queue
Eric Payne created YARN-2592:
Summary: Preemption can kill containers to fulfil need of already
over-capacity queue.
Key: YARN-2592
URL: https://issues.apache.org/jira/browse/YARN-2592
Project: Hadoop
Eric Payne created YARN-2024:
Summary: IOException in AppLogAggregatorImpl does not give
stacktrace and leaves aggregated TFile in a bad state.
Key: YARN-2024
URL: https://issues.apache.org/jira/browse/YARN-2024
Eric Payne created YARN-1115:
Summary: Provide optional means for a scheduler to check real user
ACLs
Key: YARN-1115
URL: https://issues.apache.org/jira/browse/YARN-1115
Project: Hadoop YARN
84 matches
Mail list logo