+1 (binding)
- Built from source
- Brought up a non-secure virtual cluster w/ NN, 1 DN, RM, AHS, JHS, and 3 NMs
- Validated inter- and intra-queue preemption
- Validated exclusive node labels
Thanks a lot Chao for your diligence and hard work on this release.
Eric
On Wednesday,
jobs are running
in labelled queues (YARN-4751).
- Ensure that a yarn distributed shell application can be launched and complete
successfully.
Eric Payne
From: Vinod Kumar Vavilapalli
To: "common-...@hadoop.apache.org" ;
hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org;
How do we come to a resolution regarding whether or not re-cut branch-2.8 or
release it as it is (after fixing blockers)?
There are some things in branch-2 that I would like to pull back into
branch-2.8, so a resolution to this question will affect how I proceed.
Thanks,-Eric
From: Karthik
Hey does anyone know why the precommit builds seem broken? I try to kick one
and it hangs for a long time and then produces the following:
#16288 (pending—H14 doesn’t have label Hadoop&&!H9&&!H5&&!H6; H15 doesn’t
have label Hadoop&&!H9&&!H5&&!H6; H16 doesn’t have label Hadoop&&!H9&&!H5&&!H6;
Thanks Andrew.
I downloaded the source, built it, and installed it onto a pseudo distributed
4-node cluster.
I ran mapred and streaming test cases, including sleep and wordcount.
+1 (non-binding)
-Eric
From: Andrew Wang
To: "common-...@hadoop.apache.org" ;
"hdfs-...@hadoop.apache.org"
+1 (binding)
Tested the following:
- Application History Server
-- Apps can be observed from UI
-- App and container metadata can be retrieved via REST APIs
- RM UI
-- Can kill an app from the RM UI
- Apps run in different frameworks. Frameworks tested: MR and yarn shell
-- In yarn shel
) preemption, both USERFIRST and PRIORITYFIRST
o User weights not equals to 1
o User weights in conjunction with in-queue preemption.
-Eric Payne
From: Andrew Wang
To: "common-...@hadoop.apache.org" ;
"hdfs-...@hadoop.apache.org" ;
"mapreduce-...@hadoop
+1 (binding)
Thanks a lot, Junping!
I built and installed the source on a 6-node pseudo cluster. I simple sleep and
streaming jobs that exercised intra-queue and inter-queue preemption, and used
user weights.
-Eric
From: Junping Du
To: "common-...@hadoop.apache.org" ;
"hdfs-...@hadoop.a
erformance degradation is observed with this branch work. SLS
and UT performance
tests are done.
There were 11 subtasks completed for this feature.
Huge thanks to everyone who helped with reviews, commits, guidance, and
technical discussion/design, including Wangda Tan, Vinod Vavilapalli,
Rohi
Thanks for the hard work on this release, Konstantin.
+1 (binding)
- Built from source
- Verified that refreshing of queues works as expected.
- Verified can run multiple users in a single queue
- Ran terasort test
- Verified that cross-queue preemption works as expected
Thanks. Eric Payne
, which is
the behavior I would expect if no node had the specified label. I will also add
that this procedure works fine in 2.7.
Thanks,-Eric Payne
From: Junping Du
To: "hdfs-...@hadoop.apache.org" ;
"yarn-dev@hadoop.apache.org" ;
"mapreduce-...@hado
e.org; Eric Payne
Cc: "common-...@hadoop.apache.org" ;
"hdfs-...@hadoop.apache.org" ;
"yarn-dev@hadoop.apache.org"
Sent: Monday, February 8, 2016 1:01 PM
Subject: Re: [VOTE] Release Apache Hadoop 2.6.4 RC0
+1 (non binding)
* Downloaded hadoop-2.6.4-RC0-src.tar
Thanks for working on this release!
+1 (binding)
I tested the following:
- yarn distributed shell job
- yarn streaming job
- inter-queue preemption
- compared behavior of fair and fifo ordering policy
- both userlimit_first mode and priority_first mode of intra-queue preemption
Eric Payne
- Tested simple inter-queue preemption
- Tested priority first intra-queue preemption
- Tested userlimit first intra-queue preemption
Thanks,Eric Payne
===
On Thursday, March 29, 2018, 11:15:51 PM CDT, Wangda Tan
wrote:
Hi folks
weights with FifoOrderingPolicy to ensure that weights were
assigned to users as expected.
Eric Payne
On Friday, April 6, 2018, 1:17:10 PM CDT, Lei Xu wrote:
Hi, All
I've created release candidate RC-0 for Apache Hadoop 3.0.2.
Please note: this is an amendment for Apache Hadoop
and without size-based
weight
- tested user weights
Thanks!
Eric Payne
On Monday, April 16, 2018, 7:00:03 PM CDT, Lei Xu wrote:
Hi, All
I've created release candidate RC-1 for Apache Hadoop 3.0.2, to
address missing source jars in the maven repository in RC-0.
Thanks Ajay Kuma
Sorry, Yongjun. My +1 is also binding+1 (binding)-Eric Payne
On Friday, June 1, 2018, 12:25:36 PM CDT, Eric Payne
wrote:
Thanks a lot, Yongjun, for your hard work on this release.
+1
- Built from source
- Installed on 6 node pseudo cluster
Tested the following in the Capacity
Thanks Wangda for creating this release.
+1 (binding)
Tested:
- Built from source
- Deployed to 6-node, multi-tennant, unsecured pseudo cluster with hierarchical
queue structure (CS)
- Refreshed queue (CS) properties
- Intra-queue preemption (CS)
- inter-queue preemption (CS)
- User weights (CS)
+1 (binding)
-- Built from source-- Installed on 6-node pseudo cluster-- Tested intra-
inter-queue preemption, user weights-- Ran streaming jobs, word count, and tara
gen/sort tests
Thanks Akira for all of the hard work.-Eric Payne
On Tuesday, November 13, 2018, 7:02:51 PM CST, Akira
1:47:06 AM CST, Eric Payne
wrote:
Sunil, thanks for all of the hard work on this release.
I have discovered that queue refresh doesn't work in some cases. For example,
when I change yarn.scheduler.capacity.root.default.disable_preemption, it
doesn't take effect unless I restart
locker for
release. I'm on the fence.
Thanks,
-Eric
On Wednesday, November 28, 2018, 4:58:50 PM CST, Eric Payne
wrote:
Sunil,
So, the basic symptoms are that if preemption is enabled on any queue, the
preemption is disabled after a 'yarn rm -refreshQueues'. In additio
+1 (binding)
- RM refresh updates values as expected
- Streaming jobs complete successfully
- Moving apps between queues succeeds
- Inter-queue preemption works as expected
- Successfully ran selected yarn unit tests.
===
Eric Payne
===
On Tuesday, January 8, 2019, 5:42:46 AM CST
It is my pleasure to announce that Eric Badger has accepted an invitation to
become a Hadoop Core committer.
Congratulations, Eric! This is well-deserved!
-Eric Payne
Eric Payne created YARN-10935:
-
Summary: AM Total Queue Limit goes below per-uwer AM Limit if
parent is full.
Key: YARN-10935
URL: https://issues.apache.org/jira/browse/YARN-10935
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-9975?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9975.
--
Resolution: Duplicate
I'm closing this as a dup of YARN-1115. Please reopen if you disagree.
>
[
https://issues.apache.org/jira/browse/YARN-10848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-10848.
---
Resolution: Not A Problem
I am closing this JIRA based on the above discussion.
> Vcore allocat
Eric Payne created YARN-9756:
Summary: Create metric that sums total memory/vcores preempted per
round
Key: YARN-9756
URL: https://issues.apache.org/jira/browse/YARN-9756
Project: Hadoop YARN
Eric Payne created YARN-9894:
Summary: CapacitySchedulerPerf test for measuring hundreds of apps
in a large number of queues.
Key: YARN-9894
URL: https://issues.apache.org/jira/browse/YARN-9894
Project
Eric Payne created YARN-9911:
Summary: Backport YARN-9773 (Add QueueMetrics for Custom
Resources) to branch-2 and branch-2.10
Key: YARN-9911
URL: https://issues.apache.org/jira/browse/YARN-9911
Project
[
https://issues.apache.org/jira/browse/YARN-9773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9773.
--
Fix Version/s: 3.1.4
3.2.2
3.3.0
Resolution: Fixed
Thanks
Eric Payne created YARN-10009:
-
Summary: DRF can treat minimum user limit percent as a max when
custom resource is defined
Key: YARN-10009
URL: https://issues.apache.org/jira/browse/YARN-10009
Project
Eric Payne created YARN-10033:
-
Summary: TestProportionalCapacityPreemptionPolicy not initializing
vcores for effective max resources
Key: YARN-10033
URL: https://issues.apache.org/jira/browse/YARN-10033
Eric Payne created YARN-10084:
-
Summary: Allow inheritance of max app lifetime / default app
lifetime
Key: YARN-10084
URL: https://issues.apache.org/jira/browse/YARN-10084
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-9790?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9790.
--
Fix Version/s: 2.10.1
3.1.4
3.2.2
Resolution: Fixed
Eric Payne created YARN-10164:
-
Summary: Allow NM to start even when custom resource type not
defined
Key: YARN-10164
URL: https://issues.apache.org/jira/browse/YARN-10164
Project: Hadoop YARN
Eric Payne created YARN-10251:
-
Summary: Show extended resources on legacy RM UI.
Key: YARN-10251
URL: https://issues.apache.org/jira/browse/YARN-10251
Project: Hadoop YARN
Issue Type
[
https://issues.apache.org/jira/browse/YARN-9767?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9767.
--
Resolution: Duplicate
> PartitionQueueMetrics Iss
Eric Payne created YARN-10343:
-
Summary: Legacy RM UI should include labeled metrics for
allocated, total, and reserved resources.
Key: YARN-10343
URL: https://issues.apache.org/jira/browse/YARN-10343
[
https://issues.apache.org/jira/browse/YARN-1741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-1741.
--
Resolution: Won't Fix
bq. Since branch-2.8 is EOL, I propose that we close this as Won'
Eric Payne created YARN-10451:
-
Summary: RM (v1) UI NodesPage can NPE when yarn.io/gpu resource
type is defined.
Key: YARN-10451
URL: https://issues.apache.org/jira/browse/YARN-10451
Project: Hadoop YARN
Eric Payne created YARN-10456:
-
Summary: RM PartitionQueueMetrics records are named QueueMetrics
in Simon metrics registry
Key: YARN-10456
URL: https://issues.apache.org/jira/browse/YARN-10456
Project
Eric Payne created YARN-10471:
-
Summary: Prevent logs for any container from becoming larger than
a configurable size.
Key: YARN-10471
URL: https://issues.apache.org/jira/browse/YARN-10471
Project
[
https://issues.apache.org/jira/browse/YARN-10164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-10164.
---
Resolution: Won't Do
> Allow NM to start even when custom resource type not
Eric Payne created YARN-10613:
-
Summary: Config to allow Intra-queue preemption to enable/disable
conservativeDRF
Key: YARN-10613
URL: https://issues.apache.org/jira/browse/YARN-10613
Project: Hadoop
Eric Payne created YARN-10834:
-
Summary: Intra-queue preemption: apps that don't use defined
custom resource won't be preempted.
Key: YARN-10834
URL: https://issues.apache.org/jira/browse/
Eric Payne created YARN-:
Summary: Scheduler UI: "% of Queue" is inaccurate if leaf queue is
hierarchically nested.
Key: YARN-
URL: https://issues.apache.org/jira/browse/YARN-
Proje
Eric Payne created YARN-5973:
Summary: TestCapacitySchedulerSurgicalPreemption sometimes fails
Key: YARN-5973
URL: https://issues.apache.org/jira/browse/YARN-5973
Project: Hadoop YARN
Issue Type
Eric Payne created YARN-6165:
Summary: Intra-queue preemption occurs even when preemption is
turned off for a specific queue.
Key: YARN-6165
URL: https://issues.apache.org/jira/browse/YARN-6165
Project
Eric Payne created YARN-6248:
Summary: Killing an app with pending container requests leaves the
user in UsersManager
Key: YARN-6248
URL: https://issues.apache.org/jira/browse/YARN-6248
Project: Hadoop
Eric Payne created YARN-6585:
Summary: RM fails to start when upgrading from 2.7 to 2.8 for
clusters with node labels.
Key: YARN-6585
URL: https://issues.apache.org/jira/browse/YARN-6585
Project: Hadoop
Eric Payne created YARN-7051:
Summary: FifoIntraQueuePreemptionPlugin can get concurrent
modification exception/
Key: YARN-7051
URL: https://issues.apache.org/jira/browse/YARN-7051
Project: Hadoop YARN
Eric Payne created YARN-7052:
Summary: RM SchedulingMonitor should use HadoopExecutors when
creating ScheduledExecutorService
Key: YARN-7052
URL: https://issues.apache.org/jira/browse/YARN-7052
Project
Eric Payne created YARN-7120:
Summary: CapacitySchedulerPage NPE in "Aggregate scheduler counts"
section
Key: YARN-7120
URL: https://issues.apache.org/jira/browse/YARN-7120
Project: H
Eric Payne created YARN-7149:
Summary: Cross-queue preemption sometimes starves an underserved
queue
Key: YARN-7149
URL: https://issues.apache.org/jira/browse/YARN-7149
Project: Hadoop YARN
Eric Payne created YARN-7245:
Summary: In Cap Sched UI, Max AM Resource column in Active Users
Info section should be per-user
Key: YARN-7245
URL: https://issues.apache.org/jira/browse/YARN-7245
Project
Eric Payne created YARN-7370:
Summary: Intra-queue preemption properties should be refreshable
Key: YARN-7370
URL: https://issues.apache.org/jira/browse/YARN-7370
Project: Hadoop YARN
Issue Type
Eric Payne created YARN-7424:
Summary: Capacity Scheduler Intra-queue preemption: add property
to only preempt up to configured MULP
Key: YARN-7424
URL: https://issues.apache.org/jira/browse/YARN-7424
Eric Payne created YARN-7469:
Summary: Capacity Scheduler Intra-queue preemption: User can
starve if newest app is exactly at user limit
Key: YARN-7469
URL: https://issues.apache.org/jira/browse/YARN-7469
Eric Payne created YARN-7496:
Summary: CS Intra-queue preemption user-limit calculations are not
in line with LeafQueue user-limit calculations
Key: YARN-7496
URL: https://issues.apache.org/jira/browse/YARN-7496
Eric Payne created YARN-7501:
Summary: Capacity Scheduler Intra-queue preemption should have a
"dead zone" around user limit
Key: YARN-7501
URL: https://issues.apache.org/jira/browse/YARN-7501
Eric Payne created YARN-7575:
Summary: When using absolute capacity configuration with no max
capacity, scheduler UI NPEs and can't grow queue
Key: YARN-7575
URL: https://issues.apache.org/jira/browse/YARN
Eric Payne created YARN-7619:
Summary: Max AM Resource value in CS UI is different for every user
Key: YARN-7619
URL: https://issues.apache.org/jira/browse/YARN-7619
Project: Hadoop YARN
Issue
Eric Payne created YARN-7658:
Summary: Capacity scheduler UI hangs when rendering if labels are
present
Key: YARN-7658
URL: https://issues.apache.org/jira/browse/YARN-7658
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-7658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-7658.
--
Resolution: Duplicate
> Capacity scheduler UI hangs when rendering if labels are pres
Eric Payne created YARN-7728:
Summary: Expose and expand container preemptions in Capacity
Scheduler queue metrics
Key: YARN-7728
URL: https://issues.apache.org/jira/browse/YARN-7728
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-7424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-7424.
--
Resolution: Invalid
bq. In order to create the "desired" behavior, we would have to fun
Eric Payne created YARN-4390:
Summary: Consider container request size during CS preemption
Key: YARN-4390
URL: https://issues.apache.org/jira/browse/YARN-4390
Project: Hadoop YARN
Issue Type
Eric Payne created YARN-4422:
Summary: Generic AHS sometimes doesn't show started, node, or logs
on App page
Key: YARN-4422
URL: https://issues.apache.org/jira/browse/YARN-4422
Project: Hadoop
[
https://issues.apache.org/jira/browse/YARN-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-4226.
--
Resolution: Won't Fix
Since the code works and is only slightly confusing, I am closing this ticke
[
https://issues.apache.org/jira/browse/YARN-4390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-4390.
--
Resolution: Duplicate
Closing this ticket in favor of YARN-4108
> Consider container request s
Eric Payne created YARN-7813:
Summary: Capacity Scheduler Intra-queue Preemption should be
configurable for each queue
Key: YARN-7813
URL: https://issues.apache.org/jira/browse/YARN-7813
Project: Hadoop
Eric Payne created YARN-7927:
Summary: YARN-7813 caused test failure in
TestRMWebServicesSchedulerActivities
Key: YARN-7927
URL: https://issues.apache.org/jira/browse/YARN-7927
Project: Hadoop YARN
Eric Payne created YARN-7947:
Summary: Capacity Scheduler intra-queue preemption can NPE for
non-schedulable apps
Key: YARN-7947
URL: https://issues.apache.org/jira/browse/YARN-7947
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-8425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-8425.
--
Resolution: Not A Bug
> Yarn container getting killed due to running beyond physical memory lim
[
https://issues.apache.org/jira/browse/YARN-9685?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-9685.
--
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: 3.1.3
3.2.1
Eric Payne created YARN-3769:
Summary: Preemption occurring unnecessarily because preemption
doesn't consider user limit
Key: YARN-3769
URL: https://issues.apache.org/jira/browse/YARN-3769
Pr
Eric Payne created YARN-3905:
Summary: Application History Server UI NPEs when accessing apps
run after RM restart
Key: YARN-3905
URL: https://issues.apache.org/jira/browse/YARN-3905
Project: Hadoop YARN
Eric Payne created YARN-3978:
Summary: Configurably turn off the saving of container info in
Generic AHS
Key: YARN-3978
URL: https://issues.apache.org/jira/browse/YARN-3978
Project: Hadoop YARN
Eric Payne created YARN-4217:
Summary: Failed AM attempt retries on same failed host
Key: YARN-4217
URL: https://issues.apache.org/jira/browse/YARN-4217
Project: Hadoop YARN
Issue Type
[
https://issues.apache.org/jira/browse/YARN-4217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-4217.
--
Resolution: Duplicate
bq. Eric Payne - is this a duplicate of YARN-2005?
[~vvasudev], yes it is. I did
Eric Payne created YARN-4225:
Summary: Add preemption status to {{yarn queue -status}}
Key: YARN-4225
URL: https://issues.apache.org/jira/browse/YARN-4225
Project: Hadoop YARN
Issue Type: Bug
Eric Payne created YARN-4226:
Summary: Make capacity scheduler queue's preemption status REST
API consistent with GUI
Key: YARN-4226
URL: https://issues.apache.org/jira/browse/YARN-4226
Project: H
Eric Payne created YARN-4751:
Summary: In 2.7, Labeled queue usage not shown properly in
capacity scheduler UI
Key: YARN-4751
URL: https://issues.apache.org/jira/browse/YARN-4751
Project: Hadoop YARN
[
https://issues.apache.org/jira/browse/YARN-2592?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne resolved YARN-2592.
--
Resolution: Invalid
> Preemption can kill containers to fulfil need of already over-capacity qu
Eric Payne created YARN-3275:
Summary: Preemption happening on non-preemptable queues
Key: YARN-3275
URL: https://issues.apache.org/jira/browse/YARN-3275
Project: Hadoop YARN
Issue Type: Bug
Eric Payne created YARN-3540:
Summary: Fetcher#copyMapOutput is leaking usedMemory upon
IOException during InMemoryMapOutput shuffle handler
Key: YARN-3540
URL: https://issues.apache.org/jira/browse/YARN-3540
Eric Payne created YARN-2592:
Summary: Preemption can kill containers to fulfil need of already
over-capacity queue.
Key: YARN-2592
URL: https://issues.apache.org/jira/browse/YARN-2592
Project: Hadoop
Eric Payne created YARN-2932:
Summary: Add entry for preemption setting to queue status screen
and startup/refresh logging
Key: YARN-2932
URL: https://issues.apache.org/jira/browse/YARN-2932
Project
Eric Payne created YARN-1115:
Summary: Provide optional means for a scheduler to check real user
ACLs
Key: YARN-1115
URL: https://issues.apache.org/jira/browse/YARN-1115
Project: Hadoop YARN
Eric Payne created YARN-2024:
Summary: IOException in AppLogAggregatorImpl does not give
stacktrace and leaves aggregated TFile in a bad state.
Key: YARN-2024
URL: https://issues.apache.org/jira/browse/YARN-2024
90 matches
Mail list logo
