[jira] [Resolved] (HDFS-10467) Router-based HDFS federation

[JIRA]

 [ 
https://issues.apache.org/jira/browse/HDFS-10467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Íñigo Goiri resolved HDFS-10467.

  Resolution: Fixed
Hadoop Flags: Reviewed

> Router-based HDFS federation
> 
>
> Key: HDFS-10467
> URL: https://issues.apache.org/jira/browse/HDFS-10467
> Project: Hadoop HDFS
>  Issue Type: New Feature
>  Components: fs
>Affects Versions: 2.8.1
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>  Labels: RBF
> Fix For: HDFS-10467
>
> Attachments: HDFS-10467.002.patch, HDFS-10467.PoC.001.patch, 
> HDFS-10467.PoC.patch, HDFS Router Federation.pdf, 
> HDFS-Router-Federation-Prototype.patch
>
>
> Add a Router to provide a federated view of multiple HDFS clusters.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12615) Router-based HDFS federation phase 2

[JIRA]

Íñigo Goiri created HDFS-12615:
--

 Summary: Router-based HDFS federation phase 2
 Key: HDFS-12615
 URL: https://issues.apache.org/jira/browse/HDFS-12615
 Project: Hadoop HDFS
  Issue Type: Improvement
Reporter: Íñigo Goiri
Assignee: Íñigo Goiri


This umbrella JIRA tracks set of improvements over the Router-based HDFS 
federatio (HDFS-10467).



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Merge Router-Based Federation (HDFS-10467) branch into trunk/branch-3

[Iñigo Goiri]

The 7 days has passed, and we got 3 +1s and no -1.
So the merge passes.
I'll go ahead with the merge.

It's a total of 21 patches.
Tested it locally and it works as expected with the latest trunk and
branch-3.
I'll go ahead and merge to both.

I'll close HDFS-10467 and start a new umbrella for the second phase.


Thanks for the votes!


On Tue, Oct 3, 2017 at 9:50 AM, Brahma Reddy Battula 
wrote:

> Thanks Inigo.
>
> +1 (binding)
>
> Nice feature.Involved in reviewing some jiras.
>
> On Sat, 30 Sep 2017 at 12:29 AM, Iñigo Goiri  wrote:
>
> > Hi all,
> >
> > Given that 3.0-beta1 is already cut, I’d like to call a vote for merging
> > Router-Based Federation (HDFS-10467) to trunk and branch-3.
> >
> > The vote will run for 7 days as usual.
> >
> >
> >
> > We started the discussion about merging HDFS-10467 a few weeks ago [1]
> and
> > got good feedback which we’ve incorporated already [2, 3, 4].
> >
> > There are a couple tasks left:
> >
> >- HDFS-12273 for the UI. This should be completed in the next couple
> >days.
> >- HDFS-12284 for adding security. We can move this for v2 if not
> >completed.
> >
> > We have deployed this in production for 2.7 and we did a few tests with
> > trunk a few months ago.
> >
> > This week, I’m rebasing to trunk (last one was a couple weeks ago) and
> test
> > trunk in one of our test clusters.
> >
> >
> > Finally, note that all the functionality is in the Router (a new
> component)
> > so everything is isolated.
> >
> > In addition, no new APIs have been added and we rely fully in
> > ClientProtocol.
> >
> >
> >
> > I’d like to thank the people at Microsoft (specially, Jason, Ricardo,
> > Chris, Subru, Jakob, Carlo and Giovanni), Twitter (Ming and Gera),
> LinkedIn
> > (Zhe, Erik and Konstantin), and Cloudera (Andrew and Manoj) for
> > the discussion and the ideas.
> >
> > Special thanks to Chris Douglas for the thorough reviews!
> >
> >
> >
> > Regards,
> > Inigo
> >
> >
> >
> > [1]
> >
> > http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201708.mbox/%
> 3CCAB1dGgogTu6kHtkkYeUycmNv-H3RupfPF4Cd7rpuFi6vHGdBLg%40mail.gmail.com%3E
> >
> > [2] https://issues.apache.org/jira/browse/HDFS-12381
> >
> > [3] https://issues.apache.org/jira/browse/HDFS-12430
> >
> > [4] https://issues.apache.org/jira/browse/HDFS-12450
> >
> --
>
>
>
> --Brahma Reddy Battula
>

[jira] [Created] (HDFS-12614) FSPermissionChecker#getINodeAttrs() throws NPE when INodeAttributesProvider configured

[Manoj Govindassamy (JIRA)]

Manoj Govindassamy created HDFS-12614:
-

 Summary: FSPermissionChecker#getINodeAttrs() throws NPE when 
INodeAttributesProvider configured
 Key: HDFS-12614
 URL: https://issues.apache.org/jira/browse/HDFS-12614
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Manoj Govindassamy
Assignee: Manoj Govindassamy


When INodeAttributesProvider is configured, and when resolving path (like "/") 
and checking for permission, the following code when working on 
{{pathByNameArr}} throws NullPointerException. 

{noformat}
  private INodeAttributes getINodeAttrs(byte[][] pathByNameArr, int pathIdx,
  INode inode, int snapshotId) {
INodeAttributes inodeAttrs = inode.getSnapshotINode(snapshotId);
if (getAttributesProvider() != null) {
  String[] elements = new String[pathIdx + 1];
  for (int i = 0; i < elements.length; i++) {
elements[i] = DFSUtil.bytes2String(pathByNameArr[i]);  <===
  }
  inodeAttrs = getAttributesProvider().getAttributes(elements, inodeAttrs);
}
return inodeAttrs;
  }
{noformat}

Looks like for paths like "/" where the split components based on delimiter "/" 
can be null, the pathByNameArr array can have null elements and can throw NPE.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: 2017-10-06 Hadoop 3 release status update

[Eric Yang]

Hi Allen,

What if the commands are:

yarn application -deploy –f spec.json
yarn application -stop 
yarn application -restart 
yarn application -remove 

and

yarn application –list will display both application list from RM as well as 
docker services?

I think the development team was concerned that command structure overload 
between batch applications and long running services.  In my view, there is no 
difference, they are all applications.  The only distinction is the launching 
and shutdown of services may be different from batch jobs.  I think user can 
get used to these command structures without creating additional command 
grouping.  Your input is valuable to us. 

Feedback from others can also help to improve the current work.  Thank you.

Regards,
Eric

On 10/6/17, 4:27 PM, "Jian He"  wrote:

Hi Allen,

Thanks for spending the time reviewing it.
A new patch was uploaded yesterday on YARN-7198 to address the 
documentation of missing config, you might want to check.
The api-server is basically a REST server which accepts user requests to 
deploy services, it now has an option to be run as part of RM, which eliminates 
one separate daemon.

We are open to naming suggestions. So far we used ‘service’ keyword to 
indicate this feature. E.g. 
"yarn service” sub-command is used to manage services deployed on YARN such 
as:

yarn service create -f service-spec.json
yarn service stop 

Jian

> On Oct 6, 2017, at 3:12 PM, Allen Wittenauer  
wrote:
> 
> 
>> On Oct 6, 2017, at 1:31 PM, Andrew Wang  wrote:
>> 
>>  - Still waiting on Allen to review YARN native services feature.
> 
>   Fake news.  
> 
>   I’m still -1 on it, at least prior to a patch that posted late 
yesterday. I’ll probably have a chance to play with it early next week.
> 
> 
> Key problems:
> 
>   * still haven’t been able to bring up dns daemon due to lacking 
documentation
> 
>   * it really needs better naming and command structures.  When put into 
the larger YARN context, it’s very problematic:
> 
> $ yarn —daemon start resourcemanager
> 
>   vs.
> 
> $ yarn —daemon start apiserver 
> 
>   if you awoke from a deep sleep from inside a cave, which one 
would you expect to “start YARN”? Made worse that the feature is called 
“YARN services” all over the place.
> 
> $ yarn service foo
> 
>   … what does this even mean?
> 
>   It would be great if other outsiders really looked hard at this branch 
to give the team feedback.   Once it gets released, it’s gonna be too late to 
change it….
> 
> As a sidenote:
> 
>   It’d be great if the folks working on YARN spent some time 
consolidating daemons.  With this branch, it now feels like we’re approaching 
the double digit area of daemons to turn on all the features.  It’s well past 
ridiculous, especially considering we still haven’t replaced the MRJHS’s 
feature set to the point we can turn it off.
> 
> 
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12613) Native EC coder should implement release() as idempotent function.

[Lei (Eddy) Xu (JIRA)]

Lei (Eddy) Xu created HDFS-12613:


 Summary: Native EC coder should implement release() as idempotent 
function.
 Key: HDFS-12613
 URL: https://issues.apache.org/jira/browse/HDFS-12613
 Project: Hadoop HDFS
  Issue Type: Improvement
  Components: erasure-coding
Affects Versions: 3.0.0-beta1
Reporter: Lei (Eddy) Xu


Recently, we found native EC coder crashes JVM because 
{{NativeRSDecoder#release()}} being called multiple times (HDFS-12612 and 
HDFS-12606). 

We should strength the implement the native code to make {{release()}} 
idempotent  as well.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

2.9.0 status update (10/6/2017)

[Subramaniam V K]

We have pushed out the release to 10th November 2017:
https://cwiki.apache.org/confluence/display/HADOOP/Roadmap

The *final *feature freeze date is now 20th October 2017:

   - Merge vote is ongoing for API based (Capacity) Scheduler configuration
   (YARN-5734) and HDFS Router based federation (HDFS-10467
   ). Both of them seem
   on track to pass.
   - Backport of ATS v2 (YARN-2928) and YARN Web UI v2 (YARN-3368) are in
   progress and should be done given the extension.


There are 21 outstanding blockers to be addressed by 27th October 2017:
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.9+Release

Andrew is also tracking most of these as they are common for 3.0.0 also.
We'll follow up additionally on those specifically targeted for 2.9.0.

-Subru/Arun

Re: 2017-10-06 Hadoop 3 release status update

[Andrew Wang]

Thanks for the update Allen, appreciate your continued help reviewing this
feature.

Looking at the calendar, we have three weeks from when we want to have GA
RC0 out for vote. We're already dipping into code freeze time landing HDFS
router-based federation and API-based scheduler configuration next week. If
we want to get any more features in, it means slipping the GA date.

So, my current thinking is that we should draw a line after these pending
branches merge. Like before, I'm willing to bend on this if there are
strong arguments, but the quality bar is even higher than it was for beta1,
and we've still got plenty of other blockers/criticals to work on for GA.

If you feel differently, please reach out, I can make myself very available
next week for a call.

Best,
Andrew

On Fri, Oct 6, 2017 at 3:12 PM, Allen Wittenauer 
wrote:

>
> > On Oct 6, 2017, at 1:31 PM, Andrew Wang 
> wrote:
> >
> >   - Still waiting on Allen to review YARN native services feature.
>
> Fake news.
>
> I’m still -1 on it, at least prior to a patch that posted late
> yesterday. I’ll probably have a chance to play with it early next week.
>
>
> Key problems:
>
> * still haven’t been able to bring up dns daemon due to lacking
> documentation
>
> * it really needs better naming and command structures.  When put
> into the larger YARN context, it’s very problematic:
>
> $ yarn —daemon start resourcemanager
>
> vs.
>
> $ yarn —daemon start apiserver
>
> if you awoke from a deep sleep from inside a cave, which
> one would you expect to “start YARN”? Made worse that the feature is
> called “YARN services” all over the place.
>
> $ yarn service foo
>
> … what does this even mean?
>
> It would be great if other outsiders really looked hard at this
> branch to give the team feedback.   Once it gets released, it’s gonna be
> too late to change it….
>
> As a sidenote:
>
> It’d be great if the folks working on YARN spent some time
> consolidating daemons.  With this branch, it now feels like we’re
> approaching the double digit area of daemons to turn on all the features.
> It’s well past ridiculous, especially considering we still haven’t replaced
> the MRJHS’s feature set to the point we can turn it off.
>
>

Re: 2017-10-06 Hadoop 3 release status update

[Allen Wittenauer]

> On Oct 6, 2017, at 1:31 PM, Andrew Wang  wrote:
> 
>   - Still waiting on Allen to review YARN native services feature.

Fake news.  

I’m still -1 on it, at least prior to a patch that posted late 
yesterday. I’ll probably have a chance to play with it early next week.


Key problems:

* still haven’t been able to bring up dns daemon due to lacking 
documentation

* it really needs better naming and command structures.  When put into 
the larger YARN context, it’s very problematic:

$ yarn —daemon start resourcemanager

vs.

$ yarn —daemon start apiserver 

if you awoke from a deep sleep from inside a cave, which one 
would you expect to “start YARN”? Made worse that the feature is called 
“YARN services” all over the place.

$ yarn service foo

… what does this even mean?

It would be great if other outsiders really looked hard at this branch 
to give the team feedback.   Once it gets released, it’s gonna be too late to 
change it….

As a sidenote:

It’d be great if the folks working on YARN spent some time 
consolidating daemons.  With this branch, it now feels like we’re approaching 
the double digit area of daemons to turn on all the features.  It’s well past 
ridiculous, especially considering we still haven’t replaced the MRJHS’s 
feature set to the point we can turn it off.


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12612) DFSStripedOutputStream#close will throw if called a second time with a failed streamer

[Andrew Wang (JIRA)]

Andrew Wang created HDFS-12612:
--

 Summary: DFSStripedOutputStream#close will throw if called a 
second time with a failed streamer
 Key: HDFS-12612
 URL: https://issues.apache.org/jira/browse/HDFS-12612
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: erasure-coding
Affects Versions: 3.0.0-beta1
Reporter: Andrew Wang


Found while testing with Hive. We have a cluster with 2 DNs and the XOR-2-1 
policy. If you write a file and call close() twice, it throws this exception:

{noformat}
17/10/04 16:02:14 WARN hdfs.DFSOutputStream: Cannot allocate parity 
block(index=2, policy=XOR-2-1-1024k). Not enough datanodes? Exclude nodes=[]
...
Caused by: java.io.IOException: Failed to get parity block, index=2
at 
org.apache.hadoop.hdfs.DFSStripedOutputStream.allocateNewBlock(DFSStripedOutputStream.java:500)
 ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
at 
org.apache.hadoop.hdfs.DFSStripedOutputStream.writeChunk(DFSStripedOutputStream.java:524)
 ~[hadoop-hdfs-client-3.0.0-alpha3-cdh6.x-SNAPSHOT.jar:?]
{noformat}

This is because in DFSStripedOutputStream#closeImpl, if the stream is closed, 
we throw an exception if any of the striped streamers had an exception:

{code}
  protected synchronized void closeImpl() throws IOException {
if (isClosed()) {
  final MultipleIOException.Builder b = new MultipleIOException.Builder();
  for(int i = 0; i < streamers.size(); i++) {
final StripedDataStreamer si = getStripedDataStreamer(i);
try {
  si.getLastException().check(true);
} catch (IOException e) {
  b.add(e);
}
  }
  final IOException ioe = b.build();
  if (ioe != null) {
throw ioe;
  }
  return;
}
{code}

I think this is incorrect, since we only need to throw in this situation if we 
have too many failed streamers. close should also be idempotent, so it should 
throw the first time we call close if it's going to throw at all.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12611) Ozone: SCM Cli: Use the hostname of SCM server instead of the bind address

[Elek, Marton (JIRA)]

Elek, Marton created HDFS-12611:
---

 Summary: Ozone: SCM Cli: Use the hostname of SCM server instead of 
the bind address
 Key: HDFS-12611
 URL: https://issues.apache.org/jira/browse/HDFS-12611
 Project: Hadoop HDFS
  Issue Type: Sub-task
Affects Versions: HDFS-7240
Reporter: Elek, Marton
Assignee: Elek, Marton


During tests I found that I can't use SCM CLI in multi-node environment:

{code}
2017-10-06 21:00:25 INFO  Client:939 - Retrying connect to server: 
0.0.0.0/0.0.0.0:9860. Already tried 0 time(s); retry policy is 
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
2017-10-06 21:00:26 INFO  Client:939 - Retrying connect to server: 
0.0.0.0/0.0.0.0:9860. Already tried 1 time(s); retry policy is 
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
{code}

The reason is that we should use the OZONE_SCM_CLIENT_ADDRESS_KEY instead of 
OZONE_SCM_BIND_HOST_KEY as the bind host could be 0.0.0.0 and the 
SCM_CLIENT_ADDRESS should contain the external address (in my case scm was on a 
different host)



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-12610) Ozone: OzoneClient: RpcClient list calls throw NPE when iterating over empty list

[Nandakumar (JIRA)]

Nandakumar created HDFS-12610:
-

 Summary: Ozone: OzoneClient: RpcClient list calls throw NPE when 
iterating over empty list
 Key: HDFS-12610
 URL: https://issues.apache.org/jira/browse/HDFS-12610
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar


While doing a list on empty volume/bucket and iterating over the result 
iterator throws NullPointerException.

Reported by [~elek]



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

2017-10-06 Hadoop 3 release status update

[Andrew Wang]

https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3+release+status+updates

2017-10-06

The beta1 RC0 vote passed, and beta1 is out! Now tracking GA features.

Highlights:

   - 3.0.0-beta1 has been released!
   - Router-based federation merge vote should be about to pass
   - API-based scheduler configuration merge vote is out, has the votes so
   far

Red flags:

   - Still need to nail down whether we're going to try and merge resource
   profiles. I've been emailing with Wangda and Daniel about this, we need to
   reach a decision ASAP (might already be too late).
   - Still waiting on Allen to review YARN native services feature.

Previously tracked GA blockers that have been resolved or dropped:

   - YARN-7134
    -
AppSchedulingInfo
   has a dependency on capacity schedulerOPEN:  Wangda downgraded this to
   "Major", dropping from list.

GA blockers:

   - YARN-6623
    - Add
   support to turn off launching privileged containers in the
   container-executor PATCH AVAILABLE: Actively being reviewed
   - Change of ExecutionType
  - YARN-7275
   - NM
  Statestore cleanup for Container updatesPATCH AVAILABLE: Kartheek has
  posted a patch, waiting for review
  - YARN-7178
   - Add
  documentation for Container Update API OPEN : No update from Arun,
  though it's just a docs patch
   - ReservationSystem
  - YARN-4859
   - [Bug]
  Unable to submit a job to a reservation when using FairScheduler OPEN:
  Yufei has picked this up
  - YARN-4827
  
- Document
  configuration of ReservationSystem for FairScheduler OPEN: Yufei has
  picked this up, just a docs patch
   - Rolling upgrade
  - YARN-6142
   - Support
  rolling upgrade between 2.x and 3.x OPEN : Ray is still going through
  JACC and proto output
  - HDFS-11096
  
- Support
  rolling upgrade between 2.x and 3.xPATCH AVAILABLE: Sean has revved
  the patch and is waiting on reviews from Ray, Allen

Features merged for GA:

   - Erasure coding
  - Continued bug reporting and fixing based on testing at Cloudera.
  - Still need to finish the 3.0 must-do's
   - Classpath isolation (HADOOP-11656)
   - HADOOP-14771 is still floating, along with adding documentation.
   - Compat guide (HADOOP-13714
   )
  - Synced with Daniel, he plans to wrap up the remaining  stuff next
  week
   - TSv2 alpha 2
   - This was merged, no problems thus far [image: (smile)]

Unmerged features:

   - Resource types / profiles (YARN-3926
    and YARN-7069
   ) (Wangda Tan)
  - This has been merged for 3.1.0, YARN-7069 tracks follow on work
  - Wangda said that he's okay waiting for 3.1.0 for this, we're
  waiting on Daniel. I synced with Daniel earlier this week, and
he wants to
  try and get some of it into 3.0.0. Waiting on an update.
  - I still need a JIRA query for tracking the state of this.
   - HDFS router-based federation (HDFS-10467
   ) (Inigo Goiri and
   Chris Douglas)
   - Merge vote should close any minute now
   - API-based scheduler configuration (Jonathan Hung)
  - Merge vote is out, will close next week
   - YARN native services (YARN-5079
   ) (Jian He)
  - Subtasks were filed to address Allen's review comments from the
  previous merge vote, only one pending
  - We need to confirm with Allen that this is ready to go, he hasn't
  been reviewing

Re: [VOTE] Merge feature branch YARN-5734 (API based scheduler configuration) to trunk, branch-3.0, branch-2

[Konstantin Shvachko]

+1 (binding)

I was following this effort closely from the side.
The feature proved to be very useful based on our internal use of it.
I think it will be beneficial for the community as well.
Thanks for working on this Jonathan, Wangda, and everybody involved.

--Konstantin

On Mon, Oct 2, 2017 at 11:09 AM, Jonathan Hung  wrote:

> Hi all,
>
> From discussion at [1], I'd like to start a vote to merge feature branch
> YARN-5734 to trunk, branch-3.0, and branch-2. Vote will be 7 days, ending
> Monday Oct 9 at 11:00AM PDT.
>
> This branch adds a framework to the scheduler to allow scheduler
> configuration mutation on the fly, including a REST and CLI interface, and
> an interface for the scheduler configuration backing store. Currently the
> capacity scheduler implements this framework.
>
> Umbrella is here (YARN-5734
> ), jenkins build is here
> (
> YARN-7241 ). All required
> tasks for this feature are committed. Since this feature changes RM only,
> we have tested this on a local RM setup with a suite of configuration
> changes with no issue so far.
>
> Key points:
> - The feature is turned off by default, and must be explicitly configured
> to turn on. When turned off, the behavior reverts back to the original file
> based mechanism for changing scheduler configuration (i.e. yarn rmadmin
> -refreshQueues).
> - The framework was designed in a way to be extendable to other schedulers
> (most notably FairScheduler).
> - A pluggable ACL policy (YARN-5949
> ) allows admins
> fine-grained control for who can change what configurations.
> - The configuration storage backend is also pluggable. Currently an
> in-memory, leveldb, and zookeeper implementation are supported.
>
> There were 15 subtasks completed for this feature.
>
> Huge thanks to everyone who helped with reviews, commits, guidance, and
> technical discussion/design, including Carlo Curino, Xuan Gong, Subru
> Krishnan, Min Shen, Konstantin Shvachko, Carl Steinbach, Wangda Tan, Vinod
> Kumar Vavilapalli, Suja Viswesan, Zhe Zhang, Ye Zhou.
>
> [1]
> http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201709.mbox/%
> 3CCAHzWLgfEAgczjcEOUCg-03ma3ROtO=pkec9dpggyx9rzf3n...@mail.gmail.com%3E
>
> Jonathan Hung
>

[jira] [Created] (HDFS-12609) Ozone: SCM: Refactoring of chill mode logic in NodeManager

[Nandakumar (JIRA)]

Nandakumar created HDFS-12609:
-

 Summary: Ozone: SCM: Refactoring of chill mode logic in NodeManager
 Key: HDFS-12609
 URL: https://issues.apache.org/jira/browse/HDFS-12609
 Project: Hadoop HDFS
  Issue Type: Sub-task
  Components: ozone
Reporter: Nandakumar
Assignee: Nandakumar


As per the discussion in HDFS-12467, this jira is for refactoring chill mode 
logic in SCM's {{NodeManager}}.

The following methods will be supported to manipulate chill mode status of SCM
* {{forceExitChillMode}} - will bring SCM out of chill mode (both startup and 
manual)
* {{enterChillMode}} - will set manual chill mode flag to true
* {{exitChillMode}} - will set manual chill mode flag to false
* {{getChillModeStatus}} - will return chill mode status string
* {{isOutOfChillMode}} - will return true if SCM is out of chill mode, else 
false



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/

[Oct 5, 2017 2:05:23 PM] (stevel) HADOOP-14845. Azure wasb: getFileStatus not 
making any auth check.
[Oct 5, 2017 4:58:04 PM] (Arun Suresh) YARN-7258. Add Node and Rack Hints to 
Opportunistic Scheduler. (Kartheek
[Oct 5, 2017 5:38:41 PM] (ctrezzo) YARN-2960. Add documentation for the YARN 
shared cache.
[Oct 5, 2017 6:37:53 PM] (naganarasimha_gr) YARN-7285. ContainerExecutor always 
launches with priorities due to
[Oct 5, 2017 10:26:52 PM] (arp) HDFS-12420. Add an option to disallow 'namenode 
format -force'.
[Oct 5, 2017 10:50:13 PM] (haibochen) YARN-7009. 
TestNMClient.testNMClientNoCleanupOnStop is flaky by design. 
[Oct 5, 2017 11:58:43 PM] (wang) HDFS-12567. 
BlockPlacementPolicyRackFaultTolerant fails with racks with
[Oct 6, 2017 2:46:47 AM] (xiao) HADOOP-14521. KMS client needs retry logic. 
Contributed by Rushabh S




-1 overall


The following subsystems voted -1:
unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.net.TestDNS 
   hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting 
   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestIncreaseAllocationExpirer
 
   hadoop.yarn.server.resourcemanager.metrics.TestSystemMetricsPublisher 
   hadoop.yarn.server.resourcemanager.scheduler.fair.TestFSAppStarvation 
   hadoop.yarn.server.TestContainerManagerSecurity 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-compile-javac-root.txt
  [288K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-checkstyle-root.txt
  [17M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-patch-pylint.txt
  [20K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/whitespace-eol.txt
  [11M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/whitespace-tabs.txt
  [1.2M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/diff-javadoc-javadoc-root.txt
  [1.9M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [256K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [40K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/549/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [12K]

Powered by Apache Yetus 0.6.0-SNAPSHOT   http://yetus.apache.org

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org