[jira] [Created] (HADOOP-15720) rpcTimeout may not have been applied correctly

2018-09-04 Thread Yongjun Zhang (JIRA) 
Yongjun Zhang created HADOOP-15720:
--

 Summary: rpcTimeout may not have been applied correctly
 Key: HADOOP-15720
 URL: https://issues.apache.org/jira/browse/HADOOP-15720
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Reporter: Yongjun Zhang


org.apache.hadoop.ipc.Client send multiple RPC calls to server synchronously 
via the same connection as in the following synchronized code block:
{code:java}
  synchronized (sendRpcRequestLock) {
Future senderFuture = sendParamsExecutor.submit(new Runnable() {
  @Override
  public void run() {
try {
  synchronized (Connection.this.out) {
if (shouldCloseConnection.get()) {
  return;
}

if (LOG.isDebugEnabled()) {
  LOG.debug(getName() + " sending #" + call.id
  + " " + call.rpcRequest);
}
 
byte[] data = d.getData();
int totalLength = d.getLength();
out.writeInt(totalLength); // Total Length
out.write(data, 0, totalLength);// RpcRequestHeader + RpcRequest
out.flush();
  }
} catch (IOException e) {
  // exception at this point would leave the connection in an
  // unrecoverable state (eg half a call left on the wire).
  // So, close the connection, killing any outstanding calls
  markClosed(e);
} finally {
  //the buffer is just an in-memory buffer, but it is still polite 
to
  // close early
  IOUtils.closeStream(d);
}
  }
});
  
try {
  senderFuture.get();
} catch (ExecutionException e) {
  Throwable cause = e.getCause();
  
  // cause should only be a RuntimeException as the Runnable above
  // catches IOException
  if (cause instanceof RuntimeException) {
throw (RuntimeException) cause;
  } else {
throw new RuntimeException("unexpected checked exception", cause);
  }
}
  }
{code}
And it then waits for result asynchronously via
{code:java}
/* Receive a response.
 * Because only one receiver, so no synchronization on in.
 */
private void receiveRpcResponse() {
  if (shouldCloseConnection.get()) {
return;
  }
  touch();
  
  try {
int totalLen = in.readInt();
RpcResponseHeaderProto header = 
RpcResponseHeaderProto.parseDelimitedFrom(in);
checkResponse(header);

int headerLen = header.getSerializedSize();
headerLen += CodedOutputStream.computeRawVarint32Size(headerLen);

int callId = header.getCallId();
if (LOG.isDebugEnabled())
  LOG.debug(getName() + " got value #" + callId);

Call call = calls.get(callId);
RpcStatusProto status = header.getStatus();
..
{code}
However, we can see that the {{call}} returned by {{receiveRpcResonse()}} above 
may be in any order.

The following code
{code:java}
int totalLen = in.readInt();
{code}
eventually calls one of the following two methods, where rpcTimeOut is checked 
against:
{code:java}
  /** Read a byte from the stream.
   * Send a ping if timeout on read. Retries if no failure is detected
   * until a byte is read.
   * @throws IOException for any IO problem other than socket timeout
   */
  @Override
  public int read() throws IOException {
int waiting = 0;
do {
  try {
return super.read();
  } catch (SocketTimeoutException e) {
waiting += soTimeout;
handleTimeout(e, waiting);
  }
} while (true);
  }

  /** Read bytes into a buffer starting from offset off
   * Send a ping if timeout on read. Retries if no failure is detected
   * until a byte is read.
   * 
   * @return the total number of bytes read; -1 if the connection is closed.
   */
  @Override
  public int read(byte[] buf, int off, int len) throws IOException {
int waiting = 0;
do {
  try {
return super.read(buf, off, len);
  } catch (SocketTimeoutException e) {
waiting += soTimeout;
handleTimeout(e, waiting);
  }
} while (true);
  }
{code}
But the waiting time is always initialized to 0 for each of the above read 
calls, so each call can take up to rpcTimeout. And the real time to time out a 
call appears to be accumulative.

For example, if the client issue call1, call2, then it waits for result, if the 
first call1 took (rpcTimeout - 1), thus no time out, the second took 
(rpcTimeout -1), thus no timeout, but it

Re: HADOOP-14163 proposal for new hadoop.apache.org

2018-09-04 Thread Arpit Agarwal 
Requested a new git repo for the site: 
https://gitbox.apache.org/repos/asf/hadoop-site.git


On 9/4/18, 1:33 PM, "Mingliang Liu"  wrote:

It might be late but I'm +1 on the new site and transition proposal.

Thanks Marton.

On Fri, Aug 31, 2018 at 1:07 AM Elek, Marton  wrote:

> Bumping this thread at last time.
>
> I have the following proposal:
>
> 1. I will request a new git repository hadoop-site.git and import the
> new site to there (which has exactly the same content as the existing
> site).
>
> 2. I will ask infra to use the new repository as the source of
> hadoop.apache.org
>
> 3. I will sync manually all of the changes in the next two months back
> to the svn site from the git (release announcements, new committers)
>
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
>
> If no-one objects within three days, I'll assume lazy consensus and
> start with this plan. Please comment if you have objections.
>
> Again: it allows immediate fallback at any time as svn repo will be kept
> as is (+ I will keep it up-to-date in the next 2 months)
>
> Thanks,
> Marton
>
>
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
> >
> > Thank you very much to bump up this thread.
> >
> >
> > About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> >
> > About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and
> > powerful tools.
> >
> > Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> >
> > Hugo has much more powerful compared to "mvn site". Easier to create/use
> > more modern layout/theme, and easier to handle the content (for example
> > new release announcements could be generated as part of the release
> > process)
> >
> > I think it's very low risk to try out a new approach for the site (and
> > easy to rollback in case of problems)
> >
> > Marton
> >
> > ps: I just updated the patch/preview site with the recent releases:
> >
> > ***
> > * http://hadoop.anzix.net *
> > ***
> >
> > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> >> Got pinged about this offline.
> >>
> >> Thanks for keeping at it, Marton!
> >>
> >> I think there are two road-blocks here
> >>   (1) Is the mechanism using which the website is built good enough -
> >> mvn-site / hugo etc?
> >>   (2) Is the new website good enough?
> >>
> >> For (1), I just think we need more committer attention and get
> >> feedback rapidly and get it in.
> >>
> >> For (2), how about we do it in a different way in the interest of
> >> progress?
> >>   - We create a hadoop.apache.org/new-site/ where this new site goes.
> >>   - We then modify the existing web-site to say that there is a new
> >> site/experience that folks can click on a link and navigate to
> >>   - As this new website matures and gets feedback & fixes, we finally
> >> pull the plug at a later point of time when we think we are good to go.
> >>
> >> Thoughts?
> >>
> >> +Vinod
> >>
> >>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I would like to bump this thread up.
> >>>
> >>> TLDR; There is a proposed version of a new hadoop site which is
> >>> available from here: https://elek.github.io/hadoop-site-proposal/ and
> >>> https://issues.apache.org/jira/browse/HADOOP-14163
> >>>
> >>> Please let me know what you think about it.
> >>>
> >>>
> >>> Longer version:
> >>>
> >>> This thread started long time ago to use a more modern hadoop site:
> >>>
> >>> Goals were:
> >>>
> >>> 1. To make it easier to manage it (the release entries could be
> >>> created by a script as part of the release process)
> >>> 2. To use a better look-and-feel
> >>> 3. Move it out from svn to git
> >>>
> >>> I proposed to:
> >>>
> >>> 1. Move the existing site to git and generate it with hugo (which is
> >>> a single, standalone binary)
> >>> 2. Move both the rendered and source branches to git.
> >>> 3. (Create a jenkins job to generate the site automatically)
> >>>
> >>> NOTE: this is just about forrest based hadoop.apache.org, NOT about
> >>> the documentation which is generated by mvn-site (as before)
> >>>
> >>>
> >>> I got multiple valuable feedback and I improved the proposed site
> >>> according to the comments. Allen had some concerns about the used
> >>> technologies (hugo vs. mvn-site) and I answered all

[jira] [Created] (HADOOP-15719) Fail-fast when using OAuth over http

2018-09-04 Thread Sean Mackrory (JIRA) 
Sean Mackrory created HADOOP-15719:
--

 Summary: Fail-fast when using OAuth over http
 Key: HADOOP-15719
 URL: https://issues.apache.org/jira/browse/HADOOP-15719
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Sean Mackrory


If you configure OAuth and then use abfs:// instead of abfss:// it will fail, 
but it takes a very long time, and still isn't very clear why. Good place to 
put a good human-readable exception message and fail fast.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-15666) ABFS: Compatibility tests can fail

2018-09-04 Thread Sean Mackrory (JIRA) 


 [ 
https://issues.apache.org/jira/browse/HADOOP-15666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sean Mackrory resolved HADOOP-15666.

Resolution: Not A Problem

Nope - I noticed at the end of last week it wasn't showing up any more.

> ABFS: Compatibility tests can fail
> --
>
> Key: HADOOP-15666
> URL: https://issues.apache.org/jira/browse/HADOOP-15666
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Sean Mackrory
>Priority: Major
>
> Sounds like other folks aren't hitting this, but I'm seeing failures in 2 
> compatibility test classes:
> {code}
> [ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 5.417 
> s <<< FAILURE! - in 
> org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat
> [ERROR] 
> testBlobBackCompat(org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat)
>   Time elapsed: 4.631 s  <<< ERROR!
> java.io.FileNotFoundException: 
> GET 
> http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false
> StatusCode=404
> StatusDescription=The specified path does not exist.
> ErrorCode=PathNotFound
> ErrorMessage=The specified path does not exist.
> RequestId:cc2d3e74-d01f-007f-4911-30829500
> Time:2018-08-09T18:50:19.0627297Z
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException(AzureBlobFileSystem.java:559)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:285)
> at 
> org.apache.hadoop.fs.azurebfs.ITestAzureBlobFileSystemBackCompat.testBlobBackCompat(ITestAzureBlobFileSystemBackCompat.java:54)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at 
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> Caused by: GET 
> http://ACCOUNT_NAME.dfs.core.windows.net/abfs-testcontainer-24b93630-e947-4139-8bbe-fcb96e61b7e4?resource=filesystem=5000=test/10=90=false
> StatusCode=404
> StatusDescription=The specified path does not exist.
> ErrorCode=PathNotFound
> ErrorMessage=The specified path does not exist.
> RequestId:cc2d3e74-d01f-007f-4911-30829500
> Time:2018-08-09T18:50:19.0627297Z
> at 
> org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute(AbfsRestOperation.java:126)
> at 
> org.apache.hadoop.fs.azurebfs.services.AbfsClient.listPath(AbfsClient.java:151)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.listStatus(AzureBlobFileSystemStore.java:404)
> at 
> org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.listStatus(AzureBlobFileSystem.java:282)
> ... 13 more
> {code}
> {code}
> [ERROR] Tests run: 5, Failures: 1, Errors: 2, Skipped: 0, Time elapsed: 
> 16.478 s <<< FAILURE! - in 
> org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility
> [ERROR] 
> testListFileStatus(org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility)  
> Time elapsed: 6.334 s  <<< FAILURE!
> java.lang.AssertionError: expected:<2> but was:<1>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:743)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:555)
> at org.junit.Assert.assertEquals(Assert.java:542)
> at 
> org.apache.hadoop.fs.azurebfs.ITestWasbAbfsCompatibility.testListFileStatus(ITestWasbAbfsCompatibility.java:80)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
>

Re: HADOOP-14163 proposal for new hadoop.apache.org

2018-09-04 Thread Mingliang Liu 
It might be late but I'm +1 on the new site and transition proposal.

Thanks Marton.

On Fri, Aug 31, 2018 at 1:07 AM Elek, Marton  wrote:

> Bumping this thread at last time.
>
> I have the following proposal:
>
> 1. I will request a new git repository hadoop-site.git and import the
> new site to there (which has exactly the same content as the existing
> site).
>
> 2. I will ask infra to use the new repository as the source of
> hadoop.apache.org
>
> 3. I will sync manually all of the changes in the next two months back
> to the svn site from the git (release announcements, new committers)
>
> IN CASE OF ANY PROBLEM we can switch back to the svn without any problem.
>
> If no-one objects within three days, I'll assume lazy consensus and
> start with this plan. Please comment if you have objections.
>
> Again: it allows immediate fallback at any time as svn repo will be kept
> as is (+ I will keep it up-to-date in the next 2 months)
>
> Thanks,
> Marton
>
>
> On 06/21/2018 09:00 PM, Elek, Marton wrote:
> >
> > Thank you very much to bump up this thread.
> >
> >
> > About [2]: (Just for the clarification) the content of the proposed
> > website is exactly the same as the old one.
> >
> > About [1]. I believe that the "mvn site" is perfect for the
> > documentation but for website creation there are more simple and
> > powerful tools.
> >
> > Hugo has more simple compared to jekyll. Just one binary, without
> > dependencies, works everywhere (mac, linux, windows)
> >
> > Hugo has much more powerful compared to "mvn site". Easier to create/use
> > more modern layout/theme, and easier to handle the content (for example
> > new release announcements could be generated as part of the release
> > process)
> >
> > I think it's very low risk to try out a new approach for the site (and
> > easy to rollback in case of problems)
> >
> > Marton
> >
> > ps: I just updated the patch/preview site with the recent releases:
> >
> > ***
> > * http://hadoop.anzix.net *
> > ***
> >
> > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> >> Got pinged about this offline.
> >>
> >> Thanks for keeping at it, Marton!
> >>
> >> I think there are two road-blocks here
> >>   (1) Is the mechanism using which the website is built good enough -
> >> mvn-site / hugo etc?
> >>   (2) Is the new website good enough?
> >>
> >> For (1), I just think we need more committer attention and get
> >> feedback rapidly and get it in.
> >>
> >> For (2), how about we do it in a different way in the interest of
> >> progress?
> >>   - We create a hadoop.apache.org/new-site/ where this new site goes.
> >>   - We then modify the existing web-site to say that there is a new
> >> site/experience that folks can click on a link and navigate to
> >>   - As this new website matures and gets feedback & fixes, we finally
> >> pull the plug at a later point of time when we think we are good to go.
> >>
> >> Thoughts?
> >>
> >> +Vinod
> >>
> >>> On Feb 16, 2018, at 3:10 AM, Elek, Marton  wrote:
> >>>
> >>> Hi,
> >>>
> >>> I would like to bump this thread up.
> >>>
> >>> TLDR; There is a proposed version of a new hadoop site which is
> >>> available from here: https://elek.github.io/hadoop-site-proposal/ and
> >>> https://issues.apache.org/jira/browse/HADOOP-14163
> >>>
> >>> Please let me know what you think about it.
> >>>
> >>>
> >>> Longer version:
> >>>
> >>> This thread started long time ago to use a more modern hadoop site:
> >>>
> >>> Goals were:
> >>>
> >>> 1. To make it easier to manage it (the release entries could be
> >>> created by a script as part of the release process)
> >>> 2. To use a better look-and-feel
> >>> 3. Move it out from svn to git
> >>>
> >>> I proposed to:
> >>>
> >>> 1. Move the existing site to git and generate it with hugo (which is
> >>> a single, standalone binary)
> >>> 2. Move both the rendered and source branches to git.
> >>> 3. (Create a jenkins job to generate the site automatically)
> >>>
> >>> NOTE: this is just about forrest based hadoop.apache.org, NOT about
> >>> the documentation which is generated by mvn-site (as before)
> >>>
> >>>
> >>> I got multiple valuable feedback and I improved the proposed site
> >>> according to the comments. Allen had some concerns about the used
> >>> technologies (hugo vs. mvn-site) and I answered all the questions why
> >>> I think mvn-site is the best for documentation and hugo is best for
> >>> generating site.
> >>>
> >>>
> >>> I would like to finish this effort/jira: I would like to start a
> >>> discussion about using this proposed version and approach as a new
> >>> site of Apache Hadoop. Please let me know what you think.
> >>>
> >>>
> >>> Thanks a lot,
> >>> Marton
> >>>
> >>> -
> >>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >>>
> >>
> >>
> >>

RE: [Discuss] Merge discussion for Node attribute support feature YARN-3409

2018-09-04 Thread Bibinchundatt 
+1 for merge. Fetaure would be a good addition to 3.2 release.

--
Bibin A Chundatt
M: +91-9742095715
E: bibin.chund...@huawei.com
2012实验室-印研IT BU分部
2012 Laboratories-IT BU Branch Dept.
From:Naganarasimha Garla
To:common-dev@hadoop.apache.org,Hdfs-dev,yarn-...@hadoop.apache.org,mapreduce-...@hadoop.apache.org,
Date:2018-08-29 20:00:44
Subject:[Discuss] Merge discussion for Node attribute support feature YARN-3409

Hi All,

We would like to hear your thoughts on merging “Node Attributes Support in
YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
it in for HADOOP 3.2.

*Major work happened in this branch*

YARN-6858. Attribute Manager to store and provide node attributes in RM
YARN-7871. Support Node attributes reporting from NM to RM( distributed
node attributes)
YARN-7863. Modify placement constraints to support node attributes
YARN-7875. Node Attribute store for storing and recovering attributes

*Detailed Design:*

Please refer [1] for detailed design document.

*Testing Efforts:*

We did detailed tests for the feature in the last few weeks.
This feature will be enabled only when Node Attributes constraints are
specified through SchedulingRequest from AM.
Manager implementation will help to store and recover Node Attributes. This
works with existing placement constraints.

*Regarding to API stability:*

All newly added @Public APIs are @Unstable.

Documentation jira [3] could help to provide detailed configuration
details. This feature works from end-to-end and we tested this in our local
cluster. Branch code is run against trunk and tracked via [4].

We would love to get your thoughts before opening a voting thread.

Special thanks to a team of folks who worked hard and contributed towards
this efforts including design discussion / patch / reviews, etc.: Weiwei
Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.

[1] :
https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
[2] : https://issues.apache.org/jira/browse/YARN-3409
[3] : https://issues.apache.org/jira/browse/YARN-7865
[4] : https://issues.apache.org/jira/browse/YARN-8718

Thanks,
+ Naga & Sunil Govindan

[jira] [Created] (HADOOP-15718) Hadoop --daemon status is not working as excepted

2018-09-04 Thread Harshakiran Reddy (JIRA) 
Harshakiran Reddy created HADOOP-15718:
--

 Summary: Hadoop --daemon status is not working as excepted 
 Key: HADOOP-15718
 URL: https://issues.apache.org/jira/browse/HADOOP-15718
 Project: Hadoop Common
  Issue Type: Bug
  Components: common
Affects Versions: 3.1.1
Reporter: Harshakiran Reddy


{noformat}
install/hadoop/namenode/bin> jps
76385 KMSWebServer
66977 ThriftServer
66886 RESTServer
65608 DFSZKFailoverController
65610 NameNode
64946 QuorumPeerMain
76437 Jps
66647 NodeManager
65079 JournalNode
66809 HMaster
75163 DataNode
66203 ResourceManager
67230 HRegionServer
install/hadoop/namenode/bin> ./hadoop --daemon status namenode
WARNING: Use of this script to execute namenode is deprecated.
WARNING: Attempting to execute replacement "hdfs namenode" instead.

namenode is running as process 65610.  Stop it first.
install/hadoop/namenode/bin>
{noformat}

{{Expected Output:-}}

it should give the status of service like running or stopped and exit code 
should be 0



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15717) TGT renewal thread does not log IOException

2018-09-04 Thread Szilard Nemeth (JIRA) 
Szilard Nemeth created HADOOP-15717:
---

 Summary: TGT renewal thread does not log IOException
 Key: HADOOP-15717
 URL: https://issues.apache.org/jira/browse/HADOOP-15717
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Szilard Nemeth
Assignee: Szilard Nemeth


See here: 
https://github.com/apache/hadoop/blob/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L918




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [Discuss] Merge discussion for Node attribute support feature YARN-3409

2018-09-04 Thread Sunil G 
+1 for merge.

Quickly checked all basic test runs in the branch
- Add/replace/remove of attributes work good.
- scheduler can now handle attribute based placement constraints.
- tested DS shell with various constructs like java=1.8, python!=3, AND/OR
constraints etc.
- Documentation on attributes is also looks good

Thanks Naga, Weiwei, Bibin and all the folks who supported in designing and
reviewing the branch.

- Sunil



On Wed, Aug 29, 2018 at 8:00 PM Naganarasimha Garla <
naganarasimha...@apache.org> wrote:

> Hi All,
>
> We would like to hear your thoughts on merging “Node Attributes Support in
> YARN” branch (YARN-3409) [2] into trunk in a few weeks. The goal is to get
> it in for HADOOP 3.2.
>
> *Major work happened in this branch*
>
> YARN-6858. Attribute Manager to store and provide node attributes in RM
> YARN-7871. Support Node attributes reporting from NM to RM( distributed
> node attributes)
> YARN-7863. Modify placement constraints to support node attributes
> YARN-7875. Node Attribute store for storing and recovering attributes
>
> *Detailed Design:*
>
> Please refer [1] for detailed design document.
>
> *Testing Efforts:*
>
> We did detailed tests for the feature in the last few weeks.
> This feature will be enabled only when Node Attributes constraints are
> specified through SchedulingRequest from AM.
> Manager implementation will help to store and recover Node Attributes. This
> works with existing placement constraints.
>
> *Regarding to API stability:*
>
> All newly added @Public APIs are @Unstable.
>
> Documentation jira [3] could help to provide detailed configuration
> details. This feature works from end-to-end and we tested this in our local
> cluster. Branch code is run against trunk and tracked via [4].
>
> We would love to get your thoughts before opening a voting thread.
>
> Special thanks to a team of folks who worked hard and contributed towards
> this efforts including design discussion / patch / reviews, etc.: Weiwei
> Yang, Bibin Chundatt, Wangda Tan, Vinod Kumar Vavilappali, Konstantinos
> Karanasos, Arun Suresh, Varun Saxena, Devaraj Kavali, Lei Guo, Chong Chen.
>
> [1] :
>
> https://issues.apache.org/jira/secure/attachment/12937633/Node-Attributes-Requirements-Design-doc_v2.pdf
> [2] : https://issues.apache.org/jira/browse/YARN-3409
> [3] : https://issues.apache.org/jira/browse/YARN-7865
> [4] : https://issues.apache.org/jira/browse/YARN-8718
>
> Thanks,
> + Naga & Sunil Govindan
>