[jira] [Created] (YARN-7398) LICENSE.txt is broken in branch-2 by YARN-4849

2017-10-25 Thread Subru Krishnan (JIRA) 
Subru Krishnan created YARN-7398:


 Summary: LICENSE.txt is broken in branch-2 by YARN-4849
 Key: YARN-7398
 URL: https://issues.apache.org/jira/browse/YARN-7398
 Project: Hadoop YARN
  Issue Type: Bug
Affects Versions: 2.9.0
Reporter: Subru Krishnan
Assignee: Wangda Tan
Priority: Blocker


YARN-4849 (commit sha id 56654d8820f345fdefd6a3f81836125aa67adbae) seems to 
have been based out of stale version of LICENSE.txt, for e.g: HSQLDB, gtest 
etc, so I have reverted it. 

[~leftnoteasy]/[~sunilg], can you guys take a look and fix the UI v2 licenses 
asap.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop 2.8.2 (RC1)

2017-10-25 Thread Junping Du 
??Thanks again for all who verified and voted!

I give my binding +1 to conclude the vote for 2.8.2 RC1, based on:
- Build from source and verify signatures
- Deploy pseudo-distributed cluster and run some simple job, like: pi, sleep, 
etc.
- Verify UI of daemons, like: NameNode, ResourceManager, NodeManager, etc.

Now, we have:

7 binding +1s, from:
 John Zhuge, Jason Lowe, Chris Douglas, Wangda Tan, Ravi Prakash, Eric 
Payne, Junping Du

10 non-binding +1s, from:
Hanisha Koneru, Wei Yan, Brahma Reddy Battula, Shane Kumpf, Ajay Kumar, 
Bharat Viswanadham, Mukul Kumar Singh, Eric Badger, Bibinchundatt, Rakesh 
Radhakrishnan

and no -1s.

So I am glad to announce that the vote of 2.8.2 RC1 passes.

Thanks everyone listed above who tried the release candidate and vote. Also, 
kudos to all who ever help with 2.8.2 release effort in all kinds of ways- 
especially the Yahoo! guys who deployed 2.8 in production environment and 
identify many issues with fixes. Also, Shane, Miklos and others to help with 
docker container effort during RC stage.

I'll push the release bits and send out an announcement for 2.8.2 soon.


Thanks,

Junping?


From: Eric Payne 
Sent: Tuesday, October 24, 2017 3:29 PM
To: Junping Du; common-...@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
mapreduce-...@hadoop.apache.org; yarn-dev@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

+1 (binding)

Thanks a lot, Junping!

I built and installed the source on a 6-node pseudo cluster. I simple sleep and 
streaming jobs that exercised intra-queue and inter-queue preemption, and used 
user weights.

-Eric


From: Junping Du 
To: "common-...@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org" ; 
"yarn-dev@hadoop.apache.org" 
Sent: Thursday, October 19, 2017 7:43 PM
Subject: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

Hi folks,
I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.

Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line and will 
be the latest stable/production release for Apache Hadoop - it includes 315 new 
fixed issues since 2.8.1 and 69 fixes are marked as blocker/critical issues.

  More information about the 2.8.2 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

  New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.2-RC1

  The RC tag in git is: release-2.8.2-RC1, and the latest commit id is: 
66c47f2a01ad9637879e95f80c41f798373828fb

  The maven artifacts are available via 
repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1064

  Please try the release and vote; the vote will run for the usual 5 days, 
ending on 10/24/2017 6pm PST time.

Thanks,

Junping

[jira] [Created] (YARN-7397) Reduce lock contention in FairScheduler#getAppWeight()

2017-10-25 Thread Daniel Templeton (JIRA) 
Daniel Templeton created YARN-7397:
--

 Summary: Reduce lock contention in FairScheduler#getAppWeight()
 Key: YARN-7397
 URL: https://issues.apache.org/jira/browse/YARN-7397
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler
Affects Versions: 3.0.0-beta1
Reporter: Daniel Templeton
Assignee: Daniel Templeton


In profiling the fair scheduler, a large amount of time is spent waiting to get 
the lock in {{FairScheduler.getAppWeight()}}, when the lock isn't actually 
needed.  This patch reduces the scope of the lock to eliminate that contention.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7396) NPE when accessing container logs due to null dirsHandler

2017-10-25 Thread Jonathan Hung (JIRA) 
Jonathan Hung created YARN-7396:
---

 Summary: NPE when accessing container logs due to null dirsHandler
 Key: YARN-7396
 URL: https://issues.apache.org/jira/browse/YARN-7396
 Project: Hadoop YARN
  Issue Type: Bug
Reporter: Jonathan Hung


{noformat}java.lang.NullPointerException
at 
org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.containerLogPageRedirectPath(NMWebAppFilter.java:96)
at 
org.apache.hadoop.yarn.server.nodemanager.webapp.NMWebAppFilter.doFilter(NMWebAppFilter.java:62)
at 
com.sun.jersey.spi.container.servlet.ServletContainer.doFilter(ServletContainer.java:829){noformat}
In YARN-6620 the NMContext creation in {{NodeManager#serviceInit}} was moved. 
It's now created before the dirsHandler is initialized. So when 
{{nmContext.getLocalDirsHandler}} is called, it's null.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7395) NM fails to successfully kill tasks that run over their memory limit

2017-10-25 Thread Eric Badger (JIRA) 
Eric Badger created YARN-7395:
-

 Summary: NM fails to successfully kill tasks that run over their 
memory limit
 Key: YARN-7395
 URL: https://issues.apache.org/jira/browse/YARN-7395
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Eric Badger


The NM correctly notes that the container is over its configured limit, but 
then fails to successfully kill the process. So the Docker container AM stays 
around and the job keeps running



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7394) Merge code paths for Reservation/Plan queues and Auto Created queues

2017-10-25 Thread Suma Shivaprasad (JIRA) 
Suma Shivaprasad created YARN-7394:
--

 Summary: Merge code paths for Reservation/Plan queues and Auto 
Created queues
 Key: YARN-7394
 URL: https://issues.apache.org/jira/browse/YARN-7394
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Suma Shivaprasad
Assignee: Suma Shivaprasad


The initialization/reinitialization logic for ReservationQueue and AutoCreated 
Leaf queues are similar. The proposal is to rename ReservationQueue to a more 
generic name AutoCreatedLeafQueue which are either managed by PlanQueue(already 
exists) or AutoCreatedEnabledParentQueue (new class). 





--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7393) RegistryDNS doesn't work in tcp channel

2017-10-25 Thread Jian He (JIRA) 
Jian He created YARN-7393:
-

 Summary: RegistryDNS doesn't work in tcp channel
 Key: YARN-7393
 URL: https://issues.apache.org/jira/browse/YARN-7393
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Jian He






--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] Feature Branch Merge and Security Audits

2017-10-25 Thread larry mccay 
Terrific additions, Mike!
I will spin a new revision and incorporate your additions.

#8 is a great topic - given that Hadoop is insecure by default.
Actual movement to Secure by Default would be a challenge both technically
(given the need for kerberos) and discussion-wise.
Asking whether you have considered any settings of configurations that can
be secure by default is an interesting idea.

Can you provide an example though?


On Wed, Oct 25, 2017 at 2:14 PM, Michael Yoder  wrote:

> On Sat, Oct 21, 2017 at 8:47 AM, larry mccay  wrote:
>
>> New Revision...
>>
>
> These lists are wonderful. I appreciate the split between the Tech Preview
> and the GA Readiness lists, with the emphasis on the former being "don't
> enable by default" or at least "don't enable if security is on".  I don't
> have any comments on that part.
>
> Additions inline below. If some of the additions are items covered by
> existing frameworks that any code would use, please forgive my ignorance.
> Also, my points aren't as succinct as yours. Feel free to reword.
>
> *GA Readiness Security Audit*
>> At this point, we are merging full or partial security model
>> implementations.
>> Let's inventory what is covered by the model at this point and whether
>> there are future merges required to be full.
>>
>> *1. UIs*
>>
>> 1.1. What sort of validation is being done on any accepted user input?
>> (pointers to code would be appreciated)
>> 1.2. What explicit protections have been built in for (pointers to code
>> would be appreciated):
>>   1.2.1. cross site scripting
>>   1.2.2. cross site request forgery
>>   1.2.3. click jacking (X-Frame-Options)
>>
>
> 1.2.4 If using cookies, is the secure flag for cookies
>  turned on?
>
>
>> 1.3. What sort of authentication is required for access to the UIs?
>>   1.3.1. Kerberos
>> 1.3.1.1. has TGT renewal been accounted for
>> 1.3.1.2. SPNEGO support?
>> 1.3.1.3. Delegation token?
>>   1.3.2. Proxy User ACL?
>> 1.4. What authorization is available for determining who can access what
>> capabilities of the UIs for either viewing, modifying data and/or related
>> processes?
>> 1.5. Is there any input that will ultimately be persisted in
>> configuration for executing shell commands or processes?
>> 1.6. Do the UIs support the trusted proxy pattern with doas impersonation?
>> 1.7. Is there TLS/SSL support?
>>
>
> 1.7.1 Is it possible to configure TLS protocols and cipher suites?
> 1.7.2 Is it possible to configure support for HTTP Strict Transport
> Security
> 
> (HSTS)?
> 1.8 Are accesses to the UI audited? ("User X logged into Y from IP address
> Z", etc)
>
>
>> *2. REST APIs*
>>
>> 2.1. Do the REST APIs support the trusted proxy pattern with doas
>> impersonation capabilities?
>> 2.2. What explicit protections have been built in for:
>>   2.2.1. cross site scripting (XSS)
>>   2.2.2. cross site request forgery (CSRF)
>>   2.2.3. XML External Entity (XXE)
>> 2.3. What is being used for authentication - Hadoop Auth Module?
>> 2.4. Are there separate processes for the HTTP resources (UIs and REST
>> endpoints) or are they part of existing processes?
>> 2.5. Is there TLS/SSL support?
>> 2.6. Are there new CLI commands and/or clients for accessing the REST
>> APIs?
>> 2.7. What authorization enforcement points are there within the REST APIs?
>>
>
> The TLS and audit comments above apply here, too.
>
>
>> *3. Encryption*
>>
>> 3.1. Is there any support for encryption of persisted data?
>> 3.2. If so, is KMS and the hadoop key command used for key management?
>> 3.3. KMS interaction with Proxy Users?
>>
>
> 3.4 Cryptography is hard. There are more obscure pitfalls in crypto than
> any other in computer science. Standard cryptographic libraries should
> always be used. Does this work attempt to create an encryption scheme or
> protocol? Does it have a "novel" or "unique" use of normal crypto?  There
> be dragons. Even normal-looking use of cryptography must be carefully
> reviewed.
> 3.5 If you need random bits for a security purpose, such as for a session
> token or a cryptographic key, you need a cryptographically approved place
> to acquire said bits. Use the SecureRandom class.
>
> *4. Configuration*
>>
>> 4.1. Are there any passwords or secrets being added to configuration?
>> 4.2. If so, are they accessed via Configuration.getPassword() to allow
>> for provisioning to credential providers?
>> 4.3. Are there any settings that are used to launch docker containers or
>> shell out command execution, etc?
>>
>
> +1. So good.
>
>
>> *5. HA*
>>
>> 5.1. Are there provisions for HA?
>> 5.2. Are there any single point of failures?
>>
>> *6. CVEs*
>>
>> Dependencies need to have been checked for known issues before we merge.
>> We don't however want to list any CVEs that have been fixed but not
>> released yet.
>>
>> 6.1. All dependencies

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2017-10-25 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/569/

[Oct 25, 2017 5:17:56 AM] (haibochen) YARN-7389. Make TestResourceManager 
Scheduler agnostic. (Robert Kanter
[Oct 25, 2017 8:17:21 AM] (aajisaka) HADOOP-14030. PreCommit TestKDiag failure. 
Contributed by Wei-Chiu


[Error replacing 'FILE' - Workspace is not accessible]

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7392) Render cluster information on the new yarn web ui

2017-10-25 Thread Vasudevan Skm (JIRA) 
Vasudevan Skm created YARN-7392:
---

 Summary: Render cluster information on the new yarn web ui
 Key: YARN-7392
 URL: https://issues.apache.org/jira/browse/YARN-7392
 Project: Hadoop YARN
  Issue Type: Sub-task
  Components: webapp
Reporter: Vasudevan Skm
Assignee: Vasudevan Skm


Cluster information is missing in the new web UI. 



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-7391) Consider square root instead of natural log for size-based weight

2017-10-25 Thread Steven Rand (JIRA) 
Steven Rand created YARN-7391:
-

 Summary: Consider square root instead of natural log for 
size-based weight
 Key: YARN-7391
 URL: https://issues.apache.org/jira/browse/YARN-7391
 Project: Hadoop YARN
  Issue Type: Improvement
  Components: fairscheduler
Affects Versions: 3.0.0-beta1
Reporter: Steven Rand


Currently for size-based weight, we compute the weight of an app using this 
code from 
https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java#L377:

{code}
  if (sizeBasedWeight) {
// Set weight based on current memory demand
weight = Math.log1p(app.getDemand().getMemorySize()) / Math.log(2);
  }
{code}

Because the natural log function grows slowly, the weights of two apps with 
hugely different memory demands can be quite similar. For example, {{weight}} 
evaluates to 14.3 for an app with a demand of 20 GB, and evaluates to 19.9 for 
an app with a demand of 1000 GB. The app with the much larger demand will still 
have a higher weight, but not by a large amount relative to the sum of those 
weights.

I think it's worth considering a switch to a square root function, which will 
grow more quickly. In the above example, the app with a demand of 20 GB now has 
a weight of 143, while the app with a demand of 1000 GB now has a weight of 
1012. These weights seem more reasonable relative to each other given the 
difference in demand between the two apps.

The above example is admittedly a bit extreme, but I believe that a square root 
function would also produce reasonable results in general.

The code I have in mind would look something like:

{code}
  if (sizeBasedWeight) {
// Set weight based on current memory demand
weight = Math.sqrt(app.getDemand().getMemorySize());
  }
{code}

Would people be comfortable with this change?



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org