Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Kuhu Shukla]

+1 (non-binding).- Downloaded tarball (source and binary)
- Verified signatures.
- Compiled, built source code and deployed on a single node cluster
- Ran sample MR jobs (Sleep, Wordcount) and  some "hadoop fs" commands.
Thanks a lot Vinod for your work on this release!
Regards,Kuhu Shukla 

On Friday, August 19, 2016 5:32 PM, Eric Payne 
 wrote:
 

 Thanks, Vinod, for working so hard on each 2.7 release.

+1 (non-binding)

Here's what I did:

- Built native 
- Installed on 3-node unsecure cluster
- Configured 2 queues with 2 separate label partitions 
- Verified that a job will successfully run on the correctly labelled node by 
specifying a non-default (but queue-accessible) label. 
- Verified that a distributed shell job would keep non-AM containers running 
across an App Master attempt restart. 
- Verified that preemption happens as expected (sort of). I say "sort of" 
because about twice as many containers were preempted as I thought should have 
been, but once the other underserved app began to run, it stopped preempting. 
Also, it didn't preempt between 2 queues with the same partition label. 
Partition preemption may not be supported in 2.7, so this is probably also okay.


Thanks!
Eric Payne





From: Vinod Kumar Vavilapalli 
To: "common-dev@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
Sent: Wednesday, August 17, 2016 9:05 PM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2


Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
- few issues with RC0 forced a RC1 [1]
- few more issues with RC1 forced a RC2 [2]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org



   

Re: Merge IPv6

[Elliott Clark]

HADOOP-11890 Is the related jira.

On Fri, Aug 19, 2016 at 10:09 AM, Elliott Clark  wrote:
> Is it time to merge IPv6 branch?
>
> We have been running the patches in several production and test
> clusters. We have tested it on ipv4 only clusters, on clusters where
> DFS is ipv4 and HBase is using ipv6, and we've tested it on ipv6 only
> clusters. We've seen no performance degredation, and have had no
> issues with network name resolution.
>
> Thoughts?
>
> Thanks

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13523) Implement a json format audit logger for KMS

[Xiao Chen (JIRA)]

Xiao Chen created HADOOP-13523:
--

 Summary: Implement a json format audit logger for KMS
 Key: HADOOP-13523
 URL: https://issues.apache.org/jira/browse/HADOOP-13523
 Project: Hadoop Common
  Issue Type: Improvement
  Components: kms
Reporter: Xiao Chen
Assignee: Xiao Chen






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13522) Permissions on fs stat added

[Alex Garbarini (JIRA)]

Alex Garbarini created HADOOP-13522:
---

 Summary: Permissions on fs stat added
 Key: HADOOP-13522
 URL: https://issues.apache.org/jira/browse/HADOOP-13522
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Reporter: Alex Garbarini
Priority: Minor


This patch adds to fs/shell/Stat.java the missing options of %a and %A. 

FileStatus already contains the getPermission() method required for returning 
symbolic permissions. FsPermission contains the method to return the binary 
short, but nothing to present in standard Octal format. 

Most UNIX admins base their work on such standard octal permissions. Hence, 
this patch also introduces one tiny method to translate the toShort() return 
into octal.

Build has already passed unit tests and javadoc.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Eric Payne]

Thanks, Vinod, for working so hard on each 2.7 release.

+1 (non-binding)

Here's what I did:

- Built native 
- Installed on 3-node unsecure cluster
- Configured 2 queues with 2 separate label partitions 
- Verified that a job will successfully run on the correctly labelled node by 
specifying a non-default (but queue-accessible) label. 
- Verified that a distributed shell job would keep non-AM containers running 
across an App Master attempt restart. 
- Verified that preemption happens as expected (sort of). I say "sort of" 
because about twice as many containers were preempted as I thought should have 
been, but once the other underserved app began to run, it stopped preempting. 
Also, it didn't preempt between 2 queues with the same partition label. 
Partition preemption may not be supported in 2.7, so this is probably also okay.


Thanks!
Eric Payne





From: Vinod Kumar Vavilapalli 
To: "common-dev@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
Sent: Wednesday, August 17, 2016 9:05 PM
Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2


Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
- few issues with RC0 forced a RC1 [1]
- few more issues with RC1 forced a RC2 [2]
- a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13521) SampleQuantile does not perform well under load

[Mark Wagner (JIRA)]

Mark Wagner created HADOOP-13521:


 Summary: SampleQuantile does not perform well under load
 Key: HADOOP-13521
 URL: https://issues.apache.org/jira/browse/HADOOP-13521
 Project: Hadoop Common
  Issue Type: Bug
  Components: metrics
Affects Versions: 2.3.0
Reporter: Mark Wagner


After adding quantile collection to one of our clusters we saw much higher 
latency for RPCs. This was traced down to the quantile collection. Samples are 
being buffered and inserted in groups of 500. After the buffered samples are 
inserted, the entire set of samples for this time period (600 seconds at the 
longest for us) is "compressed".

All operations for RPC metrics are synchronized. Usually this isn't an issue 
but it seems that this compression operation is taking a significant amount of 
time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13520) Document HADOOP-10158 for multi home/SPN support of HTTP principles

[Xiaoyu Yao (JIRA)]

Xiaoyu Yao created HADOOP-13520:
---

 Summary: Document HADOOP-10158 for multi home/SPN support of HTTP 
principles
 Key: HADOOP-13520
 URL: https://issues.apache.org/jira/browse/HADOOP-13520
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao


With HADOOP-10158, it is possible to load multiple principal names or all HTTP 
principals in the key tab by specifying “*” for multi home/realm scenarios 
where multiple SPNs exist for HTTP. 

This ticket is opened to 

1) Document the newly supported option * for HTTP principles such as 
hadoop.http.authentication.kerberos.principal.  

2) Change the hadoop.http.authentication.kerberos.principal in core-default.xml 
from 
{code}

  hadoop.http.authentication.kerberos.principal
  HTTP/_HOST@LOCALHOST
  
Indicates the Kerberos principal to be used for HTTP endpoint.
The principal MUST start with 'HTTP/' as per Kerberos HTTP SPNEGO 
specification.
  

{code}

to save one more configuration key needed.
{code}

  hadoop.http.authentication.kerberos.principal
  *
  
Indicates using all the Kerberos principals from 
hadoop.http.authentication.kerberos.keytab for Kerberos HTTP SPNEGO 
authentication, which is needed for multiple interface/SPN scenarios.
  

{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-13519) Make Path serializable

[Steve Loughran (JIRA)]

Steve Loughran created HADOOP-13519:
---

 Summary: Make Path serializable
 Key: HADOOP-13519
 URL: https://issues.apache.org/jira/browse/HADOOP-13519
 Project: Hadoop Common
  Issue Type: Improvement
  Components: io
Affects Versions: 2.7.2
Reporter: Steve Loughran
Assignee: Steve Loughran
Priority: Minor


If you could make Hadoop Paths serializable, you can use them in Spark 
operations without having to convert them to and from URIs.

It's trivial for paths to support this; as well as the OS code we need to add a 
check that there's no null URI coming in over the wire, and test to validate 
round tripping



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Merge IPv6

[Elliott Clark]

Is it time to merge IPv6 branch?

We have been running the patches in several production and test
clusters. We have tested it on ipv4 only clusters, on clusters where
DFS is ipv4 and HBase is using ipv6, and we've tested it on ipv6 only
clusters. We've seen no performance degredation, and have had no
issues with network name resolution.

Thoughts?

Thanks

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Updated 2.8.0-SNAPSHOT artifact

[Vinod Kumar Vavilapalli]

Jon,

That is around the time when I branched 2.8, so I guess you were getting 
SNAPSHOT artifacts till then from the branch-2 nightly builds.

If you need it, we can set up SNAPSHOT builds. Or just wait for the first RC, 
which is around the corner.

+Vinod

> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles  wrote:
> 
> Latest snapshot is uploaded in Nov 2015, but checkins are still coming in
> quite frequently.
> https://repository.apache.org/content/repositories/snapshots/org/apache/hadoop/hadoop-yarn-api/
> 
> Are there any plans to start producing updated SNAPSHOT artifacts for
> current hadoop development lines?


-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Akira Ajisaka]

Thanks Vinod for cutting another RC.

+1

- Downloaded source tarball and binary tarball
- Verified signatures and checksums
- Compiled and built a single node cluster
- Compiled Hive 2.1.0/1.2.1 and Tez 0.8.4/0.7.1 using Hadoop 2.7.3 pom 
successfully

- Ran some Hive on Tez queries successfully

Thanks,
Akira

On 8/18/16 11:05, Vinod Kumar Vavilapalli wrote:

Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted this at 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 





-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org