Re: Hadoop encryption module as Apache Chimera incubator project

Thanks guys for the opinions. Below are my responses for some questions or thoughts. On 2/3/16, 12:07 AM, "Chen, Haifeng" wrote: >Thanks Chris and Colin for your opinions. > >>> [Chris] If Chimera is not successful as an independent project or >>>stalls, Hadoop and/or

Build failed in Jenkins: Hadoop-Hdfs-trunk #2796

See Changes: [junping_du] Adding release 2.6.5 to CHANGES.txt [vinayakumarb] HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed -- [...truncated 7198 lines...] Tests

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

Hi Junping, Thanks a lot for leading the effort on 2.6.4. We were discussing in HDFS-9629, and have agreement on adding an additional step in the release process to update the year in the webui footer. See

Re: Hadoop encryption module as Apache Chimera incubator project

> On 3 Feb 2016, at 08:07, Chen, Haifeng wrote: > >>> [Colin] I do think there will be some challenges splitting this >>> functionality out into a separate jar, because of the way our CLASSPATH >>> works right now. > Yes, this challenges are common for shared libraries

Hadoop-Hdfs-trunk - Build # 2796 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2796/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 7391 lines...] [INFO] --- maven-clean-plugin:2.5:clean

Re: Hadoop encryption module as Apache Chimera incubator project

On Wed, Feb 3, 2016 at 12:48 AM, Gangumalla, Uma wrote: >>Standing in the point of shared fundamental piece of code like this, I do >>think Apache Commons might be the best direction which we can try as the >>first effort. In this direction, we still need to work with

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #862

See Changes: [junping_du] Adding release 2.6.5 to CHANGES.txt [vinayakumarb] HDFS-9715. Check storage ID uniqueness on datanode startup (Contributed -- [...truncated 6308 lines...] Java

Hadoop-Hdfs-trunk-Java8 - Build # 862 - Failure

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/862/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6501 lines...] [INFO] ---

[jira] [Created] (HDFS-9748) When addExpectedReplicasToPending is called twice, pendingReplications should avoid duplication

Walter Su created HDFS-9748: --- Summary: When addExpectedReplicasToPending is called twice, pendingReplications should avoid duplication Key: HDFS-9748 URL: https://issues.apache.org/jira/browse/HDFS-9748

RE: Hadoop encryption module as Apache Chimera incubator project

Thanks Chris and Colin for your opinions. >> [Chris] If Chimera is not successful as an independent project or stalls, >> Hadoop and/or Spark and/or $project will have to reabsorb it as maintainers. Understand the concern. One point to consider, Chimera dedicates in a specific domain of

[jira] [Created] (HDFS-9752) Permanent write failures may happen to slow writers during datanode rolling upgrades

Kihwal Lee created HDFS-9752: Summary: Permanent write failures may happen to slow writers during datanode rolling upgrades Key: HDFS-9752 URL: https://issues.apache.org/jira/browse/HDFS-9752 Project:

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #865

See Changes: [sjlee] HADOOP-12761. incremental maven build is not really incremental (sjlee) [jlowe] YARN-4594. container-executor fails to remove directory tree when chmod --

Hadoop-Hdfs-trunk-Java8 - Build # 865 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/865/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 5901 lines...] [INFO] ---

Re: Hadoop encryption module as Apache Chimera incubator project

I went through the repository, and now understand the reasoning that would locate this code in Apache Commons. This isn't proposing to extract much of the implementation and it takes none of the integration. It's limited to interfaces to crypto libraries and streams/configuration. It might be a

Re: Hadoop encryption module as Apache Chimera incubator project

Let's do one step at a time. There is a clear need for common encryption, and let's focus on making that happen. On Wed, Feb 3, 2016 at 4:48 PM, Zheng, Kai wrote: > I thought this discussion would switch to common-dev@ now? > > >> Would it make sense to also package some of

Re: [Release thread] 2.8.0 release activities

Thanks Vinod. +1 for 2.8 release start. Regards, Uma On 2/3/16, 3:53 PM, "Vinod Kumar Vavilapalli" wrote: >Seems like all the features listed in the Roadmap wiki are in. I¹m going >to try cutting an RC this weekend for a first/non-stable release off of >branch-2.8. > >Let

RE: Hadoop encryption module as Apache Chimera incubator project

I thought this discussion would switch to common-dev@ now? >> Would it make sense to also package some of the compression libraries, and >> maybe some of the text processing from MapReduce? Evolving some of this code >> to a common library with few/no dependencies would be generally useful. As

[jira] [Created] (HDFS-9753) libhdfs++: Client fails to pass TokenProto from LocatedBlockProto to server when reading a block

James Clampffer created HDFS-9753: - Summary: libhdfs++: Client fails to pass TokenProto from LocatedBlockProto to server when reading a block Key: HDFS-9753 URL: https://issues.apache.org/jira/browse/HDFS-9753

Build failed in Jenkins: Hadoop-Hdfs-trunk #2798

See Changes: [sjlee] HADOOP-12761. incremental maven build is not really incremental (sjlee) [jlowe] YARN-4594. container-executor fails to remove directory tree when chmod -- [...truncated

Hadoop-Hdfs-trunk - Build # 2798 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2798/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6193 lines...] [INFO] --- maven-clean-plugin:2.5:clean

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #866

See Changes: [jing9] HDFS-9748. Avoid duplication in pendingReplications when -- [...truncated 5816 lines...] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 15.483 sec -

Hadoop-Hdfs-trunk-Java8 - Build # 866 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/866/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 6009 lines...] [INFO] ---

Build failed in Jenkins: Hadoop-Hdfs-trunk #2799

See Changes: [jing9] HDFS-9748. Avoid duplication in pendingReplications when [shv] Move HDFS-9601 to branch-2.8.0 in CHANGES.txt. (shv) [zhezhang] HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping [shv]

Hadoop-Hdfs-trunk - Build # 2799 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2799/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 5295 lines...] [INFO] --- maven-antrun-plugin:1.7:run

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

I modified hbase pom.xml (0.98 branch) to point to staged maven artifacts. All unit tests passed. Cheers On Tue, Feb 2, 2016 at 11:01 PM, Junping Du wrote: > Hi community folks, >I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next > maintenance

RE: Hadoop encryption module as Apache Chimera incubator project

>> Let's do one step at a time. There is a clear need for common encryption, >> and let's focus on making that happen. Strongly agree. -Original Message- From: Reynold Xin [mailto:r...@databricks.com] Sent: Thursday, February 4, 2016 8:50 AM To: hdfs-dev@hadoop.apache.org Subject: Re:

Re: [Release thread] 2.8.0 release activities

Thanks Vinod. Not labeling 2.8.0 stable sounds perfectly reasonable to me. Let us not call it alpha or beta though, it is quite confusing. :) On Wed, Feb 3, 2016 at 8:17 PM, Gangumalla, Uma wrote: > Thanks Vinod. +1 for 2.8 release start. > > Regards, > Uma > > On

RE: Hadoop encryption module as Apache Chimera incubator project

The encryption or security thing is surely a good starting as the current focus. Considering or having other things like compression would help to determine how to vision, position and layout the new project, in Hadoop side, apache common project, or a new TLP, containing the candidate modules.

[jira] [Created] (HDFS-9754) Avoid unnecessary getBlockCollection calls in BlockManager

Jing Zhao created HDFS-9754: --- Summary: Avoid unnecessary getBlockCollection calls in BlockManager Key: HDFS-9754 URL: https://issues.apache.org/jira/browse/HDFS-9754 Project: Hadoop HDFS Issue

RE: Hadoop encryption module as Apache Chimera incubator project

Thanks Chris. >> I went through the repository, and now understand the reasoning that would >> locate this code in Apache Commons. This isn't proposing to extract much of >> the implementation and it takes none of the integration. It's limited to >> interfaces to crypto libraries and

[jira] [Created] (HDFS-9749) libhdfs++: RPC engine will attempt to close an asio socket before it's been opened

James Clampffer created HDFS-9749: - Summary: libhdfs++: RPC engine will attempt to close an asio socket before it's been opened Key: HDFS-9749 URL: https://issues.apache.org/jira/browse/HDFS-9749

Re: Hadoop encryption module as Apache Chimera incubator project

Why is this discussion taking place on hdfs-dev when other parts of Hadoop use encryption code? That alone makes me uncomfortable moving this code outside of the Hadoop umbrella. Also, since Yetus was invoked, it’s worthwhile pointing out that we had _many_ projects,

Hadoop-Hdfs-trunk - Build # 2797 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2797/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 5301 lines...] [INFO] --- maven-clean-plugin:2.5:clean

Build failed in Jenkins: Hadoop-Hdfs-trunk #2797

See Changes: [vinayakumarb] HDFS-9739. DatanodeStorage.isValidStorageId() is broken (Contributed by [vinayakumarb] HDFS-9740. Use a reasonable limit in DFSTestUtil.waitForMetric() [vvasudev] YARN-4625. Make

Re: [Release thread] 2.8.0 release activities

Seems like all the features listed in the Roadmap wiki are in. I’m going to try cutting an RC this weekend for a first/non-stable release off of branch-2.8. Let me know if anyone has any objections/concerns. Thanks +Vinod > On Nov 25, 2015, at 5:59 PM, Vinod Kumar Vavilapalli

Hadoop-Hdfs-trunk-Java8 - Build # 867 - Still Failing

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/867/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 5911 lines...] [INFO] ---

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #867

See Changes: [shv] Move HDFS-9601 to branch-2.8.0 in CHANGES.txt. (shv) [zhezhang] HDFS-9658. Erasure Coding: allow to use multiple EC policies in striping [shv] HDFS-9503. Use generic option -fs for NNThroughputBenchmark

Hadoop-Hdfs-trunk-Java8 - Build # 864 - Failure

See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/864/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 5999 lines...] [INFO] ---

Build failed in Jenkins: Hadoop-Hdfs-trunk-Java8 #864

See Changes: [vvasudev] YARN-4625. Make ApplicationSubmissionContext and -- [...truncated 5806 lines...] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 3.523 sec - in

[jira] [Resolved] (HDFS-9750) Document -source option for balancer

[ https://issues.apache.org/jira/browse/HDFS-9750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao resolved HDFS-9750. -- Resolution: Duplicate > Document -source option for balancer > > >

[jira] [Created] (HDFS-9750) Document -source option for balancer

Xiaoyu Yao created HDFS-9750: Summary: Document -source option for balancer Key: HDFS-9750 URL: https://issues.apache.org/jira/browse/HDFS-9750 Project: Hadoop HDFS Issue Type: Sub-task

[jira] [Created] (HDFS-9751) Add a step in the release process to update the release year in Web UI footer

Yongjun Zhang created HDFS-9751: --- Summary: Add a step in the release process to update the release year in Web UI footer Key: HDFS-9751 URL: https://issues.apache.org/jira/browse/HDFS-9751 Project: