[jira] [Reopened] (HDFS-14084) Need for more stats in DFSClient

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-14084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened HDFS-14084:
---

I reverted this from trunk, branch-3.2, branch-3.1, branch-3.1.2, and 
branch-3.0.  Heads up to [~leftnoteasy] as this will impact the 3.1.2 release 
process and require a new release candidate to be built if one was already 
created.


> Need for more stats in DFSClient
> 
>
> Key: HDFS-14084
> URL: https://issues.apache.org/jira/browse/HDFS-14084
> Project: Hadoop HDFS
>  Issue Type: Improvement
>Affects Versions: 3.0.0
>Reporter: Pranay Singh
>Assignee: Pranay Singh
>Priority: Minor
> Fix For: 3.0.4, 3.1.2, 3.3.0, 3.2.1
>
> Attachments: HDFS-14084.001.patch, HDFS-14084.002.patch, 
> HDFS-14084.003.patch, HDFS-14084.004.patch, HDFS-14084.005.patch, 
> HDFS-14084.006.patch, HDFS-14084.007.patch, HDFS-14084.008.patch, 
> HDFS-14084.009.patch, HDFS-14084.010.patch, HDFS-14084.011.patch
>
>
> The usage of HDFS has changed from being used as a map-reduce filesystem, now 
> it's becoming more of like a general purpose filesystem. In most of the cases 
> there are issues with the Namenode so we have metrics to know the workload or 
> stress on Namenode.
> However, there is a need to have more statistics collected for different 
> operations/RPCs in DFSClient to know which RPC operations are taking longer 
> time or to know what is the frequency of the operation.These statistics can 
> be exposed to the users of DFS Client and they can periodically log or do 
> some sort of flow control if the response is slow. This will also help to 
> isolate HDFS issue in a mixed environment where on a node say we have Spark, 
> HBase and Impala running together. We can check the throughput of different 
> operation across client and isolate the problem caused because of noisy 
> neighbor or network congestion or shared JVM.
> We have dealt with several problems from the field for which there is no 
> conclusive evidence as to what caused the problem. If we had metrics or stats 
> in DFSClient we would be better equipped to solve such complex problems.
> List of jiras for reference:
> -
>  HADOOP-15538 HADOOP-15530 ( client side deadlock)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.2.0 - RC0

[Jason Lowe]

Thanks for driving this release, Sunil!

+1 (binding)

- Verified signatures and digests
- Successfully performed a native build
- Deployed a single-node cluster
- Ran some sample jobs

Jason

On Fri, Nov 23, 2018 at 6:07 AM Sunil G  wrote:

> Hi folks,
>
>
>
> Thanks to all contributors who helped in this release [1]. I have created
>
> first release candidate (RC0) for Apache Hadoop 3.2.0.
>
>
> Artifacts for this RC are available here:
>
> http://home.apache.org/~sunilg/hadoop-3.2.0-RC0/
>
>
>
> RC tag in git is release-3.2.0-RC0.
>
>
>
> The maven artifacts are available via repository.apache.org at
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1174/
>
>
> This vote will run 7 days (5 weekdays), ending on Nov 30 at 11:59 pm PST.
>
>
>
> 3.2.0 contains 1079 [2] fixed JIRA issues since 3.1.0. Below feature
> additions
>
> are the highlights of this release.
>
> 1. Node Attributes Support in YARN
>
> 2. Hadoop Submarine project for running Deep Learning workloads on YARN
>
> 3. Support service upgrade via YARN Service API and CLI
>
> 4. HDFS Storage Policy Satisfier
>
> 5. Support Windows Azure Storage - Blob file system in Hadoop
>
> 6. Phase 3 improvements for S3Guard and Phase 5 improvements S3a
>
> 7. Improvements in Router-based HDFS federation
>
>
>
> Thanks to Wangda, Vinod, Marton for helping me in preparing the release.
>
> I have done few testing with my pseudo cluster. My +1 to start.
>
>
>
> Regards,
>
> Sunil
>
>
>
> [1]
>
>
> https://lists.apache.org/thread.html/68c1745dcb65602aecce6f7e6b7f0af3d974b1bf0048e7823e58b06f@%3Cyarn-dev.hadoop.apache.org%3E
>
> [2] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.2.0)
> AND fixVersion not in (3.1.0, 3.0.0, 3.0.0-beta1) AND status = Resolved
> ORDER BY fixVersion ASC
>

Re: [VOTE] Release Apache Hadoop 2.9.2 (RC0)

[Jason Lowe]

Thanks for driving this release, Akira!

+1 (binding)

- Verified signatures and digests
- Successfully performed native build from source
- Deployed a single-node cluster and ran some sample jobs

Jason

On Tue, Nov 13, 2018 at 7:02 PM Akira Ajisaka  wrote:

> Hi folks,
>
> I have put together a release candidate (RC0) for Hadoop 2.9.2. It
> includes 204 bug fixes and improvements since 2.9.1. [1]
>
> The RC is available at http://home.apache.org/~aajisaka/hadoop-2.9.2-RC0/
> Git signed tag is release-2.9.2-RC0 and the checksum is
> 826afbeae31ca687bc2f8471dc841b66ed2c6704
> The maven artifacts are staged at
> https://repository.apache.org/content/repositories/orgapachehadoop-1166/
>
> You can find my public key at:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Please try the release and vote. The vote will run for 5 days.
>
> [1] https://s.apache.org/2.9.2-fixed-jiras
>
> Thanks,
> Akira
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HDFS-13975) TestBalancer#testMaxIterationTime fails sporadically

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-13975:
-

 Summary: TestBalancer#testMaxIterationTime fails sporadically
 Key: HDFS-13975
 URL: https://issues.apache.org/jira/browse/HDFS-13975
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.2.0
Reporter: Jason Lowe


A number of precommit builds have seen this test fail like this:
{noformat}
java.lang.AssertionError: Unexpected iteration runtime: 4021ms > 3.5s
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.hdfs.server.balancer.TestBalancer.testMaxIterationTime(TestBalancer.java:1649)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{noformat}




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Next Hadoop Contributors Meetup on September 25th

[Jason Lowe]

I am happy to announce that Oath will be hosting the next Hadoop
Contributors meetup on Tuesday, September 25th at Yahoo Building G, 589
Java Drive, Sunnyvale CA from 8:00AM to 6:00PM.

The agenda will look roughly as follows:

08:00AM - 08:30AM Arrival and Check-in
08:30AM - 12:00PM A series of brief talks with some of the topics including:
  - HDFS scalability and security
  - Use cases and future directions for Docker on YARN
  - Submarine (Deep Learning on YARN)
  - Hadoop in the cloud
  - Oath's use of machine learning, Vespa, and Storm
11:45PM - 12:30PM Lunch Break
12:30PM - 02:00PM Brief talks series resume
02:00PM - 04:30PM Parallel breakout sessions to discuss topics suggested by
attendees.  Some proposed topics include:
  - Improved security credentials management for long-running YARN
applications
  - Improved management of parallel development lines
  - Proposals for the next bug bash
  - Tez shuffle handler and DAG aware scheduler overview
 04:30PM - 06:00PM Closing Reception

RSVP at https://www.meetup.com/Hadoop-Contributors/events/254012512/ is
REQUIRED to attend and spots are limited.  Security will be checking the
attendee list as you enter the building.

We will host a Google Hangouts/Meet so people who are interested but unable
to attend in person can participate remotely.  Details will be posted to
the meetup event.

Hope to see you there!

Jason

Re: [VOTE] Release Apache Hadoop 2.8.5 (RC0)

[Jason Lowe]

Thanks for driving the release, Junping!

+1 (binding)

- Verified signatures and digests
- Successfully performed a native build from source
- Successfully deployed a single-node cluster with the timeline server
- Ran some sample jobs and examined the web UI and job logs

Jason

On Mon, Sep 10, 2018 at 7:00 AM, 俊平堵  wrote:

> Hi all,
>
>  I've created the first release candidate (RC0) for Apache
> Hadoop 2.8.5. This is our next point release to follow up 2.8.4. It
> includes 33 important fixes and improvements.
>
>
> The RC artifacts are available at:
> http://home.apache.org/~junping_du/hadoop-2.8.5-RC0
>
>
> The RC tag in git is: release-2.8.5-RC0
>
>
>
> The maven artifacts are available via repository.apache.org<
> http://repository.apache.org> at:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1140
>
>
> Please try the release and vote; the vote will run for the usual 5
> working
> days, ending on 9/15/2018 PST time.
>
>
> Thanks,
>
>
> Junping
>

Re: [VOTE] Release Apache Hadoop 2.7.6 (RC0)

[Jason Lowe]

Thanks for driving the release, Konstatin!

+1 (binding)

- Verified signatures and digests
- Completed a native build from source
- Deployed a single-node cluster
- Ran some sample jobs

Jason

On Mon, Apr 9, 2018 at 6:14 PM, Konstantin Shvachko
 wrote:
> Hi everybody,
>
> This is the next dot release of Apache Hadoop 2.7 line. The previous one 2.7.5
> was released on December 14, 2017.
> Release 2.7.6 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.6-RC0/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.6-RC0/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 04/16/2018.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Jason Lowe]

Is it necessary to cut the branch so far ahead of the release?  branch-3.0
is already a maintenance line for 3.0.x releases.  Is there a known
feature/improvement planned to go into branch-3.0 that is not desirable for
the 3.0.1 release?

I have found in the past that branching so early leads to many useful fixes
being unnecessarily postponed to future releases because committers forget
to pick to the new, relatively long-lived patch branch.  This becomes
especially true if blockers end up dragging out the ultimate release date,
which has historically been quite common.  My preference would be to cut
this branch as close to the RC as possible.

Jason


On Tue, Jan 9, 2018 at 1:17 PM, Lei Xu  wrote:

> Hi, All
>
> We have released Apache Hadoop 3.0.0 in December [1]. To further
> improve the quality of release, we plan to cut branch-3.0.1 branch
> tomorrow for the preparation of Apache Hadoop 3.0.1 release. The focus
> of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug fixes
> [2].  No new features and improvement should be included.
>
> We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on Feb
> 1st, targeting for Feb 9th release.
>
> Please feel free to share your insights.
>
> [1] https://www.mail-archive.com/general@hadoop.apache.org/msg07757.html
> [2] https://issues.apache.org/jira/issues/?filter=12342842
>
> Best,
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>

[jira] [Reopened] (HDFS-12881) Output streams closed with IOUtils suppressing write errors

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened HDFS-12881:
---

Reopening to get a precommit run on the branch-2 patch.

> Output streams closed with IOUtils suppressing write errors
> ---
>
> Key: HDFS-12881
> URL: https://issues.apache.org/jira/browse/HDFS-12881
> Project: Hadoop HDFS
>  Issue Type: Bug
>    Reporter: Jason Lowe
>Assignee: Ajay Kumar
> Fix For: 3.1.0, 3.0.1
>
> Attachments: HDFS-12881-branch-2.10.0.001.patch, 
> HDFS-12881.001.patch, HDFS-12881.002.patch, HDFS-12881.003.patch, 
> HDFS-12881.004.patch
>
>
> There are a few places in HDFS code that are closing an output stream with 
> IOUtils.cleanupWithLogger like this:
> {code}
>   try {
> ...write to outStream...
>   } finally {
> IOUtils.cleanupWithLogger(LOG, outStream);
>   }
> {code}
> This suppresses any IOException that occurs during the close() method which 
> could lead to partial/corrupted output without throwing a corresponding 
> exception.  The code should either use try-with-resources or explicitly close 
> the stream within the try block so the exception thrown during close() is 
> properly propagated as exceptions during write operations are.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Resolved] (HDFS-12924) Port HDFS-12881 to branch-2 (Output streams closed with IOUtils suppressing write errors)

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-12924.
---
Resolution: Duplicate

> Port HDFS-12881 to branch-2 (Output streams closed with IOUtils suppressing 
> write errors)
> -
>
> Key: HDFS-12924
> URL: https://issues.apache.org/jira/browse/HDFS-12924
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
> Fix For: 2.10.0
>
>
> Port HDFS-12881 to branch-2



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.5 (RC1)

[Jason Lowe]

Thanks for driving the release, Konstantin!

+1 (binding)

- Verified signatures and digests
- Successfully performed a native build from source
- Deployed a single-node cluster
- Ran some sample jobs and checked the logs

Jason


On Thu, Dec 7, 2017 at 9:22 PM, Konstantin Shvachko 
wrote:

> Hi everybody,
>
> I updated CHANGES.txt and fixed documentation links.
> Also committed  MAPREDUCE-6165, which fixes a consistently failing test.
>
> This is RC1 for the next dot release of Apache Hadoop 2.7 line. The
> previous one 2.7.4 was release August 4, 2017.
> Release 2.7.5 includes critical bug fixes and optimizations. See more
> details in Release Note:
> http://home.apache.org/~shv/hadoop-2.7.5-RC1/releasenotes.html
>
> The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.5-RC1/
>
> Please give it a try and vote on this thread. The vote will run for 5 days
> ending 12/13/2017.
>
> My up to date public key is available from:
> https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>
> Thanks,
> --Konstantin
>

Re: [VOTE] Release Apache Hadoop 2.8.3 (RC0)

[Jason Lowe]

Thanks for driving this release, Junping!

+1 (binding)

- Verified signatures and digests
- Successfully performed native build from source
- Deployed a single-node cluster
- Ran some test jobs and examined the logs

Jason

On Tue, Dec 5, 2017 at 3:58 AM, Junping Du  wrote:

> Hi all,
>  I've created the first release candidate (RC0) for Apache Hadoop
> 2.8.3. This is our next maint release to follow up 2.8.2. It includes 79
> important fixes and improvements.
>
>   The RC artifacts are available at: http://home.apache.org/~
> junping_du/hadoop-2.8.3-RC0
>
>   The RC tag in git is: release-2.8.3-RC0
>
>   The maven artifacts are available via repository.apache.org at:
> https://repository.apache.org/content/repositories/orgapachehadoop-1072
>
>   Please try the release and vote; the vote will run for the usual 5
> working days, ending on 12/12/2017 PST time.
>
> Thanks,
>
> Junping
>

[jira] [Created] (HDFS-12881) Output streams closed with IOUtils suppressing write errors

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-12881:
-

 Summary: Output streams closed with IOUtils suppressing write 
errors
 Key: HDFS-12881
 URL: https://issues.apache.org/jira/browse/HDFS-12881
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Jason Lowe


There are a few places in HDFS code that are closing an output stream with 
IOUtils.cleanupWithLogger like this:
{code}
  try {
...write to outStream...
  } finally {
IOUtils.cleanupWithLogger(LOG, outStream);
  }
{code}
This suppresses any IOException that occurs during the close() method which 
could lead to partial/corrupted output without throwing a corresponding 
exception.  The code should either use try-with-resources or explicitly close 
the stream within the try block so the exception thrown during close() is 
properly propagated as exceptions during write operations are.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.9.0 (RC3)

[Jason Lowe]

Thanks for putting this release together!

+1 (binding)

- Verified signatures and digests
- Successfully built from source including native
- Deployed to single-node cluster and ran some test jobs

Jason


On Mon, Nov 13, 2017 at 6:10 PM, Arun Suresh  wrote:

> Hi Folks,
>
> Apache Hadoop 2.9.0 is the first release of Hadoop 2.9 line and will be the
> starting release for Apache Hadoop 2.9.x line - it includes 30 New Features
> with 500+ subtasks, 407 Improvements, 790 Bug fixes new fixed issues since
> 2.8.2.
>
> More information about the 2.9.0 release plan can be found here:
> *https://cwiki.apache.org/confluence/display/HADOOP/
> Roadmap#Roadmap-Version2.9
>  Roadmap#Roadmap-Version2.9>*
>
> New RC is available at: *https://home.apache.org/~
> asuresh/hadoop-2.9.0-RC3/
> *
>
> The RC tag in git is: release-2.9.0-RC3, and the latest commit id is:
> 756ebc8394e473ac25feac05fa493f6d612e6c50.
>
> The maven artifacts are available via repository.apache.org at:
>  apache.org%2Fcontent%2Frepositories%2Forgapachehadoop-1066&sa=D&
> sntz=1&usg=AFQjCNFcern4uingMV_sEreko_zeLlgdlg>*https://
> repository.apache.org/content/repositories/orgapachehadoop-1068/
>  >*
>
> We are carrying over the votes from the previous RC given that the delta is
> the license fix.
>
> Given the above - we are also going to stick with the original deadline for
> the vote : ending on Friday 17th November 2017 2pm PT time.
>
> Thanks,
> -Arun/Subru
>

[jira] [Resolved] (HDFS-12817) Support multiple storages in DataNodeCluster / SimulatedFSDataset

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-12817.
---
Resolution: Duplicate

> Support multiple storages in DataNodeCluster / SimulatedFSDataset
> -
>
> Key: HDFS-12817
> URL: https://issues.apache.org/jira/browse/HDFS-12817
> Project: Hadoop HDFS
>  Issue Type: Improvement
>  Components: datanode, test
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Minor
>
> Currently {{SimulatedFSDataset}} (and thus, {{DataNodeCluster}} with 
> {{-simulated}}) only supports a single storage per {{DataNode}}. Given that 
> the number of storages can have important implications on the performance of 
> block report processing, it would be useful for these classes to support a 
> multiple storage configuration.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Jason Lowe]

My apologies, false alarm on the CHANGES.md and RELEASENOTES.md.  I was in
the process of reviewing the release and was interrupted, and when I
resumed I thought I had already downloaded the CHANGES and RELEASENOTES,
but in fact they were the old versions from a prior review of 2.8.0.  I
reviewed both of them for 2.8.2 (for real this time!) and they look
correct.  Again my apologies for the confusion.

Jason

On Mon, Oct 23, 2017 at 3:26 PM, Jason Lowe  wrote:

> +1 (binding)
>
> - Verified signatures and digests
> - Performed a native build from source
> - Deployed to a single-node cluster
> - Ran some sample jobs
>
> The CHANGES.md and RELEASENOTES.md both refer to release 2.8.0 instead of
> 2.8.2, and I do not see the list of JIRAs in CHANGES.md that have been
> committed since 2.8.1.  Since we're voting on the source bits rather than
> the change log I kept my vote as a +1 as I do see the 2.8.2 changes in the
> source code.
>
> Jason
>
>
> On Thu, Oct 19, 2017 at 7:42 PM, Junping Du  wrote:
>
>> Hi folks,
>>  I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.
>>
>>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line
>> and will be the latest stable/production release for Apache Hadoop - it
>> includes 315 new fixed issues since 2.8.1 and 69 fixes are marked as
>> blocker/critical issues.
>>
>>   More information about the 2.8.2 release plan can be found here:
>> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>>
>>   New RC is available at: http://home.apache.org/~junpin
>> g_du/hadoop-2.8.2-RC1<http://home.apache.org/~junping_du/hadoop-2.8.2-RC0
>> >
>>
>>   The RC tag in git is: release-2.8.2-RC1, and the latest commit id
>> is: 66c47f2a01ad9637879e95f80c41f798373828fb
>>
>>   The maven artifacts are available via repository.apache.org<
>> http://repository.apache.org/> at: https://repository.apache.org/
>> content/repositories/orgapachehadoop-1064<https://repository
>> .apache.org/content/repositories/orgapachehadoop-1062>
>>
>>   Please try the release and vote; the vote will run for the usual 5
>> days, ending on 10/24/2017 6pm PST time.
>>
>> Thanks,
>>
>> Junping
>>
>>
>

Re: [VOTE] Release Apache Hadoop 2.8.2 (RC1)

[Jason Lowe]

+1 (binding)

- Verified signatures and digests
- Performed a native build from source
- Deployed to a single-node cluster
- Ran some sample jobs

The CHANGES.md and RELEASENOTES.md both refer to release 2.8.0 instead of
2.8.2, and I do not see the list of JIRAs in CHANGES.md that have been
committed since 2.8.1.  Since we're voting on the source bits rather than
the change log I kept my vote as a +1 as I do see the 2.8.2 changes in the
source code.

Jason


On Thu, Oct 19, 2017 at 7:42 PM, Junping Du  wrote:

> Hi folks,
>  I've created our new release candidate (RC1) for Apache Hadoop 2.8.2.
>
>  Apache Hadoop 2.8.2 is the first stable release of Hadoop 2.8 line
> and will be the latest stable/production release for Apache Hadoop - it
> includes 315 new fixed issues since 2.8.1 and 69 fixes are marked as
> blocker/critical issues.
>
>   More information about the 2.8.2 release plan can be found here:
> https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release
>
>   New RC is available at: http://home.apache.org/~
> junping_du/hadoop-2.8.2-RC1 du/hadoop-2.8.2-RC0>
>
>   The RC tag in git is: release-2.8.2-RC1, and the latest commit id
> is: 66c47f2a01ad9637879e95f80c41f798373828fb
>
>   The maven artifacts are available via repository.apache.org repository.apache.org/> at: https://repository.apache.org/
> content/repositories/orgapachehadoop-1064 repository.apache.org/content/repositories/orgapachehadoop-1062>
>
>   Please try the release and vote; the vote will run for the usual 5
> days, ending on 10/24/2017 6pm PST time.
>
> Thanks,
>
> Junping
>
>

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

[Jason Lowe]

+1 (binding)

I participated in the review for the reader authorization and verified that
ATSv2 has no significant impact when disabled.  Looking forward to seeing
the next increment in functionality in a release.  A big thank you to
everyone involved in this effort!

Jason


On Tue, Aug 22, 2017 at 1:32 AM, Vrushali Channapattan <
vrushalic2...@gmail.com> wrote:

> Hi folks,
>
> Per earlier discussion [1], I'd like to start a formal vote to merge
> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote will
> run for 7 days, and will end August 29 11:00 PM PDT.
>
> We have previously completed one merge onto trunk [3] and Timeline Service
> v2 has been part of Hadoop release 3.0.0-alpha1.
>
> Since then, we have been working on extending the capabilities of Timeline
> Service v2 in a feature branch [2] for a while, and we are reasonably
> confident that the state of the feature meets the criteria to be merged
> onto trunk and we'd love folks to get their hands on it in a test capacity
> and provide valuable feedback so that we can make it production-ready.
>
> In a nutshell, Timeline Service v.2 delivers significant scalability and
> usability improvements based on a new architecture. What we would like to
> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> complete end-to-end read/write flow with security and read level
> authorization via whitelists. You should be able to start setting it up and
> testing it.
>
> At a high level, the following are the key features that have been
> implemented since alpha1:
> - Security via Kerberos Authentication and delegation tokens
> - Read side simple authorization via whitelist
> - Client configurable entity sort ordering
> - Richer REST APIs for apps, app attempts, containers, fetching metrics by
> timerange, pagination, sub-app entities
> - Support for storing sub-application entities (entities that exist outside
> the scope of an application)
> - Configurable TTLs (time-to-live) for tables, configurable table prefixes,
> configurable hbase cluster
> - Flow level aggregations done as dynamic (table level) coprocessors
> - Uses latest stable HBase release 1.2.6
>
> There are a total of 82 subtasks that were completed as part of this
> effort.
>
> We paid close attention to ensure that once disabled Timeline Service v.2
> does not impact existing functionality when disabled (by default).
>
> Special thanks to a team of folks who worked hard and contributed towards
> this effort with patches, reviews and guidance: Rohith Sharma K S, Varun
> Saxena, Haibo Chen, Sangjin Lee, Li Lu, Vinod Kumar Vavilapalli, Joep
> Rottinghuis, Jason Lowe, Jian He, Robert Kanter, Micheal Stack.
>
> Regards,
> Vrushali
>
> [1] http://www.mail-archive.com/yarn-dev@hadoop.apache.org/msg27383.html
> [2] https://issues.apache.org/jira/browse/YARN-5355
> [3] https://issues.apache.org/jira/browse/YARN-2928
> [4] https://github.com/apache/hadoop/commits/YARN-5355
>

Re: [DISCUSS] Branches and versions for Hadoop 3

[Jason Lowe]

Allen Wittenauer wrote:


> > On Aug 25, 2017, at 1:23 PM, Jason Lowe  wrote:
> >
> > Allen Wittenauer wrote:
> >
> > > Doesn't this place an undue burden on the contributor with the first
> incompatible patch to prove worthiness?  What happens if it is decided that
> it's not good enough?
> >
> > It is a burden for that first, "this can't go anywhere else but 4.x"
> change, but arguably that should not be a change done lightly anyway.  (Or
> any other backwards-incompatible change for that matter.)  If it's worth
> committing then I think it's perfectly reasonable to send out the dev
> announce that there's reason for trunk to diverge from 3.x, cut branch-3,
> and move on.  This is no different than Andrew's recent announcement that
> there's now a need for separating trunk and the 3.0 line based on what's
> about to go in.
>
> So, by this definition as soon as a patch comes in to remove
> deprecated bits there will be no issue with a branch-3 getting created,
> correct?
>

I think this gets back to the "if it's worth committing" part.  I feel the
community should collectively decide when it's worth taking the hit to
maintain the separate code line.  IMHO removing deprecated bits alone is
not reason enough to diverge the code base and the additional maintenance
that comes along with the extra code line.  A new feature is traditionally
the reason to diverge because that's something users would actually care
enough about to take the compatibility hit when moving to the version that
has it.  That also helps drive a timely release of the new code line
because users want the feature that went into it.


> >  Otherwise if past trunk behavior is any indication, it ends up mostly
> enabling people to commit to just trunk, forgetting that the thing they are
> committing is perfectly valid for branch-3.
>
> I'm not sure there was any "forgetting" involved.  We likely
> wouldn't be talking about 3.x at all if it wasn't for the code diverging
> enough.
>

I don't think it was the myriad of small patches that went only into trunk
over the last 6 years that drove this.  Instead I think it was simply that
an "important enough" feature went in, like erasure coding, that gathered
momentum behind this release.  Trunk sat ignored for basically 5+ years,
and plenty of patches went into just trunk that should have gone into at
least branch-2 as well.  I don't think we as a community did the
contributors any favors by putting their changes into a code line that
didn't see a release for a very long time.  Yes 3.x could have released
sooner to help solve that issue, but given the complete lack of excitement
around 3.x until just recently is there any reason this won't happen again
with 4.x?  Seems to me 4.x will need to have something "interesting enough"
to drive people to release it relative to 3.x, which to me indicates we
shouldn't commit things only to there until we have an interest to do so.

> > Given the number of committers that openly ignore discussions like
> this, who is going to verify that incompatible changes don't get in?
> >
> > The same entities who are verifying other bugs don't get in, i.e.: the
> committers and the Hadoop QA bot running the tests.
> >  Yes, I know that means it's inevitable that compatibility breakages
> will happen, and we can and should improve the automation around
> compatibility testing when possible.
>
> The automation only goes so far.  At least while investigating
> Yetus bugs, I've seen more than enough blatant and purposeful ignored
> errors and warnings that I'm not convinced it will be effective. ("That
> javadoc compile failure didn't come from my patch!"  Um, yes, yes it did.)
> PR for features has greatly trumped code correctness for a few years now.
>

I totally agree here.  We can and should do better about this outside of
automation.  I brought up automation since I see it as a useful part of the
total solution along with better developer education, oversight, etc.  I'm
thinking specifically about tools that can report on public API signature
changes, but that's just one aspect of compatibility.  Semantic behavior is
not something a static analysis tool can automatically detect, and the only
way to automate some of that is something like end-to-end compatibility
testing.  Bigtop may cover some of this with testing of older versions of
downstream projects like HBase, Hive, Oozie, etc., and we could setup some
tests that standup two different Hadoop clusters and run tests that verify
interop between them.  But the tests will never be exhaustive and we will
still need educated commit

Re: [DISCUSS] Branches and versions for Hadoop 3

[Jason Lowe]

Allen Wittenauer wrote:


> Doesn't this place an undue burden on the contributor with the first
> incompatible patch to prove worthiness?  What happens if it is decided that
> it's not good enough?


It is a burden for that first, "this can't go anywhere else but 4.x"
change, but arguably that should not be a change done lightly anyway.  (Or
any other backwards-incompatible change for that matter.)  If it's worth
committing then I think it's perfectly reasonable to send out the dev
announce that there's reason for trunk to diverge from 3.x, cut branch-3,
and move on.  This is no different than Andrew's recent announcement that
there's now a need for separating trunk and the 3.0 line based on what's
about to go in.

I do not think it makes sense to pay for the maintenance overhead of two
nearly-identical lines with no backwards-incompatible changes between them
until we have the need.  Otherwise if past trunk behavior is any
indication, it ends up mostly enabling people to commit to just trunk,
forgetting that the thing they are committing is perfectly valid for
branch-3.  If we can agree that trunk and branch-3 should be equivalent
until an incompatible change goes into trunk, why pay for the commit
overhead and potential for accidentally missed commits until it is really
necessary?

How many will it take before the dam will break?  Or is there a timeline
> going to be given before trunk gets set to 4.x?


I think the threshold count for the dam should be 1.  As soon as we have a
JIRA that needs to be committed to move the project forward and we cannot
ship it in a 3.x release then we create branch-3 and move trunk to 4.x.
As for a timeline going to 4.x, again I don't see it so much as a "baking
period" as a "when we need it" criteria.  If we need it in a week then we
should cut it in a week.  Or a year then a year.  It all depends upon when
that 4.x-only change is ready to go in.

Given the number of committers that openly ignore discussions like this,
> who is going to verify that incompatible changes don't get in?
>

The same entities who are verifying other bugs don't get in, i.e.: the
committers and the Hadoop QA bot running the tests.  Yes, I know that means
it's inevitable that compatibility breakages will happen, and we can and
should improve the automation around compatibility testing when possible.
But I don't think there's a magic bullet for preventing all compatibility
bugs from being introduced, just like there isn't one for preventing
general bugs.  Does having a trunk branch separate but essentially similar
to branch-3 make this any better?

Longer term:  what is the PMC doing to make sure we start doing major
> releases in a timely fashion again?  In other words, is this really an
> issue if we shoot for another major in (throws dart) 2 years?
>

If we're trying to do semantic versioning then we shouldn't have a regular
cadence for major releases unless we have a regular cadence of changes that
break compatibility.  I'd hope that's not something we would strive
towards.  I do agree that we should try to be better about shipping
releases, major or minor, in a more timely manner, but I don't agree that
we should cut 4.0 simply based on a duration since the last major release.
The release contents and community's desire for those contents should
dictate the release numbering and schedule, respectively.

Jason


On Fri, Aug 25, 2017 at 2:16 PM, Allen Wittenauer 
wrote:

>
> > On Aug 25, 2017, at 10:36 AM, Andrew Wang 
> wrote:
>
> > Until we need to make incompatible changes, there's no need for
> > a Hadoop 4.0 version.
>
> Some questions:
>
> Doesn't this place an undue burden on the contributor with the
> first incompatible patch to prove worthiness?  What happens if it is
> decided that it's not good enough?
>
> How many will it take before the dam will break?  Or is there a
> timeline going to be given before trunk gets set to 4.x?
>
> Given the number of committers that openly ignore discussions like
> this, who is going to verify that incompatible changes don't get in?
>
> Longer term:  what is the PMC doing to make sure we start doing
> major releases in a timely fashion again?  In other words, is this really
> an issue if we shoot for another major in (throws dart) 2 years?
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

Re: Branch merges and 3.0.0-beta1 scope

[Jason Lowe]

Andrew Wang wrote:


> This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.


I can see a need for branch-3.0, but please do not create branch-3.  Doing
so will relegate trunk back to the "patch purgatory" branch, a place where
patches won't see a release for years.  Unless something is imminently
going in that will break backwards compatibility and warrant a new 4.x
release, I don't see the need to distinguish trunk from the 3.x line.
Leaving trunk as the 3.x line means less branches to commit patches through
and more testing of every patch since trunk would remain an active area for
testing and releasing.  If we separate trunk and branch-3 then it's almost
certain only-trunk patches will start to accumulate and never get any
"real" testing until someone eventually decides it's time to go to Hadoop
4.x.  Looking back at trunk-as-3.x for an example, patches committed there
in the early days after branch-2 was cut didn't see a release for almost 6
years.

My apologies if I've missed a feature that is just going to miss the 3.0
release and will break compatibility when it goes in.  If so then we need
to cut branch-3, but if not then here's my plea to hold off until we do
need it.

Jason


On Thu, Aug 24, 2017 at 3:33 PM, Andrew Wang 
wrote:

> Glad to see the discussion continued in my absence :)
>
> From a release management perspective, it's *extremely* reasonable to block
> the inclusion of new features a month from the planned release date. A
> typical software development lifecycle includes weeks of feature freeze and
> weeks of code freeze. It is no knock on any developer or any feature to say
> that we should not include something in 3.0.0.
>
> I've been very open and clear about the goals, schedule, and scope of 3.0.0
> over the last year plus. The point of the extended alpha process was to get
> all our features in during alpha, and the alpha merge window has been open
> for a year. I'm unmoved by arguments about how long a feature has been
> worked on. None of these were not part of the original 3.0.0 scope, and our
> users have been waiting even longer for big-ticket 3.0 items like JDK8 and
> HDFS EC that were part of the discussed scope.
>
> I see that two VOTEs have gone out since I was out. I still plan to follow
> the proposal in my original email. This means I'll cut branch-3 and
> branch-3.0, and move trunk to 4.0.0 before these VOTEs end. This will open
> up development for Hadoop 3.1.0 and 4.0.0.
>
> I'm reaching out to the lead contributor of each of these features
> individually to discuss. We need to close on this quickly, and email is too
> low bandwidth at this stage.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 2.7.4 (RC0)

[Jason Lowe]

Thanks for driving the 2.7.4 release!
+1 (binding)
- Verified signatures and digests- Successfully built from source including 
native- Deployed to a single-node cluster and ran sample MapReduce jobs
Jason 

On Saturday, July 29, 2017 6:29 PM, Konstantin Shvachko 
 wrote:
 

 Hi everybody,

Here is the next release of Apache Hadoop 2.7 line. The previous stable
release 2.7.3 was available since 25 August, 2016.
Release 2.7.4 includes 264 issues fixed after release 2.7.3, which are
critical bug fixes and major optimizations. See more details in Release
Note:
http://home.apache.org/~shv/hadoop-2.7.4-RC0/releasenotes.html

The RC0 is available at: http://home.apache.org/~shv/hadoop-2.7.4-RC0/

Please give it a try and vote on this thread. The vote will run for 5 days
ending 08/04/2017.

Please note that my up to date public key are available from:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
Please don't forget to refresh the page if you've been there recently.
There are other place on Apache sites, which may contain my outdated key.

Thanks,
--Konstantin


   

[jira] [Reopened] (HDFS-12217) HDFS snapshots doesn't capture all open files when one of the open files is deleted

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-12217?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe reopened HDFS-12217:
---

I reverted this from branch-2 because the commit broke the build.
{noformat}
[ERROR] COMPILATION ERROR : 
[INFO] -
[ERROR] 
/home/jlowe/hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java:[395,40]
 unreported exception java.io.IOException; must be caught or declared to be 
thrown
[ERROR] 
/home/jlowe/hadoop/apache/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestLeaseManager.java:[397,40]
 unreported exception java.io.IOException; must be caught or declared to be 
thrown
[INFO] 2 errors 
{noformat}

The patch will need to be updated for branch-2 and recommitted.

> HDFS snapshots doesn't capture all open files when one of the open files is 
> deleted
> ---
>
> Key: HDFS-12217
> URL: https://issues.apache.org/jira/browse/HDFS-12217
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: snapshots
>Affects Versions: 3.0.0-alpha1
>Reporter: Manoj Govindassamy
>Assignee: Manoj Govindassamy
> Fix For: 2.9.0, 3.0.0-beta1
>
> Attachments: HDFS-12217.01.patch, HDFS-12217.02.patch, 
> HDFS-12217.03.patch, HDFS-12217.04.patch, HDFS-12217.05.patch
>
>
> With the fix for HDFS-11402, HDFS Snapshots can additionally capture all the 
> open files. Just like all other files, these open files in the snapshots will 
> remain immutable. But, sometimes it is found that snapshots fail to capture 
> all the open files in the system.
> Under the following conditions, LeaseManager will fail to find INode 
> corresponding to an active lease 
> * a file is opened for writing (LeaseManager allots a lease), and
> * the same file is deleted while it is still open for writing and having 
> active lease, and
> * the same file is not referenced in any other Snapshots/Trash
> {{INode[] LeaseManager#getINodesWithLease()}} can thus return null for few 
> leases there by causing the caller to trip over and not return all the open 
> files needed by the snapshot manager.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apache Hadoop 2.8.2 Release Plan

[Jason Lowe]

+1 to base the 2.8.2 release off of the more recent activity on branch-2.8.  
Because branch-2.8.2 was cut so long ago it is missing a lot of fixes that are 
in branch-2.8.  There also are a lot of JIRAs that claim they are fixed in 
2.8.2 but are not in branch-2.8.2.  Having the 2.8.2 release be based on recent 
activity in branch-2.8 would solve both of these issues, and we'd only need to 
move the handful of JIRAs that have marked themselves correctly as fixed in 
2.8.3 to be fixed in 2.8.2.

Jason
 

On Friday, July 21, 2017 10:01 AM, Kihwal Lee 
 wrote:
 

 Thanks for driving the next 2.8 release, Junping. While I was committing a 
blocker for 2.7.4, I noticed some of the jiras are back-ported to 2.7, but 
missing in branch-2.8.2.  Perhaps it is safer and easier to simply rebranch 
2.8.2.
Thanks,Kihwal

On Thursday, July 20, 2017, 3:32:16 PM CDT, Junping Du  
wrote:

Hi all,
    Per Vinod's previous email, we just announce Apache Hadoop 2.8.1 get 
released today which is a special security release. Now, we should work towards 
2.8.2 release which aim for production deployment. The focus obviously is to 
fix blocker/critical issues [2], bug-fixes and *no* features / improvements. We 
currently have 13 blocker/critical issues, and 10 of them are Patch Available.

  I plan to cut an RC in a month - target for releasing before end of Aug., to 
give enough time for outstanding blocker / critical issues. Will start moving 
out any tickets that are not blockers and/or won't fit the timeline. For 
progress of releasing effort, please refer our release wiki [2].

  Please share thoughts if you have any. Thanks!

Thanks,

Junping

[1] 2.8.2 release Blockers/Criticals: https://s.apache.org/JM5x
[2] 2.8 Release wiki: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release


From: Vinod Kumar Vavilapalli 
Sent: Thursday, July 20, 2017 1:05 PM
To: gene...@hadoop.apache.org
Subject: [ANNOUNCE] Apache Hadoop 2.8.1 is released

Hi all,

The Apache Hadoop PMC has released version 2.8.1. You can get it from this 
page: http://hadoop.apache.org/releases.html#Download
This is a security release in the 2.8.0 release line. It consists of 2.8.0 plus 
security fixes. Users on 2.8.0 are encouraged to upgrade to 2.8.1.

Please note that 2.8.x release line continues to be not yet ready for 
production use. Critical issues are being ironed out via testing and downstream 
adoption. Production users should wait for a subsequent release in the 2.8.x 
line.

Thanks
+Vinod


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

   

Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Jason Lowe]

Thanks for the pointers, Sean!  According to the infrastructure team, 
apparently it was a typo in the protection scheme that allowed the trunk force 
push to go through.  
 
https://issues.apache.org/jira/browse/INFRA-13902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15971643#comment-15971643
   
Jason
 On Monday, April 17, 2017 3:05 PM, Sean Busbey  wrote:
 

 disallowing force pushes to trunk was done back in:

* August 2014: INFRA-8195
* February 2016: INFRA-11136

On Mon, Apr 17, 2017 at 11:18 AM, Jason Lowe
 wrote:
> I found at least one commit that was dropped, MAPREDUCE-6673.  I was able to 
> cherry-pick the original commit hash since it was recorded in the commit 
> email.
> This begs the question of why we're allowing force pushes to trunk.  I 
> thought we asked to have that disabled the last time trunk was accidentally 
> clobbered?
> Jason
>
>
>    On Monday, April 17, 2017 10:18 AM, Arun Suresh  wrote:
>
>
>  Hi
>
> I had the Apr-14 eve version of trunk on my local machine. I've pushed that.
> Don't know if anything was committed over the weekend though.
>
> Cheers
> -Arun
>
> On Mon, Apr 17, 2017 at 7:17 AM, Anu Engineer 
> wrote:
>
>> Hi Allen,
>>
>> https://issues.apache.org/jira/browse/INFRA-13902
>>
>> That happened with ozone branch too. It was an inadvertent force push.
>> Infra has advised us to force push the latest branch if you have it.
>>
>> Thanks
>> Anu
>>
>>
>> On 4/17/17, 7:10 AM, "Allen Wittenauer"  wrote:
>>
>> >Looks like someone reset HEAD back to Mar 31.
>> >
>> >Sent from my iPad
>> >
>> >> On Apr 16, 2017, at 12:08 AM, Apache Jenkins Server <
>> jenk...@builds.apache.org> wrote:
>> >>
>> >> For more details, see https://builds.apache.org/job/
>> hadoop-qbt-trunk-java8-linux-x86/378/
>> >>
>> >>
>> >>
>> >>
>> >>
>> >> -1 overall
>> >>
>> >>
>> >> The following subsystems voted -1:
>> >>    docker
>> >>
>> >>
>> >> Powered by Apache Yetus 0.5.0-SNAPSHOT  http://yetus.apache.org
>> >>
>> >>
>> >>
>> >> -
>> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>> >
>> >
>> >-
>> >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>> >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>> >
>> >
>>
>>
>
>
>



-- 
busbey


   

Re: Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Jason Lowe]

I found at least one commit that was dropped, MAPREDUCE-6673.  I was able to 
cherry-pick the original commit hash since it was recorded in the commit email.
This begs the question of why we're allowing force pushes to trunk.  I thought 
we asked to have that disabled the last time trunk was accidentally clobbered?
Jason
 

On Monday, April 17, 2017 10:18 AM, Arun Suresh  wrote:
 

 Hi

I had the Apr-14 eve version of trunk on my local machine. I've pushed that.
Don't know if anything was committed over the weekend though.

Cheers
-Arun

On Mon, Apr 17, 2017 at 7:17 AM, Anu Engineer 
wrote:

> Hi Allen,
>
> https://issues.apache.org/jira/browse/INFRA-13902
>
> That happened with ozone branch too. It was an inadvertent force push.
> Infra has advised us to force push the latest branch if you have it.
>
> Thanks
> Anu
>
>
> On 4/17/17, 7:10 AM, "Allen Wittenauer"  wrote:
>
> >Looks like someone reset HEAD back to Mar 31.
> >
> >Sent from my iPad
> >
> >> On Apr 16, 2017, at 12:08 AM, Apache Jenkins Server <
> jenk...@builds.apache.org> wrote:
> >>
> >> For more details, see https://builds.apache.org/job/
> hadoop-qbt-trunk-java8-linux-x86/378/
> >>
> >>
> >>
> >>
> >>
> >> -1 overall
> >>
> >>
> >> The following subsystems voted -1:
> >>    docker
> >>
> >>
> >> Powered by Apache Yetus 0.5.0-SNAPSHOT  http://yetus.apache.org
> >>
> >>
> >>
> >> -
> >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> >-
> >To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> >For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> >
> >
>
>


   

Re: [VOTE] Release Apache Hadoop 2.8.0 (RC3)

[Jason Lowe]

+1 (binding)
- Verfied signatures and digests- Performed a native build from the release 
tag- Deployed to a single node cluster- Ran some sample jobs
Jason
 

On Friday, March 17, 2017 4:18 AM, Junping Du  wrote:
 

 Hi all,
    With fix of HDFS-11431 get in, I've created a new release candidate (RC3) 
for Apache Hadoop 2.8.0.

    This is the next minor release to follow up 2.7.0 which has been released 
for more than 1 year. It comprises 2,900+ fixes, improvements, and new 
features. Most of these commits are released for the first time in branch-2.

      More information about the 2.8.0 release plan can be found here: 
https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+2.8+Release

      New RC is available at: 
http://home.apache.org/~junping_du/hadoop-2.8.0-RC3

      The RC tag in git is: release-2.8.0-RC3, and the latest commit id is: 
91f2b7a13d1e97be65db92ddabc627cc29ac0009

      The maven artifacts are available via repository.apache.org at: 
https://repository.apache.org/content/repositories/orgapachehadoop-1057

      Please try the release and vote; the vote will run for the usual 5 days, 
ending on 03/22/2017 PDT time.

Thanks,

Junping

   

[jira] [Resolved] (HDFS-11501) Is there a way of loading cvs files to create hive tables with desired lengths for columns?

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-11501?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-11501.
---
Resolution: Invalid

JIRA is for tracking bugs against the Hadoop project and not for general user 
support.  Please post your question to either the [Hive user mailing 
list|http://hive.apache.org/mailing_lists.html] or the [Hadoop user mailing 
list|http://hadoop.apache.org/mailing_lists.html].

> Is there a way of loading cvs files to create hive tables with desired 
> lengths for columns?
> ---
>
> Key: HDFS-11501
> URL: https://issues.apache.org/jira/browse/HDFS-11501
> Project: Hadoop HDFS
>  Issue Type: Bug
>Reporter: Wen Lin
>
> We just got on Hadoop environment. Our data sources are cvs files. The hive 
> tables created from the sources are seen all character /string columns have 
> same length of 255 bytes, even for gender which has value with one byte. Is 
> there a way of loading cvs files to create hive tables with desired lengths 
> for string columns instead of 255 across all tables? Thank you for your help!



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11252) TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with BindException

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-11252:
-

 Summary: 
TestFileTruncate#testTruncateWithDataNodesRestartImmediately can fail with 
BindException
 Key: HDFS-11252
 URL: https://issues.apache.org/jira/browse/HDFS-11252
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Jason Lowe


testTruncateWithDataNodesRestartImmediately can fail with a BindException.  The 
setup for TestFileTruncate has been fixed in the past to solve a bind 
exception, but this is occurring after the minicluster comes up and the 
datanodes are being restarted.  Maybe there's a race condition there?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-11251) ConcurrentModificationException during DataNode#refreshVolumes

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-11251:
-

 Summary: ConcurrentModificationException during 
DataNode#refreshVolumes
 Key: HDFS-11251
 URL: https://issues.apache.org/jira/browse/HDFS-11251
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0-alpha2
Reporter: Jason Lowe


The testAddVolumesDuringWrite case failed with a ReconfigurationException which 
appears to have been caused by a ConcurrentModificationException.  Stacktrace 
details to follow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Updated 2.8.0-SNAPSHOT artifact

[Jason Lowe]

At this point my preference would be to do the most expeditious thing to 
release 2.8, whether that's sticking with the branch-2.8 we have today or 
re-cutting it on branch-2.  Doing a quick JIRA query, there's been almost 2,400 
JIRAs resolved in 2.8.0 (1).  For many of them, it's well-past time they saw a 
release vehicle.  If re-cutting the branch means we have to wrap up a few extra 
things that are still in-progress on branch-2 or add a few more blockers to the 
list before we release then I'd rather stay where we're at and ship it ASAP.

Jason
(1) 
https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2C%20yarn%2C%20mapreduce%2C%20hdfs%29%20and%20resolution%20%3D%20Fixed%20and%20fixVersion%20%3D%202.8.0





On Tuesday, October 25, 2016 5:31 PM, Karthik Kambatla  
wrote:
 

 Is there value in releasing current branch-2.8? Aren't we better off
re-cutting the branch off of branch-2?

On Tue, Oct 25, 2016 at 12:20 AM, Akira Ajisaka 
wrote:

> It's almost a year since branch-2.8 has cut.
> I'm thinking we need to release 2.8.0 ASAP.
>
> According to the following list, there are 5 blocker and 6 critical issues.
> https://issues.apache.org/jira/issues/?filter=12334985
>
> Regards,
> Akira
>
>
> On 10/18/16 10:47, Brahma Reddy Battula wrote:
>
>> Hi Vinod,
>>
>> Any plan on first RC for branch-2.8 ? I think, it has been long time.
>>
>>
>>
>>
>> --Brahma Reddy Battula
>>
>> -Original Message-
>> From: Vinod Kumar Vavilapalli [mailto:vino...@apache.org]
>> Sent: 20 August 2016 00:56
>> To: Jonathan Eagles
>> Cc: common-...@hadoop.apache.org
>> Subject: Re: Updated 2.8.0-SNAPSHOT artifact
>>
>> Jon,
>>
>> That is around the time when I branched 2.8, so I guess you were getting
>> SNAPSHOT artifacts till then from the branch-2 nightly builds.
>>
>> If you need it, we can set up SNAPSHOT builds. Or just wait for the first
>> RC, which is around the corner.
>>
>> +Vinod
>>
>> On Jul 28, 2016, at 4:27 PM, Jonathan Eagles  wrote:
>>>
>>> Latest snapshot is uploaded in Nov 2015, but checkins are still coming
>>> in quite frequently.
>>> https://repository.apache.org/content/repositories/snapshots/org/apach
>>> e/hadoop/hadoop-yarn-api/
>>>
>>> Are there any plans to start producing updated SNAPSHOT artifacts for
>>> current hadoop development lines?
>>>
>>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>
> -
> To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
>
>


   

Re: [VOTE] Release Apache Hadoop 2.6.5 (RC1)

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Built native from source- Deployed to a 
single-node cluster and ran some sample jobs
Jason
 

On Sunday, October 2, 2016 7:13 PM, Sangjin Lee  wrote:
 

 Hi folks,

I have pushed a new release candidate (R1) for the Apache Hadoop 2.6.5
release (the next maintenance release in the 2.6.x release line). RC1
contains fixes to CHANGES.txt, and is otherwise identical to RC0.

Below are the details of this release candidate:

The RC is available for validation at:
http://home.apache.org/~sjlee/hadoop-2.6.5-RC1/.

The RC tag in git is release-2.6.5-RC1 and its git commit is
e8c9fe0b4c252caf2ebf1464220599650f119997.

The maven artifacts are staged via repository.apache.org at:
https://repository.apache.org/content/repositories/orgapachehadoop-1050/.

You can find my public key at
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS.

Please try the release and vote. The vote will run for the usual 5 days. I
would greatly appreciate your timely vote. Thanks!

Regards,
Sangjin


   

Re: [VOTE] Release Apache Hadoop 2.7.3 RC2

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Successfully built from source with native 
support- Deployed a single-node cluster- Ran some sample jobs successfully

Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Wednesday, August 17, 2016 9:05 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC2
   
Hi all,

I've created a new release candidate RC2 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/ 


The RC tag in git is: release-2.7.3-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1046 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - few more issues with RC1 forced a RC2 [2]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [3].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2] [VOTE] Release Apache Hadoop 2.7.3 RC1: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg26336.html 

[3] 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


   

Re: [VOTE] Release Apache Hadoop 2.7.3 RC1

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Built from source with native support- 
Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Friday, August 12, 2016 11:45 AM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC1
   
Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC1/ 


The RC tag in git is: release-2.7.3-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1045/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at home.apache.org/~vinodkv/hadoop-2.7.3-RC1/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - few issues with RC0 forced a RC1 [1]
 - a very long fix-cycle for the License & Notice issues (HADOOP-12893) caused 
2.7.3 (along with every other Hadoop release) to slip by quite a bit. This 
release's related discussion thread is linked below: [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1] [VOTE] Release Apache Hadoop 2.7.3 RC0: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/index.html#26106 

[2]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


   

Re: [Release thread] 2.6.5 release activities

[Jason Lowe]

Thanks for organizing this, Chris!
I don't believe HADOOP-13362 is needed since it's related to ContainerMetrics.  
ContainerMetrics weren't added until 2.7 by YARN-2984.
YARN-4794 looks applicable to 2.6.  The change drops right in except it has 
JDK7-isms (multi-catch clause), so it needs a slight change.

Jason

  From: Chris Trezzo 
 To: "common-...@hadoop.apache.org" ; 
hdfs-dev@hadoop.apache.org; "mapreduce-...@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 
 
 Sent: Tuesday, August 9, 2016 7:32 PM
 Subject: [Release thread] 2.6.5 release activities
   
Based on the sentiment in the "[DISCUSS] 2.6.x line releases" thread, I
have moved forward with some of the initial effort in creating a 2.6.5
release. I am forking this thread so we have a dedicated 2.6.5 release
thread.

I have gone through the git logs and gathered a list of JIRAs that are in
branch-2.7 but are missing from branch-2.6. I limited the diff to issues
with a commit date after 1/26/2016. I did this because 2.6.4 was cut from
branch-2.6 around that date (http://markmail.org/message/xmy7ebs6l3643o5e)
and presumably issues that were committed to branch-2.7 before then were
already looked at as part of 2.6.4.

I have collected these issues in a spreadsheet and have given them an
initial triage on whether they are candidates for a backport to 2.6.5. The
spreadsheet is sorted by the status of the issues with the potential
backport candidates at the top. Here is a link to the spreadsheet:
https://docs.google.com/spreadsheets/d/1lfG2CYQ7W4q3olWpOCo6EBAey1WYC8hTRUemHvYPPzY/edit?usp=sharing

As of now, I have identified 16 potential backport candidates. Please take
a look at the list and let me know if there are any that you think should
not be on the list, or ones that you think I have missed. This was just an
initial high-level triage, so there could definitely be issues that are
miss-labeled.

As a side note: we still need to look at the pre-commit build for 2.6 and
follow up with an addendum for HADOOP-12800.

Thanks everyone!
Chris Trezzo


  

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Jason Lowe]

Both sound like real problems to me, and I think it's appropriate to file JIRAs 
to track them.
Jason


  From: Andrew Wang 
 To: Karthik Kambatla  
Cc: larry mccay ; Vinod Kumar Vavilapalli 
; "common-...@hadoop.apache.org" 
; "hdfs-dev@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 
; "mapreduce-...@hadoop.apache.org" 

 Sent: Thursday, August 4, 2016 5:56 PM
 Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC0
   
Could a YARN person please comment on these two issues, one of which Vinay
also hit? If someone already triaged or filed JIRAs, I missed it.

On Mon, Jul 25, 2016 at 11:52 AM, Andrew Wang 
wrote:

> I'll also add that, as a YARN newbie, I did hit two usability issues.
> These are very unlikely to be regressions, and I can file JIRAs if they
> seem fixable.
>
> * I didn't have SSH to localhost set up (new laptop), and when I tried to
> run the Pi job, it'd exit my window manager session. I feel there must be a
> more developer-friendly solution here.
> * If you start the NodeManager and not the RM, the NM has a handler for
> SIGTERM and SIGINT that blocked my Ctrl-C and kill attempts during startup.
> I had to kill -9 it.
>
> On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
> wrote:
>
>> I got asked this off-list, so as a reminder, only PMC votes are binding
>> on releases. Everyone is encouraged to vote on releases though!
>>
>> +1 (binding)
>>
>> * Downloaded source, built
>> * Started up HDFS and YARN
>> * Ran Pi job which as usual returned 4, and a little teragen
>>
>> On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
>> wrote:
>>
>>> +1 (binding)
>>>
>>> * Downloaded and build from source
>>> * Checked LICENSE and NOTICE
>>> * Pseudo-distributed cluster with FairScheduler
>>> * Ran MR and HDFS tests
>>> * Verified basic UI
>>>
>>> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
>>>
>>> > +1 binding
>>> >
>>> > * downloaded and built from source
>>> > * checked LICENSE and NOTICE files
>>> > * verified signatures
>>> > * ran standalone tests
>>> > * installed pseudo-distributed instance on my mac
>>> > * ran through HDFS and mapreduce tests
>>> > * tested credential command
>>> > * tested webhdfs access through Apache Knox
>>> >
>>> >
>>> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
>>> > vino...@apache.org> wrote:
>>> >
>>> > > Hi all,
>>> > >
>>> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>>> > >
>>> > > As discussed before, this is the next maintenance release to follow
>>> up
>>> > > 2.7.2.
>>> > >
>>> > > The RC is available for validation at:
>>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
>>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>> > >
>>> > > The RC tag in git is: release-2.7.3-RC0
>>> > >
>>> > > The maven artifacts are available via repository.apache.org <
>>> > > http://repository.apache.org/> at
>>> > > https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1040/
>>> > <
>>> > > https://repository.apache.org/content/repositories/
>>> orgapachehadoop-1040/
>>> > >
>>> > >
>>> > > The release-notes are inside the tar-balls at location
>>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html.
>>> I
>>> > > hosted this at
>>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
>>> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html
>>> >
>>> > for
>>> > > your quick perusal.
>>> > >
>>> > > As you may have noted, a very long fix-cycle for the License & Notice
>>> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
>>> > release)
>>> > > to slip by quite a bit. This release's related discussion thread is
>>> > linked
>>> > > below: [1].
>>> > >
>>> > > Please try the release and vote; the vote will run for the usual 5
>>> days.
>>> > >
>>> > > Thanks,
>>> > > Vinod
>>> > >
>>> > > [1]: 2.7.3 release plan:
>>> > > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/
>>> msg24439.html
>>> > <
>>> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
>>> >
>>>
>>
>>
>


   

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Built from source with native support- 
Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Friday, July 22, 2016 9:15 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 


The RC tag in git is: release-2.7.3-RC0

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
 for your 
quick perusal.

As you may have noted, a very long fix-cycle for the License & Notice issues 
(HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by 
quite a bit. This release's related discussion thread is linked below: [1].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


   

Re: [VOTE] Release Apache Hadoop 2.6.4 RC0

[Jason Lowe]

+1 (binding)
- verified signatures and digests- built native from source- deployed a 
single-node cluster and ran some sample MapReduce jobs.
Jason


  From: Junping Du 
 To: "hdfs-dev@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org" ; 
"common-...@hadoop.apache.org"  
 Sent: Wednesday, February 3, 2016 1:01 AM
 Subject: [VOTE] Release Apache Hadoop 2.6.4 RC0
   
Hi community folks,
  I've created a release candidate RC0 for Apache Hadoop 2.6.4 (the next 
maintenance release to follow up 2.6.3.) according to email thread of release 
plan 2.6.4 [1]. Below is details of this release candidate:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.4-RC0/
*

The RC tag in git is: release-2.6.4-RC0

The maven artifacts are staged via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1028/?
*

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks!


Cheers,

Junping


[1]: 2.6.4 release plan: http://markmail.org/message/fk3ud3c665lscvx5?


  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Jason Lowe]

That's reasonable, especially if we don't take nearly as long for 2.7.3.  Note 
that there are almost 50 JIRAs already committed to 2.7.3, so hopefully we'll 
have a plan for that soon.
+1 (binding) for 2.7.2 RC2.
Jason


  From: Vinod Kumar Vavilapalli 
 To: mapreduce-...@hadoop.apache.org; Jason Lowe  
Cc: Hadoop Common ; "hdfs-dev@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 

 Sent: Tuesday, January 19, 2016 5:25 PM
 Subject: Re: [VOTE] Release Apache Hadoop 2.7.2 RC2
   
The JIRA YARN-4610 links YARN-3434 as the one causing the breakage, and 
YARN-3434 already exists in 2.7.1 itself. That categorizes the new issue as an 
existing bug.
If you agree with that sentiment, and given that there is a clear work-around, 
in the interest of progress of 2.7.2 (we have spent > 2 months on this now), 
I’d like to move forward.
Please LMK what you think.
Thanks+Vinod


On Jan 19, 2016, at 3:13 PM, Jason Lowe  wrote:
-1 (binding)
We have been running a release derived from 2.7 on some of our clusters, and we 
recently hit a bug where an application making large container requests can 
drastically slow down container allocations for other users in the same queue.  
See YARN-4610 for details.  Since 
yarn.scheduler.capacity.reservations-continue-look-all-nodes is on by default, 
I think we should fix this.  If we decide to ship 2.7.2 without that fix then 
the release notes should call out that JIRA and mention the workaround of 
setting yarn.scheduler.capacity.reservations-continue-look-all-nodes to false.
Jason


  From: Vinod Kumar Vavilapalli 
 To: Hadoop Common ; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
 Sent: Thursday, January 14, 2016 10:57 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2

Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
<http://repository.apache.org/> at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 
<https://repository.apache.org/content/repositories/orgapachehadoop-1027>

The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
<http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for your 
quick perusal.

As you may have noted,
 - I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 
<http://markmail.org/message/oozq3gvd4nhzsaes>
[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 
<http://markmail.org/message/iktqss2qdeykgpqk>
[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 
<http://markmail.org/message/5txhvr2qdiqglrwc>
[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw





  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC2

[Jason Lowe]

-1 (binding)
We have been running a release derived from 2.7 on some of our clusters, and we 
recently hit a bug where an application making large container requests can 
drastically slow down container allocations for other users in the same queue.  
See YARN-4610 for details.  Since 
yarn.scheduler.capacity.reservations-continue-look-all-nodes is on by default, 
I think we should fix this.  If we decide to ship 2.7.2 without that fix then 
the release notes should call out that JIRA and mention the workaround of 
setting yarn.scheduler.capacity.reservations-continue-look-all-nodes to false.
Jason


  From: Vinod Kumar Vavilapalli 
 To: Hadoop Common ; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
 Sent: Thursday, January 14, 2016 10:57 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC2
   
Hi all,

I've created an updated release candidate RC2 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/

The RC tag in git is: release-2.7.2-RC2

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1027 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC2/releasenotes.html 
 for your 
quick perusal.

As you may have noted,
 - I terminated the RC1 related voting thread after finding out that we didn’t 
have a bunch of patches that are already in the released 2.6.3 version. After a 
brief discussion, we decided to keep the parallel 2.6.x and 2.7.x releases 
incremental, see [4] for this discussion.
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 

[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 

[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc 

[4] Retracted [VOTE] Release Apache Hadoop 2.7.2 RC1: 
http://markmail.org/thread/n7ljbsnquihn3wlw

  

Re: [VOTE] Release Apache Hadoop 2.7.2 RC1

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Spot checked CHANGES.txt files- Successfully 
performed a native build from source- Deployed to a single node cluster and ran 
sample jobs
We have been running with the fix for YARN-4354 on two of our clusters for some 
time with no issues, so I feel confident that prior blocker is now fixed.
Jason
 

  From: Vinod Kumar Vavilapalli 
 To: Hadoop Common ; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
Cc: Vinod Kumar Vavilapalli 
 Sent: Wednesday, December 16, 2015 8:49 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC1
   
Hi all,

I've created a release candidate RC1 for Apache Hadoop 2.7.2.

As discussed before, this is the next maintenance release to follow up 2.7.1.

The RC is available for validation at: 
http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/ 


The RC tag in git is: release-2.7.2-RC1

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1026/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html 
for quick perusal.

As you may have noted,
 - The RC0 related voting thread got halted due to some critical issues. It 
took a while again for getting all those blockers out of the way. See the 
previous voting thread [3] for details.
 - Before RC0, an unusually long 2.6.3 release caused 2.7.2 to slip by quite a 
bit. This release's related discussion threads are linked below: [1] and [2].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes 

[2]: Planning Apache Hadoop 2.7.2 http://markmail.org/message/iktqss2qdeykgpqk 

[3]: [VOTE] Release Apache Hadoop 2.7.2 RC0: 
http://markmail.org/message/5txhvr2qdiqglrwc


   

Re: [VOTE] Release Apache Hadoop 2.6.3 RC0

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Successfully built from source with native 
code support- Deployed to a single-node cluster and ran some test jobs
Jason

  From: Junping Du 
 To: Hadoop Common ; "hdfs-dev@hadoop.apache.org" 
; "mapreduce-...@hadoop.apache.org" 
; "yarn-...@hadoop.apache.org" 
 
Cc: "junping...@apache.org" 
 Sent: Friday, December 11, 2015 6:16 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.3 RC0
   

Hi all developers in hadoop community,
  I've created a release candidate RC0 for Apache Hadoop 2.6.3 (the next 
maintenance release to follow up 2.6.2.) according to email thread of release 
plan 2.6.3 [1]. Sorry for this RC coming a bit late as several blocker issues 
were getting committed until yesterday. Below is the details:

The RC is available for validation at:
*http://people.apache.org/~junping_du/hadoop-2.6.3-RC0/
*

The RC tag in git is: release-2.6.3-RC0

The maven artifacts are staged via repository.apache.org at:
*https://repository.apache.org/content/repositories/orgapachehadoop-1025/?
*

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the usual 5 days.

Thanks and happy weekend!


Cheers,

Junping


[1]: 2.6.3 release plan: http://markmail.org/thread/nc2jogbgni37vu6y


 

Re: [VOTE] Release Apache Hadoop 2.7.2 RC0

[Jason Lowe]

-1 (binding)
Ran into public localization issues and filed YARN-4354. We need that resolved 
before the release is ready.  We will either need a timely fix or may have to 
revert YARN-2902 to unblock the release if my root-cause analysis is correct.  
I'll dig into this more today.

Jason

  From: Vinod Kumar Vavilapalli 
 To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Wednesday, November 11, 2015 10:31 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.2 RC0
   
Hi all,


I've created a release candidate RC0 for Apache Hadoop 2.7.2.


As discussed before, this is the next maintenance release to follow up
2.7.1.


The RC is available for validation at:

*http://people.apache.org/~vinodkv/hadoop-2.7.2-RC0/

*


The RC tag in git is: release-2.7.2-RC0


The maven artifacts are available via repository.apache.org at

*https://repository.apache.org/content/repositories/orgapachehadoop-1023/

*


As you may have noted, an unusually long 2.6.3 release caused 2.7.2 to slip
by quite a bit. This release's related discussion threads are linked below:
[1] and [2].


Please try the release and vote; the vote will run for the usual 5 days.


Thanks,

Vinod


[1]: 2.7.2 release plan: http://markmail.org/message/oozq3gvd4nhzsaes

[2]: Planning Apache Hadoop 2.7.2
http://markmail.org/message/iktqss2qdeykgpqk


  

Re: [VOTE] Release Apache Hadoop 2.6.2

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Performed native build from source- Deployed 
a single-node cluster and ran some test jobs

Jason
  From: Sangjin Lee 
 To: "common-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" ; 
"hdfs-dev@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli  
 Sent: Thursday, October 22, 2015 4:14 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.2
   
Hi all,

I have created a release candidate (RC0) for Hadoop 2.6.2.

The RC is available at: http://people.apache.org/~sjlee/hadoop-2.6.2-RC0/

The RC tag in git is: release-2.6.2-RC0

The list of JIRAs committed for 2.6.2:
https://issues.apache.org/jira/browse/YARN-4101?jql=project%20in%20(HADOOP%2C%20HDFS%2C%20YARN%2C%20MAPREDUCE)%20AND%20fixVersion%20%3D%202.6.2

The maven artifacts are staged at
https://repository.apache.org/content/repositories/orgapachehadoop-1022/

Please try out the release candidate and vote. The vote will run for 5 days.

Thanks,
Sangjin


   

Re: [VOTE] Release Apache Hadoop 2.7.1 RC0

[Jason Lowe]

+1 (binding)
- Verified signatures and digests
- Successfully performed a native build from source- Deployed a single-node 
cluster- Ran sample MapReduce jobs

Jason
  From: Vinod Kumar Vavilapalli 
 To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Monday, June 29, 2015 3:45 AM
 Subject: [VOTE] Release Apache Hadoop 2.7.1 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.1.

As discussed before, this is the next stable release to follow up 2.6.0,
and the first stable one in the 2.7.x line.

The RC is available for validation at:
*http://people.apache.org/~vinodkv/hadoop-2.7.1-RC0/
*

The RC tag in git is: release-2.7.1-RC0

The maven artifacts are available via repository.apache.org at
*https://repository.apache.org/content/repositories/orgapachehadoop-1019/
*

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

PS: It took 2 months instead of the planned [1] 2 weeks in getting this
release out: post-mortem in a separate thread.

[1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw


   

Re: [VOTE] Release Apache Hadoop 2.7.0 RC0

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Built from source with native support- 
Deployed to a single-node cluster and ran sample jobs
Jason

  From: Vinod Kumar Vavilapalli 
 To: common-...@hadoop.apache.org; hdfs-dev@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org 
Cc: vino...@apache.org 
 Sent: Friday, April 10, 2015 6:44 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.0 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.0.

 The RC is available at: http://people.apache.org/~vinodkv/hadoop-2.7.0-RC0/

The RC tag in git is: release-2.7.0-RC0

 The maven artifacts are available via repository.apache.org at
https://repository.apache.org/content/repositories/orgapachehadoop-1017/

As discussed before
 - This release will only work with JDK 1.7 and above
 - I’d like to use this as a starting release for 2.7.x [1], depending on
how it goes, get it stabilized and potentially use a 2.7.1 in a few
weeks as the stable release.

 Please try the release and vote; the vote will run for the usual 5 days.

 Thanks,
 Vinod

 [1]: A 2.7.1 release to follow up 2.7.0
http://markmail.org/thread/zwzze6cqqgwq4rmw

  

Re: Looking to a Hadoop 3 release

[Jason Lowe]

I'm OK with a 3.0.0 release as long as we are minimizing the pain of 
maintaining yet another release line and conscious of the incompatibilities 
going into that release line.
For the former, I would really rather not see a branch-3 cut so soon.  It's yet 
another line onto which to cherry-pick, and I don't see why we need to add this 
overhead at such an early phase.  We should only create branch-3 when there's 
an incompatible change that the community wants and it should _not_ go into the 
next major release (i.e.: it's for Hadoop 4.0).  We can develop 3.0 alphas and 
betas on trunk and release from trunk in the interim.  IMHO we need to stop 
treating trunk as a place to exile patches.

For the latter, I think as a community we need to evaluate the benefits of 
breaking compatibility against the costs of migrating.  Each time we break 
compatibility we create a hurdle for people to jump when they move to the new 
release, and we should make those hurdles worth their time.  For example, 
wire-compatibility has been mentioned as part of this.  Any feature that breaks 
wire compatibility better be absolutely amazing, as it creates a huge hurdle 
for people to jump.
To summarize:+1 for a community-discussed roadmap of what we're breaking in 
Hadoop 3 and why it's worth it for users
-1 for creating branch-3 now, we can release from trunk until the next 
incompatibility for Hadoop 4 arrives
+1 for baking classpath isolation as opt-in on 2.x and eventually default on in 
3.0
Jason
  From: Andrew Wang 
 To: "hdfs-dev@hadoop.apache.org"  
Cc: "common-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org"  
 Sent: Wednesday, March 4, 2015 12:15 PM
 Subject: Re: Looking to a Hadoop 3 release
   
Let's not dismiss this quite so handily.

Sean, Jason, and Stack replied on HADOOP-11656 pointing out that while we
could make classpath isolation opt-in via configuration, what we really
want longer term is to have it on by default (or just always on). Stack in
particular points out the practical difficulties in using an opt-in method
in 2.x from a downstream project perspective. It's not pretty.

The plan that both Sean and Jason propose (which I support) is to have an
opt-in solution in 2.x, bake it there, then turn it on by default
(incompatible) in a new major release. I think this lines up well with my
proposal of some alphas and betas leading up to a GA 3.x. I'm also willing
to help with 2.x release management if that would help with testing this
feature.

Even setting aside classpath isolation, a new major release is still
justified by JDK8. Somehow this is being ignored in the discussion. Allen,
historically the voice of the user in our community, just highlighted it as
a major compatibility issue, and myself and Tucu have also expressed our
very strong concerns about bumping this in a minor release. 2.7's bump is a
unique exception, but this is not something to be cited as precedent or
policy.

Where does this resistance to a new major release stem from? As I've
described from the beginning, this will look basically like a 2.x release,
except for the inclusion of classpath isolation by default and target
version JDK8. I've expressed my desire to maintain API and wire
compatibility, and we can audit the set of incompatible changes in trunk to
ensure this. My proposal for doing alpha and beta releases leading up to GA
also gives downstreams a nice amount of time for testing and validation.

Regards,
Andrew



On Tue, Mar 3, 2015 at 2:32 PM, Arun Murthy  wrote:

> Awesome, looks like we can just do this in a compatible manner - nothing
> else on the list seems like it warrants a (premature) major release.
>
> Thanks Vinod.
>
> Arun
>
> 
> From: Vinod Kumar Vavilapalli 
> Sent: Tuesday, March 03, 2015 2:30 PM
> To: common-...@hadoop.apache.org
> Cc: hdfs-dev@hadoop.apache.org; mapreduce-...@hadoop.apache.org;
> yarn-...@hadoop.apache.org
> Subject: Re: Looking to a Hadoop 3 release
>
> I started pitching in more on that JIRA.
>
> To add, I think we can and should strive for doing this in a compatible
> manner, whatever the approach. Marking and calling it incompatible before
> we see proposal/patch seems premature to me. Commented the same on JIRA:
> https://issues.apache.org/jira/browse/HADOOP-11656?focusedCommentId=14345875&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14345875
> .
>
> Thanks
> +Vinod
>
> On Mar 2, 2015, at 8:08 PM, Andrew Wang  andrew.w...@cloudera.com>> wrote:
>
> Regarding classpath isolation, based on what I hear from our customers,
> it's still a big problem (even after the MR classloader work). The latest
> Jackson version bump was quite painful for our downstream projects, and the
> HDFS client still leaks a lot of dependencies. Would welcome more
> discussion of this on HADOOP-11656, Steve, Colin, and Haohui have already
> chimed in.
>
>


  

[jira] [Created] (HDFS-7816) Unable to open webhdfs paths with escape characters

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-7816:


 Summary: Unable to open webhdfs paths with escape characters
 Key: HDFS-7816
 URL: https://issues.apache.org/jira/browse/HDFS-7816
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: webhdfs
Affects Versions: 2.7.0
Reporter: Jason Lowe
Priority: Blocker


webhdfs requests to open files with % characters in the filename fail because 
the filename is not being decoded properly.  For example:

$ hadoop fs -cat 'webhdfs://nn/user/somebody/abc%def'
cat: File does not exist: /user/somebody/abc%25def




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.6.0

[Jason Lowe]

+1 (binding)
- verified signatures and digests- verified late-arriving fixes for YARN-2846 
and MAPREDUCE-6156 were present
- built from source- deployed to a single-node cluster 
- ran some sample MapReduce jobs
Jason
  From: Arun C Murthy 
 To: "common-...@hadoop.apache.org" ; 
"hdfs-dev@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" ; 
"mapreduce-...@hadoop.apache.org"  
 Sent: Thursday, November 13, 2014 5:08 PM
 Subject: [VOTE] Release Apache Hadoop 2.6.0 
   
Folks,

I've created another release candidate (rc1) for hadoop-2.6.0 based on the 
feedback.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.6.0-rc1
The RC tag in git is: release-2.6.0-rc1

The maven artifacts are available via repository.apache.org at 
https://repository.apache.org/content/repositories/orgapachehadoop-1013.

Please try the release and vote; the vote will run for the usual 5 days.

thanks,
Arun


-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.


   

Re: [VOTE] Release Apache Hadoop 2.6.0

[Jason Lowe]

I just committed 2.6 blockes YARN-2846 and MAPREDUCE-6156 which should also be 
in the 2.6.0 rc1 build.
Jason
  From: Arun C Murthy 
 To: yarn-...@hadoop.apache.org 
Cc: mapreduce-...@hadoop.apache.org; Ravi Prakash ; 
"hdfs-dev@hadoop.apache.org" ; 
"common-...@hadoop.apache.org"  
 Sent: Wednesday, November 12, 2014 10:58 AM
 Subject: Re: [VOTE] Release Apache Hadoop 2.6.0
   
Sounds good. I'll create an rc1. Thanks.

Arun

On Nov 11, 2014, at 2:06 PM, Robert Kanter  wrote:

> Hi Arun,
> 
> We were testing the RC and ran into a problem with the recent fixes that
> were done for POODLE for Tomcat (HADOOP-11217 for KMS and HDFS-7274 for
> HttpFS).  Basically, in disabling SSLv3, we also disabled SSLv2Hello, which
> is required for older clients (e.g. Java 6 with openssl 0.9.8x) so they
> can't connect without it.  Just to be clear, it does not mean SSLv2, which
> is insecure.  This also affects the MR shuffle in HADOOP-11243.
> 
> The fix is super simple, so I think we should reopen these 3 JIRAs and put
> in addendum patches and get them into 2.6.0.
> 
> thanks
> - Robert
> 
> On Tue, Nov 11, 2014 at 1:04 PM, Ravi Prakash  wrote:
> 
>> Hi Arun!
>> We are very close to completion on YARN-1964 (DockerContainerExecutor).
>> I'd also like HDFS-4882 to be checked in. Do you think these issues merit
>> another RC?
>> ThanksRavi
>> 
>> 
>>    On Tuesday, November 11, 2014 11:57 AM, Steve Loughran <
>> ste...@hortonworks.com> wrote:
>> 
>> 
>> +1 binding
>> 
>> -patched slider pom to build against 2.6.0
>> 
>> -verified build did download, which it did at up to ~8Mbps. Faster than a
>> local build.
>> 
>> -full clean test runs on OS/X & Linux
>> 
>> 
>> Windows 2012:
>> 
>> Same thing. I did have to first build my own set of the windows native
>> binaries, by checking out branch-2.6.0; doing a native build, copying the
>> binaries and then purging the local m2 repository of hadoop artifacts to be
>> confident I was building against. For anyone who wants those native libs
>> they will be up on
>> https://github.com/apache/incubator-slider/tree/develop/bin/windows/ once
>> it syncs with the ASF repos.
>> 
>> afterwords: the tests worked!
>> 
>> 
>> On 11 November 2014 02:52, Arun C Murthy  wrote:
>> 
>>> Folks,
>>> 
>>> I've created a release candidate (rc0) for hadoop-2.6.0 that I would like
>>> to see released.
>>> 
>>> The RC is available at:
>>> http://people.apache.org/~acmurthy/hadoop-2.6.0-rc0
>>> The RC tag in git is: release-2.6.0-rc0
>>> 
>>> The maven artifacts are available via repository.apache.org at
>>> https://repository.apache.org/content/repositories/orgapachehadoop-1012.
>>> 
>>> Please try the release and vote; the vote will run for the usual 5 days.
>>> 
>>> thanks,
>>> Arun
>>> 
>>> 
>>> --
>>> CONFIDENTIALITY NOTICE
>>> NOTICE: This message is intended for the use of the individual or entity
>> to
>>> which it is addressed and may contain information that is confidential,
>>> privileged and exempt from disclosure under applicable law. If the reader
>>> of this message is not the intended recipient, you are hereby notified
>> that
>>> any printing, copying, dissemination, distribution, disclosure or
>>> forwarding of this communication is strictly prohibited. If you have
>>> received this communication in error, please contact the sender
>> immediately
>>> and delete it from your system. Thank You.
>>> 
>> 
>> --
>> CONFIDENTIALITY NOTICE
>> NOTICE: This message is intended for the use of the individual or entity to
>> which it is addressed and may contain information that is confidential,
>> privileged and exempt from disclosure under applicable law. If the reader
>> of this message is not the intended recipient, you are hereby notified that
>> any printing, copying, dissemination, distribution, disclosure or
>> forwarding of this communication is strictly prohibited. If you have
>> received this communication in error, please contact the sender immediately
>> and delete it from your system. Thank You.
>> 
>> 
>> 
>> 

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/





-- 
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to 
which it is addressed and may contain information that is confidential, 
privileged and exempt from disclosure under applicable law. If the reader 
of this message is not the intended recipient, you are hereby notified that 
any printing, copying, dissemination, distribution, disclosure or 
forwarding of this communication is strictly prohibited. If you have 
received this communication in error, please contact the sender immediately 
and delete it from your system. Thank You.


  

[jira] [Created] (HDFS-7199) DFSOutputStream can silently drop data if DataStreamer crashes with a non-I/O exception

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-7199:


 Summary: DFSOutputStream can silently drop data if DataStreamer 
crashes with a non-I/O exception
 Key: HDFS-7199
 URL: https://issues.apache.org/jira/browse/HDFS-7199
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs-client
Affects Versions: 2.5.0
Reporter: Jason Lowe
Priority: Critical


If the DataStreamer thread encounters a non-I/O exception then it closes the 
output stream but does not set lastException.  When the client later calls 
close on the output stream then it will see the stream is already closed with 
lastException == null, mistakently think this is a redundant close call, and 
fail to report any error to the client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.5.1 RC0

[Jason Lowe]

+1 (binding)

- verified signatures and digests
- built from source
- examined CHANGES.txt for items fixed in 2.5.1
- deployed to a single-node cluster and ran some sample MR jobs

Jason

On 09/05/2014 07:18 PM, Karthik Kambatla wrote:

Hi folks,

I have put together a release candidate (RC0) for Hadoop 2.5.1.

The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.1-RC0/
The RC git tag is release-2.5.1-RC0
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1010/

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the now usual 5
days.

Thanks
Karthik

[jira] [Resolved] (HDFS-6907) Source files missing license headers

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-6907.
--

Resolution: Duplicate

Dup of HDFS-6905.

> Source files missing license headers
> 
>
> Key: HDFS-6907
> URL: https://issues.apache.org/jira/browse/HDFS-6907
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0, 2.6.0
>Reporter: Arpit Agarwal
>
> The following files were committed without license headers as flagged by 
> Jenkins.
> {code}
>  !? 
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionFaultInjector.java
>  !? 
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/EncryptionZoneManager.java
>  !? 
> /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocol/EncryptionZoneWithId.java
> Lines that start with ? in the release audit report indicate files that 
> do not have an Apache license header.
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (HDFS-6840) Clients are always sent to the same datanode when read is off rack

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-6840:


 Summary: Clients are always sent to the same datanode when read is 
off rack
 Key: HDFS-6840
 URL: https://issues.apache.org/jira/browse/HDFS-6840
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Jason Lowe
Priority: Critical


After HDFS-6268 the sorting order of block locations is deterministic for a 
given block and locality level (e.g.: local, rack. off-rack), so off-rack 
clients all see the same datanode for the same block.  This leads to very poor 
behavior in distributed cache localization and other scenarios where many 
clients all want the same block data at approximately the same time.  The one 
datanode is crushed by the load while the other replicas only handle local and 
rack-local requests.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.5.0 RC2

[Jason Lowe]

+1 (binding)

- verified signatures and digests
- built from source
- deployed a single-node cluster
- ran some sample jobs

Jason

On 08/06/2014 03:59 PM, Karthik Kambatla wrote:

Hi folks,

I have put together a release candidate (rc2) for Hadoop 2.5.0.

The RC is available at: http://people.apache.org/~kasha/hadoop-2.5.0-RC2/
The RC tag in svn is here:
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.5.0-rc2/
The maven artifacts are staged at:
https://repository.apache.org/content/repositories/orgapachehadoop-1009/

You can find my public key at:
http://svn.apache.org/repos/asf/hadoop/common/dist/KEYS

Please try the release and vote. The vote will run for the now usual 5
days.

Thanks

Re: [VOTE] Migration from subversion to git for version control

[Jason Lowe]

+1

Jason

On 08/08/2014 09:57 PM, Karthik Kambatla wrote:

I have put together this proposal based on recent discussion on this topic.

Please vote on the proposal. The vote runs for 7 days.

1. Migrate from subversion to git for version control.
2. Force-push to be disabled on trunk and branch-* branches. Applying
changes from any of trunk/branch-* to any of branch-* should be through
"git cherry-pick -x".
3. Force-push on feature-branches is allowed. Before pulling in a
feature, the feature-branch should be rebased on latest trunk and the
changes applied to trunk through "git rebase --onto" or "git cherry-pick
".
4. Every time a feature branch is rebased on trunk, a tag that
identifies the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
the feature is pulled into trunk and the tags are no longer useful.
5. The relevance/use of tags stay the same after the migration.

Thanks
Karthik

PS: Per Andrew Wang, this should be a "Adoption of New Codebase" kind of
vote and will be Lazy 2/3 majority of PMC members.

Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated

[Jason Lowe]

I think that's a reasonable proposal as long as we understand it changes 
the burden from finding all the things that should be marked @Private to 
finding all the things that should be marked @Public. As Tom Graves 
pointed out in an earlier discussion about @LimitedPrivate, it may be 
impossible to do a straightforward task and use only interfaces marked 
@Public.  If users can't do basic things without straying from @Public 
interfaces then tons of code can break if we assume it's always fair 
game to change anything not marked @Public.  The "well you shouldn't 
have used a non-@Public interface" argument is not very useful in that 
context.


So as long as we're good about making sure officially supported features 
have corresponding @Public interfaces to wield them then I agree it will 
be easier to track those rather than track all the classes that should 
be @Private.  Hopefully if users understand that's how things work 
they'll help file JIRAs for interfaces that need to be @Public to get 
their work done.


Jason

On 07/22/2014 04:54 PM, Karthik Kambatla wrote:

Hi devs

As you might have noticed, we have several classes and methods in them that
are not annotated at all. This is seldom intentional. Avoiding incompatible
changes to all these classes can be considerable baggage.

I was wondering if we should add an explicit disclaimer in our
compatibility guide that says, "Classes without annotations are to
considered @Private"

For methods, is it reasonable to say - "Class members without specific
annotations inherit the annotations of the class"?

Thanks
Karthik

Re: [VOTE] Release Apache Hadoop 2.4.1

[Jason Lowe]

+1

- Verified signatures and digests
- Built from source, installed on single-node cluster and ran some 
sample jobs


Jason

On 06/21/2014 01:51 AM, Arun C Murthy wrote:

Folks,

I've created another release candidate (rc1) for hadoop-2.4.1 based on the 
feedback that I would like to push out.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1
The RC tag in svn is here: 
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun



--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/hdp/

Re: [VOTE] Release Apache Hadoop 0.23.11

[Jason Lowe]

+1 (binding)

- Verified signatures and digests
- Deployed binary tarball to a single-node cluster and ran some MR 
example jobs
- Built from source, deployed to a single-node cluster and ran some MR 
example jobs


Jason

On 06/19/2014 10:14 AM, Thomas Graves wrote:

Hey Everyone,

There have been various bug fixes that have went into
branch-0.23 since the 0.23.10 release.  We think its time to do a 0.23.11.

This is also the last planned release off of branch-0.23 we plan on doing.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.11-candidate-0/


The RC Tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.11-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days
til June 26th.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Jason Lowe]

+1 (binding)

Jason

On 06/24/2014 03:53 AM, Arun C Murthy wrote:

Folks,

  As discussed, I'd like to call a vote on changing our by-laws to change 
release votes from 7 days to 5.

  I've attached the change to by-laws I'm proposing.

  Please vote, the vote will the usual period of 7 days.

thanks,
Arun



[main]$ svn diff
Index: author/src/documentation/content/xdocs/bylaws.xml
===
--- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
+++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
@@ -344,7 +344,16 @@
  Votes are open for a period of 7 days to allow all active
  voters time to consider the vote. Votes relating to code
  changes are not subject to a strict timetable but should be
-made as timely as possible.
+made as timely as possible.
+
+ 
+  Product Release - Vote Timeframe
+   Release votes, alone, run for a period of 5 days. All other
+ votes are subject to the above timeframe of 7 days.
+ 
+   
+   
+
 
 
  

[jira] [Created] (HDFS-6421) RHEL4 fails to compile vecsum.c

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-6421:


 Summary: RHEL4 fails to compile vecsum.c
 Key: HDFS-6421
 URL: https://issues.apache.org/jira/browse/HDFS-6421
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: libhdfs
Affects Versions: 2.5.0
 Environment: RHEL4
Reporter: Jason Lowe


After HDFS-6287 RHEL4 builds fail trying to compile vecsum.c since they don't 
have RUSAGE_THREAD.  RHEL4 is ancient, but we use it in a 32-bit compatibility 
environment.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.4.0

[Jason Lowe]

Here's my late +1, was just finishing up looking at the release.

- Verified signatures and digests
- Examined LICENSE file
- Installed binary distribution, ran some sample MapReduce jobs and 
examined logs and job history

- Built from source

Jason

On 04/07/2014 03:04 PM, Arun C Murthy wrote:

With 11 +1s (4 binding) and no -1s the vote passes. Thanks to everyone who 
tried out the release and passed their feedback along.

I'll send a note out once I actually get the bits out and the site updated etc.

thanks,
Arun

On Mar 31, 2014, at 2:22 AM, Arun C Murthy  wrote:


Folks,

I've created a release candidate (rc0) for hadoop-2.4.0 that I would like to 
get released.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.4.0-rc0
The RC tag in svn is here: 
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.0-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun


--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

[jira] [Resolved] (HDFS-6021) NPE in FSImageFormatProtobuf upgrading from layout -52 to -53

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-6021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-6021.
--

Resolution: Duplicate

Resolving as a dup of HDFS-5988 since that seems like the most likely culprit.  
I'll reopen if it occurs again.  Thanks Andrew!

> NPE in FSImageFormatProtobuf upgrading from layout -52 to -53
> -
>
> Key: HDFS-6021
> URL: https://issues.apache.org/jira/browse/HDFS-6021
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Jason Lowe
>
> While updating my trunk instance the namenode refused to startup because the 
> layout version needed to be upgraded.  When attempting the upgrade ran into 
> an NPE in FSImageFormatProtobuf.  Full stacktrace to follow.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

[jira] [Created] (HDFS-6021) NPE in FSImageFormatProtobuf upgrading from layout -52 to -53

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-6021:


 Summary: NPE in FSImageFormatProtobuf upgrading from layout -52 to 
-53
 Key: HDFS-6021
 URL: https://issues.apache.org/jira/browse/HDFS-6021
 Project: Hadoop HDFS
  Issue Type: Bug
Affects Versions: 3.0.0
Reporter: Jason Lowe


While updating my trunk instance the namenode refused to startup because the 
layout version needed to be upgraded.  When attempting the upgrade ran into an 
NPE in FSImageFormatProtobuf.  Full stacktrace to follow.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: [VOTE] Release Apache Hadoop 2.3.0

[Jason Lowe]

+1 (binding)

- Verified signatures and digests
- Built from source with native support
- Deployed a single-node cluster and ran some sample jobs

Jason

On 02/11/2014 08:49 AM, Arun C Murthy wrote:

Folks,

I've created a release candidate (rc0) for hadoop-2.3.0 that I would like to 
get released.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
The RC tag in svn is here: 
https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

PS: Thanks to Andrew, Vinod & Alejandro for all their help in various release 
activities.

Re: Re-swizzle 2.3

[Jason Lowe]

I noticed that somehow the target version field in JIRA was invisibly 
cleared on most of the Blocker/Critical JIRAs that were originally 
targeted for 2.3.0/2.4.0.  I happened to have an old browser tab lying 
around from an earlier query for these and I tried to fix them up, 
marking some for 2.4.0 that IMHO weren't show-stoppers for the 2.3.0 
release.  It is a bit concerning that the JIRA history showed that the 
target version was set at some point in the past but no record of it 
being cleared.


Jason

On 01/29/2014 07:58 AM, Arun C Murthy wrote:

Mostly ready for a jira perspective.

Committers - Henceforth, please use extreme caution while committing to 
branch-2.3. Please commit *only* blockers to 2.3.

thanks,
Arun

On Jan 28, 2014, at 3:30 PM, Arun C Murthy  wrote:


Fixing up stuff now, thanks to Andrew for volunteering to help with Common/HDFS.

Arun

On Jan 28, 2014, at 10:54 AM, Arun C Murthy  wrote:


Sorry, missed this. Go ahead, I'll fix things up at the back end. Thanks.

On Jan 28, 2014, at 12:11 AM, Sandy Ryza  wrote:


Going forward with commits because it seems like others have been doing so


On Mon, Jan 27, 2014 at 1:31 PM, Sandy Ryza  wrote:


We should hold off commits until that's done, right?


On Mon, Jan 27, 2014 at 1:07 PM, Arun C Murthy wrote:


Yep, on it as we speak. :)


Arun

On Jan 27, 2014, at 12:36 PM, Jason Lowe  wrote:


Thanks, Arun.  Are there plans to update the Fix Versions and

CHANGES.txt accordingly?  There are a lot of JIRAs that are now going to
ship in 2.3.0 but the JIRA and CHANGES.txt says they're not fixed until
2.4.0.

Jason

On 01/27/2014 08:47 AM, Arun C Murthy wrote:

Done. I've re-created branch-2.3 from branch-2.

thanks,
Arun

On Jan 23, 2014, at 2:40 AM, Arun Murthy  wrote:


Based on the discussion at common-dev@, we've decided to target 2.3
off the tip of branch-2 based on the 2 major HDFS features which are
Heterogenous Storage (HDFS-2832) and HDFS Cache (HDFS-4949).

I'll create a new branch-2.3 on (1/24) at 6pm PST.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/




--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity
to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.




--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/



--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: Re-swizzle 2.3

[Jason Lowe]

Thanks, Arun.  Are there plans to update the Fix Versions and 
CHANGES.txt accordingly?  There are a lot of JIRAs that are now going to 
ship in 2.3.0 but the JIRA and CHANGES.txt says they're not fixed until 
2.4.0.


Jason

On 01/27/2014 08:47 AM, Arun C Murthy wrote:

Done. I've re-created branch-2.3 from branch-2.

thanks,
Arun

On Jan 23, 2014, at 2:40 AM, Arun Murthy  wrote:


Based on the discussion at common-dev@, we've decided to target 2.3
off the tip of branch-2 based on the 2 major HDFS features which are
Heterogenous Storage (HDFS-2832) and HDFS Cache (HDFS-4949).

I'll create a new branch-2.3 on (1/24) at 6pm PST.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 0.23.10

[Jason Lowe]

+1 (binding)

- verified signatures and digests
- deployed binary tarball to a single-node cluster and ran some jobs
- built from source
- deployed source build to a single-node cluster and ran some jobs

Jason

On 12/03/2013 12:22 AM, Thomas Graves wrote:

Hey Everyone,

There have been lots of improvements and bug fixes that have went into
branch-0.23 since the 0.23.9 release.  We think its time to do a 0.23.10
so I have created a release candidate (rc0) for a Hadoop-0.23.10 release.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.10-rc0/


The RC Tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.10-rc0/


The maven artifacts are available via repository.apache.org.


Please try the release and vote; the vote will run for the usual 7 days
til December 9th.

I am +1 (binding).

thanks,
Tom Graves

Re: Next releases

[Jason Lowe]

I think a lot of confusion comes from the fact that the 2.x line is 
starting to mature.  Before this there wasn't such a big contention of 
what went into patch vs. minor releases and often the lines were blurred 
between the two.  However now we have significant customers and products 
starting to use 2.x as a base, which means we need to start treating it 
like we treat 1.x.  That means getting serious about what we should put 
into a patch release vs. what we postpone to a minor release.


Here's my $0.02 on recent proposals:

+1 to releasing more often in general.  A lot of the rush to put changes 
into a patch release is because it can be a very long time between any 
kind of release.  If minor releases are more frequent then I hope there 
would be less of a need to rush something or hold up a release.


+1 to limiting checkins of patch releases to Blockers/Criticals.  If 
necessary committers check into trunk/branch-2 only and defer to the 
patch release manager for the patch release merge.  Then there should be 
fewer surprises for everyone what ended up in a patch release and less 
likely the patch release becomes destabilized from the sheer amount of 
code churn.  Maybe this won't be necessary if everyone understands that 
the patch release isn't the only way to get a change out in timely manner.


As for 2.2.1, again I think it's expectations for what that release 
means.  If it's really just a patch release then there shouldn't be 
features in it and tons of code churn, but I think many were treating it 
as the next vehicle to deliver changes in general.  If we think 2.2.1 is 
just as good or better than 2.2.0 then let's wrap it up and move to a 
more disciplined approach for subsequent patch releases and more 
frequent minor releases.


Jason

On 11/13/2013 12:10 PM, Arun C Murthy wrote:

On Nov 12, 2013, at 1:54 PM, Todd Lipcon  wrote:


On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe wrote:


To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
there.  However, I have only been following the HDFS and common side
of things so I may not have the full picture.  Arun, can you give a
specific example of something you'd like to "blow away"?

There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
similarly in HDFS a cursory glance showed up some *enhancements*/*improvements* 
in CHANGES.txt which aren't necessary for a patch release, plus things like:

HADOOP-9623 
Update jets3t dependency to 0.9.0







  


Having said that, the HDFS devs know their code the best.


I agree with Colin. If we've been backporting things into a patch release
(third version component) which don't belong, we should explicitly call out
those patches, so we can learn from our mistakes and have a discussion
about what belongs.

Good point.

Here is a straw man proposal:


A patch (third version) release should only include *blocker* bugs which are 
critical from an operational, security or data-integrity issues.

This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
is always release-able, and more importantly, deploy-able at any point in time.



Sandy did bring up a related point about timing of releases and the urge for 
everyone to cram features/fixes into a dot release.

So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 2.4 
etc.) and keep the patch releases limited to blocker bugs.

Thoughts?

thanks,
Arun

Re: test-patch failing with OOM errors in javah

[Jason Lowe]

Pinged Bobby who has access to the machines, and this is indeed what is 
happening.  There were two cases:


1) When TestUberAM times out the minicluster it is running along with 
its MRAppMaster process can escape.  There are a ton of threads in those 
processes, so it doesn't take very many of these to leak before the 
process ulimit is hit.


2) There were a couple of other surefire processes that had leaked but 
they had no discernable state left that would identify which test it was 
running other than it was something inside of mapreduce-client-jobclient 
(which could still be TestUberAM).  The main thread and most other 
non-daemon threads were gone, but there was a lone SocketReader thread 
that was still hanging around.  It wasn't a daemon thread and was 
apparently the only thread keeping the JVM alive.


So we need to prioritize fixing the TestUberAM hang, currently tracked 
by MAPREDUCE-5481 <https://issues.apache.org/jira/browse/MAPREDUCE-5481> 
and/or find a way to keep it from escaping during builds.  There might 
be another issue where SocketReader threads can prevent the JVM from 
shutting down completely in some cases.


Jason

On 10/31/2013 08:19 AM, Jason Lowe wrote:
I don't think that OOM error below indicates it needs more heap space, 
as it's complaining about the ability to create a new native thread.  
That usually is caused by lack of available virtual address space or 
hitting process ulimits.


What's most likely going on is the jenkins user is hitting a process 
ulimit.  This can occur if processes have "leaked" from previous 
build/test runs and are using a large number of threads, or a large 
number of processes have leaked overall.  Could someone with access to 
the build machines check if that is indeed the case?  If it has, bonus 
points for indentifying the source of the leak. ;-)


Thanks!

Jason

On 10/30/2013 05:39 PM, Roman Shaposhnik wrote:

I can take a look sometime later today. Meantime I can only
say that I've been running into 1Gb limit in a few builds as
of late. These days -- I just go with 2G by default.

Thanks,
Roman.

On Wed, Oct 30, 2013 at 3:33 PM, Alejandro Abdelnur 
 wrote:

The following is happening in builds for MAPREDUCE and YARN patches.
I've seen the failures in hadoop5 and hadoop7 machines. I've increased
Maven memory to 1GB (export MAVEN_OPTS="-Xmx1024m" in the jenkins
jobs) but still some failures persist:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4159/

Does anybody has an idea of what may be going on?



thx


[INFO] --- native-maven-plugin:1.0-alpha-7:javah (default) @ 
hadoop-common ---

[INFO] /bin/sh -c cd
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common 


&& /home/jenkins/tools/java/latest/bin/javah -d
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common/target/native/javah 

-classpath 
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common/target/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-annotations/target/classes:/home/jenkins/tools/java/jdk1.6.0_26/jre/../lib/tools.jar:/home/jenkins/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/home/jenkins/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/jenkins/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/jenkins/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/home/jenkins/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/jenkins/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/jenkins/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/jenkins/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/jenkins/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/home/jenkins/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/jenkins/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/home/jenkins/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/jenkins/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/jenkins/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/jenkins/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/jenkins/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/home/jenkins/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-

Re: test-patch failing with OOM errors in javah

[Jason Lowe]

I don't think that OOM error below indicates it needs more heap space, 
as it's complaining about the ability to create a new native thread.  
That usually is caused by lack of available virtual address space or 
hitting process ulimits.


What's most likely going on is the jenkins user is hitting a process 
ulimit.  This can occur if processes have "leaked" from previous 
build/test runs and are using a large number of threads, or a large 
number of processes have leaked overall.  Could someone with access to 
the build machines check if that is indeed the case?  If it has, bonus 
points for indentifying the source of the leak. ;-)


Thanks!

Jason

On 10/30/2013 05:39 PM, Roman Shaposhnik wrote:

I can take a look sometime later today. Meantime I can only
say that I've been running into 1Gb limit in a few builds as
of late. These days -- I just go with 2G by default.

Thanks,
Roman.

On Wed, Oct 30, 2013 at 3:33 PM, Alejandro Abdelnur  wrote:

The following is happening in builds for MAPREDUCE and YARN patches.
I've seen the failures in hadoop5 and hadoop7 machines. I've increased
Maven memory to 1GB (export MAVEN_OPTS="-Xmx1024m" in the jenkins
jobs) but still some failures persist:
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/4159/

Does anybody has an idea of what may be going on?



thx


[INFO] --- native-maven-plugin:1.0-alpha-7:javah (default) @ hadoop-common ---
[INFO] /bin/sh -c cd
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common
&& /home/jenkins/tools/java/latest/bin/javah -d
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common/target/native/javah
-classpath 
/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-common/target/classes:/home/jenkins/jenkins-slave/workspace/PreCommit-MAPREDUCE-Build/trunk/hadoop-common-project/hadoop-annotations/target/classes:/home/jenkins/tools/java/jdk1.6.0_26/jre/../lib/tools.jar:/home/jenkins/.m2/repository/com/google/guava/guava/11.0.2/guava-11.0.2.jar:/home/jenkins/.m2/repository/com/google/code/findbugs/jsr305/1.3.9/jsr305-1.3.9.jar:/home/jenkins/.m2/repository/commons-cli/commons-cli/1.2/commons-cli-1.2.jar:/home/jenkins/.m2/repository/org/apache/commons/commons-math/2.1/commons-math-2.1.jar:/home/jenkins/.m2/repository/xmlenc/xmlenc/0.52/xmlenc-0.52.jar:/home/jenkins/.m2/repository/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar:/home/jenkins/.m2/repository/commons-codec/commons-codec/1.4/commons-codec-1.4.jar:/home/jenkins/.m2/repository/commons-io/commons-io/2.1/commons-io-2.1.jar:/home/jenkins/.m2/repository/commons-net/commons-net/3.1/commons-net-3.1.jar:/home/jenkins/.m2/repository/javax/servlet/servlet-api/2.5/servlet-api-2.5.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty/6.1.26/jetty-6.1.26.jar:/home/jenkins/.m2/repository/org/mortbay/jetty/jetty-util/6.1.26/jetty-util-6.1.26.jar:/home/jenkins/.m2/repository/com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar:/home/jenkins/.m2/repository/com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar:/home/jenkins/.m2/repository/org/codehaus/jettison/jettison/1.1/jettison-1.1.jar:/home/jenkins/.m2/repository/stax/stax-api/1.0.1/stax-api-1.0.1.jar:/home/jenkins/.m2/repository/com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar:/home/jenkins/.m2/repository/javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar:/home/jenkins/.m2/repository/javax/activation/activation/1.1/activation-1.1.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-jaxrs/1.8.8/jackson-jaxrs-1.8.8.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-xc/1.8.8/jackson-xc-1.8.8.jar:/home/jenkins/.m2/repository/com/sun/jersey/jersey-server/1.9/jersey-server-1.9.jar:/home/jenkins/.m2/repository/asm/asm/3.2/asm-3.2.jar:/home/jenkins/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar:/home/jenkins/.m2/repository/log4j/log4j/1.2.17/log4j-1.2.17.jar:/home/jenkins/.m2/repository/net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar:/home/jenkins/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar:/home/jenkins/.m2/repository/commons-configuration/commons-configuration/1.6/commons-configuration-1.6.jar:/home/jenkins/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar:/home/jenkins/.m2/repository/commons-digester/commons-digester/1.8/commons-digester-1.8.jar:/home/jenkins/.m2/repository/commons-beanutils/commons-beanutils/1.7.0/commons-beanutils-1.7.0.jar:/home/jenkins/.m2/repository/commons-beanutils/commons-beanutils-core/1.8.0/commons-beanutils-core-1.8.0.jar:/home/jenkins/.m2/repository/org/slf4j/slf4j-api/1.7.5/slf4j-api-1.7.5.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-core-asl/1.8.8/jackson-core-asl-1.8.8.jar:/home/jenkins/.m2/repository/org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar:/home/jenkins/.m2/repository/org/

Re: [VOTE] Release Apache Hadoop 2.2.0

[Jason Lowe]

+1 (binding)

- Verified signatures and checksums
- Deployed binary tarball to a single-node cluster and successfully ran 
sample jobs
- Built source, deployed to a single-node cluster and successfully ran 
sample jobs


Jason

On 10/07/2013 02:00 AM, Arun C Murthy wrote:

Folks,

I've created a release candidate (rc0) for hadoop-2.2.0 that I would like to 
get released - this release fixes a small number of bugs and some protocol/api 
issues which should ensure they are now stable and will not change in 
hadoop-2.x.

The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail down 
the symlinks-related issues. I'll release note the fact that we have disabled 
it in 2.2. Also, thanks to Vinod for some heavy-lifting on the YARN side in the 
last couple of weeks.





--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)

[Jason Lowe]

+1 (binding)

- Verified signatures and checksums
- Built source
- Deployed single-node cluster and ran some test jobs

Jason

On 08/16/2013 12:29 AM, Konstantin Boudnik wrote:

All,

I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I would
like to release.

This is a stabilization release that includes fixed for a couple a of issues
as outlined on the security list.

The RC is available at: http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1

The maven artifacts are available via repository.apache.org.

The only difference between rc0 and rc1 is ASL added to releasenotes.html and
updated release dates in CHANGES.txt files.

Please try the release bits and vote; the vote will run for the usual 7 days.

Thanks for your voting
   Cos

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Jason Lowe]

+1 (binding)

- verified signatures and checksums
- built from source
- ran some simple jobs on a single-node cluster

On 08/15/2013 04:15 PM, Arun C Murthy wrote:

Folks,

I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would like 
to get released - this fixes the bugs we saw since the last go-around (rc1).

The RC is available at: 
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 0.23.9

[Jason Lowe]

+1

- Verified checksums and signatures
- Booted single-node cluster from binary tarball and ran a few sample jobs
- Built source distribution, installed a single-node cluster and ran a 
few sample jobs


Jason

On 07/01/2013 12:20 PM, Thomas Graves wrote:

I've created a release candidate (RC0) for hadoop-0.23.9 that I would like
to release.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.9-candidate-0/
The RC tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.9-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days til 
July 8th.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Jason Lowe]

I committed MAPREDUCE-5358 to branch-2 but did not commit it to 
branch-2.1-beta since it wasn't a blocker and Arun was in the middle of 
cutting the release.


Arun, if you feel it's appropriate to put this in branch-2.1-beta feel 
free to pull it in or let me know.  Thanks!


Jason

On 07/03/2013 02:24 AM, Tsuyoshi OZAWA wrote:

Hi Arun,

Some bug fixes about MapReduce should be included in next release.
MAPREDUCE-5221 should be merged because it's a lost feature of MRv2
compared to MRv1.
MAPREDUCE-5358 and MAPREDUCE-5335 are very trivial fixes, so it can be
merged immediately.

# I sent the previous email only to hdfs-dev, so I resending it.
# I apologize for my mistake.

On Wed, Jun 26, 2013 at 5:17 PM, Arun C Murthy  wrote:

Folks,

I've created a release candidate (rc0) for hadoop-2.1.0-beta that I would like 
to get released.

This release represents a *huge* amount of work done by the community (639 
fixes) which includes several major advances including:
# HDFS Snapshots
# Windows support
# YARN API stabilization
# MapReduce Binary Compatibility with hadoop-1.x
# Substantial amount of integration testing with rest of projects in the 
ecosystem

The RC is available at: 
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc0/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc0

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 0.23.8

[Jason Lowe]

+1

- Verified signatures and checksums
- Verified MAPREDUCE-5211 was present in CHANGES.txt and source
- Built from source, deployed single-node cluster, ran example jobs

Jason

On 05/28/2013 11:00 AM, Thomas Graves wrote:

I've created a release candidate (RC0) for hadoop-0.23.8 that I would like
to release.

This release is a sustaining release with several important bug fixes in
it.  The most critical one is MAPREDUCE-5211.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.8-candidate-0/
The RC tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.8-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

I am +1 (binding).

thanks,
Tom Graves

Re: [VOTE] Plan to create release candidate for 0.23.8

[Jason Lowe]

+1

On 05/17/2013 04:10 PM, Thomas Graves wrote:

Hello all,

We've had a few critical issues come up in 0.23.7 that I think warrants a
0.23.8 release. The main one is MAPREDUCE-5211.  There are a couple of
other issues that I want finished up and get in before we spin it.  Those
include HDFS-3875, HDFS-4805, and HDFS-4835.  I think those are on track
to finish up early next week.   So I hope to spin 0.23.8 soon after this
vote completes.

Please vote '+1' to approve this plan. Voting will close on Friday May
24th at 2:00pm PDT.

Thanks,
Tom Graves

Re: [VOTE] Release Apache Hadoop 2.0.4-alpha

[Jason Lowe]

+1 (binding)

- verified signatures and checksums
- installed single-node cluster from binaries and ran sample jobs
- built and installed single-node cluster from source and ran sample jobs

Jason

On 04/12/2013 04:56 PM, Arun C Murthy wrote:

Folks,

I've created a release candidate (RC2) for hadoop-2.0.4-alpha that I would like 
to release.

The RC is available at: 
http://people.apache.org/~acmurthy/hadoop-2.0.4-alpha-rc2/
The RC tag in svn is here: 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.4-alpha-rc2

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun


--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

Re: [VOTE] Release Apache Hadoop 0.23.7

[Jason Lowe]

+1 (binding)

- Verified signatures and checksums
- Installed single-node cluster from binary tarball and ran sample jobs
- Built from source, installed single-node cluster from resulting 
binaries, and ran sample jobs


Jason

On 04/11/2013 02:55 PM, Thomas Graves wrote:

I've created a release candidate (RC0) for hadoop-0.23.7 that I would like
to release.

This release is a sustaining release with several important bug fixes in
it.

The RC is available at:
http://people.apache.org/~tgraves/hadoop-0.23.7-candidate-0/
The RC tag in svn is here:
http://svn.apache.org/viewvc/hadoop/common/tags/release-0.23.7-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Tom Graves

Re: [VOTE] Release hadoop-2.0.3-alpha

[Jason Lowe]

+1 (non-binding)

- Verified checksums and signatures on src and binary tarballs
- Built from source
- Deployed pseudo-distributed cluster and ran some example jobs

Jason

On 02/06/2013 09:59 PM, Arun C Murthy wrote:

Folks,

I've created a release candidate (rc0) for hadoop-2.0.3-alpha that I would like 
to release.

This release contains several major enhancements such as QJM for HDFS HA, 
multi-resource scheduling for YARN, YARN ResourceManager restart etc.
Also YARN has achieved significant stability at scale (more details from Y! 
folks here: http://s.apache.org/VYO).

The RC is available at: 
http://people.apache.org/~acmurthy/hadoop-2.0.3-alpha-rc0/
The RC tag in svn is here: 
http://svn.apache.org/viewvc/hadoop/common/tags/release-2.0.3-alpha-rc0/

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

thanks,
Arun



--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

[jira] [Created] (HDFS-4427) start-dfs.sh generates malformed ssh command when not running with native libs

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-4427:


 Summary: start-dfs.sh generates malformed ssh command when not 
running with native libs
 Key: HDFS-4427
 URL: https://issues.apache.org/jira/browse/HDFS-4427
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: scripts
Reporter: Jason Lowe


After HADOOP-8712 the start-dfs.sh script is generating malformed ssh commands 
when the native hadoop libraries are not present.  This is because {{hdfs 
getconf}} is printing a warning, and that warning is accidentally interpreted 
as one of the machines to target for ssh.

Here's an example output of hdfs getconf:

{noformat}
$ hdfs getconf -namenodes 2>/dev/null
2013-01-22 21:03:59,543 WARN  util.NativeCodeLoader 
(NativeCodeLoader.java:(62)) - Unable to load native-hadoop library for 
your platform... using builtin-java classes where applicable
localhost
{noformat}


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-4426) Secondary namenode shuts down immediately after startup

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-4426:


 Summary: Secondary namenode shuts down immediately after startup
 Key: HDFS-4426
 URL: https://issues.apache.org/jira/browse/HDFS-4426
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.0.3-alpha, 0.23.6
Reporter: Jason Lowe
Priority: Blocker


After HADOOP-9181 went in, the secondary namenode immediately shuts down after 
it is started.  From the startup logs:

{noformat}
2013-01-22 19:54:28,826 INFO  namenode.SecondaryNameNode 
(SecondaryNameNode.java:initialize(299)) - Checkpoint Period   :3600 secs (60 
min)
2013-01-22 19:54:28,826 INFO  namenode.SecondaryNameNode 
(SecondaryNameNode.java:initialize(301)) - Log Size Trigger:4 txns
2013-01-22 19:54:28,845 INFO  namenode.SecondaryNameNode 
(StringUtils.java:run(616)) - SHUTDOWN_MSG: 
/
SHUTDOWN_MSG: Shutting down SecondaryNameNode at xx
/
{noformat}

I looked into the issue, and it's shutting down because SecondaryNameNode.main 
starts a bunch of daemon threads then returns.  With nothing but daemon threads 
remaining, the JVM sees no reason to keep going and proceeds to shutdown.  
Apparently we were implicitly relying on the fact that the HttpServer 
QueuedThreadPool threads were not daemon threads to keep the secondary namenode 
process up.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-4328) TestLargeBlock#testLargeBlockSize is timing out

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-4328:


 Summary: TestLargeBlock#testLargeBlockSize is timing out
 Key: HDFS-4328
 URL: https://issues.apache.org/jira/browse/HDFS-4328
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 3.0.0
Reporter: Jason Lowe


For some time now TestLargeBlock#testLargeBlockSize has been timing out on 
trunk.  It is getting hung up during cluster shutdown, and after 15 minutes 
surefire kills it and causes the build to fail since it exited uncleanly.

In addition to fixing the hang, we should consider adding a timeout parameter 
to the @Test decorator for this test.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3831) Failure to renew tokens due to test-sources left in classpath

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3831:


 Summary: Failure to renew tokens due to test-sources left in 
classpath
 Key: HDFS-3831
 URL: https://issues.apache.org/jira/browse/HDFS-3831
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: security
Affects Versions: 0.23.3, 2.1.0-alpha
Reporter: Jason Lowe
Priority: Critical




--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3688) Namenode loses datanode hostname if datanode re-registers

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3688:


 Summary: Namenode loses datanode hostname if datanode re-registers
 Key: HDFS-3688
 URL: https://issues.apache.org/jira/browse/HDFS-3688
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: data-node
Affects Versions: 0.23.3
Reporter: Jason Lowe


If a datanode ever re-registers with the namenode (e.g.: namenode restart, 
temporary network cut, etc.) then the datanode ends up registering with an IP 
address as the datanode name rather than the hostname.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HDFS-3648) TestRaidNode.testDistRaid fails

[Jason Lowe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-3648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Lowe resolved HDFS-3648.
--

Resolution: Duplicate

> TestRaidNode.testDistRaid fails
> ---
>
> Key: HDFS-3648
> URL: https://issues.apache.org/jira/browse/HDFS-3648
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 3.0.0
>Reporter: Eli Collins
>
> Per MAPREDUCE-3868 TestRaidNode fails consistently, here's a recent example 
> from HDFS-3641.
> Error Message
> expected:<0> but was:<2>
> Stacktrace
> junit.framework.AssertionFailedError: expected:<0> but was:<2>
>   at junit.framework.Assert.fail(Assert.java:47)
>   at junit.framework.Assert.failNotEquals(Assert.java:283)
>   at junit.framework.Assert.assertEquals(Assert.java:64)
>   at junit.framework.Assert.assertEquals(Assert.java:130)
>   at junit.framework.Assert.assertEquals(Assert.java:136)
>   at 
> org.apache.hadoop.raid.TestRaidNode.testDistRaid(TestRaidNode.java:583)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3603) TestHDFSTrash is failing

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3603:


 Summary: TestHDFSTrash is failing
 Key: HDFS-3603
 URL: https://issues.apache.org/jira/browse/HDFS-3603
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Affects Versions: 0.23.3, 2.0.1-alpha, 3.0.0
Reporter: Jason Lowe
Priority: Blocker


TestHDFSTrash is failing pretty regularly during test builds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3563) Fix findbug warnings in raid

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3563:


 Summary: Fix findbug warnings in raid
 Key: HDFS-3563
 URL: https://issues.apache.org/jira/browse/HDFS-3563
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/raid
Affects Versions: 3.0.0
Reporter: Jason Lowe


MAPREDUCE-3868 re-enabled raid but introduced 31 new findbugs warnings.  Those 
warnings should be fixed or appropriate items placed in an exclude file.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3554) TestRaidNode is failing

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3554:


 Summary: TestRaidNode is failing
 Key: HDFS-3554
 URL: https://issues.apache.org/jira/browse/HDFS-3554
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: contrib/raid, test
Affects Versions: 3.0.0
Reporter: Jason Lowe


After MAPREDUCE-3868 re-enabled raid, TestRaidNode has been failing in Jenkins 
builds.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3549) dist tar build fails in hadoop-hdfs-raid project

[Jason Lowe (JIRA)]

Jason Lowe created HDFS-3549:


 Summary: dist tar build fails in hadoop-hdfs-raid project
 Key: HDFS-3549
 URL: https://issues.apache.org/jira/browse/HDFS-3549
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 3.0.0
Reporter: Jason Lowe
Priority: Critical


Trying to build the distribution tarball in a clean tree via {{mvn install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip}} fails with this error:

{noformat}
main:
 [exec] tar: hadoop-hdfs-raid-3.0.0-SNAPSHOT: Cannot stat: No such file or 
directory
 [exec] tar: Exiting with failure status due to previous errors
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3195) Compilation error in FSNamesystem

[Jason Lowe (Created) (JIRA)]

Compilation error in FSNamesystem
-

 Key: HDFS-3195
 URL: https://issues.apache.org/jira/browse/HDFS-3195
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 2.0.0
Reporter: Jason Lowe
Priority: Blocker


Build fails when trying to build branch-2:

[ERROR] 
/hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32]
 unreported exception java.io.IOException; must be caught or declared to be 
thrown


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-3136) Multiple SLF4J binding warning

[Jason Lowe (Created) (JIRA)]

Multiple SLF4J binding warning
--

 Key: HDFS-3136
 URL: https://issues.apache.org/jira/browse/HDFS-3136
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.23.0
Reporter: Jason Lowe
Assignee: Jason Lowe


This is the HDFS portion of HADOOP-8005.  HDFS no longer depends upon slf4j, so 
removing it from the assembly will eliminate the HDFS-portion of the multiple 
SLF4J warnings.


--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HDFS-2649) eclipse:eclipse build fails for hadoop-hdfs-httpfs

[Jason Lowe (Created) (JIRA)]

eclipse:eclipse build fails for hadoop-hdfs-httpfs
--

 Key: HDFS-2649
 URL: https://issues.apache.org/jira/browse/HDFS-2649
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: build
Affects Versions: 0.24.0
Reporter: Jason Lowe


Building the eclipse:eclipse target fails in the hadoop-hdfs-httpfs project 
with this error:

[ERROR] Failed to execute goal 
org.apache.maven.plugins:maven-eclipse-plugin:2.8:eclipse (default-cli) on 
project hadoop-hdfs-httpfs: Request to merge when 'filtering' is not identical. 
Original=resource src/main/resources: output=target/classes, 
include=[httpfs.properties], exclude=[**/*.java], test=false, filtering=true, 
merging with=resource src/main/resources: output=target/classes, include=[], 
exclude=[httpfs.properties|**/*.java], test=false, filtering=false -> [Help 1]

This appears to be the same type of issue as reported in HADOOP-7567.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira