Re: [DISCUSS] Retire BKJM from trunk?

[Gangumalla, Uma]

For Huawei, Vinay/Brahma should know about their usage. I think after QJM
stabilized and ready they also adopted to QJM is what I know, but they
should know more than me as I left that employer while ago.

If no one is using it, It is ok to remove.

Regards,
Uma

On 7/27/16, 9:49 PM, "Rakesh Radhakrishnan"  wrote:

>If I remember correctly, Huawei also adopted QJM component. I hope @Vinay
>might have discussed internally in Huawei before starting this e-mail
>discussion thread. I'm +1, for removing the bkjm contrib from the trunk
>code.
>
>Also, there are quite few open sub-tasks under HDFS-3399 umbrella jira,
>which was used for the BKJM implementation time. How about closing these
>jira by marking as "Won't Fix"?
>
>Thanks,
>Rakesh
>Intel
>
>On Thu, Jul 28, 2016 at 1:53 AM, Sijie Guo  wrote:
>
>> + Rakesh and Uma
>>
>> Rakesh and Uma might have a better idea on this. I think Huawei was
>>using
>> it when Rakesh and Uma worked there.
>>
>> - Sijie
>>
>> On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth
>>
>> wrote:
>>
>> > I recommend including the BookKeeper community in this discussion.
>>I¹ve
>> > added their user@ and dev@ lists to this thread.
>> >
>> > I do not see BKJM being used in practice.  Removing it from trunk
>>would
>> be
>> > attractive in terms of less code for Hadoop to maintain and build,
>>but if
>> > we find existing users that want to keep it, I wouldn¹t object.
>> >
>> > --Chris Nauroth
>> >
>> > On 7/26/16, 11:14 PM, "Vinayakumar B" 
>>wrote:
>> >
>> > Hi All,
>> >
>> >BKJM was Active and made much stable when the NameNode HA was
>> > implemented and there was no QJM implemented.
>> >Now QJM is present and is much stable which is adopted by many
>> > production environment.
>> >I wonder whether it would be a good time to retire BKJM from
>> trunk?
>> >
>> >Are there any users of BKJM exists?
>> >
>> > -Vinay
>> >
>> >
>> >
>>


-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Retire BKJM from trunk?

[Rakesh Radhakrishnan]

If I remember correctly, Huawei also adopted QJM component. I hope @Vinay
might have discussed internally in Huawei before starting this e-mail
discussion thread. I'm +1, for removing the bkjm contrib from the trunk
code.

Also, there are quite few open sub-tasks under HDFS-3399 umbrella jira,
which was used for the BKJM implementation time. How about closing these
jira by marking as "Won't Fix"?

Thanks,
Rakesh
Intel

On Thu, Jul 28, 2016 at 1:53 AM, Sijie Guo  wrote:

> + Rakesh and Uma
>
> Rakesh and Uma might have a better idea on this. I think Huawei was using
> it when Rakesh and Uma worked there.
>
> - Sijie
>
> On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth 
> wrote:
>
> > I recommend including the BookKeeper community in this discussion.  I’ve
> > added their user@ and dev@ lists to this thread.
> >
> > I do not see BKJM being used in practice.  Removing it from trunk would
> be
> > attractive in terms of less code for Hadoop to maintain and build, but if
> > we find existing users that want to keep it, I wouldn’t object.
> >
> > --Chris Nauroth
> >
> > On 7/26/16, 11:14 PM, "Vinayakumar B"  wrote:
> >
> > Hi All,
> >
> >BKJM was Active and made much stable when the NameNode HA was
> > implemented and there was no QJM implemented.
> >Now QJM is present and is much stable which is adopted by many
> > production environment.
> >I wonder whether it would be a good time to retire BKJM from
> trunk?
> >
> >Are there any users of BKJM exists?
> >
> > -Vinay
> >
> >
> >
>

[jira] [Created] (HDFS-10700) I increase the value of the GC_OPTS on namenode. After I modified the value ,namenode start failed.

[Liu Guannan (JIRA)]

Liu Guannan created HDFS-10700:
--

 Summary: I increase the value of the GC_OPTS on namenode. After I 
modified the value ,namenode start failed.
 Key: HDFS-10700
 URL: https://issues.apache.org/jira/browse/HDFS-10700
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: namenode
Affects Versions: 2.7.2
 Environment: Linux Suse 11 SP3
Reporter: Liu Guannan


I increase the value of the GC_OPTS on namenode. After I modified the value 
,namenode start failed.The reasion is that Datanodes reported  block status to 
the namenode, resulting in namenode update block status slowly. And then 
namenode start failed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

[jira] [Created] (HDFS-10699) Log object instance get incorrectly in TestDFSAdmin

[Yiqun Lin (JIRA)]

Yiqun Lin created HDFS-10699:


 Summary: Log object instance get incorrectly in TestDFSAdmin
 Key: HDFS-10699
 URL: https://issues.apache.org/jira/browse/HDFS-10699
 Project: Hadoop HDFS
  Issue Type: Bug
Reporter: Yiqun Lin
Assignee: Yiqun Lin
Priority: Minor


In class TestDFSAdmin, it gets a incorrect object instance. The codes:
{code}
 public class TestDFSAdmin {
   private static final Log LOG = LogFactory.getLog(DFSAdmin.class);
   private Configuration conf = null;
   private MiniDFSCluster cluster;
   private DFSAdmin admin;
   ...
{code}
Here the class name {{DFSAdmin}} should be {{TestDFSAdmin}}.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: Apply for "assign to me" permission.

[Arpit Agarwal]

Done. Thanks for your interest in contributing to Hadoop.


On 7/27/16, 6:47 PM, "hufh"  wrote:

Hi guys,

I have opened a JIRA(https://issues.apache.org/jira/browse/HDFS-10690) and
want to fix it, but i can't assign it to myself, anyone can give me a hand?
Thanks a lot!

Fenghua

Apply for "assign to me" permission.

[hufh]

Hi guys,

I have opened a JIRA(https://issues.apache.org/jira/browse/HDFS-10690) and
want to fix it, but i can't assign it to myself, anyone can give me a hand?
Thanks a lot!

Fenghua

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Sangjin Lee]

+1 (binding)

- downloaded both source and binary tarballs and verified the signatures
- set up a pseudo-distributed cluster
- ran some simple mapreduce jobs
- checked the basic web UI

Sangjin

On Wed, Jul 27, 2016 at 12:57 PM, John Zhuge  wrote:

> +1 (non-binding)
>
> - Build source with Java 1.8.0_101 on Centos 7.2 with native
> - Build source with Java 1.7.0_79 on Mac
> - Verify license and notice using the shell script in HADOOP-13374
> - Deploy a pseudo cluster
> - Run basic dfs, distcp, ACL, webhdfs commands
> - Run MapReduce workcount and pi examples
> - Start balancer
>
> Thanks,
> John
>
> John Zhuge
> Software Engineer, Cloudera
>
> On Wed, Jul 27, 2016 at 11:38 AM, Robert Kanter 
> wrote:
>
> > +1 (binding)
> >
> > - Downloaded binary tarball
> > - verified signatures
> > - setup pseudo cluster
> > - ran some of the example jobs, clicked around the UI a bit
> >
> >
> > - Robert
> >
> >
> > On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe  >
> > wrote:
> >
> > > +1 (binding)
> > > - Verified signatures and digests- Built from source with native
> support-
> > > Deployed a pseudo-distributed cluster- Ran some sample jobs
> > > Jason
> > >
> > >   From: Vinod Kumar Vavilapalli 
> > >  To: "common-...@hadoop.apache.org" ;
> > > hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; "
> > > mapreduce-...@hadoop.apache.org" 
> > > Cc: Vinod Kumar Vavilapalli 
> > >  Sent: Friday, July 22, 2016 9:15 PM
> > >  Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
> > >
> > > Hi all,
> > >
> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> > >
> > > As discussed before, this is the next maintenance release to follow up
> > > 2.7.2.
> > >
> > > The RC is available for validation at:
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> > >
> > > The RC tag in git is: release-2.7.3-RC0
> > >
> > > The maven artifacts are available via repository.apache.org <
> > > http://repository.apache.org/> at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > <
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > >
> > >
> > > The release-notes are inside the tar-balls at location
> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> > > hosted this at
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> > for
> > > your quick perusal.
> > >
> > > As you may have noted, a very long fix-cycle for the License & Notice
> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> > release)
> > > to slip by quite a bit. This release's related discussion thread is
> > linked
> > > below: [1].
> > >
> > > Please try the release and vote; the vote will run for the usual 5
> days.
> > >
> > > Thanks,
> > > Vinod
> > >
> > > [1]: 2.7.3 release plan:
> > >
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> > <
> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> > >
> > >
> > >
> >
>

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Andrew Wang]

Hi Junping, thanks for sharing your thoughts, inline,

On Wed, Jul 27, 2016 at 9:10 AM, 俊平堵  wrote:

> Thanks Vinod for bringing up this topic for discussion. I share the same
> concern here from my previous experience and I doubt some simple rules
> proposed below could make life easier.
>
> > The question now is what we do for the 2.8.0 and 3.0.0-alpha1 fix
> versions.
> > Allen's historical perspective is that we've based each minor or major
> > release off of the previous minor release. So, 2.8.0 would be based off
> of
> > 2.7.0. Assuming 3.0.0-alpha1 happens before 2.8.0, 3.0.0-alpha1 would
> also
> > be based off of 2.7.0. This also makes sense from a user POV; someone on
> a
> > 2.6.x going to 3.0.0-alpha1 can look at the 2.7.0 and 3.0.0-alpha1 notes
> to
> > see what's changed.
> This is not correct - not reflecting the past and not helpful for the
> future. There is no benefit to claim 3.0.0-alpha1 is based on 2.7.0 over
> 2.7.3 (in case 2.8.0 is not there).
> In the past, for example, when we cut off 2.7, we already have 2.6.0 and
> 2.6.1 get released, so 2.7.0 take all commits from 2.6.1 (not 2.6.0). In
> future, assume when we start the release effor of 3.1.0 and we have 3.0.1,
> 3.0.2, etc., 3.0.x should be more stable than 3.0.0-alpha, so there is
> unnecessary to do everything from scratch (3.0.0-alpha).
>

Based on the website, 2.7.0 immediately followed 2.6.0, so that's not quite
accurate. However your point does hold for 2.5.2 -> 2.6.0.

Vinod already described this earlier, where for a while we only had a
single chronological release line. Now though, we have the concurrent 2.6.x
and 2.7.x release lines, which do fix versions independently.

One goal here is a versioning scheme that extends this concurrency to
additional major and minor releases, i.e. 2.8 and 3.0. It doesn't make
sense to have a global ordering of releases, since not all patches belong
in all branches. It also poses additional coordination cost for release
management, particularly since it's hard to predict release timings.

Another goal is a scheme that is easy for users to understand. I believe
the above scheme accurately encodes the versioning system used by most
other enterprise software.


> So the rule here should be: a new major or minor release should come from
> a release:
> 1. tag with stable
> 2. released latest
> 3. with maximum version number
> If condition 2 and 3 get conflicts, we should give priority to 3. For
> example, when 3.0.0-alpha1 is about to release, assume we have 2.8.0, 2.7.4
> and 2.7.4 get released after 2.8.0, then we should claim 3.0.0-alpha is
> based on 2.8.0 instead of 2.7.4.
>
> These rules seem to require a high degree of coordination when it comes to
release ordering, but one goal is to reduce coordination. For example, the
2.7.0 notes say "Production users should wait for a 2.7.1/2.7.2 release."
We didn't know in advance if 2.7.1 or 2.7.2 would be the stable release. We
also don't know the release ordering of 2.7.4 and 2.8.0.

Given the uncertainty around release timing and stability, it's
advantageous to avoid basing versioning on these two pieces of information.
With these rules, a changes in stability or release ordering means we need
to update JIRA fix versions.

I also didn't quite follow your example, since I assume that 2.8.0 will
*not* be marked stable similar to how 2.7.0 is not stable. If this is true,
does rule 1 take precedence and we would base 3.0.0-alpha1 off of 2.7.4
(the latest stable release)? That seems confusing given the presence of
2.8.0.

Could you elaborate on what you see as the advantages of this scheme from a
user POV? It seems like users now need to be aware of the total ordering
and also stability of releases to know what changelogs to read. And as
described previously, it's not how other enterprise software versioning
works.


> > As an example, if a JIRA was committed to branch-2.6, branch-2.7,
> branch-2,
> > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5,
> 2.7.3,
> > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of
> > rule 1, and the last two fix versions come from rule 2.
> I don't think setting version tags to be more than 3 is a good practice.
> The example above means we need to backport this patch to 5 branches which
> make our committers' life really tough - it requires more effort of
> committing a patch and also increases the risky of bugs that caused by
> backport. Given realistic community review bandwidth (please check another
> thread from Chris D.), I strongly suggest we keep active release train to
> be less than 3, so we can have 2 stable release or 1 stable release + 1
> alpha release in releasing.
>
> We already need to backport to many branches, setting some more fix
versions doesn't change that. Setting more fix versions is also not related
to review bandwidth, particularly in this case since things normally land
in trunk first. It's also not related to the 

Re: [DISCUSS] Retire BKJM from trunk?

[Sijie Guo]

+ Rakesh and Uma

Rakesh and Uma might have a better idea on this. I think Huawei was using
it when Rakesh and Uma worked there.

- Sijie

On Wed, Jul 27, 2016 at 12:06 PM, Chris Nauroth 
wrote:

> I recommend including the BookKeeper community in this discussion.  I’ve
> added their user@ and dev@ lists to this thread.
>
> I do not see BKJM being used in practice.  Removing it from trunk would be
> attractive in terms of less code for Hadoop to maintain and build, but if
> we find existing users that want to keep it, I wouldn’t object.
>
> --Chris Nauroth
>
> On 7/26/16, 11:14 PM, "Vinayakumar B"  wrote:
>
> Hi All,
>
>BKJM was Active and made much stable when the NameNode HA was
> implemented and there was no QJM implemented.
>Now QJM is present and is much stable which is adopted by many
> production environment.
>I wonder whether it would be a good time to retire BKJM from trunk?
>
>Are there any users of BKJM exists?
>
> -Vinay
>
>
>

[jira] [Resolved] (HDFS-10698) Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk

[Wei-Chiu Chuang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-10698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wei-Chiu Chuang resolved HDFS-10698.

Resolution: Duplicate

thx Youngjun for filing the jira.
This is a dup of HDFS-10696 where a patch is provided.

> Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk
> -
>
> Key: HDFS-10698
> URL: https://issues.apache.org/jira/browse/HDFS-10698
> Project: Hadoop HDFS
>  Issue Type: Bug
>  Components: hdfs
>Reporter: Yongjun Zhang
>
> {code}
> Running org.apache.hadoop.cli.TestHDFSCLI
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.887 sec 
> <<< FAILURE! - in org.apache.hadoop.cli.TestHDFSCLI
> testAll(org.apache.hadoop.cli.TestHDFSCLI)  Time elapsed: 39.697 sec  <<< 
> FAILURE!
> java.lang.AssertionError: One of the tests failed. See the Detailed results 
> to identify the command that failed
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at 
> org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:263)
> at 
> org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:125)
> at org.apache.hadoop.cli.TestHDFSCLI.tearDown(TestHDFSCLI.java:87)
> Results :
> Failed tests:
>   
> TestHDFSCLI.tearDown:87->CLITestHelper.tearDown:125->CLITestHelper.displayResults:263
>  One of the tests failed. See the Detailed results to identify the command 
> that failed
> Tests run: 1, Failures: 1, Errors: 0, Skipped: 0
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[John Zhuge]

+1 (non-binding)

- Build source with Java 1.8.0_101 on Centos 7.2 with native
- Build source with Java 1.7.0_79 on Mac
- Verify license and notice using the shell script in HADOOP-13374
- Deploy a pseudo cluster
- Run basic dfs, distcp, ACL, webhdfs commands
- Run MapReduce workcount and pi examples
- Start balancer

Thanks,
John

John Zhuge
Software Engineer, Cloudera

On Wed, Jul 27, 2016 at 11:38 AM, Robert Kanter 
wrote:

> +1 (binding)
>
> - Downloaded binary tarball
> - verified signatures
> - setup pseudo cluster
> - ran some of the example jobs, clicked around the UI a bit
>
>
> - Robert
>
>
> On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe 
> wrote:
>
> > +1 (binding)
> > - Verified signatures and digests- Built from source with native support-
> > Deployed a pseudo-distributed cluster- Ran some sample jobs
> > Jason
> >
> >   From: Vinod Kumar Vavilapalli 
> >  To: "common-...@hadoop.apache.org" ;
> > hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; "
> > mapreduce-...@hadoop.apache.org" 
> > Cc: Vinod Kumar Vavilapalli 
> >  Sent: Friday, July 22, 2016 9:15 PM
> >  Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
> >
> > Hi all,
> >
> > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> >
> > As discussed before, this is the next maintenance release to follow up
> > 2.7.2.
> >
> > The RC is available for validation at:
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> >
> > The RC tag in git is: release-2.7.3-RC0
> >
> > The maven artifacts are available via repository.apache.org <
> > http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> <
> > https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >
> >
> > The release-notes are inside the tar-balls at location
> > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> > hosted this at
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for
> > your quick perusal.
> >
> > As you may have noted, a very long fix-cycle for the License & Notice
> > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> release)
> > to slip by quite a bit. This release's related discussion thread is
> linked
> > below: [1].
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> > Thanks,
> > Vinod
> >
> > [1]: 2.7.3 release plan:
> > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> <
> > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> >
> >
> >
>

[jira] [Created] (HDFS-10698) Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk

[Yongjun Zhang (JIRA)]

Yongjun Zhang created HDFS-10698:


 Summary: Test org.apache.hadoop.cli.TestHDFSCLI fails in trunk
 Key: HDFS-10698
 URL: https://issues.apache.org/jira/browse/HDFS-10698
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: hdfs
Reporter: Yongjun Zhang


{code}
Running org.apache.hadoop.cli.TestHDFSCLI
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 39.887 sec <<< 
FAILURE! - in org.apache.hadoop.cli.TestHDFSCLI
testAll(org.apache.hadoop.cli.TestHDFSCLI)  Time elapsed: 39.697 sec  <<< 
FAILURE!
java.lang.AssertionError: One of the tests failed. See the Detailed results to 
identify the command that failed
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at 
org.apache.hadoop.cli.CLITestHelper.displayResults(CLITestHelper.java:263)
at org.apache.hadoop.cli.CLITestHelper.tearDown(CLITestHelper.java:125)
at org.apache.hadoop.cli.TestHDFSCLI.tearDown(TestHDFSCLI.java:87)


Results :

Failed tests:
  
TestHDFSCLI.tearDown:87->CLITestHelper.tearDown:125->CLITestHelper.displayResults:263
 One of the tests failed. See the Detailed results to identify the command that 
failed

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

{code}




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Andrew Wang]

>
> The -alphaX versions we're using leading up to 3.0.0 GA can be treated as
>> a.b.c versions, with alpha1 being the a.b.0 release.
>>
>
> Once 3.0.0 GA goes out, a user would want to see the diff from the latest
> 2.x.0 release (say 2.9.0).
>
> Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would
> apply, and it should show up in the release notes?
>
>
Yea. It means the if you're coming from 2.7, you'd read the 3.0.0-alphas,
3.0.0-betas, and finally the 3.0.0 GA notes.

The website notes will aggregate all the alpha and beta changes leading up
to 3.0.0 GA, so is likely where users will turn to first.

>
>> As an example, if a JIRA was committed to branch-2.6, branch-2.7,
>> branch-2,
>> branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5,
>> 2.7.3,
>> 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of
>> rule 1, and the last two fix versions come from rule 2.
>>
>> I'm very eager to move this discussion forward, so feel free to reach out
>> on or off list if I can help with anything.
>>
>
>
> I think it is good practice to set multiple fix versions. However, it
> might take the committers a little bit to learn.
>
> Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the
> 3.0.0-alphaX version?
>
>
Yea, I have a script and some JIRA queries that can help with this. I'll
also plan to compare with git log contents for extra verification.

Re: [DISCUSS] Retire BKJM from trunk?

[Chris Nauroth]

I recommend including the BookKeeper community in this discussion.  I’ve added 
their user@ and dev@ lists to this thread.

I do not see BKJM being used in practice.  Removing it from trunk would be 
attractive in terms of less code for Hadoop to maintain and build, but if we 
find existing users that want to keep it, I wouldn’t object.

--Chris Nauroth

On 7/26/16, 11:14 PM, "Vinayakumar B"  wrote:

Hi All,

   BKJM was Active and made much stable when the NameNode HA was 
implemented and there was no QJM implemented.
   Now QJM is present and is much stable which is adopted by many 
production environment.
   I wonder whether it would be a good time to retire BKJM from trunk?

   Are there any users of BKJM exists?

-Vinay



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Robert Kanter]

+1 (binding)

- Downloaded binary tarball
- verified signatures
- setup pseudo cluster
- ran some of the example jobs, clicked around the UI a bit


- Robert


On Mon, Jul 25, 2016 at 3:28 PM, Jason Lowe 
wrote:

> +1 (binding)
> - Verified signatures and digests- Built from source with native support-
> Deployed a pseudo-distributed cluster- Ran some sample jobs
> Jason
>
>   From: Vinod Kumar Vavilapalli 
>  To: "common-...@hadoop.apache.org" ;
> hdfs-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; "
> mapreduce-...@hadoop.apache.org" 
> Cc: Vinod Kumar Vavilapalli 
>  Sent: Friday, July 22, 2016 9:15 PM
>  Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
>
> Hi all,
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC0
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
>
> As you may have noted, a very long fix-cycle for the License & Notice
> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
> to slip by quite a bit. This release's related discussion thread is linked
> below: [1].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
>

Re: Improving recovery performance for degraded reads

[Rakesh Radhakrishnan]

Hi Roy,

 (a) In your last email, I am sure you meant => "... submitting read
requests to fetch "any" (instead of all) the 'k' chunk (out of k+m-x
surviving chunks)  ?
 Do you have any optimization in place to decide which data-nodes will
be part of those "k" ?

Answer:-
I hope you know the write path, just adding few details here to support the
read explanation part. While writing to an EC file, dfs client writes data
stripe(e.g. 64KB cellsize) to multiple datanodes. For (k, m) schema, the
client writes data block to the first k datanodes and parity block to the
remaining m datanodes. Say, one stripe is (k * cellSize + m * cellSize)
data. While reading, client will fetch in the same order, read stripe by
stripe. The datanodes with data blocks are first fetched than the datanodes
with parity blocks because less EC block reconstruction work is needed.
Internally, dfs client reads the whole stripe one by one and contacts k
datanodes parallelly for each stripe. If there is any failures then will
contact parity datanodes and do reconstruction on the fly.
'DFSStripedInputStream' supported both positional read and read entire
buffer(e.g filesize buffer).

 (b) Are there any caching being done (as proposed for QFS in the
previously attached "PPR" paper) ?
Answer:-
HDFS-9879, there is an open jira to discuss the caching of striped blocks
at the datanode. Perhaps, caching logic could be utilized similar to the
QFS and while reconstruction choose those datanodes that have already
cached the data in memory. This is an open improvement task as of now.

 (c) When you mentioned stripping is being done, I assume it is
probably to reduce the chunk sizes and hence k*c ?
Answer:-
Yes, striping is done by dividing the block into several chunks, we call it
as cellSize (e.g. 64KB). (k * c + m * c) is one stripe. A block group
comprises of several stripes. I'd  suggest you to read the blog -
http://blog.cloudera.com/blog/2015/09/introduction-to-hdfs-erasure-coding-in-apache-hadoop/
to understand more about the stripe, cells and block group terminolgy etc
before reeading the below answer.
   blk_0  blk_1   blk_2
 | ||
 vv   v
  +--+   +--+   +--+
  |cell_0|   |cell_1|   |cell_2|
  +--+   +--+   +--+

 Now, if my object sizes are large (e.g. super HD images) where I would
have to get data from multiple stripes to rebuild the images before I can
display to the
 client, do you think stripping would still help ?
 Is there a possibility that since I know that all the segments of the
HD image would always be read together, by stripping and distributing it on
different nodes, I am ignoring
 its special/temporal locality and further increase any associated
delays ?

Answer:-
Since for each stripe it contacts all the k datanodes, assume if there are
slow datanodes or some dead datanodes in each data block stripe then it
will affect the read performance. AFAIK, for a large file contiguous layout
is suitable, this will be supported in phase-2 and design discussions are
still going on, please see HDFS-8030 jira. On the otherside, in theory I
can say there is a benefit of striping layout, which enables the client to
work with multiple data nodes in parallel, greatly enhancing the aggregate
throughput(assuming that all datanodes are good servers). But this needs to
be tested in your cluster to understand the impact.


Thanks,
Rakesh
Intel

On Sun, Jul 24, 2016 at 12:00 PM, Roy Leonard 
wrote:

> Hi Rakesh,
>
> Thanks for sharing your thoughts and updates.
>
> (a) In your last email, I am sure you meant => "... submitting read
> requests to fetch "any" (instead of all) the 'k' chunk (out of k+m-x
> surviving chunks)  ?
> Do you have any optimization in place to decide which data-nodes will be
> part of those "k" ?
>
> (b) Are there any caching being done (as proposed for QFS in the previously
> attached "PPR" paper) ?
>
> (c) When you mentioned stripping is being done, I assume it is probably to
> reduce the chunk sizes and hence k*c ?
> Now, if my object sizes are large (e.g. super HD images) where I would have
> to get data from multiple stripes to rebuild the images before I can
> display to the client, do you think stripping would still help ?
> Is there a possibility that since I know that all the segments of the HD
> image would always be read together, by stripping and distributing it on
> different nodes, I am ignoring its special/temporal locality and further
> increase any associated delays ?
>
> Just wanted to know your thoughts.
> I am looking forward to the future performance improvements in HDFS.
>
> Regards,
> R.
>
> On Fri, Jul 22, 2016 at 8:52 AM, Rakesh Radhakrishnan 
> wrote:
>
> > I'm adding one more point to the above. In my previous mail reply, I've
> > explained the striped block reconstruction task which will be triggered
> by
> > the Namenode on 

[jira] [Reopened] (HDFS-8224) Any IOException in DataTransfer#run() will run diskError thread even if it is not disk error

[Rushabh S Shah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HDFS-8224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rushabh S Shah reopened HDFS-8224:
--

Saw one more occurrence of this bug.
We should atleast add this block to the front of the scanning queue.

> Any IOException in DataTransfer#run() will run diskError thread even if it is 
> not disk error
> 
>
> Key: HDFS-8224
> URL: https://issues.apache.org/jira/browse/HDFS-8224
> Project: Hadoop HDFS
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Rushabh S Shah
>Assignee: Rushabh S Shah
> Fix For: 2.8.0
>
>
> This happened in our 2.6 cluster.
> One of the block and its metadata file were corrupted.
> The disk was healthy in this case.
> Only the block was corrupt.
> Namenode tried to copy that block to another datanode but failed with the 
> following stack trace:
> 2015-04-20 01:04:04,421 
> [org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer@11319bc4] WARN 
> datanode.DataNode: DatanodeRegistration(a.b.c.d, 
> datanodeUuid=e8c5135c-9b9f-4d05-a59d-e5525518aca7, infoPort=1006, 
> infoSecurePort=0, ipcPort=8020, 
> storageInfo=lv=-56;cid=CID-e7f736ac-158e-446e-9091-7e66f3cddf3c;nsid=358250775;c=1428471998571):Failed
>  to transfer BP-xxx-1351096255769:blk_2697560713_1107108863999 to 
> a1.b1.c1.d1:1004 got 
> java.io.IOException: Could not create DataChecksum of type 0 with 
> bytesPerChecksum 0
> at 
> org.apache.hadoop.util.DataChecksum.newDataChecksum(DataChecksum.java:125)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:175)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readHeader(BlockMetadataHeader.java:140)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockMetadataHeader.readDataChecksum(BlockMetadataHeader.java:102)
> at 
> org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:287)
> at 
> org.apache.hadoop.hdfs.server.datanode.DataNode$DataTransfer.run(DataNode.java:1989)
> at java.lang.Thread.run(Thread.java:722)
> The following catch block in DataTransfer#run method will treat every 
> IOException as disk error fault and run disk errror
> {noformat}
> catch (IOException ie) {
> LOG.warn(bpReg + ":Failed to transfer " + b + " to " +
> targets[0] + " got ", ie);
> // check if there are any disk problem
> checkDiskErrorAsync();
>   } 
> {noformat}
> This block was never scanned by BlockPoolSliceScanner otherwise it would have 
> reported as corrupt block.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[俊平堵]

Thanks Vinod for bringing up this topic for discussion. I share the same
concern here from my previous experience and I doubt some simple rules
proposed below could make life easier.

> The question now is what we do for the 2.8.0 and 3.0.0-alpha1 fix
versions.
> Allen's historical perspective is that we've based each minor or major
> release off of the previous minor release. So, 2.8.0 would be based off of
> 2.7.0. Assuming 3.0.0-alpha1 happens before 2.8.0, 3.0.0-alpha1 would also
> be based off of 2.7.0. This also makes sense from a user POV; someone on a
> 2.6.x going to 3.0.0-alpha1 can look at the 2.7.0 and 3.0.0-alpha1 notes
to
> see what's changed.
This is not correct - not reflecting the past and not helpful for the
future. There is no benefit to claim 3.0.0-alpha1 is based on 2.7.0 over
2.7.3 (in case 2.8.0 is not there).
In the past, for example, when we cut off 2.7, we already have 2.6.0 and
2.6.1 get released, so 2.7.0 take all commits from 2.6.1 (not 2.6.0). In
future, assume when we start the release effor of 3.1.0 and we have 3.0.1,
3.0.2, etc., 3.0.x should be more stable than 3.0.0-alpha, so there is
unnecessary to do everything from scratch (3.0.0-alpha). So the rule here
should be: a new major or minor release should come from a release:
1. tag with stable
2. released latest
3. with maximum version number
If condition 2 and 3 get conflicts, we should give priority to 3. For
example, when 3.0.0-alpha1 is about to release, assume we have 2.8.0, 2.7.4
and 2.7.4 get released after 2.8.0, then we should claim 3.0.0-alpha is
based on 2.8.0 instead of 2.7.4.


> As an example, if a JIRA was committed to branch-2.6, branch-2.7,
branch-2,
> branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5,
2.7.3,
> 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of
> rule 1, and the last two fix versions come from rule 2.
I don't think setting version tags to be more than 3 is a good practice.
The example above means we need to backport this patch to 5 branches which
make our committers' life really tough - it requires more effort of
committing a patch and also increases the risky of bugs that caused by
backport. Given realistic community review bandwidth (please check another
thread from Chris D.), I strongly suggest we keep active release train to
be less than 3, so we can have 2 stable release or 1 stable release + 1
alpha release in releasing.

BTW, I never see we have a clear definition for alpha release. It is
previously used as unstable in API definition (2.1-alpha, 2.2-alpha, etc.)
but sometimes means unstable in production quality (2.7.0). I think we
should clearly define it with major consensus so user won't
misunderstanding the risky here.
Also, if we treat our 3.0.0-alpha release work seriously, we should also
think about trunk's version number issue (bump up to 4.0.0-alpha?) or there
could be no room for 3.0 incompatible feature/bits soon.

Just 2 cents.

Thanks,

Junping

2016-07-27 15:34 GMT+01:00 Karthik Kambatla :

> Inline.
>
> > 1) Set the fix version for all a.b.c versions, where c > 0.
> > 2) For each major release line, set the lowest a.b.0 version.
> >
>
> Sounds reasonable.
>
>
> >
> > The -alphaX versions we're using leading up to 3.0.0 GA can be treated as
> > a.b.c versions, with alpha1 being the a.b.0 release.
> >
>
> Once 3.0.0 GA goes out, a user would want to see the diff from the latest
> 2.x.0 release (say 2.9.0).
>
> Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would
> apply, and it should show up in the release notes?
>
>
> >
> > As an example, if a JIRA was committed to branch-2.6, branch-2.7,
> branch-2,
> > branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5,
> 2.7.3,
> > 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of
> > rule 1, and the last two fix versions come from rule 2.
> >
> > I'm very eager to move this discussion forward, so feel free to reach out
> > on or off list if I can help with anything.
> >
>
>
> I think it is good practice to set multiple fix versions. However, it might
> take the committers a little bit to learn.
>
> Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the
> 3.0.0-alphaX version?
>
>
> >
> > Best,
> > Andrew
> >
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/

[Jul 26, 2016 1:30:02 PM] (stevel) Revert "HDFS-10668. Fix intermittently 
failing UT
[Jul 26, 2016 1:53:37 PM] (kai.zheng) HADOOP-13041. Adding tests for coder 
utilities. Contributed by Kai
[Jul 26, 2016 3:01:42 PM] (weichiu) HDFS-9937. Update dfsadmin command line 
help and HdfsQuotaAdminGuide.
[Jul 26, 2016 3:19:06 PM] (varunsaxena) YARN-5431. TimelineReader daemon start 
should allow to pass its own
[Jul 26, 2016 3:43:12 PM] (varunsaxena) Revert "YARN-5431. TimelineReader 
daemon start should allow to pass its
[Jul 26, 2016 7:27:46 PM] (arp) HDFS-10642.
[Jul 26, 2016 9:54:03 PM] (Arun Suresh) YARN-5392. Replace use of Priority in 
the Scheduling infrastructure with
[Jul 26, 2016 10:33:20 PM] (cnauroth) HADOOP-13422. 
ZKDelegationTokenSecretManager JaasConfig does not work
[Jul 26, 2016 11:01:50 PM] (weichiu) HDFS-10598. DiskBalancer does not execute 
multi-steps plan. Contributed
[Jul 27, 2016 1:14:09 AM] (wangda) YARN-5342. Improve non-exclusive node 
partition resource allocation in
[Jul 27, 2016 2:08:30 AM] (Arun Suresh) YARN-5351. ResourceRequest should take 
ExecutionType into account during
[Jul 27, 2016 4:22:59 AM] (wangda) YARN-5195. RM intermittently crashed with 
NPE while handling
[Jul 27, 2016 4:56:42 AM] (brahma) HDFS-10668. Fix intermittently failing UT




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.cli.TestHDFSCLI 
   hadoop.yarn.client.api.impl.TestYarnClient 
   hadoop.yarn.server.nodemanager.TestDirectoryCollection 
   hadoop.yarn.server.resourcemanager.scheduler.TestSchedulerHealth 
   
hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler 
   hadoop.yarn.server.resourcemanager.scheduler.fifo.TestFifoScheduler 
   hadoop.yarn.server.resourcemanager.TestResourceManager 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/diff-javadoc-javadoc-root.txt
  [2.3M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [144K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [16K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [64K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/115/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: 

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Karthik Kambatla]

Inline.

> 1) Set the fix version for all a.b.c versions, where c > 0.
> 2) For each major release line, set the lowest a.b.0 version.
>

Sounds reasonable.


>
> The -alphaX versions we're using leading up to 3.0.0 GA can be treated as
> a.b.c versions, with alpha1 being the a.b.0 release.
>

Once 3.0.0 GA goes out, a user would want to see the diff from the latest
2.x.0 release (say 2.9.0).

Are you suggesting 3.0.0 GA would have c = 5 (say) and hence rule 1 would
apply, and it should show up in the release notes?


>
> As an example, if a JIRA was committed to branch-2.6, branch-2.7, branch-2,
> branch-3.0.0-alpha1, and trunk, it could have fix versions of 2.6.5, 2.7.3,
> 2.8.0, 3.0.0-alpha1. The first two fix versions come from application of
> rule 1, and the last two fix versions come from rule 2.
>
> I'm very eager to move this discussion forward, so feel free to reach out
> on or off list if I can help with anything.
>


I think it is good practice to set multiple fix versions. However, it might
take the committers a little bit to learn.

Since the plan is to cut 3.0.0 off trunk, can we just bulk edit to add the
3.0.0-alphaX version?


>
> Best,
> Andrew
>

[jira] [Created] (HDFS-10696) TestHDFSCLI fails

[Akira Ajisaka (JIRA)]

Akira Ajisaka created HDFS-10696:


 Summary: TestHDFSCLI fails
 Key: HDFS-10696
 URL: https://issues.apache.org/jira/browse/HDFS-10696
 Project: Hadoop HDFS
  Issue Type: Bug
  Components: test
Reporter: Akira Ajisaka


TestHDFSCLI fails.
{noformat}2016-07-27 19:53:20,790 [main] INFO  cli.CLITestHelper 
(CLITestHelper.java:displayResults(177)) -  Comparator: 
[RegexpComparator]
2016-07-27 19:53:20,790 [main] INFO  cli.CLITestHelper 
(CLITestHelper.java:displayResults(179)) -  Comparision result:   [fail]
2016-07-27 19:53:20,791 [main] INFO  cli.CLITestHelper 
(CLITestHelper.java:displayResults(181)) - Expected output:   [^( 
|\t)*The storage type specific quota is cleared when -storageType option is 
specified.( )*]
2016-07-27 19:53:20,791 [main] INFO  cli.CLITestHelper 
(CLITestHelper.java:displayResults(183)) -   Actual output:   
[-clrSpaceQuota [-storageType ] ...: Clear the 
space quota for each directory .
For each directory, attempt to clear the quota. An error will 
be reported if
1. the directory does not exist or is a file, or
2. user is not an administrator.
It does not fault if the directory has no quota.
The storage type specific quota is cleared when -storageType 
option is specified.   Available storageTypes are 
- RAM_DISK
- DISK
- SSD
- ARCHIVE
]
{noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Wangda Tan]

Thanks Andrew for sharing your thoughts,

It looks better if we can put multiple versions on the fix version, with
that we can at least do some queries on JIRA to check the issues like "in
branch-2.6.5 but not in branch-2.7.4".

I still have a couple of questions:

*1) How CHANGES.txt (or release note) should look like?*
Not sure if this is discussed before. Previously we put the *next* release
version of earliest branch to CHANGES.txt. However, this could be confusing
and need a lot of manual works.

For example, we have two parallel release branches: branch-2.6.5 and
branch-2.7.4. When we need to backport a commit X in branch-2.7.4 to
branch-2.6.5, we will update CHANGES.txt in branch-2.7.4 to say this commit
X is included by Hadoop-2.6.5.

However, if we release Hadoop-2.7.4 before Hadoop-2.6.5, user will find the
Hadoop-2.6.5 is not released yet.

To me, we should put the fix version in CHANGES.txt to the released Hadoop
from the earliest branch, in the above example, Hadoop-2.7.4 should be the
fix version of commit X in release note of Hadoop-2.7.4.

Instead, I suggest to add a suffix ("released") to the fix version after
release is done. So the release note generator can do query easier, and
other user of JIRA can benefit from this to understand which releases
include a given JIRA.

*2) Do we need to update historical JIRAs?*

It's better to make a consistent rule for active release branches (to me
they're branch-2.6 and up). So it will be better to update fix version for
all resolved JIRAs in release branches.

Thoughts?

Wangda

On Tue, Jul 26, 2016 at 11:40 PM, Andrew Wang 
wrote:

> I think I understand a bit better, though now I ask how this date is
> different from the release date. Based on the HowToRelease instructions, we
> set the release date to when the release vote passes. So, start of release
> vote vs. end of release vote doesn't seem that different, and these dates
> are still totally ordered.
>
> For the user in this scenario, she can upgrade from 2.7.3 to any later
> 2.7.c release (found easily since a.b.c releases are ordered), and when
> jumping to a new minor or major version, any version released
> chronologically after 2.7.3. This means you need to check the website, but
> given that this is the way most enterprise software is versioned, I think
> it'll be okay by users.
>
> I think this problem is also pretty rare in practice, since users normally
> upgrade to the highest maintenance release within a major/minor. Thus
> they'll only hit this if their upgrade cycle is faster than it takes for a
> change released in e.g. 2.6.x to then also be released in a 2.7.x.
>
> Best,
> Andrew
>
> On Tue, Jul 26, 2016 at 11:13 PM, Tsuyoshi Ozawa  wrote:
>
> > > Andrew: I bet many would assume it's the release date, like how Ubuntu
> > releases are numbered.
> >
> > Good point. Maybe I confuse you because of lack of explanation.
> >
> > I assume that "branch-cut off timing" mean the timing of freezing branch
> > like when starting the release vote. It's because that the release can
> > be delayed after the release pass. Does it make sense to you?
> >
> > > Even if we have the branch-cut date in the version string, devs still
> > need to be aware of other branches and backport appropriately.
> >
> > Yes, you're right. The good point of including date is that we can
> declare
> > which version includes the latest changes. It helps users, not devs
> > basically, to decide which version users will use: e.g. if
> > 2.8.1-20160801 is released after 2.9.0-20160701 and a user uses
> > 2.7.3-20160701, she can update their cluster 2.8.1, which include bug
> fixes
> > against 2.7.3. Please let me know if I have some missing points.
> >
> > Thanks,
> > - Tsuyoshi
> >
> > On Wednesday, 27 July 2016, Andrew Wang 
> wrote:
> >
> >> Thanks for replies Akira and Tsuyoshi, inline:
> >>
> >> Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0,
> we
> >>> need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and
> we
> >>> don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a
> jira.
> >>> Is it right?
> >>
> >>
> >> Yes, correct.
> >>
> >>
> >>> Tsuyoshi: My suggestion is adding the date when branch cut is done:
> like
> >>> 3.0.0-alpha1-20160724, 2.8.0-20160730 or something.
> >>>
> >>> Pros:-) It's totally ordered. If we have a policy such as backporting
> >>> to maintainance branches after the date, users can find that which
> >>> version
> >>> is cutting edge. In the example of above, 2.8.0-20160730 can include
> bug
> >>> fixes which is not included in 3.0.0-alpha1-20160724.
> >>>
> >>> Cons:-( A bit redundant.
> >>>
> >>> Could you elaborate on the problem this scheme addresses? We always
> want
> >> our releases, when ordered chronologically, to incorporate all the known
> >> relevant bug fixes. Even if we have the branch-cut date in the version
> >> string, devs still need to be aware of 

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Tsuyoshi Ozawa]

> I think I understand a bit better, though now I ask how this date is 
> different from the release date.

OIC. I also assume that the freezing branch cannot include the changes
between freezing date and the release date. This is for strict
ordering to ensure which is the newer. If we have lots maintenance
branches, it helps us to understand which branches include fix a
problem of my cluster.

> I think this problem is also pretty rare in practice, since users normally 
> upgrade to the highest maintenance release within a major/minor.

If there will be lots maintenance branches in parallel(2.6.x, 2.7.x,
2.8.x, 2.9.x, 3.0.x), we can hit this problem more easily -  if a user
uses plan to upgrade 2.7.3 cluster to 2.8.3 or 2.9.1 or 3.0.1, which
version should the user choose? This is my concern.

However, as you mentioned, we decide to reduce the number of branches
we keep maintenance, we don't need to do that.

Best,
- Tsuyoshi

On Wed, Jul 27, 2016 at 3:40 PM, Andrew Wang  wrote:
> I think I understand a bit better, though now I ask how this date is
> different from the release date. Based on the HowToRelease instructions, we
> set the release date to when the release vote passes. So, start of release
> vote vs. end of release vote doesn't seem that different, and these dates
> are still totally ordered.
>
> For the user in this scenario, she can upgrade from 2.7.3 to any later 2.7.c
> release (found easily since a.b.c releases are ordered), and when jumping to
> a new minor or major version, any version released chronologically after
> 2.7.3. This means you need to check the website, but given that this is the
> way most enterprise software is versioned, I think it'll be okay by users.
>
> I think this problem is also pretty rare in practice, since users normally
> upgrade to the highest maintenance release within a major/minor. Thus
> they'll only hit this if their upgrade cycle is faster than it takes for a
> change released in e.g. 2.6.x to then also be released in a 2.7.x.
>
> Best,
> Andrew
>
> On Tue, Jul 26, 2016 at 11:13 PM, Tsuyoshi Ozawa  wrote:
>>
>> > Andrew: I bet many would assume it's the release date, like how Ubuntu
>> > releases are numbered.
>>
>> Good point. Maybe I confuse you because of lack of explanation.
>>
>> I assume that "branch-cut off timing" mean the timing of freezing branch
>> like when starting the release vote. It's because that the release can be
>> delayed after the release pass. Does it make sense to you?
>>
>> > Even if we have the branch-cut date in the version string, devs still
>> > need to be aware of other branches and backport appropriately.
>>
>> Yes, you're right. The good point of including date is that we can declare
>> which version includes the latest changes. It helps users, not devs
>> basically, to decide which version users will use: e.g. if 2.8.1-20160801 is
>> released after 2.9.0-20160701 and a user uses 2.7.3-20160701, she can update
>> their cluster 2.8.1, which include bug fixes against 2.7.3. Please let me
>> know if I have some missing points.
>>
>> Thanks,
>> - Tsuyoshi
>>
>> On Wednesday, 27 July 2016, Andrew Wang  wrote:
>>>
>>> Thanks for replies Akira and Tsuyoshi, inline:
>>>
 Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0,
 we need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and 
 we
 don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a jira. 
 Is
 it right?
>>>
>>>
>>> Yes, correct.
>>>

 Tsuyoshi: My suggestion is adding the date when branch cut is done: like
 3.0.0-alpha1-20160724, 2.8.0-20160730 or something.

 Pros:-) It's totally ordered. If we have a policy such as backporting
 to maintainance branches after the date, users can find that which
 version
 is cutting edge. In the example of above, 2.8.0-20160730 can include bug
 fixes which is not included in 3.0.0-alpha1-20160724.

 Cons:-( A bit redundant.

>>> Could you elaborate on the problem this scheme addresses? We always want
>>> our releases, when ordered chronologically, to incorporate all the known
>>> relevant bug fixes. Even if we have the branch-cut date in the version
>>> string, devs still need to be aware of other branches and backport
>>> appropriately.
>>>
>>> Given that branch cuts and releases might not happen in the same order
>>> (e.g. if 3.0.0-alpha1 precedes 2.8.0), I think this also would be confusing
>>> for users. I bet many would assume it's the release date, like how Ubuntu
>>> releases are numbered.
>>>
>>> Best,
>>> Andrew
>
>

-
To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Andrew Wang]

I think I understand a bit better, though now I ask how this date is
different from the release date. Based on the HowToRelease instructions, we
set the release date to when the release vote passes. So, start of release
vote vs. end of release vote doesn't seem that different, and these dates
are still totally ordered.

For the user in this scenario, she can upgrade from 2.7.3 to any later
2.7.c release (found easily since a.b.c releases are ordered), and when
jumping to a new minor or major version, any version released
chronologically after 2.7.3. This means you need to check the website, but
given that this is the way most enterprise software is versioned, I think
it'll be okay by users.

I think this problem is also pretty rare in practice, since users normally
upgrade to the highest maintenance release within a major/minor. Thus
they'll only hit this if their upgrade cycle is faster than it takes for a
change released in e.g. 2.6.x to then also be released in a 2.7.x.

Best,
Andrew

On Tue, Jul 26, 2016 at 11:13 PM, Tsuyoshi Ozawa  wrote:

> > Andrew: I bet many would assume it's the release date, like how Ubuntu
> releases are numbered.
>
> Good point. Maybe I confuse you because of lack of explanation.
>
> I assume that "branch-cut off timing" mean the timing of freezing branch
> like when starting the release vote. It's because that the release can
> be delayed after the release pass. Does it make sense to you?
>
> > Even if we have the branch-cut date in the version string, devs still
> need to be aware of other branches and backport appropriately.
>
> Yes, you're right. The good point of including date is that we can declare
> which version includes the latest changes. It helps users, not devs
> basically, to decide which version users will use: e.g. if
> 2.8.1-20160801 is released after 2.9.0-20160701 and a user uses
> 2.7.3-20160701, she can update their cluster 2.8.1, which include bug fixes
> against 2.7.3. Please let me know if I have some missing points.
>
> Thanks,
> - Tsuyoshi
>
> On Wednesday, 27 July 2016, Andrew Wang  wrote:
>
>> Thanks for replies Akira and Tsuyoshi, inline:
>>
>> Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0, we
>>> need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and we
>>> don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a jira.
>>> Is it right?
>>
>>
>> Yes, correct.
>>
>>
>>> Tsuyoshi: My suggestion is adding the date when branch cut is done: like
>>> 3.0.0-alpha1-20160724, 2.8.0-20160730 or something.
>>>
>>> Pros:-) It's totally ordered. If we have a policy such as backporting
>>> to maintainance branches after the date, users can find that which
>>> version
>>> is cutting edge. In the example of above, 2.8.0-20160730 can include bug
>>> fixes which is not included in 3.0.0-alpha1-20160724.
>>>
>>> Cons:-( A bit redundant.
>>>
>>> Could you elaborate on the problem this scheme addresses? We always want
>> our releases, when ordered chronologically, to incorporate all the known
>> relevant bug fixes. Even if we have the branch-cut date in the version
>> string, devs still need to be aware of other branches and backport
>> appropriately.
>>
>> Given that branch cuts and releases might not happen in the same order
>> (e.g. if 3.0.0-alpha1 precedes 2.8.0), I think this also would be confusing
>> for users. I bet many would assume it's the release date, like how Ubuntu
>> releases are numbered.
>>
>> Best,
>> Andrew
>>
>

[DISCUSS] Retire BKJM from trunk?

[Vinayakumar B]

Hi All,

   BKJM was Active and made much stable when the NameNode HA was implemented 
and there was no QJM implemented.
   Now QJM is present and is much stable which is adopted by many production 
environment.
   I wonder whether it would be a good time to retire BKJM from trunk?

   Are there any users of BKJM exists?

-Vinay

Re: [DISCUSS] Release numbering semantics with concurrent (>2) releases [Was Setting JIRA fix versions for 3.0.0 releases]

[Tsuyoshi Ozawa]

> Andrew: I bet many would assume it's the release date, like how Ubuntu
releases are numbered.

Good point. Maybe I confuse you because of lack of explanation.

I assume that "branch-cut off timing" mean the timing of freezing branch
like when starting the release vote. It's because that the release can
be delayed after the release pass. Does it make sense to you?

> Even if we have the branch-cut date in the version string, devs still
need to be aware of other branches and backport appropriately.

Yes, you're right. The good point of including date is that we can declare
which version includes the latest changes. It helps users, not devs
basically, to decide which version users will use: e.g. if
2.8.1-20160801 is released after 2.9.0-20160701 and a user uses
2.7.3-20160701, she can update their cluster 2.8.1, which include bug fixes
against 2.7.3. Please let me know if I have some missing points.

Thanks,
- Tsuyoshi

On Wednesday, 27 July 2016, Andrew Wang  wrote:

> Thanks for replies Akira and Tsuyoshi, inline:
>
> Akira: Assuming 3.0.0-alpha1 will be released between 2.7.0 and 2.8.0, we
>> need to add 3.0.0-alphaX if 2.8.0 is in the fix versions of a jira and we
>> don't need to add 3.0.0-alphaX if 2.7.0 is in the fix versions of a jira.
>> Is it right?
>
>
> Yes, correct.
>
>
>> Tsuyoshi: My suggestion is adding the date when branch cut is done: like
>> 3.0.0-alpha1-20160724, 2.8.0-20160730 or something.
>>
>> Pros:-) It's totally ordered. If we have a policy such as backporting
>> to maintainance branches after the date, users can find that which version
>> is cutting edge. In the example of above, 2.8.0-20160730 can include bug
>> fixes which is not included in 3.0.0-alpha1-20160724.
>>
>> Cons:-( A bit redundant.
>>
>> Could you elaborate on the problem this scheme addresses? We always want
> our releases, when ordered chronologically, to incorporate all the known
> relevant bug fixes. Even if we have the branch-cut date in the version
> string, devs still need to be aware of other branches and backport
> appropriately.
>
> Given that branch cuts and releases might not happen in the same order
> (e.g. if 3.0.0-alpha1 precedes 2.8.0), I think this also would be confusing
> for users. I bet many would assume it's the release date, like how Ubuntu
> releases are numbered.
>
> Best,
> Andrew
>