Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Zhihai Xu]

Thanks Vinod.

+1 (non-binding)

* Downloaded and built from source
* Checked LICENSE and NOTICE
* Deployed a pseudo cluster
* Ran through MR and HDFS tests
* verified basic HDFS operations and Pi job.

Zhihai

On Fri, Jul 22, 2016 at 7:15 PM, Vinod Kumar Vavilapalli  wrote:

> Hi all,
>
> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>
> As discussed before, this is the next maintenance release to follow up
> 2.7.2.
>
> The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>
> The RC tag in git is: release-2.7.3-RC0
>
> The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/>
>
> The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
>
> As you may have noted, a very long fix-cycle for the License & Notice
> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
> to slip by quite a bit. This release's related discussion thread is linked
> below: [1].
>
> Please try the release and vote; the vote will run for the usual 5 days.
>
> Thanks,
> Vinod
>
> [1]: 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>

Re: Setting JIRA fix versions for 3.0.0 releases

[Andrew Wang]

Thanks Vinod+Wangda for the comments,

Above, Allen and I discussed a proposal which I think is reasonable and
also lines up well with our current approach. If you feel something is
underspecified or needs improvement, please raise those points.

We have been doing concurrent releases with the 2.6.x and 2.7.x lines, so
it's not entirely new ground. I believe we also haven't been doing
purely-chronological changelogs, since a JIRA can have fix versions of both
2.6.x and 2.7.x. The above proposal can be viewed as an extension of what
we're already doing for concurrent minor release branches to major release
branches.

The JIRA bulk update I want do to is add 3.0.0-alpha1 versions. Since we're
only adding, it won't affect the 2.8.0 or other release notes. I offered to
do a 2.8.0 JIRA bulk update since I already have the script, but I can also
just publish it and let someone else handle 2.8.0.

We do need to agree on the ordering of 2.8.0 and 3.0.0-alpha1 so 3's "base"
version makes sense chronologically. I'd like to base 3 off of 2.7.0, since
2.8.0 still has a number of unresolved blockers.

Finally, with some hackery, we've recently been able to compile the
projects in CDH against a Hadoop 3 snapshot. As part of the HDFS EC effort,
many contributors have already been testing Hadoop 3 snapshots on
multi-node clusters. So, I have confidence that what we have now is already
a useful artifact for downstreams, since downstreams are already trying to
consume it.

Best,
Andrew

On Mon, Jul 25, 2016 at 5:57 PM, Wangda Tan  wrote:

> Hi Andrew,
> Please wait updating fix version for branch-2 committed tickets before we
> get a consensus on this. Updating fix versions for them could bring lots of
> troubles for on going two releases (2.7.3 / 2.8.0).
>
> Thanks,
> Wangda
>
> On Mon, Jul 25, 2016 at 11:02 AM, Andrew Wang 
> wrote:
>
>> If I don't hear otherwise, I'd like to go ahead and do the bulk fix
>> version
>> update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
>> tomorrow in case there's more discussion. If any one is willing to help
>> review my script and JIRA queries, that'd also be great.
>>
>> I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
>> be a very similar process.
>>
>> Best,
>> Andrew
>>
>>
>> On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer <
>> a...@effectivemachines.com>
>> wrote:
>>
>> >
>> > > On Jul 22, 2016, at 7:16 PM, Andrew Wang 
>> > wrote:
>> > >
>> > > Does this mean you find our current system of listing a JIRA as being
>> > fixed in both a 2.6.x and 2.7.x to be confusing?
>> >
>> > Nope.  I'm only confused when there isn't a .0 release in the
>> fix
>> > line.  When I see 2.6.x and 2.7.x I know that it was back ported to
>> those
>> > branches.  If I don't see a .0, I figure it's either a mistake or
>> something
>> > that was already fixed by another change in that major/minor branch.
>> It's
>> > almost always the former, however.
>> >
>> > > FWIW, my usecase is normally not "what is the earliest release that
>> has
>> > this fix?" but rather "is this fix in this release?". If it's easy to
>> query
>> > the latter, you can also determine the former. Some kind of query tool
>> > could help here.
>> >
>> > It literally becomes a grep if people commit the release data
>> into
>> > the source tree, the release data is correct, etc:
>> >
>> > $ mvn install site  -Preleasedocs -Pdocs -DskipTests
>> > $ grep issueid
>> > hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
>> >
>> > We should probably update the release process to make sure that
>> > *in progress* release data is also committed when a .0 is cut.  That's
>> > likely missing. Another choice would be to modify the pom to that runs
>> > releasedocmaker to use a range rather than single version, but that
>> gets a
>> > bit tricky with release dates, how big of a range, etc.  Not impossible,
>> > just tricky.  Probably needs to be script that gets run as part of
>> > create-release, maybe?
>> >
>> > (In reality, I do this grep against my git repo that generates
>> the
>> > change log data automatically.  This way it is always up-to-date and not
>> > dependent upon release data being committed.  But that same grep could
>> be
>> > done with a JQL query just as easily.)
>> >
>> > > For the release notes, am I correct in interpreting this as:
>> > >
>> > > * diff a.0.0 from the previous x.y.0 release
>> > > * diff a.b.0  from the previous a.0.0 or a.b.0 release
>> > > * diff a.b.c from the previous a.b.0 or a.b.c release
>> >
>> > Pretty much yes.
>> >
>> > > Ray pointed me at the changelogs of a few other enterprise software
>> > products, and this strategy seems pretty common. I like it.
>> >
>> > It's extremely common, to the point that putting every fix for
>> > every release touched is, at least to me, weird and extremely
>> > unconventional.
>> >
>> > > I realize now that this means a lot more JIRAs will need the 2

Re: Setting JIRA fix versions for 3.0.0 releases

[Wangda Tan]

Hi Andrew,
Please wait updating fix version for branch-2 committed tickets before we
get a consensus on this. Updating fix versions for them could bring lots of
troubles for on going two releases (2.7.3 / 2.8.0).

Thanks,
Wangda

On Mon, Jul 25, 2016 at 11:02 AM, Andrew Wang 
wrote:

> If I don't hear otherwise, I'd like to go ahead and do the bulk fix version
> update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
> tomorrow in case there's more discussion. If any one is willing to help
> review my script and JIRA queries, that'd also be great.
>
> I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
> be a very similar process.
>
> Best,
> Andrew
>
>
> On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer <
> a...@effectivemachines.com>
> wrote:
>
> >
> > > On Jul 22, 2016, at 7:16 PM, Andrew Wang 
> > wrote:
> > >
> > > Does this mean you find our current system of listing a JIRA as being
> > fixed in both a 2.6.x and 2.7.x to be confusing?
> >
> > Nope.  I'm only confused when there isn't a .0 release in the fix
> > line.  When I see 2.6.x and 2.7.x I know that it was back ported to those
> > branches.  If I don't see a .0, I figure it's either a mistake or
> something
> > that was already fixed by another change in that major/minor branch.
> It's
> > almost always the former, however.
> >
> > > FWIW, my usecase is normally not "what is the earliest release that has
> > this fix?" but rather "is this fix in this release?". If it's easy to
> query
> > the latter, you can also determine the former. Some kind of query tool
> > could help here.
> >
> > It literally becomes a grep if people commit the release data
> into
> > the source tree, the release data is correct, etc:
> >
> > $ mvn install site  -Preleasedocs -Pdocs -DskipTests
> > $ grep issueid
> > hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
> >
> > We should probably update the release process to make sure that
> > *in progress* release data is also committed when a .0 is cut.  That's
> > likely missing. Another choice would be to modify the pom to that runs
> > releasedocmaker to use a range rather than single version, but that gets
> a
> > bit tricky with release dates, how big of a range, etc.  Not impossible,
> > just tricky.  Probably needs to be script that gets run as part of
> > create-release, maybe?
> >
> > (In reality, I do this grep against my git repo that generates
> the
> > change log data automatically.  This way it is always up-to-date and not
> > dependent upon release data being committed.  But that same grep could be
> > done with a JQL query just as easily.)
> >
> > > For the release notes, am I correct in interpreting this as:
> > >
> > > * diff a.0.0 from the previous x.y.0 release
> > > * diff a.b.0  from the previous a.0.0 or a.b.0 release
> > > * diff a.b.c from the previous a.b.0 or a.b.c release
> >
> > Pretty much yes.
> >
> > > Ray pointed me at the changelogs of a few other enterprise software
> > products, and this strategy seems pretty common. I like it.
> >
> > It's extremely common, to the point that putting every fix for
> > every release touched is, at least to me, weird and extremely
> > unconventional.
> >
> > > I realize now that this means a lot more JIRAs will need the 2.8.0 fix
> > version, since they only have 2.6.x and 2.7.x.
> >
> > Yup.
> >
> > >   This makes the fix rules actually pretty easy:  the lowest a.b.0
> > release and all non-.0 releases.
> > >
> > > I think this needs to be amended to handle the case of multiple major
> > release branches, since we could have something committed for both 2.9.0
> > and 3.1.0. So "lowest a.b.0 release within each major version"?
> >
> > Yeah, switching to effectively trunk-based development makes the
> > rules harder.  It's one of the reasons why the two big enterprisey
> > companies I worked at prior to working on Hadoop didn't really do
> > trunk-based for the vast majority of projects.  They always cut a branch
> > (or equivalent for that SCM) to delineate a break.   Given the amount of
> > ex-Sun folks involved in the early days of Hadoop, our pre-existing
> > development processes very much reflect that culture.
> >
> > > This was true previously (no releases from trunk, trunk is versioned
> > a.0.0), but now that trunk is essentially a minor release branch, its fix
> > version needs to be treated as such.
> >
> > Yeah, I misspoke a bit when dealing with a head-of-tree model.
> > 3.0.0-alpha1 will generate different notes than 3.0.0-alpha2, obviously.
> > Every 3.0.0-(label) release is effectively a major version in that case.
> >
> >
> >
>

Re: Setting JIRA fix versions for 3.0.0 releases

[Wangda Tan]

I just filed ticket https://issues.apache.org/jira/browse/HADOOP-13423 to
track running JDIFF on trunk and analyze results for Hadoop-common. I will
work on that and keep the JIRA and this thread updated. We need to do the
same work for YARN/MR/HDFS.

On Mon, Jul 25, 2016 at 5:47 PM, Wangda Tan  wrote:

> I agree with what Vinod mentioned: we need to revisit all incompatible
> changes and revert unnecessary ones. Even if we don't have any
> compatibility guarantees between 2.x and 3.x. But make user to be less
> frustrated while trying 3.x is always a better option to me.
>
> To achieve this we need to run jdiff for trunk and look at results. I
> would suggest to do this before 3.0.0-alpha1 release.
>
> In addition, for people who will try this 3-alpha1 release artifacts, a
> guide about migration from 2.x to 3.x will be very helpful, and it can also
> help for people to better understand what have changed (Just like
> http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
> )
>
> Thoughts?
>
> Thanks,
> Wangda
>
>
> On Thu, Jul 21, 2016 at 2:41 PM, Sean Busbey  wrote:
>
>> On Thu, Jul 21, 2016 at 4:32 PM, Vinod Kumar Vavilapalli
>>  wrote:
>> >> I really, really want a 3.0.0-alpha1 ASAP, since it's basically
>> impossible for downstreams to test incompat changes and new features
>> without a release artifact. I've been doing test builds, and
>> branch-3.0.0-alpha1 is ready for an RC besides possibly this fix version
>> issue.
>> >
>> > Not arguing against the need for an alpha release, the question is if
>> it can wait till after 2.8 gets done.
>> >
>> > Orthogonally, do we have a report of the incompatible changes? Like the
>> one I generated for some of the branch-2 releases using late jdiff work
>> from Li Lu etc. We should do this and fix any inadvertant
>> incompatibilities. Without seeing this list of incompatibilities, why even
>> make an alpha release and force downstream components to discover issues
>> what can be identified through running reports.
>> >
>>
>> I can come up with this, atleast for Source / Binary API compatibility,
>> provided folks don't mind if I use the Java API Compliance Checker[1]
>> instead of jdiff.
>>
>> I'm already familiar with quickly using it, esp with Audience
>> Annotations from my work in HBase.
>>
>> Do you want this check from some particular branch-2 release? It
>> matters since the releases along branch-2 have themselves had some
>> noise[2].
>>
>> [1]: https://github.com/lvc/japi-compliance-checker
>> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/
>>
>> --
>> busbey
>>
>> -
>> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>>
>>
>

Re: Setting JIRA fix versions for 3.0.0 releases

[Wangda Tan]

I agree with what Vinod mentioned: we need to revisit all incompatible
changes and revert unnecessary ones. Even if we don't have any
compatibility guarantees between 2.x and 3.x. But make user to be less
frustrated while trying 3.x is always a better option to me.

To achieve this we need to run jdiff for trunk and look at results. I would
suggest to do this before 3.0.0-alpha1 release.

In addition, for people who will try this 3-alpha1 release artifacts, a
guide about migration from 2.x to 3.x will be very helpful, and it can also
help for people to better understand what have changed (Just like
http://hadoop.apache.org/docs/current/hadoop-mapreduce-client/hadoop-mapreduce-client-core/MapReduce_Compatibility_Hadoop1_Hadoop2.html
)

Thoughts?

Thanks,
Wangda


On Thu, Jul 21, 2016 at 2:41 PM, Sean Busbey  wrote:

> On Thu, Jul 21, 2016 at 4:32 PM, Vinod Kumar Vavilapalli
>  wrote:
> >> I really, really want a 3.0.0-alpha1 ASAP, since it's basically
> impossible for downstreams to test incompat changes and new features
> without a release artifact. I've been doing test builds, and
> branch-3.0.0-alpha1 is ready for an RC besides possibly this fix version
> issue.
> >
> > Not arguing against the need for an alpha release, the question is if it
> can wait till after 2.8 gets done.
> >
> > Orthogonally, do we have a report of the incompatible changes? Like the
> one I generated for some of the branch-2 releases using late jdiff work
> from Li Lu etc. We should do this and fix any inadvertant
> incompatibilities. Without seeing this list of incompatibilities, why even
> make an alpha release and force downstream components to discover issues
> what can be identified through running reports.
> >
>
> I can come up with this, atleast for Source / Binary API compatibility,
> provided folks don't mind if I use the Java API Compliance Checker[1]
> instead of jdiff.
>
> I'm already familiar with quickly using it, esp with Audience
> Annotations from my work in HBase.
>
> Do you want this check from some particular branch-2 release? It
> matters since the releases along branch-2 have themselves had some
> noise[2].
>
> [1]: https://github.com/lvc/japi-compliance-checker
> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/
>
> --
> busbey
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (YARN-5429) Fix @return related javadoc warnings in yarn-api

[Vrushali C (JIRA)]

Vrushali C created YARN-5429:


 Summary: Fix @return related javadoc warnings in yarn-api
 Key: YARN-5429
 URL: https://issues.apache.org/jira/browse/YARN-5429
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Vrushali C
Assignee: Vrushali C



As part of YARN-4977, filing a subtask to fix a subset of the javadoc warnings 
in yarn-api.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Jason Lowe]

+1 (binding)
- Verified signatures and digests- Built from source with native support- 
Deployed a pseudo-distributed cluster- Ran some sample jobs
Jason

  From: Vinod Kumar Vavilapalli 
 To: "common-...@hadoop.apache.org" ; 
hdfs-...@hadoop.apache.org; yarn-dev@hadoop.apache.org; 
"mapreduce-...@hadoop.apache.org"  
Cc: Vinod Kumar Vavilapalli 
 Sent: Friday, July 22, 2016 9:15 PM
 Subject: [VOTE] Release Apache Hadoop 2.7.3 RC0
   
Hi all,

I've created a release candidate RC0 for Apache Hadoop 2.7.3.

As discussed before, this is the next maintenance release to follow up 2.7.2.

The RC is available for validation at: 
http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ 


The RC tag in git is: release-2.7.3-RC0

The maven artifacts are available via repository.apache.org 
 at 
https://repository.apache.org/content/repositories/orgapachehadoop-1040/ 


The release-notes are inside the tar-balls at location 
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I hosted 
this at http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html 
 for your 
quick perusal.

As you may have noted, a very long fix-cycle for the License & Notice issues 
(HADOOP-12893) caused 2.7.3 (along with every other Hadoop release) to slip by 
quite a bit. This release's related discussion thread is linked below: [1].

Please try the release and vote; the vote will run for the usual 5 days.

Thanks,
Vinod

[1]: 2.7.3 release plan: 
https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html 


   

Re: [Release thread] 2.8.0 release activities

[Vinod Kumar Vavilapalli]

Thanks for all that great help moving this release forward, Jian, Sangjin, 
Wangda et al! Appreciate it.

The License / Notice fix is finally in. And I pushed out a 2.7.3 RC0 last week.

I only see 9 blocker / critical fixes pending for 2.8.0 - 
https://issues.apache.org/jira/issues/?filter=12334985. Let’s do this!

Thanks
+Vinod

> On May 11, 2016, at 6:04 PM, Jian He  wrote:
> 
> For MapReduce/YARN, I closed a few staled ones. Only 4 jiras needs attention 
> for 2.8
> 
> MAPREDUCE-6288
> YARN-1815
> YARN-4685
> YARN-4844
> 
> The rest are either improvements or long-standing issues and does not qualify 
> release blocker, IMO.
> I think we’ll try to get these 4 jiras in asap. The rest will be on best 
> effort, resolve as much as possible and move them out if not resolved in time.
> 
> Jian
> 
> On May 11, 2016, at 5:37 PM, Wangda Tan 
> mailto:wheele...@gmail.com>> wrote:
> 
> Sounds good to me :).
> 
> Jian and I have looked at all existing 2.8.0 blockers and criticals today.
> To me more than half of MR/YARN blockers/criticals of 2.8 should be moved
> out. Left comments on these JIRAs asked original owners, plan to update
> target version of these JIRAs early next week.
> 
> Will keep this thread updated.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 5:06 PM, Sangjin Lee 
> mailto:sj...@apache.org>> wrote:
> 
> How about this? I'll review the HADOOP/HDFS bugs in that list to come up
> with true blockers for 2.8.0 or JIRAs that are close to being ready. I'll
> report the list here. Then folks can chime in if you agree
> 
> Perhaps Wangda, you can go over the YARN/MR bugs. Sound like a plan?
> 
> Thanks,
> Sangjin
> 
> On Wed, May 11, 2016 at 4:26 PM, Wangda Tan 
> mailto:wheele...@gmail.com>> wrote:
> 
> +1, we should close such staled JIRAs to avoid doing unnecessary checks
> for
> every releases.
> 
> I'm working on reviewing YARN/MR critical/blocker patches currently, it
> gonna very helpful if someone else can help with reviewing Common/HDFS
> JIRAs.
> 
> Thanks,
> Wangda
> 
> 
> On Wed, May 11, 2016 at 4:20 PM, Sangjin Lee 
> mailto:sj...@apache.org>> wrote:
> 
> Where do we stand in terms of closing out blocker/critical issues for
> 2.8.0? I still see 50 open JIRAs in Vinod's list:
> https://issues.apache.org/jira/issues/?filter=12334985
> 
> But I see a lot of JIRAs with no patches or very stale patches. It
> would be
> a good exercise to come up with the list of JIRAs that we need to block
> 2.8.0 for and focus our attention on closing them out. Thoughts?
> 
> Thanks,
> Sangjin
> 
> On Sat, Apr 23, 2016 at 5:05 AM, Steve Loughran 
> mailto:ste...@hortonworks.com>
> 
> wrote:
> 
> 
> On 23 Apr 2016, at 01:24, Vinod Kumar Vavilapalli <
> vino...@apache.org>
> wrote:
> 
> We are not converging - there’s still 58 more. I need help from the
> community in addressing / review 2.8.0 blockers. If folks can start
> with
> reviewing Patch available tickets, that’ll be great.
> 
> 
> 
> 
> I'm still doing the s3a stuff, other people testing and reviewing this
> stuff welcome.
> 
> in particular, I could do with others playing with this patch of mine,
> which adds counters and things into S3a, based on the azure
> instrumentation
> 
> https://issues.apache.org/jira/browse/HADOOP-13028
> 
> 
> 
> 
> 
> 
> 
> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Please don’t do this for 2.8.0 against 2.7.0 till we change our existing 
release ordering conventions.

Till now, I made sure 2.8.0 only has issues that are not in 2.7.3 - per the 
2.8.0-follows-2.7.3 convention. As part of the release process, I will fix the 
remaining tickets to follow this same convention. Again till we create new 
rules.

Thanks
+Vinod

> On Jul 25, 2016, at 11:02 AM, Andrew Wang  wrote:
> 
> If I don't hear otherwise, I'd like to go ahead and do the bulk fix version
> update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
> tomorrow in case there's more discussion. If any one is willing to help
> review my script and JIRA queries, that'd also be great.
> 
> I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
> be a very similar process.
> 
> Best,
> Andrew
> 
> 
> On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer 
> wrote:
> 
>> 
>>> On Jul 22, 2016, at 7:16 PM, Andrew Wang 
>> wrote:
>>> 
>>> Does this mean you find our current system of listing a JIRA as being
>> fixed in both a 2.6.x and 2.7.x to be confusing?
>> 
>>Nope.  I'm only confused when there isn't a .0 release in the fix
>> line.  When I see 2.6.x and 2.7.x I know that it was back ported to those
>> branches.  If I don't see a .0, I figure it's either a mistake or something
>> that was already fixed by another change in that major/minor branch.  It's
>> almost always the former, however.
>> 
>>> FWIW, my usecase is normally not "what is the earliest release that has
>> this fix?" but rather "is this fix in this release?". If it's easy to query
>> the latter, you can also determine the former. Some kind of query tool
>> could help here.
>> 
>>It literally becomes a grep if people commit the release data into
>> the source tree, the release data is correct, etc:
>> 
>> $ mvn install site  -Preleasedocs -Pdocs -DskipTests
>> $ grep issueid
>> hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
>> 
>>We should probably update the release process to make sure that
>> *in progress* release data is also committed when a .0 is cut.  That's
>> likely missing. Another choice would be to modify the pom to that runs
>> releasedocmaker to use a range rather than single version, but that gets a
>> bit tricky with release dates, how big of a range, etc.  Not impossible,
>> just tricky.  Probably needs to be script that gets run as part of
>> create-release, maybe?
>> 
>>(In reality, I do this grep against my git repo that generates the
>> change log data automatically.  This way it is always up-to-date and not
>> dependent upon release data being committed.  But that same grep could be
>> done with a JQL query just as easily.)
>> 
>>> For the release notes, am I correct in interpreting this as:
>>> 
>>> * diff a.0.0 from the previous x.y.0 release
>>> * diff a.b.0  from the previous a.0.0 or a.b.0 release
>>> * diff a.b.c from the previous a.b.0 or a.b.c release
>> 
>>Pretty much yes.
>> 
>>> Ray pointed me at the changelogs of a few other enterprise software
>> products, and this strategy seems pretty common. I like it.
>> 
>>It's extremely common, to the point that putting every fix for
>> every release touched is, at least to me, weird and extremely
>> unconventional.
>> 
>>> I realize now that this means a lot more JIRAs will need the 2.8.0 fix
>> version, since they only have 2.6.x and 2.7.x.
>> 
>>Yup.
>> 
>>>  This makes the fix rules actually pretty easy:  the lowest a.b.0
>> release and all non-.0 releases.
>>> 
>>> I think this needs to be amended to handle the case of multiple major
>> release branches, since we could have something committed for both 2.9.0
>> and 3.1.0. So "lowest a.b.0 release within each major version"?
>> 
>>Yeah, switching to effectively trunk-based development makes the
>> rules harder.  It's one of the reasons why the two big enterprisey
>> companies I worked at prior to working on Hadoop didn't really do
>> trunk-based for the vast majority of projects.  They always cut a branch
>> (or equivalent for that SCM) to delineate a break.   Given the amount of
>> ex-Sun folks involved in the early days of Hadoop, our pre-existing
>> development processes very much reflect that culture.
>> 
>>> This was true previously (no releases from trunk, trunk is versioned
>> a.0.0), but now that trunk is essentially a minor release branch, its fix
>> version needs to be treated as such.
>> 
>>Yeah, I misspoke a bit when dealing with a head-of-tree model.
>> 3.0.0-alpha1 will generate different notes than 3.0.0-alpha2, obviously.
>> Every 3.0.0-(label) release is effectively a major version in that case.
>> 
>> 
>> 


-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

>> There's a plan for more 3.0.0 alphas, so there's still time to help out
> before things settle down for beta. If 2.8.0 is ready to go, it should
> happen before even alpha2.

I wasn’t talk about my irresistible urge to help (which of course is there : )) 
, it was about making sure enough eye-balls look through this. If it absolutely 
cannot wait (which I don’t still understand why), we will just have to do 
double-shifts I guess.

>> 
>> Obviously, I am not making the case that this issue won’t happen ever. In
>> fact, this already happened with the parallel 2.6.x and 2.7.x releases. And
>> we precisely avoided major confusion there by lining up 2.7.2 behind 2.6.3
>> etc.
>> 
> 
> Could you clarify how lining up releases differently avoids the fix version
> issue?

It wouldn’t avoid the issue, it reduces it by quite a bit.

The versioning discussion happened a lot of times before and we never got to 
semantic versioning or any such convention. If it helps, we can revive the 
discussion. So far, all the releases have documented as the change-list only 
the issues “that are not in the last release, date-wise, whether in this major 
number or the ones below”. It is a simple timeline ordering, and it worked okay 
with two concurrent releases. So, if 2.8.0 happens before 3.0.0-alpha1, the 
additional changes that make the change-list for 3.0.0-alpha1 will be a delta 
of trunk-only changes.

Again, lining up releses doesn’t fix the core issue of running parallel (>2) 
releases - it just continues the current order of things. For now, the only 
tool we have is timeline ordering of 2 releases. The typical question from 
anyone is “which version did this get fixed in” and the answer is always found 
from JIRA + svn/git-log. And the fact that the (>2) concurrent releases is a 
new ground for this community (yes, you are hearing it right, this didn’t ever 
happen before for an extended period of time), we need to make some new rules 
before we make more of these releases and make it harder to follow for everyone.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Wangda Tan]

Thanks Vinod.

+1 (Binding)

- Built from source and deploy a pseudo cluster locally
- Run DS on YARN
- With/Without node labels enabled.

- Wangda


On Mon, Jul 25, 2016 at 1:55 PM, Mingliang Liu  wrote:

> Thanks Vinod.
>
> +1 (non-binding)
>
> * Downloaded and built from source (with Java 8)
> * Checked LICENSE and NOTICE files
> * Verified MD5 signatures
> * Installed pseudo-distributed instance (Mac OS X 10.11)
> * Ran through HDFS and mapreduce tests
> * Operate HDFS from command line
> * Run tools like distcp/NNThroughputBenchmark
>
> L
>
> > On Jul 22, 2016, at 7:15 PM, Vinod Kumar Vavilapalli 
> wrote:
> >
> > Hi all,
> >
> > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> >
> > As discussed before, this is the next maintenance release to follow up
> 2.7.2.
> >
> > The RC is available for validation at:
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> >
> > The RC tag in git is: release-2.7.3-RC0
> >
> > The maven artifacts are available via repository.apache.org <
> http://repository.apache.org/> at
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/ <
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/>
> >
> > The release-notes are inside the tar-balls at location
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> hosted this at
> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html> for
> your quick perusal.
> >
> > As you may have noted, a very long fix-cycle for the License & Notice
> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
> to slip by quite a bit. This release's related discussion thread is linked
> below: [1].
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> > Thanks,
> > Vinod
> >
> > [1]: 2.7.3 release plan:
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
> http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: Setting JIRA fix versions for 3.0.0 releases

[Vinod Kumar Vavilapalli]

Actually, I wouldn’t trust this report as it stands today at all.

I quickly glanced the report, looking for what it highlights as incompatible. 
But the ones that I saw have private / visible for testing annotations. Other 
than acting as useless evidence for those lashing out on branch-2, this won’t 
do much good.

Whenever we start working towards switching to this tool, it should incorporate 
the same exclude-annotations logic that the jdiff code-path does today. Do you 
think that is possible?

Thanks
+Vinod

> On Jul 22, 2016, at 4:53 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
>> Do you want this check from some particular branch-2 release? It
>> matters since the releases along branch-2 have themselves had some
>> noise[2].
>> 
>> [1]: https://github.com/lvc/japi-compliance-checker 
>>  
>> > >
>> [2]: http://abi-laboratory.pro/java/tracker/timeline/hadoop/ 
>>  
>> > >
>> 
>> -- 
>> busbey

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Allen Wittenauer]

> On Jul 25, 2016, at 1:16 PM, Sangjin Lee  wrote:
> 
> Also:  right now, the non-Linux and/or non-x86 platforms have to supply their 
> own leveldbjni jar (or at least the C level library?) in order to make YARN 
> even functional.  How is that going to work with the class path manipulation?
> 
> First, the native libraries are orthogonal to this. They're not governed by 
> the java classpath.
> 
> For those platforms where users/admins need to provide their own LevelDB 
> libraries, the only requirement would be to add them to the 
> share/hadoop/.../lib directory. I don't think we would ask end users of the 
> clusters to bring in their own LevelDB library as it would not be an end-user 
> concern. I assume the administrators of clusters (still users but not end 
> users) would add it to the clusters. The classpath isolation doesn't really 
> have an impact on that.
> 

$ jar tf leveldbjni-all-1.8.jar | grep native
META-INF/native/
META-INF/native/linux32/
META-INF/native/linux32/libleveldbjni.so
META-INF/native/linux64/
META-INF/native/linux64/libleveldbjni.so
META-INF/native/osx/
META-INF/native/osx/libleveldbjni.jnilib
META-INF/native/windows32/
META-INF/native/windows32/leveldbjni.dll
META-INF/native/windows64/
META-INF/native/windows64/leveldbjni.dll



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

Re: [DISCUSS] The order of classpath isolation work and updating/shading dependencies on trunk

[Sangjin Lee]

On Fri, Jul 22, 2016 at 5:15 PM, Allen Wittenauer 
wrote:

>
> But if I don't use ApplicationClassLoader, my java app is basically
> screwed then, right?
>

If we start upgrading the libraries aggressively, then it would also mean
that the ApplicationClassLoader should be more of the default than the
other way around (i.e. opt-out rather than opt-in). If we're not willing to
go there, then we cannot be too aggressive in upgrading libraries.

I'm not sure what you mean by "my java app is basically screwed", but if
you meant whether your java app would be OK if hadoop upgraded libraries
aggressively and you don't use the ApplicationClassLoader, then yes.


>
> Also:  right now, the non-Linux and/or non-x86 platforms have to supply
> their own leveldbjni jar (or at least the C level library?) in order to
> make YARN even functional.  How is that going to work with the class path
> manipulation?
>

First, the native libraries are orthogonal to this. They're not governed by
the java classpath.

For those platforms where users/admins need to provide their own LevelDB
libraries, the only requirement would be to add them to the
share/hadoop/.../lib directory. I don't think we would ask end users of the
clusters to bring in their own LevelDB library as it would not be an
end-user concern. I assume the administrators of clusters (still users but
not end users) would add it to the clusters. The classpath isolation
doesn't really have an impact on that.


>
> > On Jul 22, 2016, at 9:57 AM, Sangjin Lee  wrote:
> >
> > The work on HADOOP-13070 and the ApplicationClassLoader are generic and
> go beyond YARN. It can be used in any JVM that uses hadoop. The current use
> cases are MR containers, hadoop's RunJar (as in "hadoop jar"), and the YARN
> node manager auxiliary services. I'm not sure if that's what you were
> asking, but I hope it helps.
> >
> > Regards,
> > Sangjin
> >
> > On Fri, Jul 22, 2016 at 9:16 AM, Sean Busbey 
> wrote:
> > My work on HADOOP-11804 *only* helps processes that sit outside of YARN.
> :)
> >
> > On Fri, Jul 22, 2016 at 10:48 AM, Allen Wittenauer
> >  wrote:
> > >
> > > Does any of this work actually help processes that sit outside of YARN?
> > >
> > >> On Jul 21, 2016, at 12:29 PM, Sean Busbey 
> wrote:
> > >>
> > >> thanks for bringing this up! big +1 on upgrading dependencies for 3.0.
> > >>
> > >> I have an updated patch for HADOOP-11804 ready to post this week. I've
> > >> been updating HBase's master branch to try to make use of it, but
> > >> could use some other reviews.
> > >>
> > >> On Thu, Jul 21, 2016 at 4:30 AM, Tsuyoshi Ozawa 
> wrote:
> > >>> Hi developers,
> > >>>
> > >>> I'd like to discuss how to make an advance towards dependency
> > >>> management in Apache Hadoop trunk code since there has been lots work
> > >>> about updating dependencies in parallel. Summarizing recent works and
> > >>> activities as follows:
> > >>>
> > >>> 0) Currently, we have merged minimum update dependencies for making
> > >>> Hadoop JDK-8 compatible(compilable and runnable on JDK-8).
> > >>> 1) After that, some people suggest that we should update the other
> > >>> dependencies on trunk(e.g. protobuf, netty, jackthon etc.).
> > >>> 2) In parallel, Sangjin and Sean are working on classpath isolation:
> > >>> HADOOP-13070, HADOOP-11804 and HADOOP-11656.
> > >>>
> > >>> Main problems we try to solve in the activities above is as follows:
> > >>>
> > >>> * 1) tries to solve dependency hell between user-level jar and
> > >>> system(Hadoop)-level jar.
> > >>> * 2) tries to solve updating old libraries.
> > >>>
> > >>> IIUC, 1) and 2) looks not related, but it's related in fact. 2) tries
> > >>> to separate class loader between client-side dependencies and
> > >>> server-side dependencies in Hadoop, so we can the change policy of
> > >>> updating libraries after doing 2). We can also decide which libraries
> > >>> can be shaded after 2).
> > >>>
> > >>> Hence, IMHO, a straight way we should go to is doing 2 at first.
> > >>> After that, we can update both client-side and server-side
> > >>> dependencies based on new policy(maybe we should discuss what kind of
> > >>> incompatibility is acceptable, and the others are not).
> > >>>
> > >>> Thoughts?
> > >>>
> > >>> Thanks,
> > >>> - Tsuyoshi
> > >>>
> > >>> -
> > >>> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > >>> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> busbey
> > >>
> > >> -
> > >> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > >> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > >>
> > >
> > >
> > > -
> > > To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: hdfs-dev-h...

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[larry mccay]

Oops - make that:

+1 (non-binding)

On Sun, Jul 24, 2016 at 4:07 PM, larry mccay  wrote:

> +1 binding
>

> * downloaded and built from source
> * checked LICENSE and NOTICE files
> * verified signatures
> * ran standalone tests
> * installed pseudo-distributed instance on my mac
> * ran through HDFS and mapreduce tests
> * tested credential command
> * tested webhdfs access through Apache Knox
>
>
> On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Hi all,
>>
>> I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>>
>> As discussed before, this is the next maintenance release to follow up
>> 2.7.2.
>>
>> The RC is available for validation at:
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>>
>> The RC tag in git is: release-2.7.3-RC0
>>
>> The maven artifacts are available via repository.apache.org <
>> http://repository.apache.org/> at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
>> > >
>>
>> The release-notes are inside the tar-balls at location
>> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>> hosted this at
>> http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
>> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
>> for your quick perusal.
>>
>> As you may have noted, a very long fix-cycle for the License & Notice
>> issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop release)
>> to slip by quite a bit. This release's related discussion thread is linked
>> below: [1].
>>
>> Please try the release and vote; the vote will run for the usual 5 days.
>>
>> Thanks,
>> Vinod
>>
>> [1]: 2.7.3 release plan:
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html <
>> http://markmail.org/thread/6yv2fyrs4jlepmmr>
>
>
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Xiao Chen]

+1 (non-binding)

   - Downloaded the binary tarball
   - Verified checksum
   - Verified all jars have LICENSE and NOTICE (using the script
   in HADOOP-13374)
   - Started Pseudo-distributed cluster and verified basic HDFS operations
   work, example Pi job succeed.
   - Started kms with a sample jceks store, verified basic hadoop key
   operations work.


-Xiao

On Mon, Jul 25, 2016 at 11:52 AM, Andrew Wang 
wrote:

> I'll also add that, as a YARN newbie, I did hit two usability issues. These
> are very unlikely to be regressions, and I can file JIRAs if they seem
> fixable.
>
> * I didn't have SSH to localhost set up (new laptop), and when I tried to
> run the Pi job, it'd exit my window manager session. I feel there must be a
> more developer-friendly solution here.
> * If you start the NodeManager and not the RM, the NM has a handler for
> SIGTERM and SIGINT that blocked my Ctrl-C and kill attempts during startup.
> I had to kill -9 it.
>
> On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
> wrote:
>
> > I got asked this off-list, so as a reminder, only PMC votes are binding
> on
> > releases. Everyone is encouraged to vote on releases though!
> >
> > +1 (binding)
> >
> > * Downloaded source, built
> > * Started up HDFS and YARN
> > * Ran Pi job which as usual returned 4, and a little teragen
> >
> > On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
> > wrote:
> >
> >> +1 (binding)
> >>
> >> * Downloaded and build from source
> >> * Checked LICENSE and NOTICE
> >> * Pseudo-distributed cluster with FairScheduler
> >> * Ran MR and HDFS tests
> >> * Verified basic UI
> >>
> >> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
> >>
> >> > +1 binding
> >> >
> >> > * downloaded and built from source
> >> > * checked LICENSE and NOTICE files
> >> > * verified signatures
> >> > * ran standalone tests
> >> > * installed pseudo-distributed instance on my mac
> >> > * ran through HDFS and mapreduce tests
> >> > * tested credential command
> >> > * tested webhdfs access through Apache Knox
> >> >
> >> >
> >> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
> >> > vino...@apache.org> wrote:
> >> >
> >> > > Hi all,
> >> > >
> >> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> >> > >
> >> > > As discussed before, this is the next maintenance release to follow
> up
> >> > > 2.7.2.
> >> > >
> >> > > The RC is available for validation at:
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> >> > >
> >> > > The RC tag in git is: release-2.7.3-RC0
> >> > >
> >> > > The maven artifacts are available via repository.apache.org <
> >> > > http://repository.apache.org/> at
> >> > >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >> > <
> >> > >
> >>
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >> > >
> >> > >
> >> > > The release-notes are inside the tar-balls at location
> >> > >
> hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> >> > > hosted this at
> >> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html
> <
> >> > >
> http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> >> > for
> >> > > your quick perusal.
> >> > >
> >> > > As you may have noted, a very long fix-cycle for the License &
> Notice
> >> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> >> > release)
> >> > > to slip by quite a bit. This release's related discussion thread is
> >> > linked
> >> > > below: [1].
> >> > >
> >> > > Please try the release and vote; the vote will run for the usual 5
> >> days.
> >> > >
> >> > > Thanks,
> >> > > Vinod
> >> > >
> >> > > [1]: 2.7.3 release plan:
> >> > >
> >> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> >> > <
> >> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> >> >
> >>
> >
> >
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Andrew Wang]

I'll also add that, as a YARN newbie, I did hit two usability issues. These
are very unlikely to be regressions, and I can file JIRAs if they seem
fixable.

* I didn't have SSH to localhost set up (new laptop), and when I tried to
run the Pi job, it'd exit my window manager session. I feel there must be a
more developer-friendly solution here.
* If you start the NodeManager and not the RM, the NM has a handler for
SIGTERM and SIGINT that blocked my Ctrl-C and kill attempts during startup.
I had to kill -9 it.

On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
wrote:

> I got asked this off-list, so as a reminder, only PMC votes are binding on
> releases. Everyone is encouraged to vote on releases though!
>
> +1 (binding)
>
> * Downloaded source, built
> * Started up HDFS and YARN
> * Ran Pi job which as usual returned 4, and a little teragen
>
> On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
> wrote:
>
>> +1 (binding)
>>
>> * Downloaded and build from source
>> * Checked LICENSE and NOTICE
>> * Pseudo-distributed cluster with FairScheduler
>> * Ran MR and HDFS tests
>> * Verified basic UI
>>
>> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
>>
>> > +1 binding
>> >
>> > * downloaded and built from source
>> > * checked LICENSE and NOTICE files
>> > * verified signatures
>> > * ran standalone tests
>> > * installed pseudo-distributed instance on my mac
>> > * ran through HDFS and mapreduce tests
>> > * tested credential command
>> > * tested webhdfs access through Apache Knox
>> >
>> >
>> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
>> > vino...@apache.org> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> > >
>> > > As discussed before, this is the next maintenance release to follow up
>> > > 2.7.2.
>> > >
>> > > The RC is available for validation at:
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> > >
>> > > The RC tag in git is: release-2.7.3-RC0
>> > >
>> > > The maven artifacts are available via repository.apache.org <
>> > > http://repository.apache.org/> at
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
>> > <
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
>> > >
>> > >
>> > > The release-notes are inside the tar-balls at location
>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
>> > > hosted this at
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
>> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
>> > for
>> > > your quick perusal.
>> > >
>> > > As you may have noted, a very long fix-cycle for the License & Notice
>> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
>> > release)
>> > > to slip by quite a bit. This release's related discussion thread is
>> > linked
>> > > below: [1].
>> > >
>> > > Please try the release and vote; the vote will run for the usual 5
>> days.
>> > >
>> > > Thanks,
>> > > Vinod
>> > >
>> > > [1]: 2.7.3 release plan:
>> > >
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
>> > <
>> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> >
>>
>
>

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Andrew Wang]

I got asked this off-list, so as a reminder, only PMC votes are binding on
releases. Everyone is encouraged to vote on releases though!

+1 (binding)

* Downloaded source, built
* Started up HDFS and YARN
* Ran Pi job which as usual returned 4, and a little teragen

On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
wrote:

> +1 (binding)
>
> * Downloaded and build from source
> * Checked LICENSE and NOTICE
> * Pseudo-distributed cluster with FairScheduler
> * Ran MR and HDFS tests
> * Verified basic UI
>
> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
>
> > +1 binding
> >
> > * downloaded and built from source
> > * checked LICENSE and NOTICE files
> > * verified signatures
> > * ran standalone tests
> > * installed pseudo-distributed instance on my mac
> > * ran through HDFS and mapreduce tests
> > * tested credential command
> > * tested webhdfs access through Apache Knox
> >
> >
> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
> > vino...@apache.org> wrote:
> >
> > > Hi all,
> > >
> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> > >
> > > As discussed before, this is the next maintenance release to follow up
> > > 2.7.2.
> > >
> > > The RC is available for validation at:
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> > >
> > > The RC tag in git is: release-2.7.3-RC0
> > >
> > > The maven artifacts are available via repository.apache.org <
> > > http://repository.apache.org/> at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > <
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> > >
> > >
> > > The release-notes are inside the tar-balls at location
> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> > > hosted this at
> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> > for
> > > your quick perusal.
> > >
> > > As you may have noted, a very long fix-cycle for the License & Notice
> > > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> > release)
> > > to slip by quite a bit. This release's related discussion thread is
> > linked
> > > below: [1].
> > >
> > > Please try the release and vote; the vote will run for the usual 5
> days.
> > >
> > > Thanks,
> > > Vinod
> > >
> > > [1]: 2.7.3 release plan:
> > >
> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> > <
> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
> >
>

[jira] [Created] (YARN-5428) Allow for specifying the docker client configuration directory

[Shane Kumpf (JIRA)]

Shane Kumpf created YARN-5428:
-

 Summary: Allow for specifying the docker client configuration 
directory
 Key: YARN-5428
 URL: https://issues.apache.org/jira/browse/YARN-5428
 Project: Hadoop YARN
  Issue Type: Sub-task
Reporter: Shane Kumpf
Assignee: Shane Kumpf


The docker client allows for specifying a configuration directory that contains 
the docker client's configuration. It is common to store "docker login" 
credentials in this config, to avoid the need to docker login on each cluster 
member. 

By default the docker client config is $HOME/.docker/config.json on Linux. 
However, this does not work with the current container executor user switching 
and it may also be desirable to centralize this configuration beyond the single 
user's home directory.

Note that the command line arg is for the configuration directory NOT the 
configuration file.

This change will be needed to allow YARN to automatically pull images at 
localization time or within container executor.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

YARN Contributor Meetup

[Karthik Kambatla]

Hi folks

Per our previous discussions, Cloudera would like to host the next YARN
meetup.

Could you please take a moment and pick all slots that work for you here
 [1]? The timings are in PT. Keep
in mind that parking is a little tight at 3 PM. 5 PM should have plenty of
parking.

I ll send out the winning time slot by the end of this week.

Thanks
Karthik

[1] http://doodle.com/poll/hu678caa7wbeeqvw

Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Karthik Kambatla]

+1 (binding)

* Downloaded and build from source
* Checked LICENSE and NOTICE
* Pseudo-distributed cluster with FairScheduler
* Ran MR and HDFS tests
* Verified basic UI

On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:

> +1 binding
>
> * downloaded and built from source
> * checked LICENSE and NOTICE files
> * verified signatures
> * ran standalone tests
> * installed pseudo-distributed instance on my mac
> * ran through HDFS and mapreduce tests
> * tested credential command
> * tested webhdfs access through Apache Knox
>
>
> On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
> > Hi all,
> >
> > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
> >
> > As discussed before, this is the next maintenance release to follow up
> > 2.7.2.
> >
> > The RC is available for validation at:
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ <
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
> >
> > The RC tag in git is: release-2.7.3-RC0
> >
> > The maven artifacts are available via repository.apache.org <
> > http://repository.apache.org/> at
> > https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> <
> > https://repository.apache.org/content/repositories/orgapachehadoop-1040/
> >
> >
> > The release-notes are inside the tar-balls at location
> > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html. I
> > hosted this at
> > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.html <
> > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.html>
> for
> > your quick perusal.
> >
> > As you may have noted, a very long fix-cycle for the License & Notice
> > issues (HADOOP-12893) caused 2.7.3 (along with every other Hadoop
> release)
> > to slip by quite a bit. This release's related discussion thread is
> linked
> > below: [1].
> >
> > Please try the release and vote; the vote will run for the usual 5 days.
> >
> > Thanks,
> > Vinod
> >
> > [1]: 2.7.3 release plan:
> > https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.html
> <
> > http://markmail.org/thread/6yv2fyrs4jlepmmr>
>

Re: Setting JIRA fix versions for 3.0.0 releases

[Andrew Wang]

If I don't hear otherwise, I'd like to go ahead and do the bulk fix version
update on JIRA for 3.0.0-alpha1, diffing from 2.7.0. Will wait until EOD
tomorrow in case there's more discussion. If any one is willing to help
review my script and JIRA queries, that'd also be great.

I can also do the 2.8.0 bulk update (diffing from 2.7.0), since it should
be a very similar process.

Best,
Andrew


On Fri, Jul 22, 2016 at 9:50 PM, Allen Wittenauer 
wrote:

>
> > On Jul 22, 2016, at 7:16 PM, Andrew Wang 
> wrote:
> >
> > Does this mean you find our current system of listing a JIRA as being
> fixed in both a 2.6.x and 2.7.x to be confusing?
>
> Nope.  I'm only confused when there isn't a .0 release in the fix
> line.  When I see 2.6.x and 2.7.x I know that it was back ported to those
> branches.  If I don't see a .0, I figure it's either a mistake or something
> that was already fixed by another change in that major/minor branch.  It's
> almost always the former, however.
>
> > FWIW, my usecase is normally not "what is the earliest release that has
> this fix?" but rather "is this fix in this release?". If it's easy to query
> the latter, you can also determine the former. Some kind of query tool
> could help here.
>
> It literally becomes a grep if people commit the release data into
> the source tree, the release data is correct, etc:
>
> $ mvn install site  -Preleasedocs -Pdocs -DskipTests
> $ grep issueid
> hadoop-common-project/hadoop-common/src/site/markdown/release/*/CHANGES*
>
> We should probably update the release process to make sure that
> *in progress* release data is also committed when a .0 is cut.  That's
> likely missing. Another choice would be to modify the pom to that runs
> releasedocmaker to use a range rather than single version, but that gets a
> bit tricky with release dates, how big of a range, etc.  Not impossible,
> just tricky.  Probably needs to be script that gets run as part of
> create-release, maybe?
>
> (In reality, I do this grep against my git repo that generates the
> change log data automatically.  This way it is always up-to-date and not
> dependent upon release data being committed.  But that same grep could be
> done with a JQL query just as easily.)
>
> > For the release notes, am I correct in interpreting this as:
> >
> > * diff a.0.0 from the previous x.y.0 release
> > * diff a.b.0  from the previous a.0.0 or a.b.0 release
> > * diff a.b.c from the previous a.b.0 or a.b.c release
>
> Pretty much yes.
>
> > Ray pointed me at the changelogs of a few other enterprise software
> products, and this strategy seems pretty common. I like it.
>
> It's extremely common, to the point that putting every fix for
> every release touched is, at least to me, weird and extremely
> unconventional.
>
> > I realize now that this means a lot more JIRAs will need the 2.8.0 fix
> version, since they only have 2.6.x and 2.7.x.
>
> Yup.
>
> >   This makes the fix rules actually pretty easy:  the lowest a.b.0
> release and all non-.0 releases.
> >
> > I think this needs to be amended to handle the case of multiple major
> release branches, since we could have something committed for both 2.9.0
> and 3.1.0. So "lowest a.b.0 release within each major version"?
>
> Yeah, switching to effectively trunk-based development makes the
> rules harder.  It's one of the reasons why the two big enterprisey
> companies I worked at prior to working on Hadoop didn't really do
> trunk-based for the vast majority of projects.  They always cut a branch
> (or equivalent for that SCM) to delineate a break.   Given the amount of
> ex-Sun folks involved in the early days of Hadoop, our pre-existing
> development processes very much reflect that culture.
>
> > This was true previously (no releases from trunk, trunk is versioned
> a.0.0), but now that trunk is essentially a minor release branch, its fix
> version needs to be treated as such.
>
> Yeah, I misspoke a bit when dealing with a head-of-tree model.
> 3.0.0-alpha1 will generate different notes than 3.0.0-alpha2, obviously.
> Every 3.0.0-(label) release is effectively a major version in that case.
>
>
>

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

[Apache Jenkins Server]

For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/

No changes




-1 overall


The following subsystems voted -1:
asflicense unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.server.datanode.TestDataNodeLifeline 
   hadoop.hdfs.TestRollingUpgrade 
   hadoop.yarn.server.nodemanager.TestDirectoryCollection 
   hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore 
   hadoop.yarn.server.TestMiniYarnClusterNodeUtilization 
   hadoop.yarn.server.TestContainerManagerSecurity 
   hadoop.yarn.client.api.impl.TestYarnClient 

Timed out junit tests :

   org.apache.hadoop.http.TestHttpServerLifecycle 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-compile-javac-root.txt
  [172K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-checkstyle-root.txt
  [16M]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-patch-pylint.txt
  [16K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-patch-shelldocs.txt
  [16K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/whitespace-eol.txt
  [12M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/whitespace-tabs.txt
  [1.3M]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/diff-javadoc-javadoc-root.txt
  [2.3M]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-common-project_hadoop-common.txt
  [116K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [148K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
  [36K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt
  [56K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-tests.txt
  [268K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-client.txt
  [12K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-nativetask.txt
  [124K]

   asflicense:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/113/artifact/out/patch-asflicense-problems.txt
  [4.0K]

Powered by Apache Yetus 0.4.0-SNAPSHOT   http://yetus.apache.org



-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org

[jira] [Created] (YARN-5427) labeled apps cant use resources in 2.8

[Feng Yuan (JIRA)]

Feng Yuan created YARN-5427:
---

 Summary: labeled apps cant use  resources in 2.8
 Key: YARN-5427
 URL: https://issues.apache.org/jira/browse/YARN-5427
 Project: Hadoop YARN
  Issue Type: Bug
  Components: capacity scheduler
Affects Versions: 2.8.0
Reporter: Feng Yuan
 Fix For: 2.8.0


Queue A use partition X,When x is used up,app cant get resource from 
.
I find this behavior is different from 2.7.x.Also i find this in the document 
of 2.7.2 as blew:
"All queues can access to nodes without 
label"--https://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/NodeLabel.html

I think it means if queue A`s partition x is used up,app can get containers 
from DEFAULT_PARTITION? Do i misstake?

And I can confirm my configs is correct.Because i can run profectlly in 2.6 use 
this configation too.

I have observed this in 2.8: " is marked as 
".Is this the reason? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org