from:"Colin McCabe"

Re: [VOTE] Merge feature branch YARN-5355 (Timeline Service v2) to trunk

2017-08-30 Thread Colin McCabe

The "git" way of doing things would be to rebase the feature branch on
master (trunk) and then commit the patch stack.

Squashing the entire feature into a 10 MB megapatch is the "svn" way of
doing things.

The svn workflow evolved because merging feature branches back to trunk
was really painful in svn.  So people preferred just to basically do an
rsync from a checkout of the feature branch, edit that up a bit to make
sure they weren't overwriting something that happened in trunk, and then
do an "svn commit" that did not tie back to the feature branch.

P.S. Merges were so painful in SVN that some of the organizations I
worked for maintained spreadsheets of patches which had been merged to
each branch, since svn was of so little help with merging

P.P.S. svn eventually got slightly smarter about merges -- I think in
svn 1.6 or something?

Colin


On Wed, Aug 30, 2017, at 14:37, Sangjin Lee wrote:
> I recall this discussion about a couple of years ago:
> https://lists.apache.org/thread.html/43cd65c6b6c3c0e8ac2b3c76afd9eff1f78b177fabe9c4a96d9b3d0b@1440189889@%3Ccommon-dev.hadoop.apache.org%3E
> 
> On Wed, Aug 30, 2017 at 2:32 PM, Steve Loughran 
> wrote:
> 
> > I'd have assumed it would have gone in as one single patch, rather than a
> > full history. I don't see why the trunk needs all the evolutionary history
> > of a build.
> >
> > What should our policy/process be here?
> >
> > I do currently plan to merge the s3guard in as one single squashed patch;
> > just getting HADOOP-14809 sorted first.
> >
> >
> > > On 30 Aug 2017, at 07:09, Vrushali C  wrote:
> > >
> > > I'm adding my +1 (binding) to conclude the vote.
> > >
> > > With 13 +1's (11 binding) and no -1's, the vote passes. We'll get on with
> > > the merge to trunk shortly. Thanks everyone!
> > >
> > > Regards
> > > Vrushali
> > >
> > >
> > > On Tue, Aug 29, 2017 at 10:54 AM, varunsax...@apache.org <
> > > varun.saxena.apa...@gmail.com> wrote:
> > >
> > >> +1 (binding).
> > >>
> > >> Kudos to all the team members for their great work!
> > >>
> > >> Being part of the ATSv2 team, I have been involved with either
> > development
> > >> or review of most of the JIRAs'.
> > >> Tested ATSv2 in both secure and non-secure mode. Also verified that
> > there
> > >> is no impact when ATSv2 is turned off.
> > >>
> > >> Regards,
> > >> Varun Saxena.
> > >>
> > >> On Tue, Aug 22, 2017 at 12:02 PM, Vrushali Channapattan <
> > >> vrushalic2...@gmail.com> wrote:
> > >>
> > >>> Hi folks,
> > >>>
> > >>> Per earlier discussion [1], I'd like to start a formal vote to merge
> > >>> feature branch YARN-5355 [2] (Timeline Service v.2) to trunk. The vote
> > >>> will
> > >>> run for 7 days, and will end August 29 11:00 PM PDT.
> > >>>
> > >>> We have previously completed one merge onto trunk [3] and Timeline
> > Service
> > >>> v2 has been part of Hadoop release 3.0.0-alpha1.
> > >>>
> > >>> Since then, we have been working on extending the capabilities of
> > Timeline
> > >>> Service v2 in a feature branch [2] for a while, and we are reasonably
> > >>> confident that the state of the feature meets the criteria to be merged
> > >>> onto trunk and we'd love folks to get their hands on it in a test
> > capacity
> > >>> and provide valuable feedback so that we can make it production-ready.
> > >>>
> > >>> In a nutshell, Timeline Service v.2 delivers significant scalability
> > and
> > >>> usability improvements based on a new architecture. What we would like
> > to
> > >>> merge to trunk is termed "alpha 2" (milestone 2). The feature has a
> > >>> complete end-to-end read/write flow with security and read level
> > >>> authorization via whitelists. You should be able to start setting it up
> > >>> and
> > >>> testing it.
> > >>>
> > >>> At a high level, the following are the key features that have been
> > >>> implemented since alpha1:
> > >>> - Security via Kerberos Authentication and delegation tokens
> > >>> - Read side simple authorization via whitelist
> > >>> - Client configurable entity sort ordering
> > >>> - Richer REST APIs for apps, app attempts, containers, fetching
> > metrics by
> > >>> timerange, pagination, sub-app entities
> > >>> - Support for storing sub-application entities (entities that exist
> > >>> outside
> > >>> the scope of an application)
> > >>> - Configurable TTLs (time-to-live) for tables, configurable table
> > >>> prefixes,
> > >>> configurable hbase cluster
> > >>> - Flow level aggregations done as dynamic (table level) coprocessors
> > >>> - Uses latest stable HBase release 1.2.6
> > >>>
> > >>> There are a total of 82 subtasks that were completed as part of this
> > >>> effort.
> > >>>
> > >>> We paid close attention to ensure that once disabled Timeline Service
> > v.2
> > >>> does not impact existing functionality when disabled (by default).
> > >>>
> > >>> Special thanks to a team of folks who worked hard and contributed
> > towards
> > >>> this effort with patches, reviews and guidance:

Re: [DISCUSS] Branches and versions for Hadoop 3

2017-08-28 Thread Colin McCabe

On Mon, Aug 28, 2017, at 14:22, Allen Wittenauer wrote:
> 
> > On Aug 28, 2017, at 12:41 PM, Jason Lowe  wrote:
> > 
> > I think this gets back to the "if it's worth committing" part.
> 
>   This brings us back to my original question:
> 
>   "Doesn't this place an undue burden on the contributor with the first 
> incompatible patch to prove worthiness?  What happens if it is decided that 
> it's not good enough?"

I feel like this line of argument is flawed by definition.  "What
happens if the patch isn't worth breaking compatibility over"?  Then we
shouldn't break compatibility over it.  We all know that most
compatibility breaks are avoidable with enough effort.  And it's an
effort we should make, for the good of our users.

Most useful features can be implemented without compatibility breaks. 
And for the few that truly can't, the community should surely agree that
it's worth breaking compatibility before we do it.  If it's a really
cool feature, that approval will surely not be hard to get (I'm tempted
to quote your earlier email about how much we love features...)

> 
>   The answer, if I understand your position, is then at least a maybe 
> leaning towards yes: a patch that prior to this branching policy change that  
> would have gone in without any notice now has a higher burden (i.e., major 
> feature) to prove worthiness ... and in the process eliminates a whole class 
> of contributors and empowers others. Thus my concern ...
> 
> > As you mentioned, people are already breaking compatibility left and right 
> > as it is, which is why I wondered if it was really any better in practice.  
> > Personally I'd rather find out about a major breakage sooner than later, 
> > since if trunk remains an active area of development at all times it's more 
> > likely the community will sit up and take notice when something crazy goes 
> > in.  In the past, trunk was not really an actively deployed area for over 5 
> > years, and all sorts of stuff went in without people really being aware of 
> > it.
> 
>   Given the general acknowledgement that the compatibility guidelines are 
> mostly useless in reality, maybe the answer is really that we're doing 
> releases all wrong.  Would it necessarily be a bad thing if we moved to a 
> model where incompatible changes gradually released instead of one big one 
> every seven?

I haven't seen anyone "acknowledge that... compatibility guidelines are
mostly useless"... even you.  Reading your posts from the past, I don't
get that impression.  On the contrary, you are often upset about
compatibility breakages.

What would be positive about allowing compatibility breaks in minor
releases?  Can you give a specific example of what would be improved?

> 
>   Yes, I lived through the "walking on glass" days at Yahoo! and realize 
> what I'm saying.  But I also think the rate of incompatible changes has 
> slowed tremendously.  Entire groups of APIs aren't getting tossed out every 
> week anymore.
> 
> > It sounds like we agree on that part but disagree on the specifics of how 
> > to help trunk remain active.
> 
>   Yup, and there is nothing wrong with that. ;)
> 
> >  Given that historically trunk has languished for years I was hoping this 
> > proposal would help reduce the likelihood of it happening again.  If we 
> > eventually decide that cutting branch-3 now makes more sense then I'll do 
> > what I can to make that work well, but it would be good to see concrete 
> > proposals on how to avoid the problems we had with it over the last 6 years.
> 
> 
>   Yup, agree. But proposals rarely seem to get much actual traction. 
> (It's kind of fun reading the Hadoop bylaws and compatibility guidelines and 
> old [VOTE] threads to realize how much stuff doesn't actually happen despite 
> everyone generally agree that abc is a good idea.)  To circle back a bit, I 
> do also agree that automation has a role to play
> 
>Before anyone can accuse or imply me of being a hypocrite (and I'm 
> sure someone eventually will privately if not publicly), I'm sure some folks 
> don't realize I've been working on this set of problems from a different 
> angle for the past few years.
> 
>   There are a handful of people that know I was going to attempt to do a 
> 3.x release a few years ago. [Andrew basically beat me to it. :) ] But I ran 
> into the release process.  What a mess.  Way too much manual work, lots of 
> undocumented bits, violation of ASF rules(!) , etc, etc.  We've all heard the 
> complaints.
> 
>   My hypothesis:  if the release process itself is easier, then getting a 
> release based on trunk is easier too. The more we automate, the more 
> non-vendors ("non traditional release managers"?) will be willing to roll 
> releases.  The more people that feel comfortable rolling a release, the more 
> likelihood releases will happen.  The more likelihood of releases happening, 
> the greater chance trunk had of

Re: [DISCUSS] Branches and versions for Hadoop 3

2017-08-28 Thread Colin McCabe

On Mon, Aug 28, 2017, at 09:58, Allen Wittenauer wrote:
> 
> > On Aug 25, 2017, at 1:23 PM, Jason Lowe  wrote:
> > 
> > Allen Wittenauer wrote:
> >  
> > > Doesn't this place an undue burden on the contributor with the first 
> > > incompatible patch to prove worthiness?  What happens if it is decided 
> > > that it's not good enough?
> > 
> > It is a burden for that first, "this can't go anywhere else but 4.x" 
> > change, but arguably that should not be a change done lightly anyway.  (Or 
> > any other backwards-incompatible change for that matter.)  If it's worth 
> > committing then I think it's perfectly reasonable to send out the dev 
> > announce that there's reason for trunk to diverge from 3.x, cut branch-3, 
> > and move on.  This is no different than Andrew's recent announcement that 
> > there's now a need for separating trunk and the 3.0 line based on what's 
> > about to go in.
> 
>   So, by this definition as soon as a patch comes in to remove deprecated 
> bits there will be no issue with a branch-3 getting created, correct?

Jason wrote that making backwards incompatible changes should not be
"done lightly."  By that definition, "a patch... to remove deprecated
bits" would not cause us to create a whole branch for it.  It should be
something where someone could reasonably make the case that breaking
backwards compatibility was worth it.

> 
> >  Otherwise if past trunk behavior is any indication, it ends up mostly 
> > enabling people to commit to just trunk, forgetting that the thing they are 
> > committing is perfectly valid for branch-3. 
> 
>   I'm not sure there was any "forgetting" involved.  We likely wouldn't 
> be talking about 3.x at all if it wasn't for the code diverging enough.
> 
> > > Given the number of committers that openly ignore discussions like this, 
> > > who is going to verify that incompatible changes don't get in?
> >  
> > The same entities who are verifying other bugs don't get in, i.e.: the 
> > committers and the Hadoop QA bot running the tests.
> >  Yes, I know that means it's inevitable that compatibility breakages will 
> > happen, and we can and should improve the automation around compatibility 
> > testing when possible.
> 
>   The automation only goes so far.  At least while investigating Yetus 
> bugs, I've seen more than enough blatant and purposeful ignored errors and 
> warnings that I'm not convinced it will be effective. ("That javadoc compile 
> failure didn't come from my patch!"  Um, yes, yes it did.) PR for features 
> has greatly trumped code correctness for a few years now.
> 
>   In any case, specifically thinking of the folks that commit maybe one 
> or two patches a year.  They generally don't pay attention to *any* of this 
> stuff and it doesn't seem like many people are actually paying attention to 
> what gets committed until it breaks their universe.
> 
> >  But I don't think there's a magic bullet for preventing all compatibility 
> > bugs from being introduced, just like there isn't one for preventing 
> > general bugs.  Does having a trunk branch separate but essentially similar 
> > to branch-3 make this any better?
> 
>   Yes: it's been the process for over a decade now.  Unless there is some 
> outreach done, it is almost a guarantee that someone will commit something to 
> trunk they shouldn't because they simply won't know (or care?) the process 
> has changed.  

This is no different than any other type of bug.  If someone commits
something that is buggy, we should revert it.  If there are too many of
these issues, then we need more review, more testing, or both.

> 
> > > Longer term:  what is the PMC doing to make sure we start doing major 
> > > releases in a timely fashion again?  In other words, is this really an 
> > > issue if we shoot for another major in (throws dart) 2 years?
> > 
> > If we're trying to do semantic versioning
> 
>   FWIW: Hadoop has *never* done semantic versioning. A large percentage 
> of our minors should really have been majors. 
> 
> > then we shouldn't have a regular cadence for major releases unless we have 
> > a regular cadence of changes that break compatibility.  
> 
>   But given that we don't follow semantic versioning

In case someone new to the community is reading this thread, Hadoop does
have compatibility guidelines for major and minor releases, and has had
them for a very long time. 
http://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/Compatibility.html
Of course, there have been bugs in the past, and they have been
frustrating.

> 
> > I'd hope that's not something we would strive towards.  I do agree that we 
> > should try to be better about shipping releases, major or minor, in a more 
> > timely manner, but I don't agree that we should cut 4.0 simply based on a 
> > duration since the last major release.
> 
>   ... the only thing we're really left with is (technically) time, either 
> in the form of a

Re: Why there are so many revert operations on trunk?

2016-06-07 Thread Colin McCabe

One anti-pattern that keeps coming up over and over again is people
trying to do big and complex features without feature branches.  This
happened with HDFS truncate as well.  This inevitably leads to
controversy because people see very big and invasive patches zooming
past and get alarmed.  Devops people see complicated new code going
directly into production branches and freak out.  Developers see new
APIs getting added without much discussion and start pushing back.

This all gets worse when it's combined with the lack of an up-to-date
design doc.  This leads to people asking the same questions over and
over, since there's no central place they can go to do background
reading about the new feature.  It also tends to lead to arguments,
since the design decisions that were made seem to come out of nowhere,
rather than being justified by careful consideration of alternatives.

When patches get committed directly to production branches, the
discussion also gets mixed up with release management discussions, which
are often contentious as well.  Release management discussions can go on
for a while at the best of times-- when you combine this with all the
above, it just becomes really messy.

My advice is, if 3 or more people ask you for a feature branch, just
create a feature branch already!  It makes things a lot simpler and less
controversial.  It decouples the feature development discussion from the
release management discussion.  It gives developers the ability to look
at the feature as a whole, rather than feeling like they have to
understand each commit in isolation.  Devops people feel like they can
relax because the decision about what stable branches to backport to
will be made after they have a chance to clearly understand the new
feature, rather than before.  The branch merge email functions as a kind
of overview of the feature which makes people more comfortable with it.

Feature branches actually speed up development for medium or large-sized
features.  You can have branch committers who are not full committers,
but can commit to the branch.  You can rewrite your API at any time
without worrying about compatibility.  You can rewrite the history of
the branch however you like to explore new approaches.  You don't need
to backport each of your patches to N stable branches.

Some people seem scared by the need for a merge vote to merge a feature
branch.  But if your feature is big, you are going to get scrutiny
anyway.  Committing stuff directly to trunk or branch-2 is a not a "get
out of jail free" card that bypasses community review.  If anything,
community review will probably be longer and more contentious because of
the reasons above.  People get frustrated when commits to production
branches continue even while big questions about the feature still
remain unanswered.

We already have rules constraining the use of the -1 vote.  Like any
other discussion, -1 votes need to be "constructive"-- that is, they
need to clearly spell out the way forward, rather than just saying no. 
In this particular case, the concerns we had were about the way the
feature was being developed, and the API.

I think that the discussion that is going on for HDFS async right now is
healthy, and will lead to a better feature.  We have people from
downstream projects such as HBase commenting on the kind of APIs they
would find useful.  We have discussions about the impact on the RPC
system, and the branches that it makes sense to target.  And now that we
have a feature branch, we have a lot more freedom to experiment.

best,
Colin

On Tue, Jun 7, 2016, at 11:02, larry mccay wrote:
> -1 needs not be a taken as a derogatory statement being a number should
> actually make it less emotional.
> It is dangerous to a community to become oversensitive to it.
> 
> I generally see language such as "I am -1 on this until this particular
> thing is fixed" or that it violates some common pattern or precedence set
> in the project. This is perfectly reasonable language and there is no
> reason to make the reviewer provide an alternative.
> 
> So, I am giving my -1 to any proposal for rule changes on -1 votes. :)
> 
> 
> On Tue, Jun 7, 2016 at 1:15 PM, Ravi Prakash 
> wrote:
> 
> > +1 on being more respectful. We seem to be having a lot of distasteful
> > discussions recently. If we fight each other, we are only helping our
> > competitors win (and trust me, its out there).
> >
> > I would also respectfully request people not to throw -1s around. I have
> > faced this a few times and its really frustrating. Every one has opinions
> > and some times different people can't fathom why someone else thinks the
> > way they do. I am pretty sure none of us is acting with malicious intent,
> > so perhaps a little more tolerance, faith and trust will help all of us
> > improve Hadoop and the ecosystem much faster. That's not to say that
> > sometimes -1s are not warranted, but we should look to it as an extreme
> >

Re: [DISCUSS] upgrade tomcat from 6.0.44 to 6.0.45 and use non apache.org mirror

2016-05-23 Thread Colin McCabe

I think the Tomcat situation is concerning in a lot of ways.

1. We are downloading without authentication, using http rather than
https.
2. We are downloading an obsolete release.
3. Our build process is violating the apache.archive.org guidelines by
downloading from the site directly, rather than from a mirror.
4. Clean builds take longer than necessary because this tar file needs
to be downloaded.

I'm not too familiar with how Tomcat works... does anyone have any ideas
for fixing this?

best,
Colin


On Tue, May 17, 2016, at 14:08, Wangda Tan wrote:
> Hi common-devs,
> 
> When I tried to build Hadoop trunk today, Maven build failed because kms
> failed to download:
> http://archive.apache.org/dist/tomcat/tomcat-6/v6.0.44/bin/apache-tomcat-6.0.44.tar.gz
> .
> 
> When I open the link from browser, it shows bandwidth limit exceeded:
> 
> ===
> Bandwidth limit exceeded
> 
> The daily allowance of 5GB for this IP has been exceeded, and downloads
> disabled until midnight, UTC (circa 21 hours from now).
> If you have any questions about this, feel free to reach out to us at
> infrastruct...@apache.org.
> ===
> 
> Then I investigated a little more, there're two issues from what I can
> see
> 1) If you look at http://archive.apache.org/dist/, it clearly says:
> *> Please do not download from apache.org !*
> 
> 2) And also, http://archive.apache.org/dist/ says:
> > Older *non-recommended* releases can be found on our archive site
> .
> 
> We're using tomcat with version=6.0.44, on other Apache mirror site, only
> 6.0.45 tomcat is downloadable. Which essentially means 6.0.44 is a
> non-recommended release.
> 
> I would like to propose to make following changes:
> 1) Update tomcat download site from apache.org to another mirror.
> 2) Upgrade tomcat from 6.0.44 to 6.0.45.
> 
> But would like to hear your thoughts before filing JIRA.
> 
> Thanks,
> Wangda

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Compatibility guidelines for toString overrides

2016-05-12 Thread Colin McCabe

Hi all,

Recently a discussion came up on HADOOP-13028 about the wisdom of
overloading S3AInputStream#toString to output statistics information. 
It's a difficult judgement for me to make, since I'm not aware of any
compatibility guidelines for InputStream#toString.  Do we have
compatibility guidelines for toString functions?

It seems like the output of toString functions is usually used as a
debugging aid, rather than as a stable format suitable for UI display or
object serialization.  Clearly, there are a few cases where we might
want to specifically declare that a toString method is a stable API. 
However, I think if we attempt to treat the toString output of all
public classes as stable, we will have greatly increased the API
surface.  Should we formalize this and declare that toString functions
are @Unstable, Evolving unless declared otherwise?

best,
Colin

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public APIs for YARN applications

2016-05-10 Thread Colin McCabe

Thanks for explaining, Chris.  I generally agree that
UserGroupInformation should be annotated as Public rather than
LimitedPrivate, although you guys have more context than I do.

However, I do think it's important that we clarify that we can break
public APIs across a major version transition such as 2.x -> 3.x.  It
would be particularly nice to remove a lot of the static or global state
in UGI, although I don't know if we'll get to that before 3.0 is
released.

best,
Colin

On Tue, May 10, 2016, at 14:46, Chris Nauroth wrote:
> Yes, I agree with you Andrew.
> 
> Sorry, I should clarify my prior response.  I didn't mean to imply a
> blind s/LimitedPrivate/Public/g across the whole codebase.  Instead, I'm
> +1 for the intent of HADOOP-10776: a transition to Public for
> UserGroupInformation, and by extension the related parts of its API like
> Credentials.
> 
> I'm in the camp that generally questions the usefulness of
> LimitedPrivate, but I agree that transitions to Public need case-by-case
> consideration.
> 
> --Chris Nauroth
> 
> From: Andrew Wang
> >
> Date: Tuesday, May 10, 2016 at 2:40 PM
> To: Chris Nauroth
> >
> Cc: Hitesh Shah >,
> "yarn-...@hadoop.apache.org"
> >,
> "mapreduce-...@hadoop.apache.org"
> >,
> "common-dev@hadoop.apache.org"
> >
> Subject: Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public
> APIs for YARN applications
> 
> Why don't we address these on a case-by-case basis, changing the
> annotations on these key classes to Public? LimitedPrivate{"YARN
> applications"} is the same thing as Public.
> 
> This way we don't need to add special exceptions to our compatibility
> policy. Keeps it simple.
> 
> Best,
> Andrew
> 
> On Tue, May 10, 2016 at 2:26 PM, Chris Nauroth
> > wrote:
> +1 for transitioning from LimitedPrivate to Public.
> 
> I view this as an extension of the need for UserGroupInformation and
> related APIs to be Public.  Regardless of the original intent behind
> LimitedPrivate, these are de facto public now, because there is no viable
> alternative for applications that want to integrate with a secured Hadoop
> cluster.
> 
> There is prior discussion of this topic on HADOOP-10776 and HADOOP-12913.
> HADOOP-10776 is a blocker for 2.8.0 to make the transition to Public.
> 
> --Chris Nauroth
> 
> 
> 
> 
> On 5/10/16, 11:34 AM, "Hitesh Shah"
> > wrote:
> 
> >There seems to be some incorrect assumptions on why the application had
> >an issue. For rolling upgrade deployments, the application bundles the
> >client-side jars that it was compiled against and uses them in its
> >classpath and expects to be able to communicate with upgraded servers.
> >Given that hadoop-common is a monolithic jar, it ends up being used on
> >both client-side and server-side. The problem in this case was caused by
> >the fact that the ResourceManager was generating the credentials file
> >with a format understood only by hadoop-common from 3.x. For an
> >application compiled against 2.x and has *only* hadoop-common from 2.x on
> >its classpath, trying to read this file fails.
> >
> >This is not about whether internal implementations can change for
> >non-public APIs. The file format for the Credential file in this scenario
> >is *not* internal implementation especially when you can have different
> >versions of the library trying to read the file. If an older client is
> >talking to a newer versioned server, the general backward compat
> >assumption is that the client should receive a response that it can parse
> >and understand. In this scenario, the credentials file provided to the
> >YARN app by the RM should have been written out with the older version or
> >at the very least been readable by the older hadoop-common.jar.
> >
> >In any case, does anyone have any specific concerns with changing
> >LimitedPrivate({"MapReduce"}) to Public?
> >
> >And sure, if we are saying that Hadoop-3.x requires all apps built
> >against it to go through a full re-compile as well as downtime as
> >existing apps may no longer work out of the box, lets call it out very
> >explicitly in the Release notes.
> >
> >‹ Hitesh
> >
> >> On May 10, 2016, at 9:24 AM, Allen Wittenauer
> >>> wrote:
> >>
> >>
> >>> On May 10, 2016, at 8:37 AM, Hitesh Shah 
> >>> > wrote:
> >>>
> >>> There have been various discussions on various JIRAs where

Re: [DISCUSS] Treating LimitedPrivate({"MapReduce"}) as Public APIs for YARN applications

2016-05-10 Thread Colin McCabe

On Tue, May 10, 2016, at 11:34, Hitesh Shah wrote:
> There seems to be some incorrect assumptions on why the application had
> an issue. For rolling upgrade deployments, the application bundles the
> client-side jars that it was compiled against and uses them in its
> classpath and expects to be able to communicate with upgraded servers.
> Given that hadoop-common is a monolithic jar, it ends up being used on
> both client-side and server-side. The problem in this case was caused by
> the fact that the ResourceManager was generating the credentials file
> with a format understood only by hadoop-common from 3.x. For an
> application compiled against 2.x and has *only* hadoop-common from 2.x on
> its classpath, trying to read this file fails.  
> 
> This is not about whether internal implementations can change for
> non-public APIs. The file format for the Credential file in this scenario
> is *not* internal implementation especially when you can have different
> versions of the library trying to read the file. If an older client is
> talking to a newer versioned server, the general backward compat
> assumption is that the client should receive a response that it can parse
> and understand. In this scenario, the credentials file provided to the
> YARN app by the RM should have been written out with the older version or
> at the very least been readable by the older hadoop-common.jar.
> 
> In any case, does anyone have any specific concerns with changing
> LimitedPrivate({"MapReduce"}) to Public?

I think Allen's concern is valid.  These APIs were annotated as
LimitedPrivate so that we could change them in the future.  If they
become public, we lose that flexibility.

> 
> And sure, if we are saying that Hadoop-3.x requires all apps built
> against it to go through a full re-compile as well as downtime as
> existing apps may no longer work out of the box, lets call it out very
> explicitly in the Release notes. 

I think it's reasonable to require all applications to be re-compiled
for Hadoop 3.x.  One reason is because we make incompatible changes to
Hadoop's dependencies in major releases.  This dependency change also
means that "existing apps may no longer work out of the box," as you
stated, since they may need to be updated to use different versions of
our dependencies.

For the same reasons, it is reasonable to require downtime when
upgrading from 2.x to 3.x (or generally, from any major release to any
other major release).  For example, I believe the HDFS rolling upgrade
document explicitly calls out the fact that the mechanism is targetted
at going to between minor releases, not major ones.

If we need to document this better, then let's file a JIRA to do that.

best,
Colin


> 
> — Hitesh
> 
> > On May 10, 2016, at 9:24 AM, Allen Wittenauer  
> > wrote:
> > 
> > 
> >> On May 10, 2016, at 8:37 AM, Hitesh Shah  wrote:
> >> 
> >> There have been various discussions on various JIRAs where upstream 
> >> projects such as YARN apps ( Tez, Slider, etc ) are called out for using 
> >> the above so-called Private APIs. A lot of YARN applications that have 
> >> been built out have picked up various bits and pieces of implementation 
> >> from MapReduce and DistributedShell to get things to work.
> >> 
> >> A recent example is a backward incompatible change introduced ( where the 
> >> API is not even directly invoked ) in the Credentials class related to the 
> >> ability to read tokens/credentials from a file.
> > 
> > Let’s be careful here.  It should be noted that the problem happened 
> > primarily because the application jar appears to have included some hadoop 
> > jars in them.   So the API invocation isn’t the problem:  it’s the fact 
> > that the implementation under the hood changed.  If the application jar 
> > didn’t bundle hadoop jars —especially given that were already on the 
> > classpath--this problem should never have happened.
> > 
> >> This functionality is required by pretty much everyone as YARN provides 
> >> the credentials to the app by writing the credentials/tokens to a local 
> >> file which is read in when UserGroupInformation.getCurrentUser() is 
> >> invoked.
> > 
> > What you’re effectively arguing is that implementations should never 
> > change for public (and in this case LimitedPrivate) APIs.  I don’t think 
> > that’s reasonable.  Hadoop is filled with changes in major branches where 
> > the implementations have changed but the internals have been reworked to 
> > perform the work in a slightly different manner.
> > 
> >> This change breaks rolling upgrades for yarn applications from 2.x to 3.x 
> >> (whether we end up supporting rolling upgrades across 2.x to 3.x is a 
> >> separate discussion )
> > 
> > 
> > At least today, according to the document attached to YARN-666 (lol), 
> > rolling upgrades are only supported within the same major version.  
> > 
> >> 
> >> I would like to change our compatibility

Re: 7 build nodes are down

2016-05-10 Thread Colin McCabe

Did INFRA have any information on this?

best,


On Fri, May 6, 2016, at 15:14, Allen Wittenauer wrote:
> 
> Anyone know why?
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] Merge Jersey upgrade(HADOOP-9613) to trunk

2016-05-10 Thread Colin McCabe

+1 for updating this in trunk.  Thanks, Tsuyoshi Ozawa.

cheers,
Colin

On Mon, May 9, 2016, at 12:12, Tsuyoshi Ozawa wrote:
> Hi developers,
> 
> We’ve worked on upgrading jersey(HADOOP-9613) for a years. It's
> essential change to support complication with JDK8. It’s almost there.
> 
> One concern to merge this to trunk is incompatibility. After the
> release of Jersey 1.13, the root element whose content is empty
> collection is changed from null to empty object({}).  Because of this
> problem, I’ve marked HADOOP-9613 as incompatible change. Is it
> acceptable for us? If it’s acceptable change in trunk, I’d like to
> merge it into trunk.
> 
> Thanks,
> - Tsuyoshi
> 
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> 

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: 2.7.3 release plan

2016-04-11 Thread Colin McCabe

Sure.  We will continue this discussion on HADOOP-12893.  It now appears
that binary tarballs may need their own license and notice files-- see
the jira for details.

Best,
C.

On Thu, Apr 7, 2016, at 09:28, Vinod Kumar Vavilapalli wrote:
> Colin / Sean, appreciate moving your feedback (and copy-pasting your
> current comments) on this issue to the JIRA: HADOOP-12893
> 
> Thanks
> +Vinod
> 
> > On Apr 7, 2016, at 7:43 AM, Sean Busbey <bus...@cloudera.com> wrote:
> > 
> > On Wed, Apr 6, 2016 at 6:26 PM, Colin McCabe <cmcc...@apache.org 
> > <mailto:cmcc...@apache.org>> wrote:
> >> In general, the only bundled native component I can see is lz4.  I guess
> >> debatably we should add tree.h to the NOTICE file as well, since it came
> >> from BSD and is licensed under that license.
> >> 
> >> Please keep in mind bundling means "included in the source tree", NOT
> >> "downloaded during the build process."  "mvn package" dumps a ton of
> >> jars in the build directory, but these dependencies aren't considered
> >> bundled since their source does not appear in our git repo.  Similarly,
> >> linking against a library doesn't make it "bundled", nor does dlopening
> >> symbols in that library.
> >> 
> >> The big omission is that we have a lot of Javascript source files in our
> >> source tree that do not appear in LICENSE or NOTICE.  I agree that we
> >> should address those before making a new release.
> >> 
> >> best,
> >> 
> >> 
> > 
> > Each artifact that the PMC publishes must abide by the ASF licensing
> > policy. That includes
> > 
> > * Source release artifact
> > * any convenience binary artifacts places on dist.apache
> > * any convenience jars put into the ASF Nexus repository
> > 
> > That likely means that we bundle much more than just what's in the source 
> > tree.
> > 
> > (Though this sounds like we're getting off topic for the 2.7.3 release 
> > plan.)
> > 
> > -- 
> > busbey
>

Re: 2.7.3 release plan

2016-04-06 Thread Colin McCabe

In general, the only bundled native component I can see is lz4.  I guess
debatably we should add tree.h to the NOTICE file as well, since it came
from BSD and is licensed under that license.

Please keep in mind bundling means "included in the source tree", NOT
"downloaded during the build process."  "mvn package" dumps a ton of
jars in the build directory, but these dependencies aren't considered
bundled since their source does not appear in our git repo.  Similarly,
linking against a library doesn't make it "bundled", nor does dlopening
symbols in that library.

The big omission is that we have a lot of Javascript source files in our
source tree that do not appear in LICENSE or NOTICE.  I agree that we
should address those before making a new release.

best,

On Wed, Apr 6, 2016, at 14:13, Allen Wittenauer wrote:
> 
>   This is probably a good time to remind/point folks to HADOOP-12893.  
> Apache Hadoop's binary artifacts (with or without native code) and source 
> artifacts are not complying with the licenses of bundled components.  I 
> fairly confident this means releases are off the table until someone audits 
> the entire codebase given how many have been added without any thought to 
> actually following the terms of the license….
> 
>

Re: Github integration for Hadoop

2016-04-05 Thread Colin McCabe

Thanks, Vinayakumar (and Daniel Gruno.)

C.

On Tue, Apr 5, 2016, at 08:48, Vinayakumar B wrote:
> INFRA-11597 has been fixed. Now all github mails are diverted to
> common-iss...@hadoop.apache.org
> 
> -Vinay
> 
> On Tue, Apr 5, 2016 at 1:05 PM, Vinayakumar B <vinayakumar...@huawei.com>
> wrote:
> 
> > https://issues.apache.org/jira/browse/INFRA-11597 has been filed for this.
> >
> > -Vinay
> >
> > -Original Message-
> > From: Colin McCabe [mailto:co...@cmccabe.xyz]
> > Sent: 05 April 2016 08:07
> > To: common-dev@hadoop.apache.org
> > Subject: Re: Github integration for Hadoop
> >
> > Yes, please.  Let's disable these mails.
> >
> > C.
> >
> > On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> > > bq. We don't spam common-dev about every time a new patch attachment
> > > gets posted to an existing JIRA.  We shouldn't do that for github
> > > either.
> > >
> > > Is there any update on this. ?
> > > Any INFRA ticket filed to disable these mails?
> > >
> > > Because, I am sure, people would have got frustrated by seeing mails
> > > generated by my recent PR submissions. And each time, when some
> > > comment gets added to PR.
> > >
> > > -Vinay
> > >
> > > On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang
> > > <andrew.w...@cloudera.com>
> > > wrote:
> > >
> > > > Hi Eric,
> > > >
> > > > My comment wrt GH permissions is available in context earlier in the
> > > > thread, pasting again here:
> > > >
> > > > ===
> > > >
> > > > Great, glad to hear it. That wasn't mentioned in the email thread or
> > > > on the INFRA ticket, and the INFRA ticket mentions these integrations:
> > > >
> > > > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull
> > > > Request Comment, and push, Push
> > > >
> > > > Are these the right set of permissions to disable integrating PRs?
> > > > Namely, the "push" permissions look unnecessary. We should also
> > > > disable GH issues since we don't want users filing issues there.
> > > >
> > > >
> > > > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe
> > > > <cmcc...@apache.org>
> > > > wrote:
> > > >
> > > > > We don't spam common-dev about every time a new patch attachment
> > > > > gets posted to an existing JIRA.  We shouldn't do that for github
> > either.
> > > > >
> > > > > +1 to Andrew and Steve's suggestion of disabling these emails (or
> > > > > +at
> > > > > least sending them to a separate list that people can opt in to).
> > > > >
> > > > > best,
> > > > > Colin
> > > > >
> > > > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang <eric...@gmail.com>
> > wrote:
> > > > > > Hi Andrew,
> > > > > >
> > > > > > Many Apache projects already have Github integration.  Chukwa
> > > > > > also has Github integration.  Pull request can be integrated into
> > JIRA, i.e.
> > > > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > > > >
> > > > > > This allows code review process to happen at per line level on
> > > > > > Github,
> > > > or
> > > > > > comment on JIRA for summary of the activities.  Micro comments
> > > > > > are
> > > > squash
> > > > > > away.  The final commit is always happening on Apache git rather
> > > > > > than Github.  Therefore, there is no disabling required for pull
> > request.
> > > > > > Primary activity is still on JIRA, and Github is only a
> > > > > > supplement to
> > > > > make
> > > > > > line by line code review easy.  Overall user experience is
> > > > > > better than
> > > > RB
> > > > > > or Gerrit in my opinion.
> > > > > >
> > > > > > regards,
> > > > > > Eric
> > > > > >
> > > > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > > > andrew.w...@cloudera.com>
> > > > > > wrote:
> > > > > >
> > > > > >> Has there been any progress on loc

Re: Github integration for Hadoop

2016-04-04 Thread Colin McCabe

Yes, please.  Let's disable these mails.

C.

On Mon, Apr 4, 2016, at 06:21, Vinayakumar B wrote:
> bq. We don't spam common-dev about every time a new patch attachment
> gets posted
> to an existing JIRA.  We shouldn't do that for github either.
> 
> Is there any update on this. ?
> Any INFRA ticket filed to disable these mails?
> 
> Because, I am sure, people would have got frustrated by seeing mails
> generated by my recent PR submissions. And each time, when some comment
> gets added to PR.
> 
> -Vinay
> 
> On Tue, Nov 24, 2015 at 3:35 AM, Andrew Wang 
> wrote:
> 
> > Hi Eric,
> >
> > My comment wrt GH permissions is available in context earlier in the
> > thread, pasting again here:
> >
> > ===
> >
> > Great, glad to hear it. That wasn't mentioned in the email thread or on the
> > INFRA ticket, and the INFRA ticket mentions these integrations:
> >
> > Commit Comment, Create, Issue Comment, Issues, Pull Request, Pull Request
> > Comment, and push, Push
> >
> > Are these the right set of permissions to disable integrating PRs? Namely,
> > the "push" permissions look unnecessary. We should also disable GH issues
> > since we don't want users filing issues there.
> >
> >
> > On Mon, Nov 23, 2015 at 1:59 PM, Colin P. McCabe 
> > wrote:
> >
> > > We don't spam common-dev about every time a new patch attachment gets
> > > posted to an existing JIRA.  We shouldn't do that for github either.
> > >
> > > +1 to Andrew and Steve's suggestion of disabling these emails (or at
> > > least sending them to a separate list that people can opt in to).
> > >
> > > best,
> > > Colin
> > >
> > > On Sat, Nov 21, 2015 at 9:19 AM, Eric Yang  wrote:
> > > > Hi Andrew,
> > > >
> > > > Many Apache projects already have Github integration.  Chukwa also has
> > > > Github integration.  Pull request can be integrated into JIRA, i.e.
> > > > https://issues.apache.org/jira/browse/CHUKWA-783
> > > >
> > > > This allows code review process to happen at per line level on Github,
> > or
> > > > comment on JIRA for summary of the activities.  Micro comments are
> > squash
> > > > away.  The final commit is always happening on Apache git rather than
> > > > Github.  Therefore, there is no disabling required for pull request.
> > > > Primary activity is still on JIRA, and Github is only a supplement to
> > > make
> > > > line by line code review easy.  Overall user experience is better than
> > RB
> > > > or Gerrit in my opinion.
> > > >
> > > > regards,
> > > > Eric
> > > >
> > > > On Thu, Nov 19, 2015 at 11:28 AM, Andrew Wang <
> > andrew.w...@cloudera.com>
> > > > wrote:
> > > >
> > > >> Has there been any progress on locking down the Github permissions
> > like
> > > I
> > > >> asked above? It's been about 3 weeks.
> > > >>
> > > >> Steve also just asked on another thread to turn off the emails to
> > > >> common-dev. Since we're sticking with JIRA as the source-of-truth, the
> > > >> common-dev emails aren't useful.
> > > >>
> > > >> On Fri, Nov 13, 2015 at 6:59 PM, Sean Busbey 
> > > wrote:
> > > >>
> > > >> > Hi Colin!
> > > >> >
> > > >> > If Yetus is working on an issue and can't tell what the intended
> > > branch
> > > >> is
> > > >> > it points folks to project specific contribution guides.
> > > >> >
> > > >> > For Hadoop, the patch naming for specific branches should be covered
> > > in
> > > >> > this section of Hadoop's contribution guide:
> > > >> >
> > > >> > http://wiki.apache.org/hadoop/HowToContribute#Naming_your_patch
> > > >> >
> > > >> > Yetus will actually support a little bit more than that guide
> > > suggests.
> > > >> If
> > > >> > a project doesn't define a URL to point people at for help in naming
> > > >> > patches we default to this guide:
> > > >> >
> > > >> > https://yetus.apache.org/documentation/latest/precommit-patchnames/
> > > >> >
> > > >> >
> > > >> >
> > > >> > On Fri, Nov 13, 2015 at 8:05 PM, Colin P. McCabe <
> > cmcc...@apache.org>
> > > >> > wrote:
> > > >> >
> > > >> > > Thanks, Allen, I wasn't aware that Yetus now supported testing for
> > > >> > > other branches.  Is there documentation about how to name the
> > branch
> > > >> > > so it gets tested?
> > > >> > >
> > > >> > > best,
> > > >> > > Colin
> > > >> > >
> > > >> > > On Fri, Nov 13, 2015 at 7:52 AM, Allen Wittenauer <
> > a...@altiscale.com
> > > >
> > > >> > > wrote:
> > > >> > > >
> > > >> > > >> On Nov 12, 2015, at 10:55 AM, Colin P. McCabe <
> > > cmcc...@apache.org>
> > > >> > > wrote:
> > > >> > > >>
> > > >> > > >> gerrit has a button on the UI to cherry-pick to different
> > > branches.
> > > >> > > >> The button creates separate "gerrit changes" which you can then
> > > >> > > >> commit.  Eventually we could hook those up to Jenkins--
> > something
> > > >> > > >> which we've never been able to do for different branches with
> > the
> > > >> > > >> patch-file-based workflow.
> > > >> > > >
> > > >> > > >
> > > >> > > > If

Re: Branch policy question

2016-03-23 Thread Colin McCabe

> On 3/22/16, 11:03 PM, "Allen Wittenauer"
>  wrote:
> 
> >> On Mar 22, 2016, at 6:46 PM, Gangumalla, Uma 
> >>wrote:
> >> 
> >>> is it possible for me to setup a branch, self review+commit to that
> >>> branch, then request a branch merge?
> >> Basically this is something like Commit-Then-Review(here review later)
> >> process right. I have not seen we followed this approach here( not sure
> >>if
> >> I missed some branches followed that).
> >
> > At least in recent times, YARN-3368 attempted to do that with the merge
> >request found in YARN-4734. The first or second commit that went into
> >that branch clearly has never been reviewed by another committer.  Since
> >a PMC member did the commit and merge request, I thought I¹d ask since it
> >implies that the policy really is CTR for merge requests. (FWIW, I was
> >the lone -1 on the merge request, otherwise it likely would have gone in
> >given it already had 2 +1¹s despite a) breaking the build b) having
> >significant license issues and c) no design doc.)

Let me clarify that the policy is NOT CTR for merging branches back to
mainline.  It is RTC, with the added proviso that you need 3 +1 votes
from committers.  There is also a 5-day waiting period before you can
merge, so that people have time to chime in and possibly ask for changes
prior to the merge, or express a -1 vote.

I'm concerned that this discussion seems to be focusing on "how can I
use procedures and rules to avoid the need to get reviews from the
community" rather than "how can I get the community interested in giving
me reviews and feedback."

We still haven't spoken a word about what this feature or improvement
does, or why we would want it.  It is hard to say whether a feature
branch makes sense without that context.

best,
Colin

Re: Branch policy question

2016-03-22 Thread Colin McCabe

If the underlying problem is lack of reviewers for these improvements,
how about a design doc giving some motivation for the improvements and
explaining how they'll be implemented?  Then we can decide if a branch
or a few JIRAs on trunk makes more sense.

The description for HADOOP-12857 is just "let's rework this to be
smarter" which could be the description of any JIRA in Hadoop (except
for the ones which make things dumber, of course).

best,
Colin

On Tue, Mar 22, 2016, at 11:23, Andrew Wang wrote:
> A branch sounds fine, but how are we going to get 3 +1's to merge it? If
> it's hard to find one reviewer, seems even harder to find two.
> 
> On Tue, Mar 22, 2016 at 10:56 AM, Allen Wittenauer <
> allenwittena...@yahoo.com.invalid> wrote:
> 
> >
> > > On Mar 22, 2016, at 10:49 AM, larry mccay  wrote:
> > >
> > > That sounds like a reasonable approach and valid use of branches to me.
> > >
> > > Perhaps a set of functional tests could be provided/identified that would
> > > help the review process by showing backward compatibility along with new
> > > extensions for things like dynamic commands?
> > >
> >
> > This is going into trunk, so no need for backward compatibility.
> >

Re: Local repo sharing for maven builds

2015-10-05 Thread Colin McCabe

On Mon, Sep 28, 2015 at 12:52 AM, Steve Loughran  wrote:
>
> the jenkins machines are shared across multiple projects; cut the executors 
> to 1/node and then everyone's performance drops, including the time to 
> complete of all jenkins patches, which is one of the goals.

Hi Steve,

Just to be clear, the proposal wasn't to cut the executors to 1 per
node, but to have multiple Docker containers per node (perhaps 3 or 4)
and run each executor in an isolated container.  At that point,
whatever badness Maven does on the .m2 stops being a problem for
concurrently running jobs.

I guess I don't feel that strongly about this, but the additional
complexity of the other solutions (like running a "find" command in
.m2, or changing artifactID) seems like a disadvantage compared to
just using multiple containers.  And there may be other race
conditions here that we're not aware of... like a TOCTOU between
checking for a jar in .m2 and downloading it, for example.  The
Dockerized solution skips all those potential failure modes and
complexity.

cheers,
Colin


>
> https://builds.apache.org/computer/
>
> Like I said before: I don't think we need one mvn repo/build. All we need is 
> a unique artifact version tag on generated files. Ivy builds do that for you, 
> maven requires the build version in all the POMs to have a -SNAPSHOT tag, 
> which tells it to poll the remote repos for updates every day.
>
> We can build local hadoop releases with whatever version number we desire, 
> simply by using "mvn version:set" to update the version before the build. Do 
> that and you can share the same repo, with different artifacts generated and 
> referenced on every build. We don't need to play with >1 repo, which can be 
> pretty expensive. A du -h ~/.m2 tells me I have an 11GB local cache.
>
>
>> On 26 Sep 2015, at 06:43, Vinayakumar B  wrote:
>>
>> Thanks Andrew,
>>
>> May be we can try making it to 1 exec, and try for sometime. i think also
>> need to check what other jobs, hadoop ecosystem jobs, run in Hadoop nodes.
>> As HADOOP-11984 and HDFS-9139 are on the way to reduce build time
>> dramatically by enabling parallel tests, HDFS and COMMON precommit builds
>> will not block other builds for much time.
>>
>> To check, I dont have access to jenkins configuration. If I can get the
>> access I can reduce it myself and verify.
>>
>>
>> -Vinay
>>
>> On Sat, Sep 26, 2015 at 7:49 AM, Andrew Wang 
>> wrote:
>>
>>> Thanks for checking Vinay. As a temporary workaround, could we reduce the #
>>> of execs per node to 1? Our build queues are pretty short right now, so I
>>> don't think it would be too bad.
>>>
>>> Best,
>>> Andrew
>>>
>>> On Wed, Sep 23, 2015 at 12:18 PM, Vinayakumar B 
>>> wrote:
>>>
 In case if we are going to have separate repo for each executor,

 I have checked, each jenkins node is allocated 2 executors. so we only
>>> need
 to create one more replica.

 Regards,
 Vinay

 On Wed, Sep 23, 2015 at 7:33 PM, Steve Loughran 
 wrote:

>
>> On 22 Sep 2015, at 16:39, Colin P. McCabe 
>>> wrote:
>>
>>> ANNOUNCEMENT: new patches which contain hard-coded ports in test
>>> runs
> will henceforth be reverted. Jenkins matters more than the 30s of your
 time
> it takes to use the free port finder methods. Same for any hard code
 paths
> in filesystems.
>>
>> +1.  Can you add this to HowToContribute on the wiki?  Or should we
>> vote on it first?
>
> I don't think we need to vote on it: hard code ports should be
>>> something
> we veto on patches anyway.
>
> In https://issues.apache.org/jira/browse/HADOOP-12143 I propose
>>> having a
> better style guide in the docs.
>
>
>

>>>
>

Re: IPv6 Feature branch

2015-08-19 Thread Colin McCabe

+1, would be great to see Hadoop get ipv6 support.

Colin

On Mon, Aug 17, 2015 at 5:04 PM, Elliott Clark ecl...@apache.org wrote:
 Nate (nkedel) and I have been working on IPv6 on Hadoop and HBase lately.
 We're getting somewhere but there are a lot of different places that make
 assumptions about network. That means there will be a good deal of follow
 on patches as we find more and more places that need some TLC.

 Would a feature branch be good here so that we can move quickly without
 hurting stability until all of the issues are done?

 Thoughts? Comments?

 Thanks

Re: [jira] [Created] (REEF-216) Protocol Buffers 2.5 no longer available for download

2015-03-30 Thread Colin McCabe

I think it might make sense to keep around a repository of third-party
open source native code that we use in Hadoop.  Nothing fancy, just a
few .tar.gz files in a git repo that we manage.  This would avoid
incidents like this in the future and ensure that we will be able to
build old verisons of Hadoop.

best,
Colin

On Mon, Mar 23, 2015 at 11:40 PM, Tsuyoshi Ozawa oz...@apache.org wrote:
 Hi, Andrew,

 Thanks for the information. I found the old repository of protobuf:

 http://protobuf.googlecode.com/svn-history/r482/trunk/

 Based on the commit log, I created v2.5.0 tag and created tar.gz file
 for the communities:

 https://github.com/oza/protobuf/releases/tag/v2.5.0

 I know this is a temporal fix, but feel free to use it.

 Thanks,
 - Tsuyoshi

 On Tue, Mar 24, 2015 at 4:03 AM, Andrew Wang andrew.w...@cloudera.com wrote:
 I dug up this old issue saying they don't want to import old tags /
 releases:

 https://github.com/google/protobuf/issues/11

 Let's see what they say now, with Google Code shuttered.

 On Sun, Mar 22, 2015 at 10:58 PM, Tsuyoshi Ozawa oz...@apache.org wrote:

 Thanks for sharing the information, Chris. Opened HADOOP-11738 for
 tracking the issue. I also opened issue on protobuf repository.

 https://github.com/google/protobuf/issues/251

 Thanks,
 - Tsuyoshi

 On Mon, Mar 23, 2015 at 11:00 AM, Brahma Reddy Battula
 brahmareddy.batt...@huawei.com wrote:
  Should we raise one jira( in hadoop ) to track this one ( like REEF
 Project)..?
 
 
  Thanks  Regards
  Brahma Reddy Battula
  ___
  From: Chris Douglas [cdoug...@apache.org]
  Sent: Monday, March 23, 2015 5:01 AM
  To: common-dev@hadoop.apache.org
  Subject: Fwd: [jira] [Created] (REEF-216) Protocol Buffers 2.5 no longer
 available for download
 
  The Hadoop build instructions[1] also no longer apply. -C
 
  [1]
 https://git-wip-us.apache.org/repos/asf?p=hadoop.git;a=blob;f=BUILDING.txt
 
 
  -- Forwarded message --
  From: Markus Weimer (JIRA) j...@apache.org
  Date: Fri, Mar 20, 2015 at 5:33 PM
  Subject: [jira] [Created] (REEF-216) Protocol Buffers 2.5 no longer
  available for download
  To: d...@reef.incubator.apache.org
 
 
  Markus Weimer created REEF-216:
  --
 
   Summary: Protocol Buffers 2.5 no longer available for
 download
   Key: REEF-216
   URL: https://issues.apache.org/jira/browse/REEF-216
   Project: REEF
Issue Type: Bug
Components: REEF
  Affects Versions: 0.10, 0.11
  Reporter: Markus Weimer
 
 
  Google recently switched off Google Code. They transferred the
  Protocol Buffers project to
  [GitHub|https://github.com/google/protobuf], and binaries are
  available from [Google's developer
  page|https://developers.google.com/protocol-buffers/docs/downloads].
  However, only the most recent version is available. We use version 2.5
  to be compatible with Hadoop. That version isn't available for
  download. Hence, our [compile
  instructions|
 https://cwiki.apache.org/confluence/display/REEF/Compiling+REEF]
  no longer apply.
 
 
 
  --
  This message was sent by Atlassian JIRA
  (v6.3.4#6332)

Re: about CHANGES.txt

2015-03-18 Thread Colin McCabe

Hmm. I'm not sure why something in the history would not be
relevant-- do you mean that the code was removed during the merge?
While that happens sometimes, it doesn't seem that common.

In any case, we have this same problem with CHANGES.txt, JIRA, and
every other issue tracking system we use. Just because an issue was
relevant to Hadoop 0.23 doesn't mean it's still relevant to Hadoop
2.6. We have a fix version but not a this stopped being relevant
at version.

I'm not really sure what you mean by a merge-forward model either.
It sounds suspiciously like something that was designed around a
subversion workflow. If a change isn't relevant to a branch, why
would you merge it into that branch? In Hadoop, we do feature
development either in trunk or in a feature branch. When it's done we
merge it into the release branches that it's relevant to.

I suppose some changes from some feature branches are sometimes
overwritten by the existing code during a merge. But even if they
are, we still pull them into CHANGES.txt. So again-- same behavior as
git log. And in practice it doesn't seem to be a problem.

best,
Colin

On Wed, Mar 18, 2015 at 1:20 PM, Sean Busbey bus...@cloudera.com wrote:
On the matter of handling merges in the history, this comes up over in
Apache Accumulo where development follows a merge-forward model (commits go
oldest first and merge into newer branches). This means that every commit
on an older-but-still-active development branch eventually ends up merged
into the history of newer branches even when the issue was only relevant to
the older branch. The easiest problem with relying on just the git history
for changes then is that there's no way to programmatically know which of
the commits that show up in the log for a given release tag are relevant to
that release and which ones were only relevant to the older development
line.

-Sean

On Wed, Mar 18, 2015 at 2:59 PM, Colin P. McCabe cmcc...@apache.org wrote:

Alan, can you forward those private conversations (or some excerpt
thereof) to the list to explain the problem that you see?

I have been using git log to track change history for years and
never had a problem. In fact, we don't even maintain CHANGES.txt in
Cloudera's distribution including Hadoop. It causes too many spurious
conflicts during cherry picks so we just discard the CHANGES.txt part
of the change when backporting things to our branches. When you are
backporting hundreds of patches, and each one has a conflict on
CHANGES.txt (and generally, ALL of them do), it's just not worth it to
hand-resolve those conflicts.

I also wrote a script to compare which JIRAs were in which branches by
doing a delta of the git commits. It works pretty well. You can even
visualize merges in git if you want, with tools like gitk (or even
plain old git log with the right options)

Colin

On Tue, Mar 17, 2015 at 11:21 AM, Allen Wittenauer a...@altiscale.com
wrote:

Nope. I’m not particularly in the mood to write a book about a
topic that I’ve beat to death in private conversations over the past 6
months other than highlighting that any solution needs to be able to work
against scenarios like we had 3 years ago with four active release branches
+ trunk.

On Mar 17, 2015, at 10:56 AM, Yongjun Zhang yzh...@cloudera.com wrote:

Thanks Ravi and Colin for the feedback.

Hi Allen,

You pointed out that git log has problem when dealing with branch that
has merges, would you please elaborate the problem?

Thanks.

--Yongjun

On Mon, Mar 16, 2015 at 7:08 PM, Colin McCabe cmcc...@alumni.cmu.edu
wrote:

Branch merges made it hard to access change history on subversion
sometimes.

You can read the tale of woe here:

http://programmers.stackexchange.com/questions/206016/maintaining-svn-history-for-a-file-when-merge-is-done-from-the-dev-branch-to-tru

Excerpt:
prior to Subversion 1.8. The files in the branch and the files in
trunk are copies and Subversion keeps track with svn log only for
specific files, not across branches.

I think that's how the custom of CHANGES.txt started, and it was
cargo-culted forward into the git era despite not serving much purpose
any more these days (in my opinion.)

best,
Colin

On Mon, Mar 16, 2015 at 4:49 PM, Ravi Prakash ravi...@ymail.com
wrote:
+1 for automating the information contained in CHANGES.txt. There are
some changes which go in without JIRAs sometimes (CVEs eg.) . I like
git
log because its the absolute source of truth (cryptographically secure,
audited, distributed, yadadada). We could always use git hooks to
force a
commit message format.
a) cherry-picks have the same message (by default) as the original)b)
I'm not sure why branch-mergers would be a problem?c) Whoops I missed
something in the previous commit wouldn't happen if our hooks were
smartishd) no identification of what type of commit it was without
hooking

Re: about CHANGES.txt

2015-03-18 Thread Colin McCabe

Thanks for that summary, Andrew. In general, I think you're right
that generating it from JIRA would be easier than generating it from
git. We've been pretty good about setting JIRA fix versions.

I do think we could generate it from git if we wanted. We'd have to
have some kind of whitelist or blacklist of commits that looked like
they were there, but weren't, typos, etc. But since we have JIRA, we
might as well use it.

I hope we can move away from doing feature backports as squashed
commits. I've been guilty of this in the past, but I think git
should make this a lot easier to avoid, as you suggested.

best,
Colin

On Wed, Mar 18, 2015 at 2:32 PM, Andrew Wang andrew.w...@cloudera.com wrote:
I don't think we should try and build CHANGES.txt out of git log output.
There are a number of issues:

- We've all fatfingered JIRA #s when writing the commit message, leading to
false positives
- If something is committed then reverted, it's a false positive
- Both of the above are a consequence of not being able to edit git log,
without rebase and force push
- Doesn't work with multiple release branches, unless we want to add
awareness of release branch lines
- Merges are hard to track properly since we've previously done the
branch-2 backport as a single squashed commit. Maybe this will improve with
the rebase workflow, but that remains to be seen.

All of this is solved by going through JIRA instead of git log. JIRA is the
single-source-of-truth for what's fixed in a release line or not, and we
already use it to generate release notes. I don't see why CHANGES.txt
should be generated from a different source.

The bigger question for me is why we have CHANGES.txt at all when we have
release notes, since the information is almost identical.

Best.
Andrew

On Wed, Mar 18, 2015 at 1:20 PM, Sean Busbey bus...@cloudera.com wrote:

On the matter of handling merges in the history, this comes up over in
Apache Accumulo where development follows a merge-forward model (commits go
oldest first and merge into newer branches). This means that every commit
on an older-but-still-active development branch eventually ends up merged
into the history of newer branches even when the issue was only relevant to
the older branch. The easiest problem with relying on just the git history
for changes then is that there's no way to programmatically know which of
the commits that show up in the log for a given release tag are relevant to
that release and which ones were only relevant to the older development
line.

-Sean

On Wed, Mar 18, 2015 at 2:59 PM, Colin P. McCabe cmcc...@apache.org
wrote:

Alan, can you forward those private conversations (or some excerpt
thereof) to the list to explain the problem that you see?

Colin

On Tue, Mar 17, 2015 at 11:21 AM, Allen Wittenauer a...@altiscale.com
wrote:

On Mar 17, 2015, at 10:56 AM, Yongjun Zhang yzh...@cloudera.com
wrote:

Thanks Ravi and Colin for the feedback.

Hi Allen,

You pointed out that git log has problem when dealing with branch
that
has merges, would you please elaborate the problem?

Thanks.

--Yongjun

On Mon, Mar 16, 2015 at 7:08 PM, Colin McCabe cmcc...@alumni.cmu.edu

wrote:

Branch merges made it hard to access change history on subversion
sometimes.

You can read the tale of woe here:

http://programmers.stackexchange.com/questions/206016/maintaining-svn-history-for-a-file-when-merge-is-done-from-the-dev-branch-to-tru

Excerpt:
prior to Subversion 1.8. The files in the branch and the files
in
trunk are copies and Subversion keeps track with svn log only for
specific files, not across branches.

I think that's how the custom of CHANGES.txt started, and it was
cargo-culted forward into the git era despite not serving much
purpose
any more

Re: upstream jenkins build broken?

2015-03-11 Thread Colin McCabe

Is there a maven plugin or setting we can use to simply remove
directories that have no executable permissions on them?  Clearly we
have the permission to do this from a technical point of view (since
we created the directories as the jenkins user), it's simply that the
code refuses to do it.

Otherwise I guess we can just fix those tests...

Colin

On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu l...@cloudera.com wrote:
 Thanks a lot for looking into HDFS-7722, Chris.

 In HDFS-7722:
 TestDataNodeVolumeFailureXXX tests reset data dir permissions in TearDown().
 TestDataNodeHotSwapVolumes reset permissions in a finally clause.

 Also I ran mvn test several times on my machine and all tests passed.

 However, since in DiskChecker#checkDirAccess():

 private static void checkDirAccess(File dir) throws DiskErrorException {
   if (!dir.isDirectory()) {
 throw new DiskErrorException(Not a directory: 
  + dir.toString());
   }

   checkAccessByFileMethods(dir);
 }

 One potentially safer alternative is replacing data dir with a regular
 file to stimulate disk failures.

 On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth cnaur...@hortonworks.com 
 wrote:
 TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
 TestDataNodeVolumeFailureReporting, and
 TestDataNodeVolumeFailureToleration all remove executable permissions from
 directories like the one Colin mentioned to simulate disk failures at data
 nodes.  I reviewed the code for all of those, and they all appear to be
 doing the necessary work to restore executable permissions at the end of
 the test.  The only recent uncommitted patch I¹ve seen that makes changes
 in these test suites is HDFS-7722.  That patch still looks fine though.  I
 don¹t know if there are other uncommitted patches that changed these test
 suites.

 I suppose it¹s also possible that the JUnit process unexpectedly died
 after removing executable permissions but before restoring them.  That
 always would have been a weakness of these test suites, regardless of any
 recent changes.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/






 On 3/10/15, 1:47 PM, Aaron T. Myers a...@cloudera.com wrote:

Hey Colin,

I asked Andrew Bayer, who works with Apache Infra, what's going on with
these boxes. He took a look and concluded that some perms are being set in
those directories by our unit tests which are precluding those files from
getting deleted. He's going to clean up the boxes for us, but we should
expect this to keep happening until we can fix the test in question to
properly clean up after itself.

To help narrow down which commit it was that started this, Andrew sent me
this info:

/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/ has
500 perms, so I'm guessing that's the problem. Been that way since 9:32
UTC
on March 5th.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe cmcc...@apache.org
wrote:

 Hi all,

 A very quick (and not thorough) survey shows that I can't find any
 jenkins jobs that succeeded from the last 24 hours.  Most of them seem
 to be failing with some variant of this message:

 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
 on project hadoop-hdfs: Failed to clean project: Failed to delete


/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-pr
oject/hadoop-hdfs/target/test/data/dfs/data/data3
 - [Help 1]

 Any ideas how this happened?  Bad disk, unit test setting wrong
 permissions?

 Colin





 --
 Lei (Eddy) Xu
 Software Engineer, Cloudera

Re: upstream jenkins build broken?

2015-03-11 Thread Colin McCabe

On Wed, Mar 11, 2015 at 2:34 PM, Chris Nauroth cnaur...@hortonworks.com wrote:
 The only thing I'm aware of is the failOnError option:

 http://maven.apache.org/plugins/maven-clean-plugin/examples/ignoring-errors
 .html


 I prefer that we don't disable this, because ignoring different kinds of
 failures could leave our build directories in an indeterminate state.  For
 example, we could end up with an old class file on the classpath for test
 runs that was supposedly deleted.

 I think it's worth exploring Eddy's suggestion to try simulating failure
 by placing a file where the code expects to see a directory.  That might
 even let us enable some of these tests that are skipped on Windows,
 because Windows allows access for the owner even after permissions have
 been stripped.

+1.  JIRA?

Colin


 Chris Nauroth
 Hortonworks
 http://hortonworks.com/






 On 3/11/15, 2:10 PM, Colin McCabe cmcc...@alumni.cmu.edu wrote:

Is there a maven plugin or setting we can use to simply remove
directories that have no executable permissions on them?  Clearly we
have the permission to do this from a technical point of view (since
we created the directories as the jenkins user), it's simply that the
code refuses to do it.

Otherwise I guess we can just fix those tests...

Colin

On Tue, Mar 10, 2015 at 2:43 PM, Lei Xu l...@cloudera.com wrote:
 Thanks a lot for looking into HDFS-7722, Chris.

 In HDFS-7722:
 TestDataNodeVolumeFailureXXX tests reset data dir permissions in
TearDown().
 TestDataNodeHotSwapVolumes reset permissions in a finally clause.

 Also I ran mvn test several times on my machine and all tests passed.

 However, since in DiskChecker#checkDirAccess():

 private static void checkDirAccess(File dir) throws DiskErrorException {
   if (!dir.isDirectory()) {
 throw new DiskErrorException(Not a directory: 
  + dir.toString());
   }

   checkAccessByFileMethods(dir);
 }

 One potentially safer alternative is replacing data dir with a regular
 file to stimulate disk failures.

 On Tue, Mar 10, 2015 at 2:19 PM, Chris Nauroth
cnaur...@hortonworks.com wrote:
 TestDataNodeHotSwapVolumes, TestDataNodeVolumeFailure,
 TestDataNodeVolumeFailureReporting, and
 TestDataNodeVolumeFailureToleration all remove executable permissions
from
 directories like the one Colin mentioned to simulate disk failures at
data
 nodes.  I reviewed the code for all of those, and they all appear to be
 doing the necessary work to restore executable permissions at the end
of
 the test.  The only recent uncommitted patch I¹ve seen that makes
changes
 in these test suites is HDFS-7722.  That patch still looks fine
though.  I
 don¹t know if there are other uncommitted patches that changed these
test
 suites.

 I suppose it¹s also possible that the JUnit process unexpectedly died
 after removing executable permissions but before restoring them.  That
 always would have been a weakness of these test suites, regardless of
any
 recent changes.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/






 On 3/10/15, 1:47 PM, Aaron T. Myers a...@cloudera.com wrote:

Hey Colin,

I asked Andrew Bayer, who works with Apache Infra, what's going on with
these boxes. He took a look and concluded that some perms are being
set in
those directories by our unit tests which are precluding those files
from
getting deleted. He's going to clean up the boxes for us, but we should
expect this to keep happening until we can fix the test in question to
properly clean up after itself.

To help narrow down which commit it was that started this, Andrew sent
me
this info:

/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/
has
500 perms, so I'm guessing that's the problem. Been that way since 9:32
UTC
on March 5th.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe cmcc...@apache.org
wrote:

 Hi all,

 A very quick (and not thorough) survey shows that I can't find any
 jenkins jobs that succeeded from the last 24 hours.  Most of them
seem
 to be failing with some variant of this message:

 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
 on project hadoop-hdfs: Failed to clean project: Failed to delete


/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs
-pr
oject/hadoop-hdfs/target/test/data/dfs/data/data3
 - [Help 1]

 Any ideas how this happened?  Bad disk, unit test setting wrong
 permissions?

 Colin





 --
 Lei (Eddy) Xu
 Software Engineer, Cloudera

Re: 2.7 status

2015-02-17 Thread Colin McCabe

+1 for starting thinking about releasing 2.7 soon.

Re: building Windows binaries. Do we release binaries for all the
Linux and UNIX architectures? I thought we didn't. It seems a little
inconsistent to release binaries just for Windows, but not for those
other architectures and OSes. I wonder if we can improve this
situation?

best,
Colin

On Fri, Feb 13, 2015 at 4:36 PM, Karthik Kambatla ka...@cloudera.com wrote:
2 weeks from now (end of Feb) sounds reasonable. The one feature I would
like for to be included is shared-cache: we are pretty close - two more
main items to take care of.

In an offline conversation, Steve mentioned building Windows binaries for
our releases. Do we want to do that for 2.7? If so, can anyone with Windows
expertise setup a Jenkins job to build these artifacts, and may be hook it
up to https://builds.apache.org/job/HADOOP2_Release_Artifacts_Builder/

On Fri, Feb 13, 2015 at 11:07 AM, Arun Murthy a...@hortonworks.com wrote:

My bad, been sorted distracted.

I agree, we should just roll fwd a 2.7 ASAP with all the goodies.

What sort of timing makes sense? 2 week hence?

thanks,
Arun

From: Jason Lowe jl...@yahoo-inc.com.INVALID
Sent: Friday, February 13, 2015 8:11 AM
To: common-dev@hadoop.apache.org
Subject: Re: 2.7 status

I'd like to see a 2.7 release sooner than later. It has been almost 3
months since Hadoop 2.6 was released, and there have already been 634 JIRAs
committed to 2.7. That's a lot of changes waiting for an official release.

https://issues.apache.org/jira/issues/?jql=project%20in%20%28hadoop%2Chdfs%2Cyarn%2Cmapreduce%29%20AND%20fixversion%3D2.7.0%20AND%20resolution%3DFixed
Jason

From: Sangjin Lee sj...@apache.org
To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org
Sent: Tuesday, February 10, 2015 1:30 PM
Subject: 2.7 status

Folks,

What is the current status of the 2.7 release? I know initially it started
out as a java-7 only release, but looking at the JIRAs that is very much
not the case.

Do we have a certain timeframe for 2.7 or is it time to discuss it?

Thanks,
Sangjin

--
Karthik Kambatla
Software Engineer, Cloudera Inc.

http://five.sentenc.es

Re: Solaris Port SOLVED!

2014-12-15 Thread Colin McCabe

 is another thread-safe implementation that I could come up with:

 #include string.h
 #include stdlib.h
 #include errno.h
 #include stdio.h

 #define MESSAGE_BUFFER_SIZE 256

 char * getSystemErrorMessage(char * buf, int buf_len, int code) {
 #if defined(_HPUX_SOURCE)
 char * msg;
 errno = 0;
 msg = strerror(code);
 if (errno == 0) {
   strncpy(buf, msg, buf_len-1);
   buf[buf_len-1] = '\0';
 } else {
   snprintf(buf, buf_len, %s %d,
   Can't get system error message for code, code);
 }
 #else
 if (strerror_r(code, buf, buf_len) != 0) {
   snprintf(buf, buf_len, %s %d,
   Can't get system error message for code, code);
 }
 #endif
 return buf;
 }

 #define TERROR(code) \
 getSystemErrorMessage(messageBuffer, sizeof(messageBuffer), code)

 int main(int argc, char ** argv) {
 if (argc  1) {
   char messageBuffer[MESSAGE_BUFFER_SIZE];
   int code = atoi(argv[1]);

   fprintf(stderr, System error for code %s: %s\n, argv[1],
 TERROR(code));
 }
 return 0;
 }


 This changes terror to a macro TERROR and requires all functions that
 call TERROR macro to declare the local variable messageBuffer. Since
 there are only five files to modify, I think it is not a big effort.
 What do you think?

 -- Asokan

 On 12/13/2014 04:29 AM, malcolm wrote:
 Colin,

 I am not sure what you mean by a thread-local buffer (in native
 code). In Java this is pretty standard, but I couldn't find any
 implementation for C code.

 Here is the terror function:

   const char* terror(int errnum)
   {
 if ((errnum  0) || (errnum = sys_nerr)) {
   return unknown error.;
 }
 return sys_errlist[errnum];
   }


 The interface is identical to strerror, but the implementation is
 actually re-entrant since it returns a pointer to a static string.

 If I understand your suggestion, the new function would look like this:

  const char* terror(int errnum)
  {
 static char result[65];

 strerror_r(errnum, result, 64);

 return result;
  }

 No need for snprintf, strerror_r  has the 'n' bounding built-in.

 Of course, this is still non-re-entrant, so unless the caller copies
 the returned buffer, before the function is called again, there is a
 problem.

 After considerable thought, I have come up with this version of
 terror, tested OK on Windows, Linux and Solaris:

  #if defined(_WIN32)
  #define strerror_r(errno,buf,len) strerror_s(buf,len,errno)
  #endif

  #define MAX_ERRORS 256
  #define MAX_ERROR_LEN 80

  char *terror(int errnum)
  {

 static char errlist[MAX_ERRORS][MAX_ERROR_LEN+1]; // cache of
  error messages

 if ( errnum = 0  errnum  MAX_ERRORS )
   {
 if ( errlist[errnum][0] == 0 )
   strerror_r( errnum, errlist[errnum], MAX_ERROR_LEN);

 return errlist[errnum];
   }
 else
   {
 return Unknown error;
   }
  }

 This version is portable and re-entrant.

 On windows, the largest errnum is 43, on Ubuntu 14.04 we have 133,
 and on Solaris 11.1 we get 151.

 If this is OK with you, I will open a jira for this.


 Thanks,
 Malcolm


 On 12/12/2014 11:10 PM, Colin McCabe wrote:
 Just use snprintf to copy the error message from strerror_r into a
 thread-local buffer of 64 bytes or so.  Then preserve the existing
 terror() interface.

 Can you open a jira for this?

 best,
 Colin

 On Thu, Dec 11, 2014 at 8:35 PM,
 malcolmmalcolm.kaval...@oracle.commailto:malcolm.kaval...@oracle.com
 wrote:
 So, turns out that if I had naively changed all calls to terror or
 references to sys_errlist, to using strerror_r, then I would have broken
 code for Windows and HPUX (and possibly other OSes).

 If we are to assume that current code runs fine on all platforms
 (maybe even
 AIX an MacOS, for example), then any change/additions made to the
 code and
 not ifdeffed appropriately can break on other OSes. On the other
 hand,  too
 many ifdefs can pollute the code source and render it less readable
 (though
 possibly less important).

 In the general case what are code contributors responsibilities to
 adding
 code regarding OSes besides Linux ?
 What OSes does jenkins test on ?
 I guess maintainers of code on non-tested platforms are responsible for
 their own testing ?

 How do we avoid the ping-pong effect, i.e. I make a generic change to
 code
 which breaks on Windows, then the Windows maintainer reverts changes to
 break on Solaris for example ? Or does this not happen in actuality ?


 On 12/11/2014 11:25 PM, Asokan, M wrote:
 Hi Malcom,
   The Windows versions of strerror() and strerror_s() functions are
 probably meant for ANSI C library functions that set errno.  For core
 Windows API calls (like UNIX system calls), one gets the error number by
 calling GetLastError() function.  In the code snippet I sent earlier,
 the
 code argument is the value returned

Re: Solaris Port

2014-12-12 Thread Colin McCabe

 libc version, is highly portable and should work regardless
 of the kernel or OS in place.

 On Dec 11, 2014, at 5:20 AM, malcolm malcolm.kaval...@oracle.com
 wrote:

 FYI, there are a couple more files that reference sys_errlist directly
 (not just terror within exception.c) , but also hdfs_http_client.c and
 NativeiO.c

 On 12/11/2014 07:38 AM, malcolm wrote:

 Hi Colin,

 Exactly, as you noticed, the problem is the thread-local buffer needed
 to return from terror.
 Currently, terror just returns a static string from an array, this is
 fast, simple and error-proof.

 In order to use strerror_r inside terror,  would require allocating a
 buffer inside terror  and depend on the caller to free the buffer after
 using it, or to pass a buffer to terrror (which is basically the same as
 strerror_r, rendering terror redundant).
 Both cases require modification outside terror itself, as far as I can
 tell, no simple fix. Unless you have an alternative which I haven't 
 thought
 of ?

 As far as I can tell, we have two choices:

 1. Remove terror and replace calls with strerror_r, passing a buffer
 from the callee.
   Advantage: a more modern portable interface.
   Disadvantage: All calls to terror need to be modified, though
 all seem to be in a few files as far as I can tell.

 2. Adding a sys_errlist array (ifdeffed for Solaris)
   Advantage: no change to any calls to terror
   Disadvantage: 2 additional files added to source tree (.c and
 .h) and some minor ifdefs only used for Solaris.

 I think it is more a question of style than anything else, so I leave
 you to make the call.

 Thanks for your patience,
 Malcolm





 On 12/10/2014 09:54 PM, Colin McCabe wrote:

 On Wed, Dec 10, 2014 at 2:31 AM, malcolm
 malcolm.kaval...@oracle.com wrote:

 Hi Colin,

 Thanks for the hints around JIRAs.

 You are correct errno still exists, however sys_errlist does not.

 Hadoop uses a function terror (defined in exception.c) which indexes
 sys_errlist by errno to return the error message from the array.
 This
 function is called 26 times in various places (in 2.2)

 Originally, I thought to replace all calls to terror with strerror,
 but
 there can be issues with multi-threading (it returns a buffer which
 can be
 overwritten), so it seemed simpler just to recreate the sys_errlist
 message
 array.

 There is also a multi-threaded version strerror_r where you pass the
 buffer
 as a parameter, but this would necessitate changing every call to
 terror
 with mutiple lines of code.

 Why don't you just use strerror_r inside terror()?

 I wrote that code originally.  The reason I didn't want to use
 strerror_r there is because GNU libc provides a non-POSIX definition
 of strerror_r, and forcing it to use the POSIX one is a pain. But you
 can do it.  You also will require a thread-local buffer to hold the
 return from strerror_r, since it is not guaranteed to be static
 (although in practice it is 99% of the time-- another annoyance with
 the API).


 



 ATTENTION: -

 The information contained in this message (including any files
 transmitted with this message) may contain proprietary, trade secret or
 other confidential and/or legally privileged information. Any pricing
 information contained in this message or in any files transmitted with this
 message is always confidential and cannot be shared with any third parties
 without prior written approval from Syncsort. This message is intended to be
 read only by the individual or entity to whom it is addressed or by their
 designee. If the reader of this message is not the intended recipient, you
 are on notice that any use, disclosure, copying or distribution of this
 message, in any form, is strictly prohibited. If you have received this
 message in error, please immediately notify the sender and/or Syncsort and
 destroy all copies of this message in your possession, custody or control.

Re: Solaris Port

2014-12-10 Thread Colin McCabe

Hi Malcolm,

In general we file JIRAs for particular issues. So if one issue is
handling errlist on Solaris, that might be one JIRA. Another issue
might be handling socket write timeouts on Solaris. And so on. Most
of these should probably be HADOOP tickets since they sound like they
are mostly in the generic hadoop-common code.

solaris does not have errno seems like a bold statement. errno is
part of POSIX, and Solaris is a POSIX os, right? Am I way off base on
this?
I googled around and one of the first results I found talked about
errno values on Solaris.
http://www.pixelstech.net/article/1413273556-A-trick-of-building-multithreaded-application-on-Solaris
Perhaps I misunderstood what you meant by this statement.

Anyway, please file JIRAs for any portability improvements you can think of!

best,
Colin

On Mon, Dec 8, 2014 at 9:09 PM, malcolm malcolm.kaval...@oracle.com wrote:
Hi Colin,

A short summary of my changes are as follows:

- Native C source files: added 5, modified 6, requiring also changes to
CMakeLists.txt. Of course, all changes are ifdeffed for Solaris
appropriately and new files, are prefixed with solaris_ as well.

For example, Solaris does not have errno, or errlist any more which are used
quite a lot in hadoop native code. I could have replaced all calls to use
strerror instead which would be compatible with Linux, however in the
interests of making minimal changes, I recreated and added these files from
a running Solaris machine instead.

Another issue is that Solaris doesn't have the timeout option for sockets,
so I had to write my own solaris_read routine with timeout and added it to
DomainSocket.c . A few issues with lz4 on Sparc needed modification, and
some other OS specific issues: getgrouplist, container-executer (from yarn).

- Some very minor changes were made to some Java source files (mainly tests
to get them to pass on Solaris)

The above changes were made to 2.2, I will recheck everything against the
latest trunk, maybe some fixes aren't needed any more.

I have generated a single patch file with all changes. Perhaps it would be
better to file multiple JIRAs for each change, perhaps grouped, one per
issue ? Or should I file a JIRA for each modified source file ?

Thank you,
Malcolm

On 12/08/2014 09:53 PM, Colin McCabe wrote:

Hi Malcolm,

It's great that you are going to contribute! Please make your patches
against trunk.

2.2 is fairly old at this point. It hasn't been the focus of
development in more than a year.

We don't use github or pull requests.

Check the section on http://wiki.apache.org/hadoop/HowToContribute
that talks about Contributing your work. Excerpt:
Finally, patches should be attached to an issue report in Jira via
the Attach File link on the issue's Jira. Please add a comment that
asks for a code review following our code review checklist. Please
note that the attachment should be granted license to ASF for
inclusion in ASF works (as per the Apache License §5).

As this says, you attach the patch file to a JIRA that you have
created, and then hit submit patch.

I don't think a branch is required for this work since it is just
build fixes, right?

best,
Colin

On Mon, Dec 8, 2014 at 3:30 AM, malcolm malcolm.kaval...@oracle.com
wrote:

I have ported Hadoop native libraries to Solaris 11 (both Sparc and
Intel )
Oracle have agreed to release my changes to the community so that Solaris
platforms can benefit.
Reading the HowToContribute and GitandHadoop documents, I am not 100%
clear
on how to get my changes into the main tree. I am also a Git(hub) newbie,
and was using svn previously.

Please let me know if I am going the correct path:

1. I forked Hadoop on Github and downloaded a clone to my development
machine.

2. The changes I made were to 2.2.0, can I still add changes to this
branch,
and hopefully get them accepted or must I migrate my changes to 2.6 ? (On
the main Hadoop download page, 2.2 is still listed as the GA version )

3. I understand that I should create a new branch for my changes, and
then
generate pull requests after uploading them to Github.

4. I also registered at Jira in the understanding that I need to
generate a
Jira number for my changes, and to name my branch accordingly ?

Does all this make sense ?

Thanks,
Malcolm

Re: Solaris Port

2014-12-10 Thread Colin McCabe

On Wed, Dec 10, 2014 at 2:31 AM, malcolm malcolm.kaval...@oracle.com wrote:
 Hi Colin,

 Thanks for the hints around JIRAs.

 You are correct errno still exists, however sys_errlist does not.

 Hadoop uses a function terror (defined in exception.c) which indexes
 sys_errlist by errno to return the error message from the array. This
 function is called 26 times in various places (in 2.2)

 Originally, I thought to replace all calls to terror with strerror, but
 there can be issues with multi-threading (it returns a buffer which can be
 overwritten), so it seemed simpler just to recreate the sys_errlist message
 array.

 There is also a multi-threaded version strerror_r where you pass the buffer
 as a parameter, but this would necessitate changing every call to terror
 with mutiple lines of code.

Why don't you just use strerror_r inside terror()?

I wrote that code originally.  The reason I didn't want to use
strerror_r there is because GNU libc provides a non-POSIX definition
of strerror_r, and forcing it to use the POSIX one is a pain.  But you
can do it.  You also will require a thread-local buffer to hold the
return from strerror_r, since it is not guaranteed to be static
(although in practice it is 99% of the time-- another annoyance with
the API).


 Sorry, I wasn't clear.

 Also, I have been requested to ensure my port is available on 2.4, perceived
 as a more stable release. If I make changes to this branch are they
 automatically available for 2.6, or will I need multiple JIRAs ?

As Steve commented, 2.4 is pretty much done.  I don't think this
kind of thing would get backported there.  Even getting it in a 2.6
release seems unlikely to me (but I haven't thought about it that
hard).  I would just get the work done, and let it show up in the
release it's ready in.

cheers,
Colin


 Thanks,
 Malcolm


 On 12/10/2014 10:45 AM, Colin McCabe wrote:

 Hi Malcolm,

 In general we file JIRAs for particular issues.  So if one issue is
 handling errlist on Solaris, that might be one JIRA.  Another issue
 might be handling socket write timeouts on Solaris.  And so on.  Most
 of these should probably be HADOOP tickets since they sound like they
 are mostly in the generic hadoop-common code.

 solaris does not have errno seems like a bold statement.  errno is
 part of POSIX, and Solaris is a POSIX os, right?  Am I way off base on
 this?
 I googled around and one of the first results I found talked about
 errno values on Solaris.

 http://www.pixelstech.net/article/1413273556-A-trick-of-building-multithreaded-application-on-Solaris
   Perhaps I misunderstood what you meant by this statement.

 Anyway, please file JIRAs for any portability improvements you can think
 of!

 best,
 Colin

 On Mon, Dec 8, 2014 at 9:09 PM, malcolm malcolm.kaval...@oracle.com
 wrote:

 Hi Colin,

 A short summary of my changes are as follows:

 - Native C source files: added 5,  modified 6, requiring also changes to
 CMakeLists.txt. Of course, all changes are ifdeffed for Solaris
 appropriately and new files, are prefixed with solaris_ as well.

 For example, Solaris does not have errno, or errlist any more which are
 used
 quite a lot in hadoop native code. I could have replaced all calls to use
 strerror instead which would be compatible with Linux, however in the
 interests of making minimal changes, I recreated and added these files
 from
 a running Solaris machine instead.

 Another issue is that Solaris doesn't have the timeout option for
 sockets,
 so I had to write my own solaris_read routine with timeout and added it
 to
 DomainSocket.c . A few issues with lz4 on Sparc needed modification, and
 some other OS specific issues: getgrouplist, container-executer (from
 yarn).

 - Some very minor changes were made to some Java source files (mainly
 tests
 to get them to pass on Solaris)

 The above changes were made to 2.2, I will recheck everything against the
 latest trunk, maybe some fixes aren't needed any more.

 I have generated a single patch file with all changes. Perhaps it would
 be
 better to file multiple JIRAs for each change, perhaps grouped, one per
 issue ? Or should I file a JIRA for each modified source file ?

 Thank you,
 Malcolm


 On 12/08/2014 09:53 PM, Colin McCabe wrote:

 Hi Malcolm,

 It's great that you are going to contribute!  Please make your patches
 against trunk.

 2.2 is fairly old at this point.  It hasn't been the focus of
 development in more than a year.

 We don't use github or pull requests.

 Check the section on http://wiki.apache.org/hadoop/HowToContribute
 that talks about Contributing your work.  Excerpt:
 Finally, patches should be attached to an issue report in Jira via
 the Attach File link on the issue's Jira. Please add a comment that
 asks for a code review following our code review checklist. Please
 note that the attachment should be granted license to ASF for
 inclusion in ASF works (as per the Apache License §5).

 As this says, you attach the patch file to a JIRA that you

Re: Solaris Port

2014-12-08 Thread Colin McCabe

Hi Malcolm,

It's great that you are going to contribute!  Please make your patches
against trunk.

2.2 is fairly old at this point.  It hasn't been the focus of
development in more than a year.

We don't use github or pull requests.

Check the section on http://wiki.apache.org/hadoop/HowToContribute
that talks about Contributing your work.  Excerpt:
Finally, patches should be attached to an issue report in Jira via
the Attach File link on the issue's Jira. Please add a comment that
asks for a code review following our code review checklist. Please
note that the attachment should be granted license to ASF for
inclusion in ASF works (as per the Apache License §5).

As this says, you attach the patch file to a JIRA that you have
created, and then hit submit patch.

I don't think a branch is required for this work since it is just
build fixes, right?

best,
Colin


On Mon, Dec 8, 2014 at 3:30 AM, malcolm malcolm.kaval...@oracle.com wrote:
 I have ported Hadoop  native libraries to Solaris 11 (both Sparc and Intel )
 Oracle have agreed to release my changes to the community so that Solaris
 platforms can benefit.
 Reading the HowToContribute and GitandHadoop documents, I am not 100% clear
 on how to get my changes into the main tree. I am also a Git(hub) newbie,
 and was using svn previously.

 Please let me know if I am going the correct path:

 1. I forked Hadoop on Github and downloaded a clone to my development
 machine.

 2. The changes I made were to 2.2.0, can I still add changes to this branch,
 and hopefully get them accepted or must I migrate my changes to 2.6 ? (On
 the main Hadoop download page, 2.2 is still listed as the GA version )

 3. I understand that I should create a new branch for my changes, and then
 generate pull requests after uploading them to Github.

 4. I also registered  at Jira in the understanding that I need to generate a
 Jira number for my changes, and to name my branch accordingly ?

 Does all this make sense ?

 Thanks,
 Malcolm

Re: Switching to Java 7

2014-12-08 Thread Colin McCabe

On Mon, Dec 8, 2014 at 7:46 AM, Steve Loughran ste...@hortonworks.com wrote:
 On 8 December 2014 at 14:58, Ted Yu yuzhih...@gmail.com wrote:

 Looks like there was still OutOfMemoryError :


 https://builds.apache.org/job/Hadoop-Hdfs-trunk/1964/testReport/junit/org.apache.hadoop.hdfs.server.namenode.snapshot/TestRenameWithSnapshots/testRenameDirAcrossSnapshottableDirs/


 Well, I'm going to ignore that for now as it's a java 8 problem, surfacing
 this weekend once the builds were actually switched to Java 8. memory size
 tuning can continue.

 I have now committed the Java 7+ only patch to branch-2 and up: new code
 does not have to worry about java 6 compatibility unless they plan to
 backport to Java 2.6 or earlier. Having written some Java 7 code, the 
 constructor for typed classes are a convenience, the multiple-catch entries
 more useful, as they eliminate duplicate code in exception handling.

 Getting this patch in has revealed that the Jenkins builds of hadoop are
 (a) a bit of a mess and (b) prone to race conditions related to the m2
 repository if 1 project builds simultaneously. The way the nightly builds
 are staggered means this doesn't usually surface, but it may show up during
 precommit/postcommit builds.

It would be nice if we could have a separate .m2 directory per test executor.

It seems like that would eliminate these race conditions once and for
all, at the cost of storing a few extra jars (proportional to the # of
simultaneous executors)

best,
Colin



 The switch to Java 7 as the underlying JDK appears to be triggering
 failures, these are things that the projects themselves are going to have
 to look at.


 This then, is where we are with builds right now. This is not a consequence
 of the changes to the POM; this list predates that patch. This is Jenkins
 running Hadoop builds and tests with Java 7u55


 *Working: *

 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-branch2/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk/

 *failing tests*

 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-2-Build/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk/
 https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk/

 *failing tests on Java 8 (may include OOM)*

 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-common-trunk-Java8/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Hdfs-trunk-Java8/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Mapreduce-trunk-Java8/
 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Yarn-trunk-Java8/


 *failing with maven internal dependency problems*

 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/


 *failing even though it appears to work in the logs*

 https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-Common-trunk/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: a friendly suggestion for developers when uploading patches

2014-12-01 Thread Colin McCabe

On Wed, Nov 26, 2014 at 2:58 PM, Karthik Kambatla ka...@cloudera.com
wrote:

 Yongjun, thanks for starting this thread. I personally like Steve's
 suggestions, but think two digits should be enough.

 I propose we limit the restrictions to versioning the patches with version
 numbers and .patch extension. People have their own preferences for the
 rest of the name (e.g. MAPREDUCE, MapReduce, MR, mr, mapred) and I don't
 see a gain in forcing everyone to use one.

 Putting the suggestions (tight and loose) on the wiki would help new
 contributors as well.


+1

best,
Colin


 On Wed, Nov 26, 2014 at 2:43 PM, Eric Payne erichadoo...@yahoo.com.invalid
 
 wrote:

  +1.The different color for newest patch doesn't work very well if you
  are color blind, so I do appreciate a revision number in the name.
 
From: Yongjun Zhang yzh...@cloudera.com
   To: common-dev@hadoop.apache.org
   Sent: Tuesday, November 25, 2014 11:37 PM
   Subject: Re: a friendly suggestion for developers when uploading patches
 
  Thanks Harsh for the info and Andrew for sharing the script. It looks
 that
  the script is intelligent enough to pick the latest attachment even if
 all
  attachments have the same name.
 
  Yet, I hope we use the following as the guideline for patch names:
 
  *projectName*-*jiraNum*-*revNum*.patch
 
 
  So we can easily identify individual patch revs.
 
  Thanks.
 
  --Yongjun
 
  On Tue, Nov 25, 2014 at 5:54 PM, Andrew Wang andrew.w...@cloudera.com
  wrote:
 
   This might be a good time to mention my fetch-patch script, I use it to
   easily download the latest attachment on a jira:
  
   https://github.com/umbrant/dotfiles/blob/master/bin/fetch-patch
  
   On Tue, Nov 25, 2014 at 5:44 PM, Harsh J ha...@cloudera.com wrote:
  
For the same filename, you can observe also that the JIRA colors the
latest one to be different than the older ones automatically - this
 is
what I rely on.
   
On Sat, Nov 22, 2014 at 12:36 AM, Yongjun Zhang yzh...@cloudera.com
 
wrote:
 Hi,

 When I look at patches uploaded to jiras, from time to time I
 notice
   that
 different revisions of the patch is uploaded with the same patch
 file
name,
 some time for quite a few times. It's confusing which is which.

 I'd suggest that as a guideline, we do the following when
 uploading a
patch:

- include a revision number in the patch file name.A
- include a comment, stating that a new patch is uploaded,
  including
the
revision number of the patch in the comment.

 This way, it's easier to refer to a specific version of a patch,
 and
  to
 know which patch a comment is made about.

 Hope that makes sense to you.

 Thanks.

 --Yongjun
   
   
   
--
Harsh J

Re: submitting a hadoop patch doesn't trigger jenkins test run

2014-11-25 Thread Colin McCabe

+1 for increasing the test timeout for tests spanning multiple sub-projects.

I can see the value in what Steve L. suggested... if you make a major
change that touches a particular subproject, you should try to get the
approval of a committer who knows that subproject.  But I don't think that
forcing artificial patch splits is the way to do this...  There are also
some patches that are completely mechanical and don't really require the
involvement of YARN / HDFS committer, even if they change that project.
For example, fixing a misspelling in the name of a hadoop-common API.

Colin

On Tue, Nov 25, 2014 at 8:45 AM, Yongjun Zhang yzh...@cloudera.com wrote:

 Thanks all for the feedback. To summarize (and I have a suggestion at the
 end of this email), there are two scenarios:

1. A change that span multiple *bigger* projects. r.g. hadoop, hbase.
2. A change that span multiple *sub* projects* within hadoop, e.g.,
common, hdfs, yarn

 For 1, it's required for the change to be backward compatible, thus
 splitting change for multiple *bigger* projects is a must.

 For 2, there are two sub types,

- 2.1 those changes that can be made within hadoop sub-projects, and
there is no external impact
- 2.2 those changes that have external impact, that is, the changes
involve adding new APIs and marking old API deprecated, and
 corresponding
changes in other *bigger* projects will have to be made independently.
 *But
the changes within hadoop subjects can still be done altogether.*

 I think (Please correct me if I'm wrong):

- What Colin referred to is 2.1 and changes within hadoop sub-subjects
for 2.2;
- Steve's not for changes across hadoop-common and hdfs, or
hadoop-common and yarn means 2.1, Steve's  changes that only
span hdfs-and-yarn would be fairly doubtful too. implies his doubt of
existence of 2.1.

 For changes of 2.1 (if any) and *hadoop* changes of 2.2, we do have an
 option of making the change across all hadoop sub-projects altogether, to
 save the multiple steps Colin referred to.

 If this option is feasible, should we consider increasing the jenkins
 timeout for this kind of changes (I mean making the timeout adjustable, if
 it's for single sub-project, use the old timeout; otherwise, increase
 accordingly)  so that we have at least this option when needed?

 Thanks.

 --Yongjun


 On Tue, Nov 25, 2014 at 2:28 AM, Steve Loughran ste...@hortonworks.com
 wrote:

  On 25 November 2014 at 00:58, Bernd Eckenfels e...@zusammenkunft.net
  wrote:
 
   Hello,
  
   Am Mon, 24 Nov 2014 16:16:00 -0800
   schrieb Colin McCabe cmcc...@alumni.cmu.edu:
  
Conceptually, I think it's important to support patches that modify
multiple sub-projects.  Otherwise refactoring things in common
becomes a multi-step process.
  
   This might be rather philosophical (and I dont want to argue the need
   to have the patch infrastructure work for the multi-project case),
   howevere if a multi-project change cannot be applied in multiple steps
   it is probably also not safe at runtime (unless the multiple projects
   belong to a single instance/artifact). And then beeing forced to
   commit/compile/test in multiple steps actually increases the
   dependencies topology.
  
 
  +1 for changes that span, say hadoop and hbase. but not for changes
 across
  hadoop-common and hdfs, or hadoop-common and yarn. changes that only span
  hdfs-and-yarn would be fairly doubtful too.
 
  there is a dependency graph in hadoop's own jars —and cross module (not
  cross project) changes do need to happen.
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.

Re: Guava

2014-11-10 Thread Colin McCabe

I'm usually an advocate for getting rid of unnecessary dependencies
(cough, jetty, cough), but a lot of the things in Guava are really
useful.

Immutable collections, BiMap, Multisets, Arrays#asList, the stuff for
writing hashCode() and equals(), String#Joiner, the list goes on.  We
particularly use the Cache/CacheBuilder stuff a lot in HDFS to get
maps with LRU eviction without writing a lot of boilerplate.  The QJM
stuff uses ListenableFuture a lot, although perhaps we could come up
with our own equivalent for that.

On Mon, Nov 10, 2014 at 9:26 AM, Alejandro Abdelnur tuc...@gmail.com wrote:
 IMO we should:

 1* have a clean and thin client API JAR (which does not drag any 3rd party
 dependencies, or a well defined small set -i.e. slf4j  log4j-)
 2* have a client implementation that uses a classloader to isolate client
 impl 3rd party deps from app dependencies.

 #2 can be done using a stock URLClassLoader (i would just subclass it to
 forbid packages in the API JAR and exposed 3rd parties to be loaded from
 the app JAR)

 #1 is the tricky thing as our current API modules don't have a clean
 API/impl separation.

 thx
 PS: If folks are interested in pursing this, I can put together a prototype
 of how  #2 would work (I don't think it will be more than 200 lines of code)

Absolutely, I agree that we should not be using Guava types in public
APIs.  Guava has not been very responsible with backwards
compatibility, that much is clear.

A client / server jar separation is an interesting idea.  But then we
still have to get rid of Guava and other library deps in the client
jars.  I think it would be more work than it seems.  For example, the
HDFS client uses Guava Cache a lot, so we'd have to write our own
version of this.

Can't we just shade this stuff?  Has anyone tried shading Hadoop's Guava?

best,
Colin




 On Mon, Nov 10, 2014 at 5:18 AM, Steve Loughran ste...@hortonworks.com
 wrote:

 Yes, Guava is a constant pain; there's lots of open JIRAs related to it, as
 its the one we can't seamlessly upgrade. Not unless we do our own fork and
 reinsert the missing classes.

 The most common uses in the code are

 @VisibleForTesting (easily replicated)
 and the Precondition.check() operations

 The latter is also easily swapped out, and we could even add the check they
 forgot:
 Preconditions.checkArgNotNull(argname, arg)


 These are easy; its the more complex data structures that matter more.

 I think for Hadoop 2.7  java 7 we need to look at this problem and do
 something. Even if we continue to ship Guava 11 so that the HBase team
 don't send any (more) death threats, we can/should rework Hadoop to build
 and run against Guava 16+ too. That's needed to fix some of the recent java
 7/8+ changes.

 -Everything in v11 dropped from v16 MUST  to be implemented with our own
 versions.
 -anything tagged as deprecated in 11+ SHOULD be replaced by newer stuff,
 wherever possible.

 I think for 2.7+ we should add some new profiles to the POM, for Java 8 and
 9 alongside the new baseline java 7. For those later versions we could
 perhaps mandate Guava 16.



 On 10 November 2014 00:42, Arun C Murthy a...@hortonworks.com wrote:

  … has been a constant pain w.r.t compatibility etc.
 
  Should we consider adopting a policy to not use guava in
 Common/HDFS/YARN?
 
  MR doesn't matter too much since it's application-side issue, it does
 hurt
  end-users though since they still might want a newer guava-version, but
 at
  least they can modify MR.
 
  Thoughts?
 
  thanks,
  Arun
 
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Build failed in Jenkins: Hadoop-Common-trunk due to missing header file

2014-10-06 Thread Colin McCabe

On Thu, Oct 2, 2014 at 1:15 PM, Ted Yu yuzhih...@gmail.com wrote:
 On my Mac and on Linux, I was able to
 find /usr/include/openssl/opensslconf.h

 However the file is absent on Jenkins machine(s).

 Just want to make sure that the file is needed for native build before
 filing INFRA ticket.

opensslconf.h is part of the openssl-devel package (at least on my
machine) and if it is missing, I would suspect that openssl is either
not installed or incorrectly installed.

We need it for the native build to have coverage for the
openssl-related things (like random number generation and encryption).

Colin


 Cheers

 On Thu, Oct 2, 2014 at 9:09 AM, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com
 wrote:

 Hi Ted,

 On my local, the build of trunk with mvn package -Pnative,dist works
 well. I'm not certain whether this problem is related, but some build
 fails on YARN(e.g. YARN-2562, YARN-2615, YARN-2640).

 The version information of OS and libssl-dev on my local environment
 is as follows:

 $ uname -a
 Linux ip-172-31-4-83 3.13.0-24-generic #46-Ubuntu SMP Thu Apr 10
 19:11:08 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux

 $ apt-cache show libssl-dev
 Package: libssl-dev
 Priority: optional
 Section: libdevel
 Installed-Size: 6162
 Maintainer: Ubuntu Developers ubuntu-devel-disc...@lists.ubuntu.com
 Original-Maintainer: Debian OpenSSL Team
 pkg-openssl-de...@lists.alioth.debian.org
 Architecture: amd64
 Source: openssl
 Version: 1.0.1f-1ubuntu2
 Depends: libssl1.0.0 (= 1.0.1f-1ubuntu2), zlib1g-dev
 Recommends: libssl-doc
 Filename: pool/main/o/openssl/libssl-dev_1.0.1f-1ubuntu2_amd64.deb
 Size: 1066212
 MD5sum: 321724885048f9a78d0e93887a7eb296
 SHA1: e337538bed6e5765a0a85c4ca2af1d0deefd6ce0
 SHA256: ed199dc9131923fa3c911202f165402b1310f50dcdfab987f6f5c2669fc698cc

 Cheers,
 - Tsuyoshi

 On Thu, Oct 2, 2014 at 11:43 PM, Ted Yu yuzhih...@gmail.com wrote:
  Hadoop-Common-trunk build failed due to missing opensslconf.h
 
  Is this environment issue or due to recent commits ?
 
  Cheers
 
  On Thu, Oct 2, 2014 at 7:31 AM, Apache Jenkins Server 
  jenk...@builds.apache.org wrote:
 
  See https://builds.apache.org/job/Hadoop-Common-trunk/1257/
 
  --
   [exec] /usr/bin/cmake -E cmake_progress_report 
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native/CMakeFiles
 
  8
   [exec] [ 16%] Building C object
 
 CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o
   [exec] /usr/bin/cc  -Dhadoop_EXPORTS -m32 -g -Wall -O2 -D_REENTRANT
  -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -fPIC -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native/javah
 
  -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src
 
  -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src
 
  -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/src
 
  -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native
 
  -I/home/jenkins/tools/java/latest/include
  -I/home/jenkins/tools/java/latest/include/linux -I
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/util
 
-o
 
 CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o
   -c 
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
  
   [exec] In file included from 
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/nat
  
   [exec]
  ive/src/org/apache/hadoop/crypto/org_apache_hadoop_crypto.h:33:0,
   [exec]  from 
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c
  :19:
   [exec] /usr/incl
   [exec] ude/openssl/aes.h:55:33: fatal error: openssl/opensslconf.h:
  No such file or directory
   [exec]  #include openssl/opensslconf.h
   [exec]  ^
   [exec] compilation terminated.
   [exec] make[2]: ***
 
 [CMakeFiles/hadoop.dir/main/native/src/org/apache/hadoop/crypto/OpensslCipher.c.o]
  Error 1
   [exec] make[1]: *** [CMakeFiles/hadoop.dir/all] Error 2
   [exec] make: *** [all] Error 2
   [exec] make[2]: Leaving directory `
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native
  '
   [exec] make[1]: Leaving directory `
 
 https://builds.apache.org/job/Hadoop-Common-trunk/ws/hadoop-common-project/hadoop-common/target/native
  '
  [INFO]
  
  [INFO]

Re: submissions with bsd code?

2014-10-03 Thread Colin McCabe

On Wed, Oct 1, 2014 at 4:30 PM, John Smith sharepockywithj...@gmail.com wrote:
 hi developers.

 i have some native code working on solaris, but my changes use getgrouplist
 from openssh.  is that ok?  do i need to do anything special?  is the
 license in file enough?

Is the license in which file enough?

Colin


 thank you for your help.

builds failing on H9 with cannot access java.lang.Runnable

2014-10-03 Thread Colin McCabe

It looks like builds are failing on the H9 host with cannot access
java.lang.Runnable

Example from 
https://builds.apache.org/job/PreCommit-HDFS-Build/8313/artifact/patchprocess/trunkJavacWarnings.txt
:

[INFO] 
[INFO] BUILD FAILURE
[INFO] 
[INFO] Total time: 03:13 min
[INFO] Finished at: 2014-10-03T18:04:35+00:00
[INFO] Final Memory: 57M/839M
[INFO] 
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
(default-testCompile) on project hadoop-mapreduce-client-app:
Compilation failure
[ERROR] 
/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java:[189,-1]
cannot access java.lang.Runnable
[ERROR] bad class file: java/lang/Runnable.class(java/lang:Runnable.class)

I don't have shell access to this, does anyone know what's going on on H9?

best,
Colin

Re: builds failing on H9 with cannot access java.lang.Runnable

2014-10-03 Thread Colin McCabe

Thanks, Andrew and Giridharan.

Colin

On Fri, Oct 3, 2014 at 1:20 PM, Andrew Bayer andrew.ba...@gmail.com wrote:
 Yeah, the other possibility is that an ansible run borks running
 slaves. If this happens again, let us know.

 A.

 On Fri, Oct 3, 2014 at 1:15 PM, Giridharan Kesavan
 gkesa...@hortonworks.com wrote:
 all the slaves are getting re-booted give it some more time

 -giri

 On Fri, Oct 3, 2014 at 1:13 PM, Ted Yu yuzhih...@gmail.com wrote:

 Adding builds@

 On Fri, Oct 3, 2014 at 1:07 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

  It looks like builds are failing on the H9 host with cannot access
  java.lang.Runnable
 
  Example from
 
 https://builds.apache.org/job/PreCommit-HDFS-Build/8313/artifact/patchprocess/trunkJavacWarnings.txt
  :
 
  [INFO]
  
  [INFO] BUILD FAILURE
  [INFO]
  
  [INFO] Total time: 03:13 min
  [INFO] Finished at: 2014-10-03T18:04:35+00:00
  [INFO] Final Memory: 57M/839M
  [INFO]
  
  [ERROR] Failed to execute goal
  org.apache.maven.plugins:maven-compiler-plugin:2.5.1:testCompile
  (default-testCompile) on project hadoop-mapreduce-client-app:
  Compilation failure
  [ERROR]
 
 /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/commit/TestCommitterEventHandler.java:[189,-1]
  cannot access java.lang.Runnable
  [ERROR] bad class file:
 java/lang/Runnable.class(java/lang:Runnable.class)
 
  I don't have shell access to this, does anyone know what's going on on
 H9?
 
  best,
  Colin
 


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Cannot access Jenkins generated test results, findbug warnings and javac warnings

2014-09-19 Thread Colin McCabe

Thanks, Steve.

Should we just put everything in patchprocess/ like before?  It seems
like renaming this directory to PreCommit-HADOOP-Build-patchprocess/
or PreCommit-YARN-Build-patchprocess/ in various builds has created
problems, and not made things any more clear.  What do you guys think?

Colin

On Fri, Sep 19, 2014 at 5:42 AM, Steve Loughran ste...@hortonworks.com wrote:
 well, I have those permissions, but not sure where things are going right
 now

 1. jenkins is set to archive patchprocess/*.*
 1  publish tests from trunk/**/target/surefire-reports/*.xml


 I've set it to use the path
 PreCommit-HADOOP-Build-patchprocess/*.*

  will do some manual triggering to see if it takes

 On 19 September 2014 10:54, Wangda Tan wheele...@gmail.com wrote:

 Hi Steve,
 I guess this problem should be also caused by wrong URL, if anybody have
 admin access to Jenkins, correct URL should be easily found.

 Thanks,
 Wangda

 On Fri, Sep 19, 2014 at 4:32 PM, Steve Loughran ste...@hortonworks.com
 wrote:

  Looks like HADOOP-11084 isn't complete —the patch to the build to get it
  working post-git
 
  before that patch the builds weren't working at all ... now its just
  getting the URLs wrong.
 
  If you can work out the right URLs we can fix this easily enough
 
  On 19 September 2014 09:24, Wangda Tan wheele...@gmail.com wrote:
 
   Hi Hadoop developers,
   I found recently, I cannot access Jenkins generated results, like:
  
  
  
   *Test
   results:
   https://builds.apache.org/job/PreCommit-YARN-Build/5039//testReport/
   https://builds.apache.org/job/PreCommit-YARN-Build/5039//testReport/
   Findbugs
   warnings:
  
 
 https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
   
  
 
 https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-yarn-client.html
   Javac
   warnings:
  
 
 https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
   
  
 
 https://builds.apache.org/job/PreCommit-YARN-Build/5039//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
   *
  
   It will report 404 when trying to access findbugs/javac warnings and it
   will redirect to info page of build when trying to access test report.
  
   I'm not sure if there's any recent changes on Jenkins configuration.
 Did
   you hit the problem like this?
  
   Thanks,
   Wangda
  
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: In hindsight... Re: Thinking ahead to hadoop-2.6

2014-09-15 Thread Colin McCabe

On Mon, Sep 15, 2014 at 10:48 AM, Allen Wittenauer a...@altiscale.com wrote:

 It’s now September.  With the passage of time, I have a lot of doubts 
 about this plan and where that trajectory takes us.

 * The list of changes that are already in branch-2 scare the crap out of any 
 risk adverse person (Hello to my fellow operations people!). Not only are the 
 number of changes extremely high, but in addition there are a lot of major, 
 blockbuster features in what is supposed to be a minor release.  Combined 
 with the fact that we’ve had to do some micro releases, it seems to hint that 
 branch-2 is getting less stable over time.

I don't see what is so scary about 2.6, can you be more concrete?  It
seems like a pretty normal release to me and most of the new features
are optional.

I also don't see why you think that branch-2 is getting less stable
over time.  Actually, I think that branch-2 has gotten more stable
over time as people have finally gotten around to upgrading from 1.x
or earlier, and contributed their efforts to addressing regressions in
branch-2.

 *  One of the plans talked about was rolling a 2.7 release that drops JDK6 
 and makes JDK7 the standard.  If 2.7 comes after 2.6 in October, date wise  
 makes it somewhere around January 2015.  JDK7 EOL’s in April 2015.  So we’ll 
 have a viable JDK7 release for exactly 3 months.  Frankly, it is too late for 
 us to talk about JDK7 and we need to start thinking about JDK8.

 * trunk is currently sitting at 3 years old.  There is a lot of stuff that 
 has been hanging around that really needs to get into people hands so that we 
 can start stabilizing it for a “real” release.

We have been pretty careful about minimizing trunk's divergence from
branch-2.  I can't think of an example of anything in trunk that
really needs to get into people's hands-- did I forget something?



 To me this all says one thing:

 Drop the 2.6.0 release, branch trunk, and start rolling a 3.0.0-alpha 
 with JDK8 as the minimum.  2.5.1 becomes the base for all sustaining work.  
 This gives the rest of the community time to move to JDK8 if they haven’t 
 already.  For downstream vendors, it gives a roadmap for their customers who 
 will be asking about JDK8 sooner rather than later.  By the time 3.0 
 stabilizes, we’re probably looking at April, which is perfect timing.

 One of the issues I’ve heard mention is that 3.0 doesn’t have 
 anything “compelling” in it.  Well, dropping 2.6 makes the feature list the 
 carrot, JDK8 support is obviously the stick.

 Thoughts?

As we've discussed before, supporting JDK8 is very different from
forcing people to use JDK8.  branch-2 and Hadoop 2.6 most certainly
should support JDK8, and most certainly NOT force people to use JDK8.
Cloudera has been using JDK7 internally for a long time, and
recommending it to customers too.  Some developers are using JDK8 as
well.  It works fine (although I'm sure there will be bugs and
workarounds that get reported and fixed as more people migrate).  I
don't see this particular issue as a reason to change the schedule.

best,
Colin






 On Aug 15, 2014, at 6:07 PM, Subramaniam Krishnan su...@apache.org wrote:

 Thanks for initiating the thread Arun.

 Can we add YARN-1051 https://issues.apache.org/jira/browse/YARN-1051 to
 the list? We have most of the patches for the sub-JIRAs under review and
 have committed a couple.

 -Subru

 -- Forwarded message --

 From: Arun C Murthy a...@hortonworks.com

 Date: Tue, Aug 12, 2014 at 1:34 PM

 Subject: Thinking ahead to hadoop-2.6

 To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org, 
 hdfs-...@hadoop.apache.org hdfs-...@hadoop.apache.org, 
 mapreduce-...@hadoop.apache.org mapreduce-...@hadoop.apache.org,

 yarn-...@hadoop.apache.org yarn-...@hadoop.apache.org





 Folks,



 With hadoop-2.5 nearly done, it's time to start thinking ahead to
 hadoop-2.6.



 Currently, here is the Roadmap per the wiki:



• HADOOP

• Credential provider HADOOP-10607

• HDFS

• Heterogeneous storage (Phase 2) - Support APIs for using
 storage tiers by the applications HDFS-5682

• Memory as storage tier HDFS-5851

• YARN

• Dynamic Resource Configuration YARN-291

• NodeManager Restart YARN-1336

• ResourceManager HA Phase 2 YARN-556

• Support for admin-specified labels in YARN YARN-796

• Support for automatic, shared cache for YARN application
 artifacts YARN-1492

• Support NodeGroup layer topology on YARN YARN-18

• Support for Docker containers in YARN YARN-1964

• YARN service registry YARN-913



 My suspicion is, as is normal, some will make the cut and some won't.

 Please do add/subtract from the list as appropriate. Ideally, it would be
 good to ship hadoop-2.6 in a 6-8 weeks (say, October) to keep up

Re: Git repo ready to use

2014-09-12 Thread Colin McCabe

It's an issue with test-patch.sh.  See
https://issues.apache.org/jira/browse/HADOOP-11084

best,
Colin

On Mon, Sep 8, 2014 at 3:38 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 We're still not seeing findbugs results show up on precommit runs. I see
 that we're archiving ../patchprocess/*, and Ted thinks that since it's
 not in $WORKSPACE it's not getting picked up. Can we get confirmation of
 this issue? If so, we could just add patchprocess to the toplevel
 .gitignore.

 On Thu, Sep 4, 2014 at 8:54 AM, Sangjin Lee sjl...@gmail.com wrote:

 That's good to know. Thanks.


 On Wed, Sep 3, 2014 at 11:15 PM, Vinayakumar B vinayakum...@apache.org
 wrote:

  I think its still pointing to old svn repository which is just read only
  now.
 
  You can use latest mirror:
  https://github.com/apache/hadoop
 
  Regards,
  Vinay
  On Sep 4, 2014 11:37 AM, Sangjin Lee sjl...@gmail.com wrote:
 
   It seems like the github mirror at
  https://github.com/apache/hadoop-common
   has stopped getting updates as of 8/22. Could this mirror have been
  broken
   by the git transition?
  
   Thanks,
   Sangjin
  
  
   On Fri, Aug 29, 2014 at 11:51 AM, Ted Yu yuzhih...@gmail.com wrote:
  
From https://builds.apache.org/job/Hadoop-hdfs-trunk/1854/console :
   
ERROR: No artifacts found that match the file pattern
trunk/hadoop-hdfs-project/*/target/*.tar.gz. Configuration
error?ERROR http://stacktrace.jenkins-ci.org/search?query=ERROR:
?trunk/hadoop-hdfs-project/*/target/*.tar.gz? doesn?t match anything,
but ?hadoop-hdfs-project/*/target/*.tar.gz? does. Perhaps that?s what
you mean?
   
   
I corrected the path to hdfs tar ball.
   
   
FYI
   
   
   
On Fri, Aug 29, 2014 at 8:48 AM, Alejandro Abdelnur 
 t...@cloudera.com
  
wrote:
   
 it seems we missed updating the HADOOP precommit job to use Git, it
  was
 still using SVN. I've just updated it.

 thx


 On Thu, Aug 28, 2014 at 9:26 PM, Ted Yu yuzhih...@gmail.com
 wrote:

  Currently patchprocess/ (contents shown below) is one level
 higher
   than
  ${WORKSPACE}
 
  diffJavadocWarnings.txt
 newPatchFindbugsWarningshadoop-hdfs.html
   patchFindBugsOutputhadoop-hdfs.txt
 patchReleaseAuditOutput.txt
   trunkJavadocWarnings.txt
  filteredPatchJavacWarnings.txt
   newPatchFindbugsWarningshadoop-hdfs.xml
  patchFindbugsWarningshadoop-hdfs.xml
 patchReleaseAuditWarnings.txt
  filteredTrunkJavacWarnings.txt  patch
  patchJavacWarnings.txttestrun_hadoop-hdfs.txt
  jirapatchEclipseOutput.txt
   patchJavadocWarnings.txt  trunkJavacWarnings.txt
 
  Under Files to archive input box of
  PreCommit-HDFS-Build/configure, I
 saw:
 
  '../patchprocess/*' doesn't match anything, but '*' does. Perhaps
that's
  what you mean?
 
  I guess once patchprocess is moved back under ${WORKSPACE}, a lot
  of
 things
  would be back to normal.
 
  Cheers
 
  On Thu, Aug 28, 2014 at 9:16 PM, Alejandro Abdelnur 
   t...@cloudera.com

  wrote:
 
   i'm also seeing broken links for javadocs warnings.
  
   Alejandro
   (phone typing)
  
On Aug 28, 2014, at 20:00, Andrew Wang 
  andrew.w...@cloudera.com
   
  wrote:
   
I noticed that the JUnit test results aren't getting picked
 up
  anymore. I
suspect we just need to update the path to the surefire
 output,
   but
  based
on a quick examination I'm not sure what that is.
   
Does someone mind taking another look?
   
   
On Thu, Aug 28, 2014 at 4:21 PM, Karthik Kambatla 
 ka...@cloudera.com
wrote:
   
Thanks Giri and Ted for fixing the builds.
   
   
On Thu, Aug 28, 2014 at 9:49 AM, Ted Yu 
 yuzhih...@gmail.com
  
 wrote:
   
Charles:
QA build is running for your JIRA:
   

 https://builds.apache.org/job/PreCommit-hdfs-Build/7828/parameters/
   
Cheers
   
   
On Thu, Aug 28, 2014 at 9:41 AM, Charles Lamb 
cl...@cloudera.com
 
wrote:
   
On 8/28/2014 12:07 PM, Giridharan Kesavan wrote:
   
Fixed all the 3 pre-commit buids. test-patch's git reset
   --hard
 is
removing
the patchprocess dir, so moved it off the workspace.
Thanks Giri. Should I resubmit HDFS-6954's patch? I've
  gotten
   3
 or 4
jenkins messages that indicated the problem so something
 is
resubmitting,
but now that you've fixed it, should I resubmit it again?
   
Charles
   
  
 



 --
 Alejandro

Re: migrating private branches to the new git repo

2014-09-08 Thread Colin McCabe

+1 for using git log instead of CHANGES.txt.

Colin

On Wed, Sep 3, 2014 at 11:07 AM, Chris Douglas cdoug...@apache.org wrote:
 On Tue, Sep 2, 2014 at 2:38 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Not to derail the conversation, but if CHANGES.txt is making backports more
 annoying, why don't we get rid of it? It seems like we should be able to
 generate it via a JIRA query, and git log can also be used for a quick
 check (way faster than svn log).

 +1

 As long as release notes and incompatible changes are recorded in each
 branch, we gain no accuracy by maintaining this manually. Commit
 messages that record the merge revisions instead of the change are
 similarly hateful. -C

Re: Updates on migration to git

2014-08-27 Thread Colin McCabe

Thanks for making this happen, Karthik and Daniel.  Great job.

best,
Colin

On Tue, Aug 26, 2014 at 5:59 PM, Karthik Kambatla ka...@cloudera.com wrote:
 Yes, we have requested for force-push disabled on trunk and branch-*
 branches. I didn't test it though :P, it is not writable yet.


 On Tue, Aug 26, 2014 at 5:48 PM, Todd Lipcon t...@cloudera.com wrote:

 Hey Karthik,

 Just to confirm, have we disabled force-push support on the repo?

 In my experience, especially when a project has committers new to git,
 force-push support causes more trouble than it's worth.

 -Todd


 On Tue, Aug 26, 2014 at 4:39 PM, Karthik Kambatla ka...@cloudera.com
 wrote:

  Looks like our git repo is good to go.
 
  On INFRA-8195, I am asking Daniel to enable writing to it. In case you
 find
  any issues, please comment on the JIRA.
 
  Thanks
  Karthik
 
 
  On Tue, Aug 26, 2014 at 3:28 PM, Arpit Agarwal aagar...@hortonworks.com
 
  wrote:
 
   I cloned the new repo, built trunk and branch-2, verified all the
  branches
   are present. Also checked a few branches and the recent commit history
   matches our existing repo. Everything looks good so far.
  
  
   On Tue, Aug 26, 2014 at 1:19 PM, Karthik Kambatla ka...@cloudera.com
   wrote:
  
The git repository is now ready for inspection. I ll take a look
  shortly,
but it would be great if a few others could too.
   
Once we are okay with it, we can ask it to be writable.
   
On Tuesday, August 26, 2014, Karthik Kambatla ka...@cloudera.com
   wrote:
   
 Hi Suresh

 There was one vote thread on whether to migrate to git, and the
 implications to the commit process for individual patches and
 feature
 branches -

  
 https://www.mail-archive.com/common-dev@hadoop.apache.org/msg13447.html
.
 Prior to that, there was a discuss thread on the same topic.

 As INFRA handles the actual migration from subversion to git, the
  vote
 didn't include those specifics. The migration is going on as we
 speak
(See
 INFRA-8195). The initial expectation was that the migration would
 be
   done
 in a few hours, but it has been several hours and the last I heard
  the
 import was still running.

 I have elaborated on the points in the vote thread and drafted up a
   wiki
 page on how-to-commit -
https://wiki.apache.org/hadoop/HowToCommitWithGit
 . We can work on improving this further and call a vote thread on
  those
 items if need be.

 Thanks
 Karthik


 On Tue, Aug 26, 2014 at 11:41 AM, Suresh Srinivas 
sur...@hortonworks.com
 javascript:_e(%7B%7D,'cvml','sur...@hortonworks.com'); wrote:

 Karthik,

 I would like to see detailed information on how this migration
 will
  be
 done, how it will affect the existing project and commit process.
  This
 should be done in a document that can be reviewed instead of in an
   email
 thread on an ad-hoc basis. Was there any voting on this in PMC and
should
 we have a vote to ensure everyone is one the same page on doing
 this
   and
 how to go about it?

 Regards,
 Suresh


 On Tue, Aug 26, 2014 at 9:17 AM, Karthik Kambatla 
  ka...@cloudera.com
 javascript:_e(%7B%7D,'cvml','ka...@cloudera.com');
 wrote:

  Last I heard, the import is still going on and appears closer to
getting
  done. Thanks for your patience with the migration.
 
  I ll update you as and when there is something. Eventually, the
  git
repo
  should be at the location in the wiki.
 
 
  On Mon, Aug 25, 2014 at 3:45 PM, Karthik Kambatla 
   ka...@cloudera.com
 javascript:_e(%7B%7D,'cvml','ka...@cloudera.com');
  wrote:
 
   Thanks for bringing these points up, Zhijie.
  
   By the way, a revised How-to-commit wiki is at:
   https://wiki.apache.org/hadoop/HowToCommitWithGit . Please
 feel
free
 to
   make changes and improve it.
  
   On Mon, Aug 25, 2014 at 11:00 AM, Zhijie Shen 
zs...@hortonworks.com
 javascript:_e(%7B%7D,'cvml','zs...@hortonworks.com');
   wrote:
  
   Do we have any convention about user.name and
 user.email?
   For
   example,
   we'd like to use @apache.org for the email.
  
  
   May be, we can ask people to use project-specific configs here
  and
use
   their real name and @apache.org address.
  
   Is there any downside to letting people use their global
 values
   for
 these
   configs?
  
  
  
  
   Moreover, do we want to use --author=Author Name 
 em...@address.com javascript:_e(%7B%7D,'cvml','em...@address.com
   ');
   when committing on behalf of a particular contributor?
  
  
   Fetching the email-address is complicated here. Should we use
  the
   contributor's email from JIRA? What if that is not their
 @apache
 address?

Re: Hortonworks scripting ...

2014-08-18 Thread Colin McCabe

This mailing list is for questions about Apache Hadoop, not commercial
Hadoop distributions.  Try asking a Hortonworks-specific mailing list.

best,
Colin

On Thu, Aug 14, 2014 at 3:23 PM, Niels Basjes ni...@basjes.nl wrote:
 Hi,

 In the core Hadoop you can on your (desktop) client have multiple clusters
 available simply by having multiple directories with setting files (i.e.
 core-site.xml etc.) and select the one you want by changing the environment
 settings (i.e. HADOOP_CONF_DIR and such) around.

 This doesn't work when I run under the Hortonworks 2.1.2 distribution.

 There I find that in all of the scripts placed in /usr/bin/ there is
 mucking about with the environment settings. Things from /etc/default are
 sourced and they override my settings.
 Now I can control part of it by directing the BIGTOP_DEFAULTS_DIR into a
 blank directory.
 But in /usr/bin/pig sourcing /etc/default/hadoop hardcoded into the script.

 Why is this done this way?

 P.S. Where is the git(?) repo located where this (apperently HW specific)
 scripting is maintained?

 --
 Best regards / Met vriendelijke groeten,

 Niels Basjes

Re: [DISCUSS] Switch to log4j 2

2014-08-18 Thread Colin McCabe

On Fri, Aug 15, 2014 at 8:50 AM, Aaron T. Myers a...@cloudera.com wrote:
Not necessarily opposed to switching logging frameworks, but I believe we
can actually support async logging with today's logging system if we wanted
to, e.g. as was done for the HDFS audit logger in this JIRA:

https://issues.apache.org/jira/browse/HDFS-5241

Yes, this is a great example of making something async without
switching logging frameworks. +1 for doing that where it is
appropriate.

--
Aaron T. Myers
Software Engineer, Cloudera

On Fri, Aug 15, 2014 at 5:44 AM, Steve Loughran ste...@hortonworks.com
wrote:

moving to SLF4J as an API is independent —it's just a better API for
logging than commons-logging, was already a dependency and doesn't force
anyone to switch to a new log back end.

Interesting idea. Did anyone do a performance comparison and/or API
comparison with SLF4j on Hadoop?

On 15 August 2014 03:34, Tsuyoshi OZAWA ozawa.tsuyo...@gmail.com wrote:

Hi,

Steve has started discussion titled use SLF4J APIs in new modules?
as a related topic.

http://mail-archives.apache.org/mod_mbox/hadoop-common-dev/201404.mbox/%3cca+4kjvv_9cmmtdqzcgzy-chslyb1wkgdunxs7wrheslwbuh...@mail.gmail.com%3E

It sounds good to me to use asynchronous logging when we log INFO. One

-1. Async logging for everything will make a lot of failures
un-debuggable. Just to give one example, what if you get a JVM out of
memory crash? You'll lose the last few log messages which could have
told you what was going on. Even if the JVM doesn't terminate, log
messages will be out of order, which is annoying, and will make
debugging harder.

The kernel already buffers the log files in memory. Not every log
message generates a disk seek. But on the other hand, if the JVM
process crashes, you've got everything. In other words, we've already
got as much buffering and asynchronicity as we need!

If the problem is that the noisy logs are overloading the disk
bandwidth, that problem can't be solved by adding Java-level async.
You need more bandwidth. A simple way of doing this is putting the
log partition on /dev/shm. We could also look into stripping some of
the boilerplate from log messages-- there are a lot of super-long log
messages that could be much more concise. Other Java logging
frameworks might have less overhead (I'm not an expert on this, but
maybe someone could post some numbers?)

best,
Colin

concern is that asynchronous logging makes debugging difficult - I
don't know log4j 2 well, but I suspect that ordering of logging can be
changed even if WARN or FATAL are logged with synchronous logger.

Thanks,
- Tsuyoshi

On Thu, Aug 14, 2014 at 6:44 AM, Arpit Agarwal aagar...@hortonworks.com

wrote:
I don't recall whether this was discussed before.

I often find our INFO logging to be too sparse for useful diagnosis. A
high
performance logging framework will encourage us to log more.
Specifically,
Asynchronous Loggers look interesting.
https://logging.apache.org/log4j/2.x/manual/async.html#Performance

What does the community think of switching to log4j 2 in a Hadoop 2.x
release?

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or
entity
to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the
reader
of this message is not the intended recipient, you are hereby notified
that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender
immediately
and delete it from your system. Thank You.

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

Re: [VOTE] Migration from subversion to git for version control

2014-08-11 Thread Colin McCabe

+1.

best,
Colin

On Fri, Aug 8, 2014 at 7:57 PM, Karthik Kambatla ka...@cloudera.com wrote:
 I have put together this proposal based on recent discussion on this topic.

 Please vote on the proposal. The vote runs for 7 days.

1. Migrate from subversion to git for version control.
2. Force-push to be disabled on trunk and branch-* branches. Applying
changes from any of trunk/branch-* to any of branch-* should be through
git cherry-pick -x.
3. Force-push on feature-branches is allowed. Before pulling in a
feature, the feature-branch should be rebased on latest trunk and the
changes applied to trunk through git rebase --onto or git cherry-pick
commit-range.
4. Every time a feature branch is rebased on trunk, a tag that
identifies the state before the rebase needs to be created (e.g.
tag_feature_JIRA-2454_2014-08-07_rebase). These tags can be deleted once
the feature is pulled into trunk and the tags are no longer useful.
5. The relevance/use of tags stay the same after the migration.

 Thanks
 Karthik

 PS: Per Andrew Wang, this should be a Adoption of New Codebase kind of
 vote and will be Lazy 2/3 majority of PMC members.

Re: Code guidelines and bash

2014-07-29 Thread Colin McCabe

On Tue, Jul 29, 2014 at 2:45 AM, 俊平堵 junping...@apache.org wrote:
 Sun's java code convention (published in year of 97) suggest 80 column per
 line for old-style terminals. It sounds pretty old, However, I saw some
 developers (not me :)) like to open multiple terminals in one screen for
 coding/debugging so 80-colum could be just fit. Google's java convention (
 https://google-styleguide.googlecode.com/svn/trunk/javaguide.html#s4.4-column-limit)
 shows some flexibility here with 80 or 100 column (and some exception
 cases).
 Like Chris mentioned early, I think this 80-column should just be a general
 guideline but not a strict limit - we can break it if it hurts legibility
 of code reading.
 btw, some research work found that CPL (characters per line) only had small
 effects on readability for news, including factors of speed and
 comprehension (
 http://psychology.wichita.edu/surl/usabilitynews/72/LineLength.asp). Not
 sure if reading code is the same (assume break lines properly).

There is a lot of contradictory research in this area.  For example,
http://eric.ed.gov/?id=EJ749012 talks about 70 characters per line as
ideal.

I think a lot of these studies don't really translate very well to
code.  (A lot of them are college students seeing how quickly they can
read a news article.)  Code with extremely long line lengths tends to
have super-deep nesting, which makes it hard to keep track of what is
going on (the so-called arrow anti-pattern).  This is especially
true when there are break and continue statements involved.
Super-long lines make diffs very difficult to do.  And it's just
unpleasant to read, forcing users to choose between horizontal
scrolling or tiny text...

Maybe it makes sense to extend the bash line length, though, if it's
tough to fit in 80 chars.  Bash is whitespace sensitive and doing the
line continuation thing is a pain.  Another option might be renaming
some variables, or using temp variables with shorter names...

best,
Colin




 2014-07-29 15:24 GMT+08:00 Andrew Purtell apurt...@apache.org:

 On Mon, Jul 28, 2014 at 12:05 PM, Doug Cutting cutt...@apache.org wrote:

  On Sun, Jul 27, 2014 at 9:28 PM, Ted Dunning tdunn...@maprtech.com
  wrote:
   I don't know of any dev environments in common use today that can't
  display 100 characters.
 
  I edit in an 80-column Emacs window that just fits beside an 80-column
  shell window on a portrait-rotated 24 monitor.
 

 You win the Internet today, Old School category! (smile)


 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)

Re: [DISCUSS] Assume Private-Unstable for classes that are not annotated

2014-07-25 Thread Colin McCabe

+1.

Colin


On Tue, Jul 22, 2014 at 2:54 PM, Karthik Kambatla ka...@cloudera.com
wrote:

 Hi devs

 As you might have noticed, we have several classes and methods in them that
 are not annotated at all. This is seldom intentional. Avoiding incompatible
 changes to all these classes can be considerable baggage.

 I was wondering if we should add an explicit disclaimer in our
 compatibility guide that says, Classes without annotations are to
 considered @Private

 For methods, is it reasonable to say - Class members without specific
 annotations inherit the annotations of the class?

 Thanks
 Karthik

Re: Porting Hadoop to FreeBSD

2014-07-14 Thread Colin McCabe

Thanks for working on this, Dmitry.  It's good to see Hadoop support
another platform.  The code changes look pretty minor, too.

best,
Colin

On Tue, Jul 8, 2014 at 7:08 AM, Dmitry Sivachenko trtrmi...@gmail.com wrote:
 Hello,

 I am trying to make hadoop usable on FreeBSD OS.  Following Steve Loughran's 
 suggestion, I created HADOOP-10796 to accumulate links to other issues 
 related to this subject.
 I added links to (almost) all patches needed to achieve this goal.  Right now 
 I have a small cluster running Hadoop-2.4.1 on FreeBSD and passing basic 
 tests (TeraSort mainly).

 I am looking forward for cooperation with you to include all relevant changes 
 into hadoop distribution.
 Please do not hesitate to contact me if you are not satisfied with the 
 proposed solutions.

 Thanks in advance!

Re: [jira] [Created] (HADOOP-10795) unale to build hadoop 2.4.1(redhat5.8 x64)

2014-07-14 Thread Colin McCabe

A unit test failed.  See my response on JIRA.

best,
Colin

On Tue, Jul 8, 2014 at 3:11 PM, Jay Vyas jayunit100.apa...@gmail.com wrote:
 these appear to be java errors related use to your jdk?
 maybe your JDK doesnt match up well with your OS.
 Consider trying red hat 6+ or Fedora 20?


 On Jul 8, 2014, at 5:45 AM, moses.wang (JIRA) j...@apache.org wrote:

 moses.wang created HADOOP-10795:
 ---

 Summary: unale to build hadoop 2.4.1(redhat5.8 x64)
 Key: HADOOP-10795
 URL: https://issues.apache.org/jira/browse/HADOOP-10795
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
 Environment: OS version rehat 5.8 x64
 maven version 3.3.1
 java version jdk 1.7_15 for x64
Reporter: moses.wang


 unale to build hadoop 2.4.1(redhat5.8 x64)
 WARNING] Some problems were encountered while building the effective model 
 for org.apache.hadoop:hadoop-project:pom:2.4.1
 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
 found duplicate declaration of plugin 
 org.apache.maven.plugins:maven-enforcer-plugin @ line 1015, column 15

 [WARNING] Some problems were encountered while building the effective model 
 for org.apache.hadoop:hadoop-common:jar:2.4.1
 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
 found duplicate declaration of plugin 
 org.apache.maven.plugins:maven-surefire-plugin @ line 479, column 15
 [WARNING] 'build.plugins.plugin.(groupId:artifactId)' must be unique but 
 found duplicate declaration of plugin 
 org.apache.maven.plugins:maven-enforcer-plugin @ 
 org.apache.hadoop:hadoop-project:2.4.1, 
 /home/software/Server/hadoop-2.4.1-src/hadoop-project/pom.xml, line 1015, 
 column 15

 WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/FastByteComparisons.java:[25,15]
  Unsafe is internal proprietary API and may be removed in a future release
 [WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java:[42,15]
  Unsafe is internal proprietary API and may be removed in a future release
 [WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[21,15]
  Signal is internal proprietary API and may be removed in a future release
 [WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/SignalLogger.java:[22,15]
  SignalHandler is internal proprietary API and may be removed in a future 
 release

 [WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[22,24]
  AlgorithmId is internal proprietary API and may be removed in a future 
 release
 [WARNING] 
 /home/software/Server/hadoop-2.4.1-src/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/security/ssl/KeyStoreTestUtil.java:[23,24]
  CertificateAlgorithmId is internal proprietary API and may be removed in a 
 future release

 testROBufferDirAndRWBufferDir[1](org.apache.hadoop.fs.TestLocalDirAllocator) 
  Time elapsed: 0.014 sec   FAILURE!

 [ERROR] Failed to execute goal 
 org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on 
 project hadoop-common: There are test failures.
 [ERROR]



 --
 This message was sent by Atlassian JIRA
 (v6.2#6252)

Re: Plans of moving towards JDK7 in trunk

2014-06-20 Thread Colin McCabe

I think the important thing to do right now is to ensure our code
works with jdk8.  This is similar to the work we did last year to fix
issues that cropped up with jdk7.  There were just a few minor things
like the error in which it ran unit tests that caused issues.

I don't think it's out of the question to bump minVersion directly
from 1.6 to 1.8.  Java 7 is EOL in less than a year, after all... at
least according to the current roadmap here:
http://www.oracle.com/technetwork/java/eol-135779.html

We should try to do things in a way that causes the least disruption
for users upgrading clusters, if possible...

cheers,
Colin

On Thu, Jun 19, 2014 at 3:31 PM, Andrew Purtell apurt...@apache.org wrote:
 There are a number of security (algorithm, not vulnerability) and
 performance improvements that landed in 8, not 7. As a runtime for the
 performance conscious, it might be recommendable. I've come across GC
 issues in 6 or 7 where, talking with some Java platform folks, the first
 suggested course of action is try again with 8. Would be be more of the
 current moment if this discussion was about setting guidelines that
 prescribe when and when not to use 8+ language features, or concurrency
 library improvements that rely on intrinsics only available in the 8
 runtime? Has the Java 6 ship sailed? Just set the minimum supported JDK and
 runtime at 7 at next release? Use of the  operator or multicatch wouldn't
 and shouldn't need be debated, they are quite minor. On the other hand, I
 would imagine discussion and debate on what 8+ language features might be
 useful to use at some future time could be a lively one.



 On Wed, Jun 18, 2014 at 3:03 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 In CDH5, Cloudera encourages people to use JDK7.  JDK6 has been EOL
 for a while now and is not something we recommend.

 As we discussed before, everyone is in favor of upgrading to JDK7.
 Every cluster operator of a reasonably modern Hadoop should do it
 whatever distro or release you run.  As developers, we run JDK7 as
 well.

 I'd just like to see a plan for when branch-2 (or some other branch)
 will create a stable release that drops support for JDK1.6.  If we
 don't have such a plan, I feel like it's too early to talk about this
 stuff.

 If we drop support for 1.6 in trunk but not in branch-2, we are
 fragmenting the project.  People will start writing unreleaseable code
 (because it doesn't work on branch-2) and we'll be back to the bad old
 days of Hadoop version fragmentation that branch-2 was intended to
 solve.  Backports will become harder.  The biggest problem is that
 trunk will start to depend on libraries or Maven plugins that branch-2
 can't even use, because they're JDK7+-only.

 Steve wrote: if someone actually did file a bug on something on
 branch-2 which didn't work on Java 6 but went away on Java7+, we'd
 probably close it as a WORKSFORME.

 Steve, if this is true, we should just bump the minimum supported
 version for branch-2 to 1.7 today and resolve this.  If we truly
 believe that there are no issues here, then let's just decide to drop
 1.6 in a specific future release of Hadoop 2.  If there are issues
 with releasing JDK1.7+ only code, then let's figure out what they are
 before proceeding.

 best,
 Colin


 On Wed, Jun 18, 2014 at 1:41 PM, Sandy Ryza sandy.r...@cloudera.com
 wrote:
  We do release warnings when we are aware of vulnerabilities in our
  dependencies.
 
  However, unless I'm grossly misunderstanding, the vulnerability that you
  point out is not a vulnerability within the context of our software.
   Hadoop doesn't try to sandbox within JVMs.  In a secure setup, any JVM
  running non-trusted user code is running as that user, so breaking out
  doesn't offer the ability to do anything malicious.
 
  -Sandy
 
  On Wed, Jun 18, 2014 at 1:30 PM, Ottenheimer, Davi 
 davi.ottenhei...@emc.com
  wrote:
 
  Andrew,
 
 
 
  “I don't see any point to switching” is an interesting perspective,
 given
  the well-known risks of running unsafe software. Clearly customer best
  interest is stability. JDK6 is in a known unsafe state. The longer
 anyone
  delays the necessary transition to safety the longer the door is left
 open
  to predictable disaster.
 
 
 
  You also said we still test and support JDK6. I searched but have not
  been able to find Cloudera critical security fixes for JDK6.
 
 
 
  Can you clarify, for example, Java 6 Update 51 for CVE-2013-2465? In
 other
  words, did you release to your customers any kind of public alert or
  warning of this CVSS 10.0 event as part of your JDK6 support?
 
 
 
  http://www.cvedetails.com/cve/CVE-2013-2465/
 
 
 
  If you are not releasing your own security fixes for JDK6 post-EOL would
  it perhaps be safer to say Cloudera is hands-off; neither supports, nor
  opposes the known insecure and deprecated/unpatched JDK?
 
 
 
  I mentioned before in this thread the Oracle support timeline:
 
 
 
  - official public EOL (end of life

Re: Plans of moving towards JDK7 in trunk

2014-06-20 Thread Colin McCabe

Er, that should read order in which it ran unit tests.

C.

On Fri, Jun 20, 2014 at 11:02 AM, Colin McCabe cmcc...@alumni.cmu.edu wrote:
 I think the important thing to do right now is to ensure our code
 works with jdk8.  This is similar to the work we did last year to fix
 issues that cropped up with jdk7.  There were just a few minor things
 like the error in which it ran unit tests that caused issues.

 I don't think it's out of the question to bump minVersion directly
 from 1.6 to 1.8.  Java 7 is EOL in less than a year, after all... at
 least according to the current roadmap here:
 http://www.oracle.com/technetwork/java/eol-135779.html

 We should try to do things in a way that causes the least disruption
 for users upgrading clusters, if possible...

 cheers,
 Colin

 On Thu, Jun 19, 2014 at 3:31 PM, Andrew Purtell apurt...@apache.org wrote:
 There are a number of security (algorithm, not vulnerability) and
 performance improvements that landed in 8, not 7. As a runtime for the
 performance conscious, it might be recommendable. I've come across GC
 issues in 6 or 7 where, talking with some Java platform folks, the first
 suggested course of action is try again with 8. Would be be more of the
 current moment if this discussion was about setting guidelines that
 prescribe when and when not to use 8+ language features, or concurrency
 library improvements that rely on intrinsics only available in the 8
 runtime? Has the Java 6 ship sailed? Just set the minimum supported JDK and
 runtime at 7 at next release? Use of the  operator or multicatch wouldn't
 and shouldn't need be debated, they are quite minor. On the other hand, I
 would imagine discussion and debate on what 8+ language features might be
 useful to use at some future time could be a lively one.



 On Wed, Jun 18, 2014 at 3:03 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:

 In CDH5, Cloudera encourages people to use JDK7.  JDK6 has been EOL
 for a while now and is not something we recommend.

 As we discussed before, everyone is in favor of upgrading to JDK7.
 Every cluster operator of a reasonably modern Hadoop should do it
 whatever distro or release you run.  As developers, we run JDK7 as
 well.

 I'd just like to see a plan for when branch-2 (or some other branch)
 will create a stable release that drops support for JDK1.6.  If we
 don't have such a plan, I feel like it's too early to talk about this
 stuff.

 If we drop support for 1.6 in trunk but not in branch-2, we are
 fragmenting the project.  People will start writing unreleaseable code
 (because it doesn't work on branch-2) and we'll be back to the bad old
 days of Hadoop version fragmentation that branch-2 was intended to
 solve.  Backports will become harder.  The biggest problem is that
 trunk will start to depend on libraries or Maven plugins that branch-2
 can't even use, because they're JDK7+-only.

 Steve wrote: if someone actually did file a bug on something on
 branch-2 which didn't work on Java 6 but went away on Java7+, we'd
 probably close it as a WORKSFORME.

 Steve, if this is true, we should just bump the minimum supported
 version for branch-2 to 1.7 today and resolve this.  If we truly
 believe that there are no issues here, then let's just decide to drop
 1.6 in a specific future release of Hadoop 2.  If there are issues
 with releasing JDK1.7+ only code, then let's figure out what they are
 before proceeding.

 best,
 Colin


 On Wed, Jun 18, 2014 at 1:41 PM, Sandy Ryza sandy.r...@cloudera.com
 wrote:
  We do release warnings when we are aware of vulnerabilities in our
  dependencies.
 
  However, unless I'm grossly misunderstanding, the vulnerability that you
  point out is not a vulnerability within the context of our software.
   Hadoop doesn't try to sandbox within JVMs.  In a secure setup, any JVM
  running non-trusted user code is running as that user, so breaking out
  doesn't offer the ability to do anything malicious.
 
  -Sandy
 
  On Wed, Jun 18, 2014 at 1:30 PM, Ottenheimer, Davi 
 davi.ottenhei...@emc.com
  wrote:
 
  Andrew,
 
 
 
  “I don't see any point to switching” is an interesting perspective,
 given
  the well-known risks of running unsafe software. Clearly customer best
  interest is stability. JDK6 is in a known unsafe state. The longer
 anyone
  delays the necessary transition to safety the longer the door is left
 open
  to predictable disaster.
 
 
 
  You also said we still test and support JDK6. I searched but have not
  been able to find Cloudera critical security fixes for JDK6.
 
 
 
  Can you clarify, for example, Java 6 Update 51 for CVE-2013-2465? In
 other
  words, did you release to your customers any kind of public alert or
  warning of this CVSS 10.0 event as part of your JDK6 support?
 
 
 
  http://www.cvedetails.com/cve/CVE-2013-2465/
 
 
 
  If you are not releasing your own security fixes for JDK6 post-EOL would
  it perhaps be safer to say Cloudera is hands-off; neither supports, nor
  opposes the known

Re: Plans of moving towards JDK7 in trunk

2014-06-18 Thread Colin McCabe

I think we should come up with a plan for when the next Hadoop release
will drop support for JDK6.  We all know that day needs to come... the
only question is when.  I agree that writing the JDK7-only code
doesn't seem very productive unless we have a plan for when it will be
released and usable.

best,
Colin

On Tue, Jun 17, 2014 at 10:08 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Reviving this thread, I noticed there's been a patch and +1 on
 HADOOP-10530, and I don't think we actually reached a conclusion.

 I (and others) have expressed concerns about moving to JDK7 for trunk.
 Summarizing a few points:

 - We can't move to JDK7 in branch-2 because of compatibility
 - branch-2 is currently the only Hadoop release vehicle, there are no plans
 for a trunk-based Hadoop 3
 - Introducing JDK7-only APIs in trunk will increase divergence with
 branch-2 and make backports harder
 - Almost all developers care only about branch-2, since it is the only
 release vehicle

 With this in mind, I struggle to see any upsides to introducing JDK7-only
 APIs to trunk. Please let's not do anything on HADOOP-10530 or related
 until we agree on this.

 Thanks,
 Andrew


 On Mon, Apr 14, 2014 at 3:31 PM, Steve Loughran ste...@hortonworks.com
 wrote:

 On 14 April 2014 17:46, Andrew Purtell apurt...@apache.org wrote:

  How well is trunk tested? Does anyone deploy it with real applications
  running on top? When will the trunk codebase next be the basis for a
  production release? An impromptu diff of hadoop-common trunk against
  branch-2 as of today is 38,625 lines. Can they be said to be the same
  animal? I ask because any disincentive toward putting code in trunk is
  beside the point, if the only target worth pursuing today is branch-2
  unless one doesn't care if the code is released for production use.
  Questions on whither JDK6 or JDK7+ (or JRE6 versus JRE7+) only matter for
  the vast majority of Hadoopers if talking about branch-2.
 
 
 I think its partly a timescale issue; its also because the 1-2 transition
 was so significant, especially at the YARN layer, that it's still taking
 time to trickle through.

 If you do want code to ship this year, branch-2 is where you are going to
 try and get it in -and like you say, that's where things get tried in the
 field. At the same time, the constraints of stability are holding us back
 -already-.

 I don't see why we should have such another major 1-2 transition in future;
 the rate that Arun is pushing out 2.x releases its almost back to the 0.1x
 timescale -though at that point most people were fending for themselves and
 expectations of stability were less. We do want smaller version increments
 in future, which branch-2 is -mostly- delivering.

 While Java 7 doesn't have some must-have features, Java 8 is a significant
 improvement in the language, and we should be looking ahead to that, maybe
 even doing some leading-edge work on the side, so the same discussion
 doesn't come up in two years time when java 7 goes EOL.


 -steve

 (personal opinions only, etc, )


 
  On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe cmcc...@alumni.cmu.edu
  wrote:
 
   I think the bottom line here is that as long as our stable release
   uses JDK6, there is going to be a very, very strong disincentive to
   put any code which can't run on JDK6 into trunk.
  
   Like I said earlier, the traditional reason for putting something in
   trunk but not the stable release is that it needs more testing.  If a
   stable release that drops support for JDK6 is more than a year away,
   does it make sense to put anything in trunk like that?  What might
   need more than a year of testing?  Certainly not changes to
   LocalFileSystem to use the new APIs.  I also don't think an upgrade to
   various libraries qualifies.
  
   It might be best to shelve this for now, like we've done in the past,
   until we're ready to talk about a stable release that requires JDK7+.
   At least that's my feeling.
  
   If we're really desperate for the new file APIs JDK7 provides, we
   could consider using loadable modules for it in branch-2.  This is
   similar to how we provide JNI versions of certain things on certain
   platforms, without dropping support for the other platforms.
  
   best,
   Colin
  
   On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata rst...@altiscale.com
   wrote:
There's an outstanding question addressed to me: Are there
 particular
features or new dependencies that you would like to contribute (or
 see
contributed) that require using the Java 1.7 APIs?  The question
misses the point: We'd figure out how to write something we wanted to
contribute to Hadoop against the APIs of Java4 if that's what it took
to get them into a stable release.  And at current course and speed,
that's how ridiculous things could get.
   
To summarize, it seems like there's a vague consensus that it might
 be
okay to eventually allow the use of Java7 in trunk, but there's

Re: Plans of moving towards JDK7 in trunk

2014-06-18 Thread Colin McCabe

 for

 themselves

and

 expectations of stability were less. We do want smaller version

increments

 in future, which branch-2 is -mostly- delivering.



 While Java 7 doesn't have some must-have features, Java 8 is a

significant

 improvement in the language, and we should be looking ahead to

 that,

maybe

 even doing some leading-edge work on the side, so the same

 discussion doesn't come up in two years time when java 7 goes EOL.





 -steve



 (personal opinions only, etc, )





 

  On Mon, Apr 14, 2014 at 9:22 AM, Colin McCabe 

   cmcc...@alumni.cmu.edumailto:cmcc...@alumni.cmu.edu

  wrote:

 

   I think the bottom line here is that as long as our stable

   release uses JDK6, there is going to be a very, very strong

   disincentive to put any code which can't run on JDK6 into
 trunk.

  

   Like I said earlier, the traditional reason for putting

   something

   in

   trunk but not the stable release is that it needs more testing.

If a

   stable release that drops support for JDK6 is more than a year

   away,

   does it make sense to put anything in trunk like that?  What

   might need more than a year of testing?  Certainly not changes

   to LocalFileSystem to use the new APIs.  I also don't think an

   upgrade

to

   various libraries qualifies.

  

   It might be best to shelve this for now, like we've done in

   the

   past,

   until we're ready to talk about a stable release that requires

   JDK7+.

   At least that's my feeling.

  

   If we're really desperate for the new file APIs JDK7 provides,

   we could consider using loadable modules for it in branch-2.

   This is similar to how we provide JNI versions of certain

   things on certain platforms, without dropping support for the
 other

  platforms.

  

   best,

   Colin

  

   On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata 

   rst...@altiscale.commailto:rst...@altiscale.com



   wrote:

There's an outstanding question addressed to me: Are there

 particular

features or new dependencies that you would like to

contribute

   (or

 see

contributed) that require using the Java 1.7 APIs?  The

question misses the point: We'd figure out how to write

something we

   wanted

to

contribute to Hadoop against the APIs of Java4 if that's

what it

took

to get them into a stable release.  And at current course

and

speed,

that's how ridiculous things could get.

   

To summarize, it seems like there's a vague consensus that

it

   might

 be

okay to eventually allow the use of Java7 in trunk, but

there's

   no

decision.  And there's been no answer to the concern that

even if

 such

dependencies were allowed in Java7, the only people using

them

would

be people who uninterested in getting their patches into a

stable release of Hadoop on any knowable timeframe, which

doesn't bode

well

for the ability to stabilize that Java7 code when it comes

time

   to

attempt to.

   

I don't have more to add, so I'll go back to lurking.  It'll

be interesting to see where we'll be standing a year from
 now.

   

On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA

ozawa.tsuyo...@gmail.commailto:ozawa.tsuyo...@gmail.com
 wrote:

Hi,

   

+1 for Karthik's idea(non-binding).

   

IMO, we should keep the compatibility between JDK 6 and JDK

7 on

 both

   branch-1

and branch-2, because users can be using them. For future

   releases

  that

   we can

declare breaking compatibility(e.g. 3.0.0 release), we can

use

JDK 7

features if we

can get benefits. However, it can increase maintenance

costs and

   distributes the

efforts of contributions to maintain branches. Then, I

think it

   is

reasonable approach

that we use limited and minimum JDK-7 APIs when we have

reasons

   we

  need

   to use

the features.

By the way, if we start to use JDK 7 APIs, we should

declare the

 basis

when to use

JDK 7 APIs on Wiki not to confuse contributors.

   

Thanks,

- Tsuyoshi

   

On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata 

rst...@altiscale.commailto:rst...@altiscale.com

 

   wrote:

It might make sense to try to enumerate the benefits of

switching

 to

Java7 APIs and dependencies.

   

  - Java7 introduced a huge number

Re: HADOOP_ROOT_LOGGER

2014-05-22 Thread Colin McCabe

It's not always practical to edit the log4j.properties file.  For one
thing, if you're using a management system, there may be many log4j
properties sprinkled around the system, and it could be difficult to figure
out which is the one you need to edit.  For another, you may not (should
not?) have permission to do this on a production cluster.

Doing something like HADOOP_ROOT_LOGGER=DEBUG,console hadoop fs -cat
/foo has helped me diagnose problems in the past.

best,
Colin


On Thu, May 22, 2014 at 6:34 AM, Robert Rati rr...@redhat.com wrote:

 In my experience the default HADOOP_ROOT_LOGGER definition will override
 any root logger defined in log4j.properties, which is where the problems
 have arisen.  If the HADOOP_ROOT_LOGGER definition in hadoop-config.sh were
 removed, wouldn't the root logger defined in the log4j.properties file be
 used?  Or do the client commands not read that configuration file?

 I'm trying to understand why the root logger should be defined outside of
 the log4j.properties file.

 Rob


 On 05/22/2014 12:53 AM, Vinayakumar B wrote:

 Hi Robert,

 I understand your confusion.

 HADOOP_ROOT_LOGGER is set to default value INFO,console if it hasn't set
 for anything and logs will be displayed on the console itself.
 This will be true for any client commands you run. For ex: hdfs dfs -ls
 /

 But for the server scripts (hadoop-daemon.sh, yarn-daemon.sh, etc)
   HADOOP_ROOT_LOGGER will be set to INFO, RFA if HADOOP_ROOT_LOGGER env
 variable is not defined.
 So that all the log messages of the server daemons goto some log files and
 this will be maintained by RollingFileAppender. If you want to override
 all
 these default and set your own loglevel then define that as env variable
 HADOOP_ROOT_LOGGER.

 For ex:
 export HADOOP_ROOT_LOGGER=DEBUG,RFA
export above env variable and then start server scripts or execute
 client
 commands, all logs goto files and will be maintained by
 RollingFileAppender.


 Regards,
 Vinay


 On Wed, May 21, 2014 at 6:42 PM, Robert Rati rr...@redhat.com wrote:

  I noticed in hadoop-config.sh there is this line:

 HADOOP_OPTS=$HADOOP_OPTS -Dhadoop.root.logger=${HADOOP_
 ROOT_LOGGER:-INFO,console}

 which is setting a root logger if HADOOP_ROOT_LOGGER isn't set.  Why is
 this here.needed?  There is a log4j.properties file provided that
 defines a
 default logger.  I believe the line above will result in overriding
 whatever is set for the root logger in the log4j.properties file.  This
 has
 caused some confusion and hacks to work around this.

 Is there a reason not to remove the above code and just have all the
 logger definitions in the log4j.properties file?  Is there maybe a
 compatibility concern?

 Rob

Re: Plans of moving towards JDK7 in trunk

2014-04-14 Thread Colin McCabe

I think the bottom line here is that as long as our stable release
uses JDK6, there is going to be a very, very strong disincentive to
put any code which can't run on JDK6 into trunk.

Like I said earlier, the traditional reason for putting something in
trunk but not the stable release is that it needs more testing.  If a
stable release that drops support for JDK6 is more than a year away,
does it make sense to put anything in trunk like that?  What might
need more than a year of testing?  Certainly not changes to
LocalFileSystem to use the new APIs.  I also don't think an upgrade to
various libraries qualifies.

It might be best to shelve this for now, like we've done in the past,
until we're ready to talk about a stable release that requires JDK7+.
At least that's my feeling.

If we're really desperate for the new file APIs JDK7 provides, we
could consider using loadable modules for it in branch-2.  This is
similar to how we provide JNI versions of certain things on certain
platforms, without dropping support for the other platforms.

best,
Colin

On Sun, Apr 13, 2014 at 10:39 AM, Raymie Stata rst...@altiscale.com wrote:
 There's an outstanding question addressed to me: Are there particular
 features or new dependencies that you would like to contribute (or see
 contributed) that require using the Java 1.7 APIs?  The question
 misses the point: We'd figure out how to write something we wanted to
 contribute to Hadoop against the APIs of Java4 if that's what it took
 to get them into a stable release.  And at current course and speed,
 that's how ridiculous things could get.

 To summarize, it seems like there's a vague consensus that it might be
 okay to eventually allow the use of Java7 in trunk, but there's no
 decision.  And there's been no answer to the concern that even if such
 dependencies were allowed in Java7, the only people using them would
 be people who uninterested in getting their patches into a stable
 release of Hadoop on any knowable timeframe, which doesn't bode well
 for the ability to stabilize that Java7 code when it comes time to
 attempt to.

 I don't have more to add, so I'll go back to lurking.  It'll be
 interesting to see where we'll be standing a year from now.

 On Sun, Apr 13, 2014 at 2:09 AM, Tsuyoshi OZAWA
 ozawa.tsuyo...@gmail.com wrote:
 Hi,

 +1 for Karthik's idea(non-binding).

 IMO, we should keep the compatibility between JDK 6 and JDK 7 on both 
 branch-1
 and branch-2, because users can be using them. For future releases that we 
 can
 declare breaking compatibility(e.g. 3.0.0 release), we can use JDK 7
 features if we
 can get benefits. However, it can increase maintenance costs and distributes 
 the
 efforts of contributions to maintain branches. Then, I think it is
 reasonable approach
 that we use limited and minimum JDK-7 APIs when we have reasons we need to 
 use
 the features.
 By the way, if we start to use JDK 7 APIs, we should declare the basis
 when to use
 JDK 7 APIs on Wiki not to confuse contributors.

 Thanks,
 - Tsuyoshi

 On Wed, Apr 9, 2014 at 11:44 AM, Raymie Stata rst...@altiscale.com wrote:
 It might make sense to try to enumerate the benefits of switching to
 Java7 APIs and dependencies.

   - Java7 introduced a huge number of language, byte-code, API, and
 tooling enhancements!  Just to name a few: try-with-resources, newer
 and stronger encyrption methods, more scalable concurrency primitives.
  See http://www.slideshare.net/boulderjug/55-things-in-java-7

   - We can't update current dependencies, and we can't add cool new ones.

   - Putting language/APIs aside, don't forget that a huge amount of effort
 goes into qualifying for Java6 (at least, I hope the folks claiming to
 support Java6 are putting in such an effort :-).  Wouldn't Hadoop
 users/customers be better served if qualification effort went into
 Java7/8 versus Java6/7?

 Getting to Java7 as a development env (and Java8 as a runtime env)
 seems like a no-brainer.  Question is: How?

 On Tue, Apr 8, 2014 at 10:21 AM, Sandy Ryza sandy.r...@cloudera.com wrote:
 It might make sense to try to enumerate the benefits of switching to Java7
 APIs and dependencies.  IMO, the ones listed so far on this thread don't
 make a compelling enough case to drop Java6 in branch-2 on any time frame,
 even if this means supporting Java6 through 2015.  For example, the change
 in RawLocalFileSystem semantics might be an incompatible change for
 branch-2 any way.


 On Tue, Apr 8, 2014 at 10:05 AM, Karthik Kambatla 
 ka...@cloudera.comwrote:

 +1 to NOT breaking compatibility in branch-2.

 I think it is reasonable to require JDK7 for trunk, if we limit use of
 JDK7-only API to security fixes etc. If we make other optimizations (like
 IO), it would be a pain to backport things to branch-2. I guess this all
 depends on when we see ourselves shipping Hadoop-3. Any ideas on that?


 On Tue, Apr 8, 2014 at 9:19 AM, Eli Collins e...@cloudera.com wrote:

  On Tue, Apr 8, 2014 at 2:00 AM, Ottenheimer, Davi

Re: [Important] Confirmation related to OpenSsl security issue

2014-04-11 Thread Colin McCabe

I took a quick glance at the build output, and I don't think openssl
is getting linked statically into libhadooppipes.a.

I see the following lines:

Linking CXX static library libhadooppipes.a
/usr/bin/cmake -P CMakeFiles/hadooppipes.dir/cmake_clean_target.cmake
/usr/bin/cmake -E cmake_link_script
CMakeFiles/hadooppipes.dir/link.txt --verbose=1
/usr/bin/ar cr libhadooppipes.a
CMakeFiles/hadooppipes.dir/main/native/pipes/impl/HadoopPipes.cc.o
/usr/bin/ranlib libhadooppipes.a

later on there are lines like this:

/usr/bin/c++-g -Wall -O2 -D_REENTRANT -D_GNU_SOURCE
-D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64
CMakeFiles/pipes-sort.dir/main/native/examples/impl/sort.cc.o  -o
examples/pipes-sort -rdynamic libhadooppipes.a libhadooputils.a -lssl
-lcrypto -lpthread

So when using libhadooppipes.a, you must supply your own copy of
libssl.so.  If you supply a vulnerable copy, you will be vulnerable.
If you supply a non-vulnerable copy, you won't be.  So I don't think
there is an impact on our build (unless I missed something here).

Just to make sure, it would be good if someone who uses
libhadooppipes.a to look at one of the versions in our release tarball
and verify that it works with the fixed openssl.

Colin

On Fri, Apr 11, 2014 at 2:27 AM, Vinayakumar B vinayakuma...@huawei.com wrote:
 Hi,

 Recently one security issue has been found in OpenSSL which has impacted so 
 many customers of different vendors.

 http://www.kb.cert.org/vuls/byvendor?searchviewQuery=FIELD+Reference=720951SearchOrder=4

 I want to ask, whether is there in impact of this on the Hadoop Release?

 Currently Hadoop-pipes are uses openssl-devel packages for building native 
 support.

 Can someone familiar with Hadoop-pipes can confirm whether is there any 
 impact of this security issue on builds of Hadoop built with defective 
 openssl?

 Regards,
Vinay

 
 This e-mail and attachments contain confidential information from HUAWEI,
 which is intended only for the person or entity whose address is listed
 above. Any use of the information contained herein in any way (including,
 but not limited to, total or partial disclosure, reproduction, or
 dissemination) by persons other than the intended recipient's) is
 prohibited. If you receive this e-mail in error, please notify the sender by
 phone or email immediately and delete it!

Re: Plans of moving towards JDK7 in trunk

2014-04-05 Thread Colin McCabe

I've been using JDK7 for Hadoop development for a while now, and I
know a lot of other folks have as well.  Correct me if I'm wrong, but
what we're talking about here is not moving towards JDK7 but
breaking compatibility with JDK6.

There are a lot of good reasons to ditch JDK6.  It would let us use
new APIs in JDK7, especially the new file APIs.  It would let us
update a few dependencies to newer versions.

I don't like the idea of breaking compatibility with JDK6 in trunk,
but not in branch-2.  The traditional reason for putting something in
trunk but not in branch-2 is that it is new code that needs some time
to prove itself.  This doesn't really apply to incrementing min.jdk--
we could do that easily whenever we like.  Meanwhile, if trunk starts
accumulating jdk7-only code and dependencies, backports from trunk to
branch-2 will become harder and harder over time.

Since we make stable releases off of branch-2, and not trunk, I don't
see any upside to this.  To be honest, I see only negatives here.
More time backporting, more issues that show up only in production
(branch-2) and not on dev machines (trunk).

Maybe it's time to start thinking about what version of branch-2 will
drop jdk6 support.  But until there is such a version, I don't think
trunk should do it.

best,
Colin


On Fri, Apr 4, 2014 at 3:15 PM, Haohui Mai h...@hortonworks.com wrote:
 I'm referring to the later case. Indeed migrating JDK7 for branch-2 is more
 difficult.

 I think one reasonable approach is to put the hdfs / yarn clients into
 separate jars. The client-side jars can only use JDK6 APIs, so that
 downstream projects running on top of JDK6 continue to work.

 The HDFS/YARN/MR servers need to be run on top of JDK7, and we're free to
 use JDK7 APIs inside them. Given the fact that there're way more code in
 the server-side compared to the client-side, having the ability to use JDK7
 in the server-side only might still be a win.

 The downside I can think of is that it might complicate the effort of
 publishing maven jars, but this should be an one-time issue.

 ~Haohui


 On Fri, Apr 4, 2014 at 2:37 PM, Alejandro Abdelnur t...@cloudera.comwrote:

 Haohui,

 Is the idea to compile/test with JDK7 and recommend it for runtime and stop
 there? Or to start using JDK7 API stuff as well? If the later is the case,
 then backporting stuff to branch-2 may break and patches may have to be
 refactored for JDK6. Given that branch-2 got GA status not so long ago, I
 assume it will be active for a while.

 What are your thoughts on this regard?

 Thanks


 On Fri, Apr 4, 2014 at 2:29 PM, Haohui Mai h...@hortonworks.com wrote:

  Hi,
 
  There have been multiple discussions on deprecating supports of JDK6 and
  moving towards JDK7. It looks to me that the consensus is that now hadoop
  is ready to drop the support of JDK6 and to move towards JDK7. Based on
 the
  consensus, I wonder whether it is a good time to start the migration.
 
  Here are my understandings of the current status:
 
  1. There is no more public updates of JDK6 since Feb 2013. Users no
 longer
  get fixes of security vulnerabilities through official public updates.
  2. Hadoop core is stuck with out-of-date dependency unless moving towards
  JDK7. (see
  http://hadoop.6.n7.nabble.com/very-old-dependencies-td71486.html)
  The implementation can also benefit from it thanks to the new
  functionalities in JDK7.
  3. The code is ready for JDK7. Cloudera and Hortonworks have successful
  stories of supporting Hadoop on JDK7.
 
 
  It seems that the real work of moving to JDK7 is minimal. We only need to
  (1) make sure the jenkins are running on top of JDK7, and (2) to update
 the
  minimum required Java version from 6 to 7. Therefore I propose that let's
  move towards JDK7 in trunk in the short term.
 
  Your feedbacks are appreciated.
 
  Regards,
  Haohui
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 



 --
 Alejandro


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please

Re: very old dependencies

2014-03-28 Thread Colin McCabe

I think we need some way of isolating YARN, MR, and HDFS clients from
the Hadoop dependencies.  Anything else just isn't sane... whatever we
may say, there will always be clients that rely on the dependencies
that we pull in, if we make those visible.  I can't really blame
clients for this.  It's sort of unclear what happens when you pull in
multiple versions of a dependency through Maven.

The easiest way to solve this would be to use something like Maven-shade.

best,
Colin

On Fri, Mar 28, 2014 at 10:30 AM, Steve Loughran ste...@hortonworks.com wrote:
 I don't disagree about version age -I've just been downgrading a project to
 use some of the versions. The issue with YARN apps is that you get these on
 your classpath.


1. There's a JIRA: https://issues.apache.org/jira/browse/HADOOP-9991
2. Nobody is working full time on these, I sporadically apply the
patches -I get to feel the pain downstream

 commons lang is at 2.6 now, which should keep you happy.

 jetty? Now that the shuffle is on netty we could think about this and jersey

 Guava is trouble. Leave it: new code doesn't work. Remove it and old code
 stops working.
 https://issues.apache.org/jira/browse/HADOOP-10101

 I think for branch-3 we should go ahead an apply the patch. For branch 2,
 it's too destructuve.

 At some point we also have to commit being java7 + only -this would
 actually help us move up to some of the later dependencies. That's clearly
 not a branch-2




 On 28 March 2014 14:59, Sangjin Lee sjl...@gmail.com wrote:

 Hi folks,

 Even as 2.3 was released, several dependencies of hadoop are quite dated.
 And more and more of them are causing friction for hadoop-related libraries
 and hadoop users in general, as these dependency versions are often
 incompatibly different than the versions most people use these days. So
 this is becoming a very real problem, if not one already.

 Some key ones include
 - guava: 11.0.2 (current 16.0.1)
 - jetty: 6.1.26 (current 9.1.3)
 - commons-lang: 2.6 (current 3.3.1)

 In particular, guava is already causing a lot of issues as many developers
 have moved onto a newer version and are using APIs that do not exist in
 11.0.2, and get NoSuchMethodErrors and such.

 Also, for jetty, 6.1.26 has been EOFed for some time now.

 Do we have a plan to review some of these dependencies and upgrade them to
 reasonably up-to-date versions? I remember seeing some JIRAs on specific
 dependencies, but has a review been done to determine a good set of
 versions to upgrade to?

 It would be great if we could upgrade some of the more common ones at least
 to modern versions.

 Thanks,
 Sangjin


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded Client + Old Server

2014-03-20 Thread Colin McCabe

+1 for making this guarantee explicit.

It also definitely seems like a good idea to test mixed versions in bigtop.

HDFS is not immune to new client, old server scenarios because the HDFS
client gets bundled into a lot of places.

Colin
On Mar 20, 2014 10:55 AM, Chris Nauroth cnaur...@hortonworks.com wrote:

 Our use of protobuf helps mitigate a lot of compatibility concerns, but
 there still can be situations that require careful coding on our part.
  When adding a new field to a protobuf message, the client might need to do
 a null check, even if the server-side implementation in the new version
 always populates the field.  When adding a whole new RPC endpoint, the
 client might need to consider the possibility that the RPC endpoint isn't
 there on an old server, and degrade gracefully after the RPC fails.  The
 original issue in MAPREDUCE-4052 concerned the script commands passed in a
 YARN container submission, where protobuf doesn't provide any validation
 beyond the fact that they're strings.

 Forward compatibility is harder than backward compatibility, and testing is
 a big challenge.  Our test suites in the Hadoop repo don't cover this.
  Does anyone know if anything in Bigtop tries to run with mixed versions?

 I agree that we need to make it clear in the language that upgrading client
 alone is insufficient to get access to new server-side features, including
 new YARN APIs.  Thanks for the suggestions, Steve.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/



 On Thu, Mar 20, 2014 at 5:53 AM, Steve Loughran ste...@hortonworks.com
 wrote:

  I'm clearly supportive of this, though of course the testing costs needed
  to back up the assertion make it more expensive than just a statement.
 
  Two issues
 
  -we'd need to make clear that new cluster features that a client can
 invoke
  won't be available. You can't expect snapshot or symlink support running
  against a -2.2.0 cluster, even if the client supports it.
 
  -in YARN, there are no guarantees that an app compiled against later YARN
  APIs will work in old clusters. Because YARN apps upload themselves to
 the
  server, and run with their hadoop, hdfs  yarn libraries. We have to do a
  bit of introspection in our code already to support this situation. The
  compatibility doc would need to be clear on that too: YARN apps that use
  new APIs (including new fields in datastructures) can expect link
  exceptions
 
 
 
 
 
  On 20 March 2014 04:25, Vinayakumar B vinayakuma...@huawei.com wrote:
 
   +1, I agree with your point Chris. It depends on the client application
   how they using the hdfs jars in their classpath.
  
   As implementation already supports the compatibility (through
 protobuf),
   No extra code changes required to support new Client + old server.
  
   I feel it will be good to explicitly mention about the compatibility of
   existing APIs in both versions.
  
   Anyway this is not applicable for the new APIs in latest client and
 this
   is understood. We can make it explicit in the document though.
  
  
   Regards,
   Vinayakumar B
  
   -Original Message-
   From: Chris Nauroth [mailto:cnaur...@hortonworks.com]
   Sent: 20 March 2014 05:36
   To: common-dev@hadoop.apache.org
   Cc: mapreduce-...@hadoop.apache.org; hdfs-...@hadoop.apache.org;
   yarn-...@hadoop.apache.org
   Subject: Re: [DISCUSS] Clarification on Compatibility Policy: Upgraded
   Client + Old Server
  
   I think this kind of compatibility issue still could surface for HDFS,
   particularly for custom applications (i.e. something not executed via
   hadoop jar on a cluster node, where the client classes ought to be
   injected into the classpath automatically).  Running DistCP between 2
   clusters of different versions could result in a 2.4.0 client calling a
   2.3.0 NameNode.  Someone could potentially pick up the 2.4.0 WebHDFS
   client as a dependency and try to use it to make HTTP calls to a 2.3.0
  HDFS
   cluster.
  
   Chris Nauroth
   Hortonworks
   http://hortonworks.com/
  
  
  
   On Wed, Mar 19, 2014 at 4:28 PM, Vinod Kumar Vavilapalli 
   vino...@apache.org
wrote:
  
It makes sense only for YARN today where we separated out the
 clients.
HDFS is still a monolithic jar so this compatibility issue is kind of
invalid there.
   
+vinod
   
On Mar 19, 2014, at 1:59 PM, Chris Nauroth cnaur...@hortonworks.com
 
wrote:
   
 I'd like to discuss clarification of part of our compatibility
  policy.
 Here is a link to the compatibility documentation for release
 2.3.0:


   
 http://hadoop.apache.org/docs/r2.3.0/hadoop-project-dist/hadoop-common
/Compatibility.html#Wire_compatibility

 For convenience, here are the specific lines in question:

 Client-Server compatibility is required to allow users to continue
 using the old clients even after upgrading the server (cluster) to
 a
 later version (or vice versa). For example, a Hadoop 2.1.0 client
 talking to

Re: [VOTE] Release Apache Hadoop 2.3.0

2014-02-11 Thread Colin McCabe

Looks good.

+1, also non-binding.

I downloaded the source tarball, checked md5, built, ran some unit
tests, ran an HDFS cluster.

cheers,
Colin

On Tue, Feb 11, 2014 at 6:53 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Thanks for putting this together Arun.

 +1 non-binding

 Downloaded source tarball
 Verified signature and digests
 Ran apache-rat:check
 Built with mvn clean package install -Pdist -Dtar
 Started a one-node cluster and ran wordcount

 Best,
 Andrew

 On Tue, Feb 11, 2014 at 5:40 PM, Ravi Prakash ravi...@ymail.com wrote:

 Thanks Arun for another release!

 +1 non-binding

 Verified signatures, deployed a single node cluster and ran sleep and
 wordcount. Everything looks fine.


 Regards
 Ravi




 On Tuesday, February 11, 2014 5:36 PM, Travis Thompson 
 tthomp...@linkedin.com wrote:

 Everything looks good so far, running on 100 nodes with security enabled.

 I've found two minor issues I've found with the new Namenode UI so far and
 will work on them over the next few days:

 HDFS-5934
 HDFS-5935

 Thanks,

 Travis


 On Feb 11, 2014, at 4:53 PM, Mohammad Islam misla...@yahoo.com
 wrote:

  Thanks Arun for the initiative.
 
  +1 non-binding.
 
 
  I tested the followings:
  1. Build package from the source tar.
  2. Verified with md5sum
  3. Verified with gpg
  4. Basic testing
 
  Overall, good to go.
 
  Regards,
  Mohammad
 
 
 
 
  On Tuesday, February 11, 2014 2:07 PM, Chen He airb...@gmail.com
 wrote:
 
  +1, non-binding
  successful compiled on MacOS 10.7
  deployed to Fedora 7 and run test job without any problem.
 
 
 
  On Tue, Feb 11, 2014 at 8:49 AM, Arun C Murthy a...@hortonworks.com
 wrote:
 
  Folks,
 
  I've created a release candidate (rc0) for hadoop-2.3.0 that I would
 like
  to get released.
 
  The RC is available at:
  http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
  The RC tag in svn is here:
  https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0
 
  The maven artifacts are available via repository.apache.org.
 
  Please try the release and vote; the vote will run for the usual 7 days.
 
  thanks,
  Arun
 
  PS: Thanks to Andrew, Vinod  Alejandro for all their help in various
  release activities.
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or
 entity to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the
 reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.

Re: Possible Length issue

2013-12-31 Thread Colin McCabe

There is a maximum length for message buffers that was introduced by
HADOOP-9676.  So messages with length 1752330339 should not be
accepted.

best,
Colin

On Sat, Dec 28, 2013 at 11:06 AM, Dhaivat Pandya
dhaivatpan...@gmail.com wrote:
 Hi,

 I've been working a lot with the Hadoop NameNode IPC protocol (while
 building a cache layer on top of Hadoop). I've noticed that for request
 packets coming from the default DFS client that do not have a method name,
 the length field is often *completely *off.

 For example, I've been looking at packets with length 1752330339. I thought
 this might have been an issue with my cache layer, so I checked with
 Wireshark, and found packets with such absurd length parameters (obviously,
 the packets themselves weren't actually that long; the length field was
 misrepresented).

 Unfortunately, I haven't had the opportunity to test this issue on other
 machines and setups (the reproducing steps should be running an ls / with
 the default DFS client and sniffing the packets to find the length
 parameter, release 1.2.1).

 Is this normal behavior, a bug or something I'm missing?

 Thank you,

 Dhaivat

Re: Next releases

2013-12-06 Thread Colin McCabe

If 2.4 is released in January, I think it's very unlikely to include
symlinks.  There is still a lot of work to be done before they're
usable.  You can look at the progress on HADOOP-10019.  For some of
the subtasks, it will require some community discussion before any
code can be written.

For better or worse, symlinks have not been requested by users as
often as features like NFS export, HDFS caching, ACLs, etc, so effort
has been focused on those instead.

For now, I think we should put the symlinks-disabling patches
(HADOOP-10020, etc) into branch-2, so that they will be part of the
next releases without additional effort.

I would like to see HDFS caching make it into 2.4.  The APIs and
implementation are beginning to stabilize, and around January it
should be ok to backport to a stable branch.

best,
Colin

On Thu, Dec 5, 2013 at 3:57 PM, Arun C Murthy a...@hortonworks.com wrote:
 Ok, I've updated https://wiki.apache.org/hadoop/Roadmap with a initial 
 strawman list for hadoop-2.4 which I feel we can get out in Jan.

 What else would folks like to see? Please keep timeframe in mind.

 thanks,
 Arun

 On Dec 2, 2013, at 10:55 AM, Arun C Murthy a...@hortonworks.com wrote:


 On Nov 13, 2013, at 1:55 PM, Jason Lowe jl...@yahoo-inc.com wrote:


 +1 to limiting checkins of patch releases to Blockers/Criticals.  If 
 necessary committers check into trunk/branch-2 only and defer to the patch 
 release manager for the patch release merge.  Then there should be fewer 
 surprises for everyone what ended up in a patch release and less likely the 
 patch release becomes destabilized from the sheer amount of code churn.  
 Maybe this won't be necessary if everyone understands that the patch 
 release isn't the only way to get a change out in timely manner.

 I've updated https://wiki.apache.org/hadoop/Roadmap to reflect that we only 
 put in Blocker/Critical bugs into Point Releases.

 Committers, from now, please exercise extreme caution when committing to a 
 point release: they should only be limited to Blocker bugs.

 thanks,
 Arun


 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Next releases

2013-11-14 Thread Colin McCabe

On Wed, Nov 13, 2013 at 10:10 AM, Arun C Murthy a...@hortonworks.com wrote:

 On Nov 12, 2013, at 1:54 PM, Todd Lipcon t...@cloudera.com wrote:

 On Mon, Nov 11, 2013 at 2:57 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 To be honest, I'm not aware of anything in 2.2.1 that shouldn't be
 there.  However, I have only been following the HDFS and common side
 of things so I may not have the full picture.  Arun, can you give a
 specific example of something you'd like to blow away?

 There are bunch of issues in YARN/MapReduce which clearly aren't *critical*, 
 similarly in HDFS a cursory glance showed up some 
 *enhancements*/*improvements* in CHANGES.txt which aren't necessary for a 
 patch release, plus things like:

 HADOOP-9623
 Update jets3t dependency to 0.9.0

I'm fine with reverting HADOOP-9623 from branch-2.2 and pushing it out
to branch-2.3.  It does bring in httpcore, a dependency that wasn't
there before.

Colin


 Here is a straw man proposal:

 
 A patch (third version) release should only include *blocker* bugs which are 
 critical from an operational, security or data-integrity issues.

 This way, we can ensure that a minor series release (2.2.x or 2.3.x or 2.4.x) 
 is always release-able, and more importantly, deploy-able at any point in 
 time.

 

 Sandy did bring up a related point about timing of releases and the urge for 
 everyone to cram features/fixes into a dot release.

 So, we could remedy that situation by doing a release every 4-6 weeks (2.3, 
 2.4 etc.) and keep the patch releases limited to blocker bugs.

 Thoughts?

 thanks,
 Arun





 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Merge HDFS-4949 to trunk

2013-10-24 Thread Colin McCabe

On Thu, Oct 24, 2013 at 1:45 PM, Chris Nauroth cnaur...@hortonworks.com wrote:
 Hi Andrew,

 I've come to the conclusion that I'm very confused about merge votes.  :-)
  It's not just about HDFS-4949.  I'm confused about all merge votes.
  Rather than muddy the waters here, I've started a separate discussion on
 common-dev.

 I do agree with the general plan outlined here, and I will comment directly
 on the HDFS-4949 jira with a binding +1 when I see that we've completed
 that plan.

Thanks, Chris.  Andrew posted a merge patch to HDFS-4949.

We're happy that this code is getting closer to getting into trunk,
since it will make it easier to integrate with the other features in
trunk (like HDFS-2832).  There are still some follow-up tasks, but we
feel that it's easier to do those in trunk.

I'm going to update the design doc in just a moment so be sure to
check it out.  Are there any other things we should do today prior to
merging?

Colin



 Chris Nauroth
 Hortonworks
 http://hortonworks.com/



 On Wed, Oct 23, 2013 at 2:18 PM, Andrew Wang andrew.w...@cloudera.comwrote:

 Hey Chris,

 Right now we're on track to have all of those things done by tomorrow.
 Since the remaining issues are either not technical or do not involve major
 changes, I was hoping we could +1 this merge vote in the spirit of +1
 pending jenkins. We've gotten clean unit test runs on upstream Jenkins as
 well, so the only fixups we should need for test-patch.sh are findbugs and
 javac (which are normally pretty trivial to clean up). Of course, all of
 your listed prereqs and test-patch would be taken care of before actually
 merging to trunk.

 So, we can reset the vote if you feel strongly about this, but it seems
 like the only real result will be delaying the merge by a week.

 Thanks,
 Andrew


 On Wed, Oct 23, 2013 at 1:03 PM, Chris Nauroth cnaur...@hortonworks.com
 wrote:

  I've received some feedback that we haven't handled this merge vote the
  same as other comparable merge votes, and that the vote should be reset
  because of this.
 
  The recent custom is that we only call for the merge vote after all
  pre-requisites have been satisfied.  This would include committing to the
  feature branch all patches that the devs deem necessary before the code
  lands in trunk, posting a test plan, posting an updated design doc in
 case
  implementation choices diverged from the original design doc, and
 getting a
  good test-patch run from Jenkins on the merge patch.  This was the
 process
  followed for other recent major features like HDFS-2802 (snapshots),
  HDFS-347 (short-circuit reads via sharing file descriptors), and
  HADOOP-8562 (Windows compatibility).  In this thread, we've diverged from
  that process by calling for a vote on a branch that hasn't yet completed
  the pre-requisites and stating a plan for work to be done before the
 merge.
 
  I still support this work, but can we please restart the vote after the
  pre-requisites have landed in the branch?
 
  Chris Nauroth
  Hortonworks
  http://hortonworks.com/
 
 
 
  On Fri, Oct 18, 2013 at 1:37 PM, Chris Nauroth cnaur...@hortonworks.com
  wrote:
 
   +1
  
   Sounds great!
  
   Regarding testing caching+federation, this is another thing that I had
   intended to pick up as part of HDFS-5149.  I'm not sure if I can get
 this
   done in the next 7 days, so I'll keep you posted.
  
   Chris Nauroth
   Hortonworks
   http://hortonworks.com/
  
  
  
   On Fri, Oct 18, 2013 at 11:15 AM, Colin McCabe cmcc...@alumni.cmu.edu
  wrote:
  
   Hi Chris,
  
   I think it's feasible to complete those tasks in the next 7 days.
   Andrew is on HDFS-5386.
  
   The test plan document is a great idea.  We'll try to get that up
   early next week.  We have a lot of unit tests now, clearly, but some
   manual testing is important too.
  
   If we discover any issues during testing, then we can push out the
   merge timeframe.  For example, one area that probably needs more
   testing is caching+federation.
  
   I would like to get HDFS-5378 and HDFS-5366 in as well.
  
   The other subtasks are nice to have but not really critical, and I
   think it would be just as easy to do them in trunk.  We're hoping that
   having this in trunk will make it easier for us to collaborate on
   HDFS-2832 and other ongoing work.
  
Also, I want to confirm that this vote only covers trunk.
I don't see branch-2 mentioned, so I assume that we're
not voting on merge to branch-2 yet.
  
   Yeah, this vote is only to merge to trunk.
  
   cheers.
   Colin
  
   On Fri, Oct 18, 2013 at 10:48 AM, Chris Nauroth
   cnaur...@hortonworks.com wrote:
I agree that the code has reached a stable point.  Colin and Andrew,
   thank
you for your contributions and collaboration.
   
Throughout development, I've watched the feature grow by running
 daily
builds in a pseudo-distributed deployment.  As of this week, the
 full
feature set is working end-to-end.  I also think we've reached

Re: [VOTE] Merge HDFS-4949 to trunk

2013-10-18 Thread Colin McCabe

Hi Chris,

I think it's feasible to complete those tasks in the next 7 days.
Andrew is on HDFS-5386.

The test plan document is a great idea.  We'll try to get that up
early next week.  We have a lot of unit tests now, clearly, but some
manual testing is important too.

If we discover any issues during testing, then we can push out the
merge timeframe.  For example, one area that probably needs more
testing is caching+federation.

I would like to get HDFS-5378 and HDFS-5366 in as well.

The other subtasks are nice to have but not really critical, and I
think it would be just as easy to do them in trunk.  We're hoping that
having this in trunk will make it easier for us to collaborate on
HDFS-2832 and other ongoing work.

 Also, I want to confirm that this vote only covers trunk.
 I don't see branch-2 mentioned, so I assume that we're
 not voting on merge to branch-2 yet.

Yeah, this vote is only to merge to trunk.

cheers.
Colin

On Fri, Oct 18, 2013 at 10:48 AM, Chris Nauroth
cnaur...@hortonworks.com wrote:
 I agree that the code has reached a stable point.  Colin and Andrew, thank
 you for your contributions and collaboration.

 Throughout development, I've watched the feature grow by running daily
 builds in a pseudo-distributed deployment.  As of this week, the full
 feature set is working end-to-end.  I also think we've reached a point of
 API stability for clients who want to control caching programmatically.

 There are several things that I'd like to see completed before the merge as
 pre-requisites:

 - HDFS-5203: Concurrent clients that add a cache directive on the same path
 may prematurely uncache from each other.
 - HDFS-5385: Caching RPCs are AtMostOnce, but do not persist client ID and
 call ID to edit log.
 - HDFS-5386: Add feature documentation for datanode caching.
 - Standard clean-ups to satisfy Jenkins pre-commit on the merge patch.
  (For example, I know we've introduced some Javadoc warnings.)
 - Full test suite run on Windows.  (The feature is not yet implemented on
 Windows.  This is just intended to catch regressions.)
 - Test plan posted to HDFS-4949, similar in scope to the snapshot test plan
 that was posted to HDFS-2802.  For my own part, I've run the new unit
 tests, and I've tested end-to-end in a pseudo-distributed deployment.  It's
 unlikely that I'll get a chance to test fully distributed before the vote
 closes, so I'm curious to hear if you've done this on your side yet.

 Also, I want to confirm that this vote only covers trunk.  I don't see
 branch-2 mentioned, so I assume that we're not voting on merge to branch-2
 yet.

 Before I cast my vote, can you please discuss whether or not it's feasible
 to complete all of the above in the next 7 days?  For the issues assigned
 to me, I do expect to complete them.

 Thanks again for all of your hard work!

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/



 On Thu, Oct 17, 2013 at 3:07 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 +1.  Thanks, guys.

 best,
 Colin

 On Thu, Oct 17, 2013 at 3:01 PM, Andrew Wang andrew.w...@cloudera.com
 wrote:
  Hello all,
 
  I'd like to call a vote to merge the HDFS-4949 branch (in-memory caching)
  to trunk. Colin McCabe and I have been hard at work the last 3.5 months
  implementing this feature, and feel that it's reached a level of
 stability
  and utility where it's ready for broader testing and integration.
 
  I'd also like to thank Chris Nauroth at Hortonworks for code reviews and
  bug fixes, and everyone who's reviewed the HDFS-4949 design doc and left
  comments.
 
  Obviously, I am +1 for the merge. The vote will run the standard 7 days,
  closing on October 24 at 11:59PM.
 
  Thanks,
  Andrew


 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Merge HDFS-4949 to trunk

2013-10-17 Thread Colin McCabe

+1.  Thanks, guys.

best,
Colin

On Thu, Oct 17, 2013 at 3:01 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Hello all,

 I'd like to call a vote to merge the HDFS-4949 branch (in-memory caching)
 to trunk. Colin McCabe and I have been hard at work the last 3.5 months
 implementing this feature, and feel that it's reached a level of stability
 and utility where it's ready for broader testing and integration.

 I'd also like to thank Chris Nauroth at Hortonworks for code reviews and
 bug fixes, and everyone who's reviewed the HDFS-4949 design doc and left
 comments.

 Obviously, I am +1 for the merge. The vote will run the standard 7 days,
 closing on October 24 at 11:59PM.

 Thanks,
 Andrew

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Colin McCabe

On Tue, Oct 1, 2013 at 8:59 PM, Arun C Murthy a...@hortonworks.com wrote:
 Yes, sorry if it wasn't clear.

 As others seem to agree, I think we'll be better getting a protocol/api 
 stable GA done and then iterating on bugs etc.

 I'm not super worried about HADOOP-9984 since symlinks just made it to 
 branch-2.1 recently.

 Currently we only have 2 blockers: HADOOP-9984  MAPREDUCE-5530. Both of 
 which are PA and I've reviewed MR-5530 and is good to go (thanks Robert). 
 Hopefully we can finish up HADOOP-9984 asap and we'll be good.

We've had several reviews for HADOOP-9984 and are currently just
waiting on a +1.

Sorry if this is a dumb question, but will the 2.2 release be made
from branch-2 or what is currently named branch-2.1-beta?  If it is
the latter, we have a few backports we'll need to do.

Colin



 thanks,
 Arun

 On Oct 1, 2013, at 4:53 PM, Alejandro Abdelnur t...@cloudera.com wrote:

 Arun,

 Does this mean that you want to skip a beta release and go straight to GA
 with the next release?

 thx


 On Tue, Oct 1, 2013 at 4:15 PM, Arun C Murthy a...@hortonworks.com wrote:

 Guys,

 I took a look at the content in 2.1.2-beta so far, other than the
 critical fixes such as HADOOP-9984 (symlinks) and few others in YARN/MR,
 there is fairly little content (unit tests fixes etc.)

 Furthermore, it's standing up well in testing too. Plus, the protocols
 look good for now (I wrote a gohadoop to try convince myself), let's lock
 them in.

 Given that, I'm thinking we can just go ahead rename it 2.2.0 rather than
 make another 2.1.x release.

 This will drop a short-lived release (2.1.2) and help us move forward on
 2.3 which has a fair bunch of content already...

 Thoughts?

 thanks,
 Arun


 On Sep 24, 2013, at 4:24 PM, Zhijie Shen zs...@hortonworks.com wrote:

 I've added MAPREDUCE-5531 to the blocker list. - Zhijie


 On Tue, Sep 24, 2013 at 3:41 PM, Arun C Murthy a...@hortonworks.com
 wrote:

 With 4 +1s (3 binding) and no -1s the vote passes. I'll push it out…
 I'll
 make it clear on the release page, that there are some known issues and
 that we will follow up very shortly with another release.

 Meanwhile, let's fix the remaining blockers (please mark them as such
 with
 Target Version 2.1.2-beta).
 The current blockers are here:
 http://s.apache.org/hadoop-2.1.2-beta-blockers

 thanks,
 Arun

 On Sep 16, 2013, at 11:38 PM, Arun C Murthy a...@hortonworks.com
 wrote:

 Folks,

 I've created a release candidate (rc0) for hadoop-2.1.1-beta that I
 would like to get released - this release fixes a number of bugs on top
 of
 hadoop-2.1.0-beta as a result of significant amounts of testing.

 If things go well, this might be the last of the *beta* releases of
 hadoop-2.x.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0
 The RC tag in svn is here:

 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7
 days.

 thanks,
 Arun


 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
 entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the
 reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.




 --
 Zhijie Shen
 Hortonworks Inc.
 http://hortonworks.com/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity
 to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified
 that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender
 immediately
 and delete it from your system. Thank You.

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received

Re: 2.1.2 (Was: Re: [VOTE] Release Apache Hadoop 2.1.1-beta)

2013-10-02 Thread Colin McCabe

I don't think HADOOP-9972 is a must-do for the next Apache release,
whatever version number it ends up having.  It's just adding a new
API, not changing any existing ones, and it can be done entirely in
generic code.  (The globber doesn't involve FileSystem or AFS
subclasses).

My understanding is that GA is about stabilizing APIs rather than
achieving feature completeness on symlinks.

Colin

On Wed, Oct 2, 2013 at 6:01 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 If we're serious about not breaking compatibility after GA, then we need to
 slow down and make sure we get these new APIs right, or can add them in a
 compatible fashion.

 HADOOP-9984 ended up being a bigger change than initially expected, and we
 need to break compatibility with out-of-tree FileSystems to do it properly.
 I would like to see HADOOP-9972 in as well (globLinkStatus), and there are
 open questions on HADOOP-9984 about changing PathFilter and
 FileStatus.getPath() semantics (which would be incompatible). Yes, we could
 just +1 HADOOP-9984 and stamp 2.2.0 on it, but I think it looks bad to then
 immediately turn around and release an incompatible 2.3.

 My preference is still for a 2.1.2 with the above API questions resolved,
 then an actual API-stable 2.2.0 GA. This is already punting out all the
 other related FS/tooling changes that we think can be done compatibly but
 are still pretty crucial: shell, distcp, webhdfs, hftp; it'd be great to
 get help on any of these.

 Thanks,
 Andrew


 On Wed, Oct 2, 2013 at 2:56 PM, Roman Shaposhnik r...@apache.org wrote:

 On Tue, Oct 1, 2013 at 5:15 PM, Vinod Kumar Vavilapalli
 vino...@hortonworks.com wrote:
  +1. We should get an RC as soon as possible so that we can get all the
 downstream components to sign off.
  The earlier the better.

 On this very note -- would there be any interest in joining efforts
 with the Bigtop integration aimed at Hadoop 2.2.x based release
 of all the Hadoop ecosystem projects?

 Our current plan is to release Bigtop 0.7.0 within a couple of weeks.
 That will be the last stable 2.0.x-based release. Bigtop 0.8.0 is supposed
 to
 be based on Hadoop 2.x that gets us (Bigtop community) as close as possible
 to the Hadoop's GA. Here's more on what we'll be doing with Bigtop 0.8.0:

 http://comments.gmane.org/gmane.comp.apache.incubator.bigtop.devel/10769

 Of course, on the Bigtop side of things we're stuck with all the necessary
 integration work anyway, but if there's anything at all that folks are
 willing
 to help us and the bigger Hadoop community with that would be very
 much appreciated. I think both communities will benefit from this type
 of collaboration.

 On a practical side of things, as soon as the branch for 2.2.0 gets cut
 Bigtop can start publishing a complete set of Hadoop ecosystem
 artifacts built against that particular version and easily install-able
 on all of our supported systems. We can also start publishing VMs
 so that folks on OSes other than Linux can help us with testing.

 Thanks,
 Roman.

Re: symlink support in Hadoop 2 GA

2013-09-19 Thread Colin McCabe

What we're trying to get to here is a consensus on whether
FileSystem#listStatus and FileSystem#globStatus should return symlinks
__as_symlinks__. If 2.1-beta goes out with these semantics, I think
we are not going to be able to change them later. That is what will
happen in the do nothing scenario.

Also see Jason Lowe's comment here:
https://issues.apache.org/jira/browse/HADOOP-9912?focusedCommentId=13772002page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13772002

Colin

On Wed, Sep 18, 2013 at 5:11 PM, J. Rottinghuis jrottingh...@gmail.com wrote:
However painful protobuf version changes are at build time for Hadoop
developers, at runtime with multiple clusters and many Hadoop users this is
a total nightmare.
Even upgrading clusters from one protobuf version to the next is going to
be very difficult. The same users will run jobs on, and/or readwrite to
multiple clusters. That means that they will have to fork their code, run
multiple instances? Or in the very least they have to do an update to their
applications. All in sync with Hadoop cluster changes. And these are not
doable in a rolling fashion.
All Hadoop and HBase clusters will all upgrade at the same time, or we'll
have to have our users fork / roll multiple versions ?
My point is that these things are much harder than just fix the (Jenkins)
build and we're done. These changes are massively disruptive.

There is a similar situation with symlinks. Having an API that lets users
create symlinks is very problematic. Some users create symlinks and as Eli
pointed out, somebody else (or automated process) tries to copy to / from
another (Hadoop 1.x?) cluster over hftp. What will happen ?
Having an API that people should not use is also a nightmare. We
experienced this with append. For a while it was there, but users were not
allowed to use it (or else there were large #'s of corrupt blocks). If
there is an API to create a symlink, then some of our users are going to
use it and others are going to trip over those symlinks. We already know
that Pig does not work with symlinks yet, and as Steve pointed out, there
is tons of other code out there that assumes that !isDir() means isFile().

I like symlink functionality, but in our migration to Hadoop 2.x this is a
total distraction. If the APIs stay in 2.2 GA we'll have to choose to:
a) Not uprev until symlink support is figured out up and down the stack,
and we've been able to migrate all our 1.x (equivalent) clusters to 2.x
(equivalent). Or
b) rip out the API altogether. Or
c) change the implementation to throw an UnsupportedOperationException
I'm not sure yet which of these I like least.

Thanks,

Joep

On Wed, Sep 18, 2013 at 9:48 AM, Arun C Murthy a...@hortonworks.com wrote:

On Sep 16, 2013, at 6:49 PM, Andrew Wang andrew.w...@cloudera.com wrote:

Hi all,

I wanted to broadcast plans for putting the FileSystem symlinks work
(HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I
think
it's pretty important we get it in since it's not a compatible change; if
it misses the GA train, we're not going to have symlinks until the next
major release.

Just catching up, is this an incompatible change, or not? The above reads
'not an incompatible change'.

Arun

However, we're still dealing with ongoing issues revealed via testing.
There's user-code out there that only handles files and directories and
will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
for a nice example where globStatus returning symlinks broke Pig; some of
us had a conference call to talk it through, and one definite conclusion
was that this wasn't solvable in a generally compatible manner.

There are also still some gaps in symlink support right now. For example,
the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
resolution, and tooling like the FsShell and Distcp still need to be
updated as well.

So, there's definitely work to be done, but there are a lot of users
interested in the feature, and symlinks really should be in GA. Would
appreciate any thoughts/input on the matter.

Thanks,
Andrew

--
Arun C. Murthy
Hortonworks Inc.
http://hortonworks.com/

--
CONFIDENTIALITY NOTICE
NOTICE: This message is intended for the use of the individual or entity to
which it is addressed and may contain information that is confidential,
privileged and exempt from disclosure under applicable law. If the reader
of this message is not the intended recipient, you are hereby notified that
any printing, copying, dissemination, distribution, disclosure or
forwarding of this communication is strictly prohibited. If you have
received this communication in error, please contact the sender immediately
and delete it from your system. Thank You.

Re: symlink support in Hadoop 2 GA

2013-09-17 Thread Colin McCabe

I think it makes sense to finish symlinks support in the Hadoop 2 GA release.

Colin

On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang andrew.w...@cloudera.com wrote:
 Hi all,

 I wanted to broadcast plans for putting the FileSystem symlinks work
 (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
 it's pretty important we get it in since it's not a compatible change; if
 it misses the GA train, we're not going to have symlinks until the next
 major release.

 However, we're still dealing with ongoing issues revealed via testing.
 There's user-code out there that only handles files and directories and
 will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
 for a nice example where globStatus returning symlinks broke Pig; some of
 us had a conference call to talk it through, and one definite conclusion
 was that this wasn't solvable in a generally compatible manner.

 There are also still some gaps in symlink support right now. For example,
 the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
 resolution, and tooling like the FsShell and Distcp still need to be
 updated as well.

 So, there's definitely work to be done, but there are a lot of users
 interested in the feature, and symlinks really should be in GA. Would
 appreciate any thoughts/input on the matter.

 Thanks,
 Andrew

Re: symlink support in Hadoop 2 GA

2013-09-17 Thread Colin McCabe

The issue is not modifying existing APIs.  The issue is that code has
been written that makes assumptions that are incompatible with the
existence of things that are not files or directories.  For example,
there is a lot of code out there that looks at FileStatus#isFile, and
if it returns false, assumes that what it is looking at is a
directory.  In the case of a symlink, this assumption is incorrect.

Faced with this, we have considered making the default behavior of
listStatus and globStatus to be fully resolving symlinks, and simply
not listing dangling symlinks. Code which is prepared to deal symlinks
can use newer versions of the listStatus and globStatus functions
which do return symlinks as symlinks.

We might consider defaulting FileSystem#listStatus and
FileSystem#globStatus to fully resolving symlinks by default and
defaulting FileContext#listStatus and FileContext#Util#globStatus to
the opposite.  This seems like the maximally compatible solution that
we're going to get.  I think this makes sense.

The alternative is kicking the can down the road to Hadoop 3, and
letting vendors of alternative (including some proprietary
alternative) systems continue to claim that Hadoop doesn't support
symlinks yet (with some justice).

P.S.  I would be fine with putting this in 2.2 or 2.3 if that seems
more appropriate.

sincerely,
Colin

On Tue, Sep 17, 2013 at 8:23 AM, Suresh Srinivas sur...@hortonworks.com wrote:
 I agree that this is an important change. However, 2.2.0 GA is getting
 ready to rollout in weeks. I am concerned that these changes will add not
 only incompatible changes late in the game, but also possibly instability.
 Java API incompatibility is some thing we have avoided for the most part
 and I am concerned that this is adding such incompatibility in FileSystem
 APIs. We should find work arounds by adding possibly newer APIs and leaving
 existing APIs as is. If this can be done, my vote is to enable this feature
 in 2.3. Even if it cannot be done, I am concerned that this is coming quite
 late and we should see if could allow some incompatible changes into 2.3
 for this feature.


 On Mon, Sep 16, 2013 at 6:49 PM, Andrew Wang andrew.w...@cloudera.comwrote:

 Hi all,

 I wanted to broadcast plans for putting the FileSystem symlinks work
 (HADOOP-8040) into branch-2.1 for the pending Hadoop 2 GA release. I think
 it's pretty important we get it in since it's not a compatible change; if
 it misses the GA train, we're not going to have symlinks until the next
 major release.

 However, we're still dealing with ongoing issues revealed via testing.
 There's user-code out there that only handles files and directories and
 will barf when given a symlink (perhaps a dangling one!). See HADOOP-9912
 for a nice example where globStatus returning symlinks broke Pig; some of
 us had a conference call to talk it through, and one definite conclusion
 was that this wasn't solvable in a generally compatible manner.

 There are also still some gaps in symlink support right now. For example,
 the more esoteric FileSystems like WebHDFS, HttpFS, and HFTP need symlink
 resolution, and tooling like the FsShell and Distcp still need to be
 updated as well.

 So, there's definitely work to be done, but there are a lot of users
 interested in the feature, and symlinks really should be in GA. Would
 appreciate any thoughts/input on the matter.

 Thanks,
 Andrew




 --
 http://hortonworks.com/download/

 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-22 Thread Colin McCabe

On Wed, Aug 21, 2013 at 3:49 PM, Stack st...@duboce.net wrote:
 On Wed, Aug 21, 2013 at 1:25 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 St.Ack wrote:

  + Once I figured where the logs were, found that JAVA_HOME was not being
  exported (don't need this in hadoop-2.0.5 for instance).  Adding an
  exported JAVA_HOME to my running shell which don't seem right but it took
  care of it (I gave up pretty quick on messing w/
  yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
  getting anywhere)

 I thought that we were always supposed to have JAVA_HOME set when
 running any of these commands.  At least, I do.  How else can the
 system disambiguate between different Java installs?  I need 2
 installs to test with JDK7.



 That is fair enough but I did not need to define this explicitly previously
 (for hadoop-2.0.5-alpha for instance) or the JAVA_HOME that was figured in
 start scripts was propagated and now is not (I have not dug in).



  + This did not seem to work for me:
  namehadoop.security.group.mapping/name
 
 valueorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback/va
  lue.

 We've seen this before.  I think your problem is that you have
 java.library.path set correctly (what System.loadLibrary checks), but
 your system library path does not include a necessary dependency of
 libhadoop.so-- most likely, libjvm.so.  Probably, we should fix
 NativeCodeLoader to actually make a function call in libhadoop.so
 before it declares everything OK.


 My expectation was that if native group lookup fails, as it does here, then
 the 'Fallback' would kick in and we'd do the Shell query.  This mechanism
 does not seem to be working.

I filed https://issues.apache.org/jira/browse/HADOOP-9895 to address this issue.

best,
Colin




 St.Ack

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

2013-08-21 Thread Colin McCabe

St.Ack wrote:

+ Once I figured where the logs were, found that JAVA_HOME was not being
exported (don't need this in hadoop-2.0.5 for instance). Adding an
exported JAVA_HOME to my running shell which don't seem right but it took
care of it (I gave up pretty quick on messing w/
yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
getting anywhere)

I thought that we were always supposed to have JAVA_HOME set when
running any of these commands. At least, I do. How else can the
system disambiguate between different Java installs? I need 2
installs to test with JDK7.

+ This did not seem to work for me:
namehadoop.security.group.mapping/name
valueorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback/va
lue.

We've seen this before. I think your problem is that you have
java.library.path set correctly (what System.loadLibrary checks), but
your system library path does not include a necessary dependency of
libhadoop.so-- most likely, libjvm.so. Probably, we should fix
NativeCodeLoader to actually make a function call in libhadoop.so
before it declares everything OK.

Colin

On Tue, Aug 20, 2013 at 5:35 PM, Stack st...@duboce.net wrote:
On Thu, Aug 15, 2013 at 2:15 PM, Arun C Murthy a...@hortonworks.com wrote:

Folks,

I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would
like to get released - this fixes the bugs we saw since the last go-around
(rc1).

The RC is available at:
http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
The RC tag in svn is here:
http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

The maven artifacts are available via repository.apache.org.

Please try the release and vote; the vote will run for the usual 7 days.

It basically works (in insecure mode), +1.

+ Checked signature.
+ Ran on small cluster w/ small load made using mapreduce interfaces.
+ Got the HBase full unit test suite to pass on top of it.

I had the following issues getting it to all work. I don't know if they are
known issues so will just list them here first.

+ I could not find documentation on how to go from tarball to running
cluster (the bundled 'cluster' and 'standalone' doc are not about how to
get this tarball off the ground).
+ I had a bit of a struggle putting this release in place under hbase unit
tests. The container would just exit w/ 127 errcode. No logs in expected
place. Tripped over where minimrcluster was actually writing. Tried to
corral it so it played nicely w/o our general test setup but found that the
new mini clusters have 'target' hardcoded as output dirs.
+ Once I figured where the logs were, found that JAVA_HOME was not being
exported (don't need this in hadoop-2.0.5 for instance). Adding an
exported JAVA_HOME to my running shell which don't seem right but it took
care of it (I gave up pretty quick on messing w/
yarn.nodemanager.env-whitelist and yarn.nodemanager.admin-env -- I wasn't
getting anywhere)
+ This did not seem to work for me:
namehadoop.security.group.mapping/name
valueorg.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback/value.
It just did this:

Caused by: java.lang.UnsatisfiedLinkError:
org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative()V
at org.apache.hadoop.security.JniBasedUnixGroupsMapping.anchorNative(Native
Method)
at
org.apache.hadoop.security.JniBasedUnixGroupsMapping.clinit(JniBasedUnixGroupsMapping.java:49)
at
org.apache.hadoop.security.JniBasedUnixGroupsMappingWithFallback.init(JniBasedUnixGroupsMappingWithFallback.java:38)

..so I replaced it
w/ org.apache.hadoop.security.ShellBasedUnixGroupsMapping on the hbase-side
to get my cluster up and running.

+ Untarring the bin dir, it undoes as hadoop-X.Y.Z-beta. Undoing the src
dir it undoes as hadoop-X.Y.Z-beta-src. I'd have thought they would undo
into the one directory overlaying each other.

St.Ack

Re: git line endings trouble since recent trunk

2013-07-01 Thread Colin McCabe

On Sat, Jun 29, 2013 at 5:18 PM, Luke Lu l...@vicaya.com wrote:
 The problem is due to relnotes.py generating the html containing some CRLF
 (from JIRA) and the release manager not using git-svn, which caused the
 html with mixed eol getting checked in. The problem would then manifest for
 git users due to text=auto in .gitattributes (see HADOOP-8912) that auto
 converts CRLF to LF, hence the persistent modified status of the
 releasenotes.html for a fresh checkout.

 Adding svn:eol-style=native would only fix half the problem. We need to fix
 relnotes.py to avoid generating html with mixed eols (by normalizing
 everything to LF or native).

While I agree that it would be nice to fix relnotes.py, it seems to me
that setting svn:eol-style=native should fix the problem completely.
Files with this attribute set are stored internally by subversion with
all newlines as LF, and converted to CRLF as needed.  After all,
eol-style=native would not be very useful if it only applied on
checkout.  Windows users would be constantly checking in CRLF in that
case.

I'm not an svn expert, though, and I haven't tested the above.

Colin




 On Fri, Jun 28, 2013 at 1:03 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 Clarification: svn:eol-style = native causes the files to contain
 whatever the native platform used to check out the code uses.  I think
 just setting this property on all the HTML files should resolve this
 and future problems.

 patch posted.
 C.

 On Fri, Jun 28, 2013 at 12:56 PM, Colin McCabe cmcc...@alumni.cmu.edu
 wrote:
  I think the fix for this is to set svn:eol-style to native on this
  file.  It's set on many other files, just not on this one:
 
  cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style
  ./hadoop-project-dist/README.txt
  native
  cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style
  ./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
  cmccabe@keter:~/hadoopST/trunk
 
  It might also be a good idea to run dos2unix on it.
 
  I thought that in general we wanted to have 'LF' everywhere, so
  perhaps we should add this to the patch.sh script to prevent this from
  re-occurring.
 
  Colin
 
 
  On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com
 wrote:
  I haven't been able to find a solution.  Just filed
  https://issues.apache.org/jira/browse/HADOOP-9675.
 
 
  On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.com
 wrote:
 
  Sandy... did you fix this? any jira to track? me too facing same
 problem..
 
  Thanks,
  Omkar Joshi
  *Hortonworks Inc.* http://www.hortonworks.com
 
 
  On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com
  wrote:
 
   Have the same problem here, have to edit the patch manually to
 exclude
  the
   changes in releasenotes.html
  
  
   On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza 
 sandy.r...@cloudera.com
   wrote:
  
Has anybody else been having trouble with line endings since
 pulling
   trunk
recently?
   
hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
shows up as modified even though I haven't touched it, and I can't
  check
   it
out or reset to a previous version to make that go away.  The only
  thing
   I
can do to neutralize it is to put it in a dummy commit, but I have
 to
  do
this every time I switch branches or rebase.
   
This appears to have began after the release notes commit
 (8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a
 line
endings change.
   
-Sandy
   
  
  
  
   --
   Zhijie Shen
   Hortonworks Inc.
   http://hortonworks.com/

Re: git line endings trouble since recent trunk

2013-06-28 Thread Colin McCabe

I think the fix for this is to set svn:eol-style to native on this
file.  It's set on many other files, just not on this one:

cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style
./hadoop-project-dist/README.txt
native
cmccabe@keter:~/hadoopST/trunk svn propget svn:eol-style
./hadoop-hdfs-project/hadoop-hdfs/src/main/docs/releasenotes.html
cmccabe@keter:~/hadoopST/trunk

It might also be a good idea to run dos2unix on it.

I thought that in general we wanted to have 'LF' everywhere, so
perhaps we should add this to the patch.sh script to prevent this from
re-occurring.

Colin


On Fri, Jun 28, 2013 at 12:27 PM, Sandy Ryza sandy.r...@cloudera.com wrote:
 I haven't been able to find a solution.  Just filed
 https://issues.apache.org/jira/browse/HADOOP-9675.


 On Fri, Jun 28, 2013 at 12:19 PM, Omkar Joshi ojo...@hortonworks.comwrote:

 Sandy... did you fix this? any jira to track? me too facing same problem..

 Thanks,
 Omkar Joshi
 *Hortonworks Inc.* http://www.hortonworks.com


 On Fri, Jun 28, 2013 at 11:54 AM, Zhijie Shen zs...@hortonworks.com
 wrote:

  Have the same problem here, have to edit the patch manually to exclude
 the
  changes in releasenotes.html
 
 
  On Fri, Jun 28, 2013 at 11:01 AM, Sandy Ryza sandy.r...@cloudera.com
  wrote:
 
   Has anybody else been having trouble with line endings since pulling
  trunk
   recently?
  
   hadoop-common-project/hadoop-common/src/main/docs/releasenotes.html
   shows up as modified even though I haven't touched it, and I can't
 check
  it
   out or reset to a previous version to make that go away.  The only
 thing
  I
   can do to neutralize it is to put it in a dummy commit, but I have to
 do
   this every time I switch branches or rebase.
  
   This appears to have began after the release notes commit
(8c5676830bb176157b2dc28c48cd3dd0a9712741), and must be due to a line
   endings change.
  
   -Sandy
  
 
 
 
  --
  Zhijie Shen
  Hortonworks Inc.
  http://hortonworks.com/

Re: bug in SequenceFile.sync()?

2013-06-24 Thread Colin McCabe

Hi Chris,

Thanks for the report.  I filed
https://issues.apache.org/jira/browse/HADOOP-9667 for this.

Colin
Software Engineer, Cloudera


On Mon, Jun 24, 2013 at 2:20 AM, Christopher Ng cng1...@gmail.com wrote:
 cross-posting this from cdh-users group where it received little interest:

 is there a bug in SequenceFile.sync()?  This is from cdh4.3.0:

 /** Seek to the next sync mark past a given position.*/
 public synchronized void sync(long position) throws IOException {
   if (position+SYNC_SIZE = end) {
 seek(end);
 return;
   }

   if (position  headerEnd) {
 // seek directly to first record
 in.seek(headerEnd); 
 should this not call seek (ie this.seek) instead?
 // note the sync marker seen in the header
 syncSeen = true;
 return;
   }

 the problem is that when you sync to the start of a compressed file, the
 noBufferedKeys and valuesDecompressed isn't reset so a block read isn't
 triggered.  When you subsequently call next() you're potentially getting
 keys from the buffer which still contains keys from the previous position
 of the file.

Re: [DISCUSS] - Committing client code to 3rd Party FileSystems within Hadoop Common

2013-05-23 Thread Colin McCabe

You might try looking at what KosmoFS (KFS) did.  They have some code in
org/apache/hadoop/fs which calls their own Java shim.

This way, the shim code in hadoop-common gets updated whenever FileSystem
changes, but there is no requirement to install KFS before building Hadoop.

You might also try asking Steve Loughran, since he did some great work
recently to try to nail down the exact semantics of FileSystem and
FileContext and improve the related unit tests (see HADOOP-9258 and related
JIRAs.)

best,
Colin



On Thu, May 23, 2013 at 2:52 PM, Stephen Watt sw...@redhat.com wrote:

 Thanks for responding Harsh.

 I agree. Hadoop Common does do a good job of maintaining a stable and
 public FS and FS Context API. The pro for maintaining client libraries
 outside of Hadoop Common is that the release owner of the library has much
 more autonomy and agility in maintaining the library. From the glusterfs
 plugin perspective, I concur with this. In contrast, if my library was
 managed inside of Hadoop Common, I'd have to spend the time to earn
 committer status to have an equivalent amount of autonomy and agility,
 which is overkill for someone just wanting to maintain 400 lines of code.

 I ruminated a bit about one con which might be that because it doesn't get
 shipped with Hadoop Common it might make it harder for the Hadoop User
 community to find out about it and obtain it. However, if you consider the
 LZO codec, the fact that its not bundled certainly doesn't hamper its
 adoption.

 You mentioned testing. I don't think regression across Hadoop releases is
 as big of an issue as (based on my understanding) you really just have two
 FileSystem interfaces (abstract class) to worry about WRT to compliance,
 namely the FileSystem interface reflected for Hadoop 1.0 and the FileSystem
 interface reflected for Hadoop 2.0. However, this is a broader topic that I
 also want to discuss so I'll tee it up in a separate thread.

 Regards
 Steve Watt


 - Original Message -
 From: Harsh J ha...@cloudera.com
 To: common-dev@hadoop.apache.org
 Sent: Thursday, May 23, 2013 1:37:30 PM
 Subject: Re: [DISCUSS] - Committing client code to 3rd Party FileSystems
 within Hadoop Common

 I think we do a fairly good work maintaining a stable and public FileSystem
 and FileContext API for third-party plugins to exist outside of Apache
 Hadoop but still be able to work well across versions.

 The question of test pops up though, specifically that of testing against
 trunk to catch regressions across various implementations, but it'd be much
 work for us to also maintain glusterfs dependencies and mechanisms as part
 of trunk.

 We do provide trunk build snapshot artifacts publicly for downstream
 projects to test against, which I think may help cover the continuous
 testing concerns, if there are those.

 Right now, I don't think the S3 FS we maintain really works all that well.
 I also recall, per recent conversations on the lists, that AMZN has started
 shipping their own library for a better implementation rather than
 perfecting the implementation we have here (correct me if am wrong but I
 think the changes were not all contributed back). I see some work going on
 for OpenStack's Swift, for which I think Steve also raised a similar
 discussion here: http://search-hadoop.com/m/W1S5h2SrxlG, but I don't
 recall
 if the conversation proceeded at the time.

 What's your perspective as the releaser though? Would you not find
 maintaining this outside easier, especially in terms of maintaining your
 code for quicker releases, for both bug fixes and features - also given
 that you can CI it against Apache Hadoop trunk at the same time?


 On Thu, May 23, 2013 at 11:47 PM, Stephen Watt sw...@redhat.com wrote:

  (Resending - I think the first time I sent this out it got lost within
 all
  the ByLaws voting)
 
  Hi Folks
 
  My name is Steve Watt and I am presently working on enabling glusterfs to
  be used as a Hadoop FileSystem. Most of the work thus far has involved
  developing a Hadoop FileSystem plugin for glusterfs. I'm getting to the
  point where the plugin is becoming stable and I've been trying to
  understand where the right place is to host/manage/version it.
 
  Steve Loughran was kind enough to point out a few past threads in the
  community (such as
 
 http://lucene.472066.n3.nabble.com/Need-to-add-fs-shim-to-use-QFS-td4012118.html
 )
  that show a project disposition to move away from Hadoop Common
 containing
  client code (plugins) for 3rd party FileSystems. This makes sense and
  allows the filesystem plugin developer more autonomy as well as reduces
  Hadoop Common's dependence on 3rd Party libraries.
 
  Before I embark down that path, can the PMC/Committers verify that the
  preference is still to have client code for 3rd Party FileSystems hosted
  and managed outside of Hadoop Common?
 
  Regards
  Steve Watt
 



 --
 Harsh J

Re: [VOTE] Plan to create release candidate Monday 3/18

2013-03-11 Thread Colin McCabe

+1 (non-binding)

best,
Colin


On Sun, Mar 10, 2013 at 8:38 PM, Matt Foley ma...@apache.org wrote:

 Hi all,
 I have created branch-1.2 from branch-1, and propose to cut the first
 release candidate for 1.2.0 on Monday 3/18 (a week from tomorrow), or as
 soon thereafter as I can achieve a stable build.

 Between 1.1.2 and the current 1.2.0, there are 176 patches!!  Draft release
 notes are available at .../branch-1.2/src/docs/releasenotes.html in the
 sources.

 Any non-destabilizing patches committed to branch-1.2 during the coming
 week (and of course also committed to branch-1) will be included in the RC.
  However, at this point I request that any big new developments not yet in
 branch-1.2 be targeted for 1.3.

 Release plans have to be voted on too, so please vote '+1' to approve this
 plan.  Voting will close on Sunday 3/17 at 8:30pm PDT.

 Thanks,
 --Matt
 (release manager)

Re: development environment for hadoop core

2013-01-21 Thread Colin McCabe

Hi Erik,

Eclipse can run junit tests very rapidly.  If you want a shorter test
cycle, that's one way to get it.

There is also Maven-shell, which reduces some of the overhead of starting
Maven.  But I haven't used it so I can't really comment.

cheers,
Colin


On Mon, Jan 21, 2013 at 8:36 AM, Erik Paulson epaul...@unit1127.com wrote:

 On Wed, Jan 16, 2013 at 7:31 AM, Glen Mazza gma...@talend.com wrote:

  On 01/15/2013 06:50 PM, Erik Paulson wrote:
 
  Hello -
 
  I'm curious what Hadoop developers use for their day-to-day hacking on
  Hadoop. I'm talking changes to the Hadoop libraries and daemons, and not
  developing Map-Reduce jobs or using using the HDFS Client libraries to
  talk
  to a filesystem from an application.
 
  I've checked out Hadoop, made minor changes and built it with Maven, and
  tracked down the resulting artifacts in a target/ directory that I could
  deploy. Is this typically how a cloudera/hortonworks/mapr/etc dev works,
  or
  are the IDEs more common?
 
  I haven't built Hadoop yet myself.  Your use of a in a target/
  directory indicates you're also kind of new with Maven itself, as that's
  the standard output folder for any Maven project.  One of many nice
 things
  about Maven is once you learn how to build one project with it you pretty
  much know how to build any project with it, as everything's standardized
  with it.
 
  Probably best to stick with the command line for building and use Eclipse
  for editing, to keep things simple, but don't forget the mvn
  eclipse:eclipse command to set up Eclipse projects that you can
  subsequently import into your Eclipse IDE:
 http://www.jroller.com/gmazza/*
  *entry/web_service_tutorial#**EclipseSetup
 http://www.jroller.com/gmazza/entry/web_service_tutorial#EclipseSetup
 
 
 
  I realize this sort of sounds like a dumb question, but I'm mostly
 curious
  what I might be missing out on if I stay away from anything other than
  vim,
  and not being entirely sure where maven might be caching jars that it
 uses
  to build,
 
 
  That will be your local Maven repository, in an .m2 hidden folder in your
  user home directory.
 
 
 
   and how careful I have to be to ensure that my changes wind up in
  the right places without having to do a clean build every time.
 
 
  Maven can detect changes (using mvn install instead of mvn clean
 install),
  but I prefer doing clean builds.  You can use the -Dmaven.test.skip
 setting
  to speed up your mvn clean installs if you don't wish to run the tests
  each time.
 

 Thanks to everyone for their advice last week, it's been helpful.

 You're spot-on that I'm new to Maven, but I'm a little confused as to what
 the different targets/goals are best to use. Here's my scenario.

 What I'd like to get working is the DataNodeCluster, which lives in the
 tests.

 Running it from hadoop-hdfs-project/hadoop-hdfs/target as
 'hadoop jar ./hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
 org.apache.hadoop.hdfs.DataNodeCluster
 -n 2'

 blows up with a NPE inside of MiniDFSCluster - the offending line is
 'dfsdir = conf.get(HDFS_MINIDFS_BASEDIR, null);' (line 2078 of
 MiniDFSCluster.java)

 I'm not worried about being able to figure out what's wrong (I'm pretty
 sure it's that conf is still null when this gets called) - what I'm trying
 to use this as is a way to understand what gets built when.

 Just to check, I added a System.out.println one line before 2078 of
 MiniDFSCluster, and recompiled from hadoop-common/hadoop-hdfs-project with

 mvn package -DskipTests

 Because I don't want to run all the tests.

 This certainly compiles the codes - if I leave the semicolon off of my
 change the compile fails, even with -DskipTests. However, it doesn't appear
 to rebuild

 target/hadoop-hdfs-3.0.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
 - the timestamp is still the old version.

 It _does_ copy

 target/hadoop-hdfs-3.0.0-SNAPSHOT/share/hadoop/hdfs/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
 to target/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar, or at least otherwise
 update the timestamp on target/hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar (unless
 it's copying or building it from somewhere else - but if it is, it's
 picking up old versions of my code)

 I only get an updated version if I ask for
 mvn package -Pdist -DskipTests

 Which is a 3 minute rebuild cycle, even for something as simple as changing
 the text in my System.out.println. (Even just a mvn package -DskipTests
 with no changes to any source code is a 40 second operation)

 I haven't sat around and waited for 'mvn package' to run and fire off the
 test suite. I don't know if that would result in an updated
 hadoop-hdfs-3.0.0-SNAPSHOT-tests.jar
 being built.

 So, my question is:

 - Is there a better maven target to use if I just want to update code in
 MiniDFSCluster.java and run DataNodeCluster, all of which wind up in
 -tests.jar? ('better' here means a shorter build cycle. I'm a terrible
 programmer so finding errors quickly is a priority for me

Re: making a hadoop-common test run if a property is set

2012-12-18 Thread Colin McCabe

On Mon, Dec 17, 2012 at 11:03 AM, Steve Loughran ste...@hortonworks.com wrote:
 On 17 December 2012 16:06, Tom White t...@cloudera.com wrote:

 There are some tests like the S3 tests that end with Test (e.g.
 Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
 start with Test. Only those that start with Test are run
 automatically (see the surefire configuration in
 hadoop-project/pom.xml). You have to run the others manually with mvn
 test -Dtest=

 The mechanism that Colin describes is probably better though, since
 the environment-specific tests can be run as a part of a full test run
 by Jenkins if configured appropriately.


 I'd like that -though one problem with the current system is that you need
 to get the s3 (and soon: openstack) credentials into
 src/test/resources/core-site.xml, which isn't the right approach. If we
 could get them into properties files things would be easier.
 That's overkill for adding a few more openstack tests -but I would like to
 make it easier to turn those and the rackspace ones without sticking my
 secrets into an XML file under SCM

I think the way to go is to have one XML file include another.

?xml version=1.0?
?xml-stylesheet type=text/xsl href=configuration.xsl?
configuration xmlns:xi=http://www.w3.org/2001/XInclude;
  property
 nameboring.config.1/name
 valueboring-value/value
 ... etc, etc...
  xi:include href=../secret-stuff.xml /
/configuration

That way, you can keep the boring configuration under version control,
and still have your password sitting in a small separate
non-version-controlled XML file.

We use this trick a bunch with the HA configuration stuff-- 99% of the
configuration is the same between the Active and Standby Namenodes,
but you can't give them the same dfs.ha.namenode.id or dfs.name.dir.
Includes help a lot here.

 another tactic could be to have specific test projects: test-s3,
 test-openstack, test-... which contain nothing but test cases. You'd set
 jenkins up those test projects too -the reason for having the separate
 names is to make it blatantly clear which tests you've not run

I dunno.  Every time a project puts unit or system tests into a
separate project, the developers never run them.  I've seen it happen
enough times that I think I can call it an anti-pattern by now.  I
like having tests alongside the code-- to the maximum extent that is
possible.

cheers,
Colin




 Tom

 On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran ste...@hortonworks.com
 wrote:
  thanks, I'l; have a look. I've always wanted to add the notion of skipped
  to test runs -all the way through to the XML and generated reports, but
  you'd have to do a new junit runner for this and tweak the reporting
 code.
  Which, if it involved going near maven source, is not something I am
  prepared to do
 
  On 14 December 2012 18:57, Colin McCabe cmcc...@alumni.cmu.edu wrote:
 
  One approach we've taken in the past is making the junit test skip
  itself when some precondition is not true.  Then, we often create a
  property which people can use to cause the skipped tests to become a
  hard error.
 
  For example, all the tests that rely on libhadoop start with these
 lines:
 
   @Test
   public void myTest() {
  Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
 ...
   }
 
  This causes them to be silently skipped when libhadoop.so is not
  available or loaded (perhaps because it hasn't been built.)
 
  However, if you want to cause this to be a hard error, you simply run
   mvn test -Drequire.test.libhadoop
 
  See TestHdfsNativeCodeLoader.java to see how this is implemented.
 
  The main idea is that your Jenkins build slaves use all the -Drequire
  lines, but people running tests locally are not inconvenienced by the
  need to build libhadoop.so in every case.  This is especially good
  because libhadoop.so isn't known to build on certain platforms like
  AIX, etc.  It seems to be a good tradeoff so far.  I imagine that s3
  could do something similar.
 
  cheers,
  Colin
 
 
  On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com
 
  wrote:
   The swiftfs tests need only to run if there's a target filesystem;
  copying
   the s3/s3n tests, something like
  
 property
   nametest.fs.swift.name/name
   valueswift://your-object-store-herel//value
 /property
  
   How does one actually go about making junit tests optional in
 mvn-land?
   Should the probe/skip logic be in the code -which can make people
 think
  the
   test passed when it didn't actually run? Or can I turn it on/off in
  maven?
  
   -steve

Re: making a hadoop-common test run if a property is set

2012-12-18 Thread Colin McCabe

On Tue, Dec 18, 2012 at 1:05 AM, Colin McCabe cmcc...@alumni.cmu.edu wrote:
 On Mon, Dec 17, 2012 at 11:03 AM, Steve Loughran ste...@hortonworks.com 
 wrote:
 On 17 December 2012 16:06, Tom White t...@cloudera.com wrote:

 There are some tests like the S3 tests that end with Test (e.g.
 Jets3tNativeS3FileSystemContractTest) - unlike normal tests which
 start with Test. Only those that start with Test are run
 automatically (see the surefire configuration in
 hadoop-project/pom.xml). You have to run the others manually with mvn
 test -Dtest=

 The mechanism that Colin describes is probably better though, since
 the environment-specific tests can be run as a part of a full test run
 by Jenkins if configured appropriately.


 I'd like that -though one problem with the current system is that you need
 to get the s3 (and soon: openstack) credentials into
 src/test/resources/core-site.xml, which isn't the right approach. If we
 could get them into properties files things would be easier.
 That's overkill for adding a few more openstack tests -but I would like to
 make it easier to turn those and the rackspace ones without sticking my
 secrets into an XML file under SCM

 I think the way to go is to have one XML file include another.

 ?xml version=1.0?
 ?xml-stylesheet type=text/xsl href=configuration.xsl?
 configuration xmlns:xi=http://www.w3.org/2001/XInclude;
   property
  nameboring.config.1/name
  valueboring-value/value
  ... etc, etc...
   xi:include href=../secret-stuff.xml /
 /configuration

 That way, you can keep the boring configuration under version control,
 and still have your password sitting in a small separate
 non-version-controlled XML file.

 We use this trick a bunch with the HA configuration stuff-- 99% of the
 configuration is the same between the Active and Standby Namenodes,
 but you can't give them the same dfs.ha.namenode.id or dfs.name.dir.
 Includes help a lot here.

 another tactic could be to have specific test projects: test-s3,
 test-openstack, test-... which contain nothing but test cases. You'd set
 jenkins up those test projects too -the reason for having the separate
 names is to make it blatantly clear which tests you've not run

 I dunno.  Every time a project puts unit or system tests into a
 separate project, the developers never run them.  I've seen it happen
 enough times that I think I can call it an anti-pattern by now.  I
 like having tests alongside the code-- to the maximum extent that is
 possible.

Just to be clear, I'm not referring to any Hadoop-related project
here, just certain other open source (and not) ones I've worked on.
System/unit tests belong with the rest of the code, otherwise they get
stale real fast.

It sometimes makes sense for integration tests to live in a separate
repo, since by their nature they're usually talking to stuff that
lives in multiple repos.

best,
Colin


 cheers,
 Colin




 Tom

 On Mon, Dec 17, 2012 at 10:06 AM, Steve Loughran ste...@hortonworks.com
 wrote:
  thanks, I'l; have a look. I've always wanted to add the notion of skipped
  to test runs -all the way through to the XML and generated reports, but
  you'd have to do a new junit runner for this and tweak the reporting
 code.
  Which, if it involved going near maven source, is not something I am
  prepared to do
 
  On 14 December 2012 18:57, Colin McCabe cmcc...@alumni.cmu.edu wrote:
 
  One approach we've taken in the past is making the junit test skip
  itself when some precondition is not true.  Then, we often create a
  property which people can use to cause the skipped tests to become a
  hard error.
 
  For example, all the tests that rely on libhadoop start with these
 lines:
 
   @Test
   public void myTest() {
  Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
 ...
   }
 
  This causes them to be silently skipped when libhadoop.so is not
  available or loaded (perhaps because it hasn't been built.)
 
  However, if you want to cause this to be a hard error, you simply run
   mvn test -Drequire.test.libhadoop
 
  See TestHdfsNativeCodeLoader.java to see how this is implemented.
 
  The main idea is that your Jenkins build slaves use all the -Drequire
  lines, but people running tests locally are not inconvenienced by the
  need to build libhadoop.so in every case.  This is especially good
  because libhadoop.so isn't known to build on certain platforms like
  AIX, etc.  It seems to be a good tradeoff so far.  I imagine that s3
  could do something similar.
 
  cheers,
  Colin
 
 
  On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com
 
  wrote:
   The swiftfs tests need only to run if there's a target filesystem;
  copying
   the s3/s3n tests, something like
  
 property
   nametest.fs.swift.name/name
   valueswift://your-object-store-herel//value
 /property
  
   How does one actually go about making junit tests optional in
 mvn-land?
   Should the probe/skip logic be in the code -which can make

Re: making a hadoop-common test run if a property is set

2012-12-14 Thread Colin McCabe

One approach we've taken in the past is making the junit test skip
itself when some precondition is not true.  Then, we often create a
property which people can use to cause the skipped tests to become a
hard error.

For example, all the tests that rely on libhadoop start with these lines:

 @Test
 public void myTest() {
Assume.assumeTrue(NativeCodeLoader.isNativeCodeLoaded());
   ...
 }

This causes them to be silently skipped when libhadoop.so is not
available or loaded (perhaps because it hasn't been built.)

However, if you want to cause this to be a hard error, you simply run
 mvn test -Drequire.test.libhadoop

See TestHdfsNativeCodeLoader.java to see how this is implemented.

The main idea is that your Jenkins build slaves use all the -Drequire
lines, but people running tests locally are not inconvenienced by the
need to build libhadoop.so in every case.  This is especially good
because libhadoop.so isn't known to build on certain platforms like
AIX, etc.  It seems to be a good tradeoff so far.  I imagine that s3
could do something similar.

cheers,
Colin


On Fri, Dec 14, 2012 at 9:56 AM, Steve Loughran ste...@hortonworks.com wrote:
 The swiftfs tests need only to run if there's a target filesystem; copying
 the s3/s3n tests, something like

   property
 nametest.fs.swift.name/name
 valueswift://your-object-store-herel//value
   /property

 How does one actually go about making junit tests optional in mvn-land?
 Should the probe/skip logic be in the code -which can make people think the
 test passed when it didn't actually run? Or can I turn it on/off in maven?

 -steve

Re: [DISCUSS] create a hadoop-build subproject (a follow up on the thread 'introduce Python as build-time...')

2012-12-10 Thread Colin McCabe

On Fri, Dec 7, 2012 at 5:31 PM, Radim Kolar h...@filez.com wrote:
 1. cmake and protoc maven plugins already exists. why you want to write a
 new ones?

This has already been discussed; see
https://groups.google.com/forum/?fromgroups=#!topic/cmake-maven-project-users/5FpfUHmg5Ho

Actually the situation is even worse than it might seem from that
thread, since it turns out that com.googlecode.cmake-maven-project has
no support for any platforms but Windows.  It also has no support for
running native unit tests, which is a big motivation behind
HADOOP-8887.

 2. Groovy accepts java syntax. Just rewrite saveVersion.sh to java (its done
 already in JIRA) and put it in pom.xml - no overhaul of build infrastructure
 needed.

Part of the reason for this thread is so that we can come up with a
solution for both branch-1 and later branches.  This would not be
accomplished by putting all the logic into a pom.xml file, since
branch-1 doesn't use Maven.

best,
Colin

Re: libhdfs on windows

2012-10-25 Thread Colin McCabe

Hi Peter,

This might be a good question for hdfs-dev?

As Harsh pointed out below, HDFS-573 was never committed.  I don't
even see a patch attached, although there is some discussion.

In the mean time, might I suggest using the webhdfs interface on
Windows?  webhdfs was intended as a stable REST interface that can be
accessed from any platform.

cheers,
Colin


On Thu, Oct 25, 2012 at 7:19 AM, Peter Marron
peter.mar...@trilliumsoftware.com wrote:
 Hi,

 I've been looking at using libhdfs and I would like to use it on windows.
 I have found HDFS-573 and the information on this page:
 http://issues.apache.org/jira/browse/HDFS-573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 which suggests that quite a lot of work was done on this way back in 2009.
 So is there some source from this effort retained somewhere? If so, where?
 Or do I have to start from scratch? Apologies if this has already been asked 
 recently.

 Any help appreciated.

 Peter Marron

Re: HADOOP-8887 port to branch-2

2012-10-18 Thread Colin McCabe

Hi Andrew,

It seems that your attachment did not appear on the mailing list.

I'm taking a look at the branch-2 build presently.

Colin


On Thu, Oct 18, 2012 at 6:20 PM, Andrew Purtell apurt...@apache.org wrote:
 The port of HADOOP-8887 to branch-2 fails the build. Please kindly see
 attached.

 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)

Re: HADOOP-8887 port to branch-2

2012-10-18 Thread Colin McCabe

Hmm.  It did build for me (in the sense of completing successfully, etc).

You've definitely found an issue, though.  The version should be
2.0.3-SNAPSHOT for those projects, not 3.0.0-SNAPSHOT.

Do you want to file a JIRA and post this patch, or should I?

cheers.
Colin


On Thu, Oct 18, 2012 at 6:46 PM, Andrew Purtell apurt...@apache.org wrote:
 Sorry, I always forget that about the mailing list software.

 From a6efe4883f3f7694ec6e8da04de982fb21eed8a9 Mon Sep 17 00:00:00 2001
 From: Andrew Purtell apurt...@apache.org
 Date: Thu, 18 Oct 2012 17:22:17 -0700
 Subject: [PATCH] Fix incorrect versions and missing module parent introduced
  by HADOOP-8887

 ---
  dev-support/cmake-maven-ng-plugin/pom.xml |7 ++-
  dev-support/pom.xml   |4 ++--
  hadoop-project/pom.xml|2 +-
  3 files changed, 9 insertions(+), 4 deletions(-)

 diff --git a/dev-support/cmake-maven-ng-plugin/pom.xml
 b/dev-support/cmake-maven-ng-plugin/pom.xml
 index e3d54bc..8652214 100644
 --- a/dev-support/cmake-maven-ng-plugin/pom.xml
 +++ b/dev-support/cmake-maven-ng-plugin/pom.xml
 @@ -17,10 +17,15 @@
xsi:schemaLocation=http://maven.apache.org/POM/4.0.0
http://maven.apache.org/maven-v4_0_0.xsd;
modelVersion4.0.0/modelVersion
 +  parent
 +groupIdorg.apache.hadoop/groupId
 +artifactIddev-support/artifactId
 +version2.0.3-SNAPSHOT/version
 +  /parent
groupIdorg.apache.hadoop.cmake.maven.ng/groupId
artifactIdcmake-ng/artifactId
packagingmaven-plugin/packaging
 -  version3.0.0-SNAPSHOT/version
 +  version2.0.3-SNAPSHOT/version
namecmake-ng Maven Mojo/name
urlhttp://maven.apache.org/url
properties
 diff --git a/dev-support/pom.xml b/dev-support/pom.xml
 index 721e231..11a96cc 100644
 --- a/dev-support/pom.xml
 +++ b/dev-support/pom.xml
 @@ -20,12 +20,12 @@
parent
  groupIdorg.apache.hadoop/groupId
  artifactIdhadoop-project/artifactId
 -version3.0.0-SNAPSHOT/version
 +version2.0.3-SNAPSHOT/version
  relativePath../hadoop-project/relativePath
/parent
groupIdorg.apache.hadoop/groupId
artifactIddev-support/artifactId
 -  version3.0.0-SNAPSHOT/version
 +  version2.0.3-SNAPSHOT/version
descriptionApache Hadoop Development Support/description
nameApache Hadoop Development Support/name
packagingpom/packaging
 diff --git a/hadoop-project/pom.xml b/hadoop-project/pom.xml
 index 568a59d..88d4676 100644
 --- a/hadoop-project/pom.xml
 +++ b/hadoop-project/pom.xml
 @@ -704,7 +704,7 @@
  plugin
groupIdorg.apache.hadoop.cmake.maven.ng/groupId
artifactIdcmake-ng/artifactId
 -  version3.0.0-SNAPSHOT/version
 +  version2.0.3-SNAPSHOT/version
  /plugin
  plugin
groupIdorg.apache.maven.plugins/groupId
 --
 1.7.9.5


 On Thu, Oct 18, 2012 at 6:35 PM, Colin McCabe cmcc...@alumni.cmu.eduwrote:

 Hi Andrew,

 It seems that your attachment did not appear on the mailing list.

 I'm taking a look at the branch-2 build presently.

 Colin


 On Thu, Oct 18, 2012 at 6:20 PM, Andrew Purtell apurt...@apache.org
 wrote:
  The port of HADOOP-8887 to branch-2 fails the build. Please kindly see
  attached.
 
  --
  Best regards,
 
 - Andy
 
  Problems worthy of attack prove their worth by hitting back. - Piet Hein
  (via Tom White)
 




 --
 Best regards,

- Andy

 Problems worthy of attack prove their worth by hitting back. - Piet Hein
 (via Tom White)

Re: test-patch and native code compile

2012-09-06 Thread Colin McCabe

We could also call uname from test-patch.sh and skip running native
tests on Mac OS X.

I also think that HADOOP-7147 should be open rather than won't fix,
as Alejandro commented.  Allen Wittenauer closed it as won't fix
because he personally did not intend to fix it, but that doesn't mean
it's not a bug.

cheers.
Colin


On Thu, Sep 6, 2012 at 8:29 AM, Eli Collins e...@cloudera.com wrote:
 Yea we want jenkins to run with native. How about adding making native
 optional in test-patch via a flag and updating the jenkins jobs to use
 it?

 On Thu, Sep 6, 2012 at 7:25 AM, Alejandro Abdelnur t...@cloudera.com wrote:
 Makes sense, though the Jenkins runs should continue to run w/ native, right?

 On Thu, Sep 6, 2012 at 12:49 AM, Hemanth Yamijala yhema...@gmail.com wrote:
 Hi,

 The test-patch script in Hadoop source runs a native compile with the
 patch. On platforms like MAC, there are issues with the native
 compile. For e.g we run into HADOOP-7147 that has been resolved as
 Won't fix.

 Hence, should we have a switch in test-patch to not run native compile
 ? Could open a JIRA and fix, if that's OK ?

 Thanks
 hemanth



 --
 Alejandro

cmake

2012-05-21 Thread Colin McCabe

Hi all,

We'd like to use CMake instead of autotools to build native (C/C++) code in
Hadoop.  There are a lot of reasons to want to do this.  For one thing, it is
not feasible to use autotools on the Windows platform, because it depends on
UNIX shell scripts, the m4 macro processor, and some other pieces of
infrastructure which are not present on Windows.

For another thing, CMake builds are substantially simpler and faster, because
there is only one layer of generated code.  With autotools, you have automake
generating m4 code which autoconf reads, which it uses to generate a UNIX shell
script, which then generates another UNIX shell script, which eventually
generates Makefiles.  CMake simply generates Makefiles out of CMakeLists.txt
files-- much simpler to understand and debug, and much faster.
CMake is a lot easier to learn.

automake error messages can be very, very confusing.  This is because you are
essentially debugging a pile of shell scripts and macros, rather than a
coherent whole.  So you see error messages like autoreconf: cannot empty
/tmp/ar0.4849: Is a directory or Can't locate object method path via
package Autom4te... and so forth.  CMake error messages come from the CMake
application and they almost always immediately point you to the problem.

From a build point of view, the net result of adopting CMake would be that you
would no longer need automake and related programs installed to build the
native parts of Hadoop.  Instead, you would need CMake installed.  CMake is
packaged by Red Hat, even in RHEL5, so it shouldn't be difficult to install
locally.  It's also available for Mac OS X and Windows, as I mentioned earlier.

The JIRA for this work is at https://issues.apache.org/jira/browse/HADOOP-8368
Thanks for reading.

sincerely,
Colin

88 matches

Mail list logo