[jira] [Resolved] (HADOOP-7675) Ant option to run disabled kerberos authentication tests.

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7675?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-7675.

Resolution: Won't Fix

This ticket is ancient history. Resolving.

> Ant option to run disabled kerberos authentication tests.
> -
>
> Key: HADOOP-7675
> URL: https://issues.apache.org/jira/browse/HADOOP-7675
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 1.1.0
>Reporter: Jitendra Nath Pandey
>Priority: Major
>
> The kerberos tests, TestKerberosAuthenticator and 
> TestKerberosAuthenticationHandler, are disabled using @Ignore. A better 
> approach would be to have an ant option to run them.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Apache Hadoop 3.0.1 Release plan

[Aaron T. Myers]

Hey Anu,

My feeling on HDFS-12990 is that we've discussed it quite a bit already and
it doesn't seem at this point like either side is going to budge. I'm
certainly happy to have a phone call about it, but I don't expect that we'd
make much progress.

My suggestion is that we simply include the patch posted to HDFS-12990 in
the 3.0.1 RC and call this issue out clearly in the subsequent VOTE thread
for the 3.0.1 release. Eddy, are you up for that?

Best,
Aaron

On Thu, Feb 1, 2018 at 1:13 PM, Lei Xu  wrote:

> +Xiao
>
> My understanding is that we will have this for 3.0.1.   Xiao, could
> you give your inputs here?
>
> On Thu, Feb 1, 2018 at 11:55 AM, Anu Engineer 
> wrote:
> > Hi Eddy,
> >
> > Thanks for driving this release. Just a quick question, do we have time
> to close this issue?
> > https://issues.apache.org/jira/browse/HDFS-12990
> >
> > or are we abandoning it? I believe that this is the last window for us
> to fix this issue.
> >
> > Should we have a call and get this resolved one way or another?
> >
> > Thanks
> > Anu
> >
> > On 2/1/18, 10:51 AM, "Lei Xu"  wrote:
> >
> > Hi, All
> >
> > I just cut branch-3.0.1 from branch-3.0.  Please make sure all
> patches
> > targeted to 3.0.1 being checked in both branch-3.0 and branch-3.0.1.
> >
> > Thanks!
> > Eddy
> >
> > On Tue, Jan 9, 2018 at 11:17 AM, Lei Xu  wrote:
> > > Hi, All
> > >
> > > We have released Apache Hadoop 3.0.0 in December [1]. To further
> > > improve the quality of release, we plan to cut branch-3.0.1 branch
> > > tomorrow for the preparation of Apache Hadoop 3.0.1 release. The
> focus
> > > of 3.0.1 will be fixing blockers (3), critical bugs (1) and bug
> fixes
> > > [2].  No new features and improvement should be included.
> > >
> > > We plan to cut branch-3.0.1 tomorrow (Jan 10th) and vote for RC on
> Feb
> > > 1st, targeting for Feb 9th release.
> > >
> > > Please feel free to share your insights.
> > >
> > > [1] https://www.mail-archive.com/general@hadoop.apache.org/
> msg07757.html
> > > [2] https://issues.apache.org/jira/issues/?filter=12342842
> > >
> > > Best,
> > > --
> > > Lei (Eddy) Xu
> > > Software Engineer, Cloudera
> >
> >
> >
> > --
> > Lei (Eddy) Xu
> > Software Engineer, Cloudera
> >
> > 
> -
> > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> >
> >
> >
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: When are incompatible changes acceptable (HDFS-12990)

[Aaron T. Myers]

Yes indeed, that's the proposal being discussed on HDFS-12990 - just to
revert the default NN RPC port change, and none of the other port changes.
The other default port changes actually do have some technical benefit, and
I believe are far less likely to be embedded in databases, scripts, tests,
etc. in real deployments.

Best,
Aaron

On Thu, Jan 11, 2018 at 11:42 AM, larry mccay <lmc...@apache.org> wrote:

> No, the proposal was to only fix the NN port change - as I understood it.
>
> On Thu, Jan 11, 2018 at 2:01 PM, Eric Yang <ey...@hortonworks.com> wrote:
>
> > If I am reading this correctly, Daryn and Larry are in favor of complete
> > revert instead of namenode only.  Please charm in if I am wrong.  This is
> > the reason that I try to explore each perspective to understand the cost
> of
> > each options.  It appears that we have a fragment of opinions, and only
> one
> > choice will serve the need of majority of the community.  It would be
> good
> > for a PMC to call the vote at reasonable pace to address this issue to
> > reduce the pain point from either side of oppositions.
> >
> >
> >
> > Regards,
> >
> > Eric
> >
> >
> >
> > *From: *Chris Douglas <cdoug...@apache.org>
> > *Date: *Wednesday, January 10, 2018 at 7:36 PM
> > *To: *Eric Yang <ey...@hortonworks.com>
> > *Cc: *"Aaron T. Myers" <a...@apache.org>, Daryn Sharp <da...@oath.com>,
> > Hadoop Common <common-dev@hadoop.apache.org>, larry mccay <
> > lmc...@apache.org>
> >
> > *Subject: *Re: When are incompatible changes acceptable (HDFS-12990)
> >
> >
> >
> > Isn't this limited to reverting the 8020 -> 9820 change? -C
> >
> >
> >
> > On Wed, Jan 10, 2018 at 6:13 PM Eric Yang <ey...@hortonworks.com> wrote:
> >
> > The fix in HDFS-9427 can potentially bring in new customers because less
> > chance for new comer to encountering “port already in use” problem.  If
> we
> > make change according to HDFS-12990, then this incompatible change does
> not
> > make incompatible change compatible.  Other ports are not reverted
> > according to HDFS-12990.  User will encounter the bad taste in the mouth
> > that HDFS-9427 attempt to solve.  Please do consider both negative side
> > effects of reverting as well as incompatible minor release change.
> Thanks
> >
> > Regards,
> > Eric
> >
> > From: larry mccay <lmc...@apache.org>
> > Date: Wednesday, January 10, 2018 at 10:53 AM
> > To: Daryn Sharp <da...@oath.com>
> > Cc: "Aaron T. Myers" <a...@apache.org>, Eric Yang <ey...@hortonworks.com
> >,
> > Chris Douglas <cdoug...@apache.org>, Hadoop Common <
> > common-dev@hadoop.apache.org>
> > Subject: Re: When are incompatible changes acceptable (HDFS-12990)
> >
> > On Wed, Jan 10, 2018 at 1:34 PM, Daryn Sharp <da...@oath.com daryn@
> > oath.com>> wrote:
> >
> > I fully agree the port changes should be reverted.  Although
> > "incompatible", the potential impact to existing 2.x deploys is huge.
> I'd
> > rather inconvenience 3.0 deploys that compromise <1% customers.  An
> > incompatible change to revert an incompatible change is called
> > compatibility.
> >
> > +1
> >
> >
> >
> >
> > Most importantly, consider that there is no good upgrade path existing
> > deploys, esp. large and/or multi-cluster environments.  It’s only
> feasible
> > for first-time deploys or simple single-cluster upgrades willing to take
> > downtime.  Let's consider a few reasons why:
> >
> >
> >
> > 1. RU is completely broken.  Running jobs will fail.  If MR on hdfs
> > bundles the configs, there's no way to transparently coordinate the
> switch
> > to the new bundle with the port changed.  Job submissions will fail.
> >
> >
> >
> > 2. Users generally do not add the rpc port number to uris so unless their
> > configs are updated they will contact the wrong port.  Seamlessly
> > coordinating the conf change without massive failures is impossible.
> >
> >
> >
> > 3. Even if client confs are updated, they will break in a multi-cluster
> > env with NNs using different ports.  Users/services will be forced to add
> > the port.  The cited hive "issue" is not a bug since it's the only way to
> > work in a multi-port env.
> >
> >
> >
> > 4. Coordinating the port add/change of uris is systems everywhere (you
> > know something will be missed), updating of

Re: When are incompatible changes acceptable (HDFS-12990)

[Aaron T. Myers]

Hey Eric,

Comments inline.

On Wed, Jan 10, 2018 at 9:06 AM, Eric Yang <ey...@hortonworks.com> wrote:

> Hi Aaron,
>
>
>
> Correct me if I am wrong, the port change is only required when making a
> new cluster due to the default value.  Existing cluster does not need to
> make the switch, if Hadoop configuration contains user defined number.
>

Certainly true that a port change isn't required, and if it's already
properly being set everywhere throughout a deployment (i.e. all clients,
client applications, scripts, etc.) it won't be an issue. I'm most worried
about *client* configs throughout a large deployment, which would be
difficult (impossible?) to coordinate an update to. Entirely possible, if
not likely, that many clients are inadvertently relying on the default
port, so when they start using the updated software they'll break because
of the default port change.


> Ambari, and Cloudera Manager are already handling user defined ports
> correctly.  Some QA tools may need to change, but it is a good exercise to
> run on non-standard port.
>

Sites which are using Ambari or Cloudera Manager are more likely to work,
but again, I worry about client configs and other places that might have
hard-coded the port number, e.g. in Hive or in scripts.

I will also say that Hadoop users which are *not* using Ambari or CM should
be considered as well. Sites like this are perhaps the most likely to break
because of this change.


> I gave my vote to keep the setting, and fully respect the community’s
> decision in this matter.
>

Thanks, Eric. I understand your argument to be that changing this default
port might not be so bad, but it also sounds like you wouldn't object if
others conclude that it's best to change it back. Is that right?

Best,
Aaron



>
>
> Regards,
>
> Eric
>
>
>
> *From: *<a...@cloudera.com> on behalf of "Aaron T. Myers" <a...@apache.org>
> *Date: *Tuesday, January 9, 2018 at 9:22 PM
> *To: *Eric Yang <ey...@hortonworks.com>
> *Cc: *Chris Douglas <cdoug...@apache.org>, larry mccay <lmc...@apache.org>,
> Hadoop Common <common-dev@hadoop.apache.org>
> *Subject: *Re: When are incompatible changes acceptable (HDFS-12990)
>
>
>
> On Tue, Jan 9, 2018 at 3:15 PM, Eric Yang <ey...@hortonworks.com> wrote:
>
> While I agree the original port change was unnecessary, I don’t think
> Hadoop NN port change is a bad thing.
>
> I worked for a Hadoop distro that NN RPC port was default to port 9000.
> When we migrate from BigInsights to IOP and now to HDP, we have to move
> customer Hive metadata to new NN RPC port.  It only took one developer
> (myself) to write the tool for the migration.  The incurring workload is
> not as bad as most people anticipated because Hadoop depends on
> configuration file for referencing namenode.  Most of the code can work
> transparently.  It helped to harden the downstream testing tools to be more
> robust.
>
>
>
> While there are of course ways to deal with this, the question really
> should be whether or not it's a desirable thing to do to our users.
>
>
>
>
> We will never know how many people are actively working on Hadoop 3.0.0.
> Perhaps, couple hundred developers or thousands.
>
>
>
> You're right that we can't know for sure, but I strongly suspect that this
> is a substantial overestimate. Given how conservative Hadoop operators tend
> to be, I view it as exceptionally unlikely that many deployments have been
> created on or upgraded to Hadoop 3.0.0 since it was released less than a
> month ago.
>
>
>
> Further, I hope you'll agree that the number of
> users/developers/deployments/applications which are currently on Hadoop
> 2.x is *vastly* greater than anyone who might have jumped on Hadoop 3.0.0
> so quickly. When all of those users upgrade to any 3.x version, they will
> encounter this needless incompatible change and be forced to work around it.
>
>
>
> I think the switch back may have saved few developers work, but there
> could be more people getting impacted at unexpected minor release change in
> the future.  I recommend keeping current values to avoid rule bending and
> future frustrations.
>
>
>
> That we allow this incompatible change now does not mean that we are
> categorically allowing more incompatible changes in the future. My point is
> that we should in all instances evaluate the merit of any incompatible
> change on a case-by-case basis. This is not an exceptional circumstance -
> we've made incompatible changes in the past when appropriate, e.g. breaking
> some clients to address a security issue. I and others believe that in this
> case the benefits greatly outweigh the downsides of changing this back to
> what it 

Re: When are incompatible changes acceptable (HDFS-12990)

[Aaron T. Myers]

On Tue, Jan 9, 2018 at 3:15 PM, Eric Yang <ey...@hortonworks.com> wrote:

> While I agree the original port change was unnecessary, I don’t think
> Hadoop NN port change is a bad thing.
>
> I worked for a Hadoop distro that NN RPC port was default to port 9000.
> When we migrate from BigInsights to IOP and now to HDP, we have to move
> customer Hive metadata to new NN RPC port.  It only took one developer
> (myself) to write the tool for the migration.  The incurring workload is
> not as bad as most people anticipated because Hadoop depends on
> configuration file for referencing namenode.  Most of the code can work
> transparently.  It helped to harden the downstream testing tools to be more
> robust.
>

While there are of course ways to deal with this, the question really
should be whether or not it's a desirable thing to do to our users.


>
> We will never know how many people are actively working on Hadoop 3.0.0.
> Perhaps, couple hundred developers or thousands.


You're right that we can't know for sure, but I strongly suspect that this
is a substantial overestimate. Given how conservative Hadoop operators tend
to be, I view it as exceptionally unlikely that many deployments have been
created on or upgraded to Hadoop 3.0.0 since it was released less than a
month ago.

Further, I hope you'll agree that the number of
users/developers/deployments/applications which are currently on Hadoop 2.x
is *vastly* greater than anyone who might have jumped on Hadoop 3.0.0 so
quickly. When all of those users upgrade to any 3.x version, they will
encounter this needless incompatible change and be forced to work around it.


> I think the switch back may have saved few developers work, but there
> could be more people getting impacted at unexpected minor release change in
> the future.  I recommend keeping current values to avoid rule bending and
> future frustrations.
>

That we allow this incompatible change now does not mean that we are
categorically allowing more incompatible changes in the future. My point is
that we should in all instances evaluate the merit of any incompatible
change on a case-by-case basis. This is not an exceptional circumstance -
we've made incompatible changes in the past when appropriate, e.g. breaking
some clients to address a security issue. I and others believe that in this
case the benefits greatly outweigh the downsides of changing this back to
what it has always been.

Best,
Aaron


>
> Regards,
> Eric
>
> On 1/9/18, 11:21 AM, "Chris Douglas" <cdoug...@apache.org> wrote:
>
> Particularly since 9820 isn't in the contiguous range of ports in
> HDFS-9427, is there any value in this change?
>
> Let's change it back to prevent the disruption to users, but
> downstream projects should treat this as a bug in their tests. Please
> open JIRAs in affected projects. -C
>
>
> On Tue, Jan 9, 2018 at 5:18 AM, larry mccay <lmc...@apache.org> wrote:
> > On Mon, Jan 8, 2018 at 11:28 PM, Aaron T. Myers <a...@apache.org>
> wrote:
> >
> >> Thanks a lot for the response, Larry. Comments inline.
> >>
> >> On Mon, Jan 8, 2018 at 6:44 PM, larry mccay <lmc...@apache.org>
> wrote:
> >>
> >>> Question...
> >>>
> >>> Can this be addressed in some way during or before upgrade that
> allows it
> >>> to only affect new installs?
> >>> Even a config based workaround prior to upgrade might make this a
> change
> >>> less disruptive.
> >>>
> >>> If part of the upgrade process includes a step (maybe even a
> script) to
> >>> set the NN RPC port explicitly beforehand then it would allow
> existing
> >>> deployments and related clients to remain whole - otherwise it
> will uptake
> >>> the new default port.
> >>>
> >>
> >> Perhaps something like this could be done, but I think there are
> downsides
> >> to anything like this. For example, I'm sure there are plenty of
> >> applications written on top of Hadoop that have tests which
> hard-code the
> >> port number. Nothing we do in a setup script will help here. If we
> don't
> >> change the default port back to what it was, these tests will
> likely all
> >> have to be updated.
> >>
> >>
> >
> > I may not have made my point clear enough.
> > What I meant to say is to fix the default port but direct folks to
> > explicitly set the port they are using in a deployment (the current
> > default) so that it doesn't change out from under them - unles

Re: When are incompatible changes acceptable (HDFS-12990)

[Aaron T. Myers]

Thanks a lot for the response, Larry. Comments inline.

On Mon, Jan 8, 2018 at 6:44 PM, larry mccay  wrote:

> Question...
>
> Can this be addressed in some way during or before upgrade that allows it
> to only affect new installs?
> Even a config based workaround prior to upgrade might make this a change
> less disruptive.
>
> If part of the upgrade process includes a step (maybe even a script) to
> set the NN RPC port explicitly beforehand then it would allow existing
> deployments and related clients to remain whole - otherwise it will uptake
> the new default port.
>

Perhaps something like this could be done, but I think there are downsides
to anything like this. For example, I'm sure there are plenty of
applications written on top of Hadoop that have tests which hard-code the
port number. Nothing we do in a setup script will help here. If we don't
change the default port back to what it was, these tests will likely all
have to be updated.


>
> Meta note: we shouldn't be so pedantic about policy that we can't back out
> something that is considered a bug or even mistake.
>

This is my bigger point. Rigidly adhering to the compat guidelines in this
instance helps almost no one, while hurting many folks.

We basically made a mistake when we decided to change the default NN port
with little upside, even between major versions. We discovered this very
quickly, and we have an opportunity to fix it now and in so doing likely
disrupt very, very few users and downstream applications. If we don't
change it, we'll be causing difficulty for our users, downstream
developers, and ourselves, potentially for years.

Best,
Aaron

When are incompatible changes acceptable (HDFS-12990)

[Aaron T. Myers]

Hello all,

Over in HDFS-12990 [1],
we're having some discussion about whether or not it's ever acceptable to
make an incompatible change in a minor or dot release. In general this is
of course undesirable and should be avoided in almost all cases. However, I
believe that each instance of someone desiring to make an incompatible
change should be evaluated on a case-by-case basis to consider the costs
and benefits of making that change. For example, I believe that we've
historically made incompatible changes in minor or dot releases which would
break older clients for security reasons.

In this particular case linked above,  I believe that given that Hadoop
3.0.0 was just released, and thus very few folks are likely to have
deployed it, it would benefit a large number of existing deployments and
downstream applications to change the default NN RPC port number back to
what it was in all previously-released versions of Apache Hadoop. I'd like
to make this change in 3.0.1, and there is no question that doing so would
should be considered an incompatible change between 3.0.0 and 3.0.1.
However, I believe this incompatible change is warranted given the
circumstances.

Would like to hear others' thoughts on this.

Thanks,
Aaron

[1] For some background, it used to be the case that many of Hadoop's
default service ports were in the ephemeral range. This could potentially
cause a service to fail to start up on a given host if some other process
had happened to have already bound to said port. As part of that effort, we
also changed the default NN RPC port from 8020 to 9820. Even though 8020
wasn't in the ephemeral range, we moved it to 9820 to be close to the new
range of the rest of the ports. At the time this change was made, though, I
and others didn't realize the substantial downsides that doing so would
introduce, for example the Hive metastore will put full HDFS paths
including the port into its database, which can be a substantial upgrade
headache.

Re: [VOTE] Release Apache Hadoop 3.0.0 RC1

[Aaron T. Myers]

+1 (binding)

- downloaded the src tarball and built the source (-Pdist -Pnative)
- verified the checksum
- brought up a secure pseudo distributed cluster
- did some basic file system operations (mkdir, list, put, cat) and
confirmed that everything was working
- confirmed that the web UI worked

Best,
Aaron

On Fri, Dec 8, 2017 at 12:31 PM, Andrew Wang 
wrote:

> Hi all,
>
> Let me start, as always, by thanking the efforts of all the contributors
> who contributed to this release, especially those who jumped on the issues
> found in RC0.
>
> I've prepared RC1 for Apache Hadoop 3.0.0. This release incorporates 302
> fixed JIRAs since the previous 3.0.0-beta1 release.
>
> You can find the artifacts here:
>
> http://home.apache.org/~wang/3.0.0-RC1/
>
> I've done the traditional testing of building from the source tarball and
> running a Pi job on a single node cluster. I also verified that the shaded
> jars are not empty.
>
> Found one issue that create-release (probably due to the mvn deploy change)
> didn't sign the artifacts, but I fixed that by calling mvn one more time.
> Available here:
>
> https://repository.apache.org/content/repositories/orgapachehadoop-1075/
>
> This release will run the standard 5 days, closing on Dec 13th at 12:31pm
> Pacific. My +1 to start.
>
> Best,
> Andrew
>

[jira] [Created] (HADOOP-14855) Hadoop scripts may errantly believe a daemon is still running, preventing it from starting

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-14855:
---

 Summary: Hadoop scripts may errantly believe a daemon is still 
running, preventing it from starting
 Key: HADOOP-14855
 URL: https://issues.apache.org/jira/browse/HADOOP-14855
 Project: Hadoop Common
  Issue Type: Bug
  Components: scripts
Affects Versions: 3.0.0-alpha4
Reporter: Aaron T. Myers


I encountered a case recently where the NN wouldn't start, with the error 
message "namenode is running as process 16769.  Stop it first." In fact the NN 
was not running at all, but rather another long-running process was running 
with this pid.

It looks to me like our scripts just check to see if _any_ process is running 
with the pid that the NN (or any Hadoop daemon) most recently ran with. This is 
clearly not a fool-proof way of checking to see if a particular type of daemon 
is now running, as some other process could start running with the same pid 
since the daemon in question was previously shut down.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0-alpha1 RC0

[Aaron T. Myers]

+1 (binding) from me. Downloaded the source, built from source, set up a
pseudo cluster, and ran a few of the sample jobs.

Thanks a lot for doing all this release work, Andrew.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Aug 30, 2016 at 8:51 AM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi all,
>
> Thanks to the combined work of many, many contributors, here's an RC0 for
> 3.0.0-alpha1:
>
> http://home.apache.org/~wang/3.0.0-alpha1-RC0/
>
> alpha1 is the first in a series of planned alpha releases leading up to GA.
> The objective is to get an artifact out to downstreams for testing and to
> iterate quickly based on their feedback. So, please keep that in mind when
> voting; hopefully most issues can be addressed by future alphas rather than
> future RCs.
>
> Sorry for getting this out on a Tuesday, but I'd still like this vote to
> run the normal 5 days, thus ending Saturday (9/3) at 9AM PDT. I'll extend
> if we lack the votes.
>
> Please try it out and let me know what you think.
>
> Best,
> Andrew
>

Re: Why there are so many revert operations on trunk?

[Aaron T. Myers]

Junping,

All of this is being discussed on HDFS-9924. Suggest you follow the
conversation there.

--
Aaron T. Myers
Software Engineer, Cloudera

On Mon, Jun 6, 2016 at 7:20 AM, Junping Du <j...@hortonworks.com> wrote:

> Hi Andrew,
>
>  I just noticed you revert 8 commits on trunk last Friday:
>
> HADOOP-13226
>
> HDFS-10430
>
> HDFS-10431
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10390
>
> HADOOP-13168
>
> HDFS-10346
>
> HADOOP-12957
>
> HDFS-10224
>
>And I didn't see you have any comments on JIRA or email discussion
> before you did this. I don't think we are legally allowed to do this even
> as committer/PMC member. Can you explain what's your intention to do this?
>
>BTW, thanks Nicolas to revert all these "illegal" revert operations.
>
>
>
> Thanks,
>
>
> Junping
>

Re: 'Target Version' field missing in Jira

[Aaron T. Myers]

Great news. Thanks a lot for looking into this, Vinod.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Feb 16, 2016 at 11:42 AM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> The data is still intact as I can see that I can continue to search on
> JIRA against target-version. But only in advanced-search though.
>
> +Vinod
>
> > On Feb 16, 2016, at 10:23 AM, Aaron T. Myers <a...@cloudera.com> wrote:
> >
> > I worry
> > that all the data for the target versions has disappeared as well...
>
>

Re: 'Target Version' field missing in Jira

[Aaron T. Myers]

Has anyone followed up with ASF Infra about getting this addressed? I worry
that all the data for the target versions has disappeared as well...

--
Aaron T. Myers
Software Engineer, Cloudera

On Fri, Feb 12, 2016 at 1:35 PM, Kihwal Lee <kih...@yahoo-inc.com.invalid>
wrote:

> It's still here:
> https://issues.apache.org/jira/plugins/servlet/project-config/HDFS/fields
> But somehow not showing up on pages.
> Kihwal
>
>
>   From: Arpit Agarwal <aagar...@hortonworks.com>
>  To: "common-dev@hadoop.apache.org" <common-dev@hadoop.apache.org>; "
> hdfs-...@hadoop.apache.org" <hdfs-...@hadoop.apache.org>
>  Sent: Friday, February 12, 2016 1:52 PM
>  Subject: 'Target Version' field missing in Jira
>
> Is it just me or has the Target Version/s field gone missing from Apache
> Hadoop Jira? I don't recall any recent discussion about it.
>
>
>
>

Re: Hadoop Common: Why not re-use the Security model offered by SELINUX?

[Aaron T. Myers]

In addition to everything Allen has already said, which I entirely agree
with, I'll also point out that much of the focus on Hadoop security has
been related to authentication, and only somewhat more recently on
providing advanced authorization capabilities. I'll readily admit to not
knowing much about SE Linux's capabilities, but my impression is that it
wouldn't do much to be able to help out with authentication within Hadoop,
and hence wouldn't have been a realistic option when Hadoop's security work
was started many years ago.

--
Aaron T. Myers
Software Engineer, Cloudera

On Thu, Mar 26, 2015 at 8:27 AM, Allen Wittenauer a...@altiscale.com wrote:


 When Hadoop was started 10 years ago and first started getting
 it’s security bits in place 8 years ago, SE Linux was immature and
 extremely broken.  (Which, in many ways, is still true.  Like a lot of
 Linux vs. commercial competitors, it’s about 80% there.)  Plus, considering
 that Hadoop was written in Java , it never made sense to integrate Hadoop
 so tightly with those interfaces.

 It’s also worth pointing out that SE Linux also doesn’t help at
 all at protecting things inside of HDFS since it is a user-land file system
 and not mounted at the VFS layer.  The number of contortions required to
 get support would be… interesting.

 Now that time has progressed and both projects have evolved,
 someone might be interested in exploring the possibilities. It’s worth
 pointing out that large portions of Hadoop’s security model is pluggable.
 So if you’d like to take this work on, feel free.

 On Mar 26, 2015, at 7:44 AM, Madhan Sundararajan 
 madhan.sundarara...@tcs.com wrote:

  Allen,
 
  Unlike you, I am no Unix veteran.
 
  However, having used Hadoop briefly I observed this anomaly.
 
  Yes, as you have highlighted, this is not applicable to non-Linux
  platforms.
 
  Hadoop's security layer can be made to re-use SELINUX' policies through
  remote policy server, to ease the application of policies from a
  centralised policy server.
 
  Further, Hadoop can be made to re-use role-based-access-controls provided
  by SELINUX.
 
  In addition, Hadoop daemons can be subjected to the fine-grained access
  policies of SELINUX to use the Linux Server's resources.
 
  Regards
  Madhan Sundararajan Devaki
 
  Tata Consultancy Services Limited
  415/21-24, Kumaran Nagar,
  Sholinganallur,
  Old Mahabalipuram,
  Chennai - 600 119,Tamil Nadu
  India
  Cell:- +91-9840141129
  Mailto: madhan.sundarara...@tcs.com
  Website: http://www.tcs.com
  
  Experience certainty.   IT Services
 Business Solutions
 Consulting
  
 
 
 
  From:   Allen Wittenauer a...@altiscale.com
  To: common-dev@hadoop.apache.org
  Date:   03/26/2015 06:51 PM
  Subject:Re: Hadoop Common: Why not re-use the Security model
  offered by SELINUX?
 
 
 
 
  How would you propose we use SELinux features to support
  security, especially in a distributed manner where clients might be under
  different administrative controls?  What about the non-Linux platforms
  that Hadoop runs on?
 
 
  On Mar 26, 2015, at 3:46 AM, Madhan Sundararajan
  madhan.sundarara...@tcs.com wrote:
 
  Team,
 
  SELINUX was introduced to bring in a robust security management in Linux
 
  OS.
 
  In all distributions of Hadoop (Cloudera/Hortonworks/...) one of the
  pre-installation checklist items is to disable SELINUX in all the nodes
  of
  the cluster.
 
  Why not re-use the security model offered by SELINUX setting instead of
  re-inventing from scratch through Sentry/Knox/etc...?
 
  Regards
  Madhan Sundararajan Devaki
 
  Tata Consultancy Services Limited
  415/21-24, Kumaran Nagar,
  Sholinganallur,
  Old Mahabalipuram,
  Chennai - 600 119,Tamil Nadu
  India
  Cell:- +91-9840141129
  Mailto: madhan.sundarara...@tcs.com
  Website: http://www.tcs.com
  
  Experience certainty.   IT Services
Business Solutions
Consulting
  
  =-=-=
  Notice: The information contained in this e-mail
  message and/or attachments to it may contain
  confidential or privileged information. If you are
  not the intended recipient, any dissemination, use,
  review, distribution, printing or copying of the
  information contained in this e-mail message
  and/or attachments to it are strictly prohibited. If
  you have received this communication in error,
  please notify us by reply e-mail or telephone and
  immediately and permanently delete the message
  and any attachments. Thank you
 
 
 
 

Re: upstream jenkins build broken?

[Aaron T. Myers]

Hey Colin,

I asked Andrew Bayer, who works with Apache Infra, what's going on with
these boxes. He took a look and concluded that some perms are being set in
those directories by our unit tests which are precluding those files from
getting deleted. He's going to clean up the boxes for us, but we should
expect this to keep happening until we can fix the test in question to
properly clean up after itself.

To help narrow down which commit it was that started this, Andrew sent me
this info:

/home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-
Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3/ has
500 perms, so I'm guessing that's the problem. Been that way since 9:32 UTC
on March 5th.

--
Aaron T. Myers
Software Engineer, Cloudera

On Tue, Mar 10, 2015 at 1:24 PM, Colin P. McCabe cmcc...@apache.org wrote:

 Hi all,

 A very quick (and not thorough) survey shows that I can't find any
 jenkins jobs that succeeded from the last 24 hours.  Most of them seem
 to be failing with some variant of this message:

 [ERROR] Failed to execute goal
 org.apache.maven.plugins:maven-clean-plugin:2.5:clean (default-clean)
 on project hadoop-hdfs: Failed to clean project: Failed to delete

 /home/jenkins/jenkins-slave/workspace/PreCommit-HDFS-Build/hadoop-hdfs-project/hadoop-hdfs/target/test/data/dfs/data/data3
 - [Help 1]

 Any ideas how this happened?  Bad disk, unit test setting wrong
 permissions?

 Colin

Re: Looking to a Hadoop 3 release

[Aaron T. Myers]

+1, this sounds like a good plan to me.

Thanks a lot for volunteering to take this on, Andrew.

Best,
Aaron

On Mon, Mar 2, 2015 at 3:19 PM, Andrew Wang andrew.w...@cloudera.com
wrote:

 Hi devs,

 It's been a year and a half since 2.x went GA, and I think we're about due
 for a 3.x release.
 Notably, there are two incompatible changes I'd like to call out, that will
 have a tremendous positive impact for our users.

 First, classpath isolation being done at HADOOP-11656, which has been a
 long-standing request from many downstreams and Hadoop users.

 Second, bumping the source and target JDK version to JDK8 (related to
 HADOOP-11090), which is important since JDK7 is EOL in April 2015 (two
 months from now). In the past, we've had issues with our dependencies
 discontinuing support for old JDKs, so this will future-proof us.

 Between the two, we'll also have quite an opportunity to clean up and
 upgrade our dependencies, another common user and developer request.

 I'd like to propose that we start rolling a series of monthly-ish series of
 3.0 alpha releases ASAP, with myself volunteering to take on the RM and
 other cat herding responsibilities. There are already quite a few changes
 slated for 3.0 besides the above (for instance the shell script rewrite) so
 there's already value in a 3.0 alpha, and the more time we give downstreams
 to integrate, the better.

 This opens up discussion about inclusion of other changes, but I'm hoping
 to freeze incompatible changes after maybe two alphas, do a beta (with no
 further incompat changes allowed), and then finally a 3.x GA. For those
 keeping track, that means a 3.x GA in about four months.

 I would also like to stress though that this is not intended to be a big
 bang release. For instance, it would be great if we could maintain wire
 compatibility between 2.x and 3.x, so rolling upgrades work. Keeping
 branch-2 and branch-3 similar also makes backports easier, since we're
 likely maintaining 2.x for a while yet.

 Please let me know any comments / concerns related to the above. If people
 are friendly to the idea, I'd like to cut a branch-3 and start working on
 the first alpha.

 Best,
 Andrew

[jira] [Created] (HADOOP-11158) KerberosAuthenticationHandler should have reasonable default for

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-11158:
---

 Summary: KerberosAuthenticationHandler should have reasonable 
default for 
 Key: HADOOP-11158
 URL: https://issues.apache.org/jira/browse/HADOOP-11158
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.5.1
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


The {{KerberosAuthenticationHandler}} currently defines no default value for 
the {{.kerberos.name.rules}} config key, which means that users who use this 
class and don't set this setting will get an NPE. We should set the default for 
this DEFAULT like it is for the other Kerberos name mapping properties in 
Hadoop.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] Release Apache Hadoop 2.4.1

[Aaron T. Myers]

I'm -0 on rc1.

Note the latest discussion on HDFS-6527 which first resulted in that patch
being reverted from branch-2.4.1 because it was believed it wasn't
necessary, and then some more discussion which indicates that in fact the
patch for HDFS-6527 should be included in 2.4.1, but with a slightly
different test case.

I believe that rc1 was actually created after the first backport of
HDFS-6527, but before the revert, so rc1 should be functionally correct,
but the test case is not quite correct in rc1, and I believe that rc1 does
not currently reflect the actual tip of branch-2.4.1. I'm not going to
consider this a deal-breaker, but seems like we should probably clean it up.

To get this all sorted out properly, if we wanted to, I believe we should
do another backport of HDFS-6527 to branch-2.4.1 including only the amended
test case, and create a new RC from that point.

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera


On Fri, Jun 20, 2014 at 11:51 PM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

 I've created another release candidate (rc1) for hadoop-2.4.1 based on the
 feedback that I would like to push out.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.4.1-rc1
 The RC tag in svn is here:
 https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.4.1-rc1

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun



 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/hdp/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: [VOTE] Change by-laws on release votes: 5 days instead of 7

[Aaron T. Myers]

+1 (binding)

--
Aaron T. Myers
Software Engineer, Cloudera


On Tue, Jun 24, 2014 at 1:53 AM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

  As discussed, I'd like to call a vote on changing our by-laws to change
 release votes from 7 days to 5.

  I've attached the change to by-laws I'm proposing.

  Please vote, the vote will the usual period of 7 days.

 thanks,
 Arun

 

 [main]$ svn diff
 Index: author/src/documentation/content/xdocs/bylaws.xml
 ===
 --- author/src/documentation/content/xdocs/bylaws.xml   (revision 1605015)
 +++ author/src/documentation/content/xdocs/bylaws.xml   (working copy)
 @@ -344,7 +344,16 @@
  pVotes are open for a period of 7 days to allow all active
  voters time to consider the vote. Votes relating to code
  changes are not subject to a strict timetable but should be
 -made as timely as possible./p/li
 +made as timely as possible./p
 +
 + ul
 + li strongProduct Release - Vote Timeframe/strong
 +   pRelease votes, alone, run for a period of 5 days. All other
 + votes are subject to the above timeframe of 7 days./p
 + /li
 +   /ul
 +   /li
 +
 /ul
 /section
  /body
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Plans of moving towards JDK7 in trunk

[Aaron T. Myers]

On Fri, Jun 20, 2014 at 5:01 PM, Andrew Wang andrew.w...@cloudera.com
wrote:

 Thanks everyone for the discussion so far. I talked with some of our other
 teams and thought about the issue some more.

 Regarding branch-2, we can't do much because of compatibility. Dropping
 support for a JDK is supposed to happen in a major release. I think we all
 understand this though, so it's not really under discussion.


+1



 Regarding trunk, I think that leapfrogging to JDK8 is the right move. JDK7
 is EOL April next year, so it'd be better to avoid going through this pain
 twice so soon. Developer momentum also seems very strong behind JDK8
 because of all the shiny new features, so I think we'll see quick adoption.
 We also need some time to clean up APIs and I'm sure people have big,
 incompatible project ideas floating around they'd like to get in.


+1, doesn't seem to be much point in targeting another JDK version that
will be EOL so soon.



 With the JDK7 EOL in mind, we need a JDK8-based 3.0 release by mid next
 year. Since I have a strong interest in all these things, I'd like to
 volunteer as release manager for this beast. This means, yep, I'll wrangle
 the builds, worry about compat, bump lib versions, and all those other fun
 tasks. There's clearly a lot to discuss logistically (let's take that to a
 different thread), but this feels like the right way forward to me.


+1, thanks a lot for volunteering to take this on.



 Best,
 Andrew

 On Fri, Jun 20, 2014 at 11:10 AM, Bryan Beaudreault 
 bbeaudrea...@hubspot.com wrote:

  As one of the customers you guys have referenced a few times, I 100%
 agree
  with Colin. :)
 
  We had to wait a few years for java7 to be supported for hadoop, or
  certified at least.  It would be great to not have to wait a few years
  again for the newly released java8.  People are already clamoring to use
 it
  at our company, but we have to tell them no if their code in any way is
  pulled in by a hadoop job.
 
  It seems an easier and more wide-reaching priority to first ensure
  compatibility with newer JVMs, then hammer out deprecation of older and
 use
  of new APIs internally.  Those are of lesser concern to many of the
  customers I'd think.
 
 
  On Fri, Jun 20, 2014 at 2:02 PM, Colin McCabe cmcc...@alumni.cmu.edu
  wrote:
 
   Er, that should read order in which it ran unit tests.
  
   C.
  
   On Fri, Jun 20, 2014 at 11:02 AM, Colin McCabe cmcc...@alumni.cmu.edu
 
   wrote:
I think the important thing to do right now is to ensure our code
works with jdk8.  This is similar to the work we did last year to fix
issues that cropped up with jdk7.  There were just a few minor things
like the error in which it ran unit tests that caused issues.
   
I don't think it's out of the question to bump minVersion directly
from 1.6 to 1.8.  Java 7 is EOL in less than a year, after all... at
least according to the current roadmap here:
http://www.oracle.com/technetwork/java/eol-135779.html
   
We should try to do things in a way that causes the least disruption
for users upgrading clusters, if possible...
   
cheers,
Colin
   
On Thu, Jun 19, 2014 at 3:31 PM, Andrew Purtell apurt...@apache.org
 
   wrote:
There are a number of security (algorithm, not vulnerability) and
performance improvements that landed in 8, not 7. As a runtime for
 the
performance conscious, it might be recommendable. I've come across
 GC
issues in 6 or 7 where, talking with some Java platform folks, the
  first
suggested course of action is try again with 8. Would be be more of
  the
current moment if this discussion was about setting guidelines that
prescribe when and when not to use 8+ language features, or
  concurrency
library improvements that rely on intrinsics only available in the 8
runtime? Has the Java 6 ship sailed? Just set the minimum supported
  JDK
   and
runtime at 7 at next release? Use of the  operator or multicatch
   wouldn't
and shouldn't need be debated, they are quite minor. On the other
  hand,
   I
would imagine discussion and debate on what 8+ language features
 might
   be
useful to use at some future time could be a lively one.
   
   
   
On Wed, Jun 18, 2014 at 3:03 PM, Colin McCabe 
 cmcc...@alumni.cmu.edu
  
wrote:
   
In CDH5, Cloudera encourages people to use JDK7.  JDK6 has been EOL
for a while now and is not something we recommend.
   
As we discussed before, everyone is in favor of upgrading to JDK7.
Every cluster operator of a reasonably modern Hadoop should do
 it
whatever distro or release you run.  As developers, we run JDK7 as
well.
   
I'd just like to see a plan for when branch-2 (or some other
 branch)
will create a stable release that drops support for JDK1.6.  If we
don't have such a plan, I feel like it's too early to talk about
 this
stuff.
   
If we drop support for 1.6 in trunk but not in branch-2, 

[jira] [Created] (HADOOP-10712) Add support for accessing the NFS gateway from the AIX NFS client

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-10712:
---

 Summary: Add support for accessing the NFS gateway from the AIX 
NFS client
 Key: HADOOP-10712
 URL: https://issues.apache.org/jira/browse/HADOOP-10712
 Project: Hadoop Common
  Issue Type: Bug
  Components: nfs
Affects Versions: 2.4.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


We've identified two issues when trying to access the HDFS NFS Gateway from an 
AIX NFS client:

# In the case of COMMITs, the AIX NFS client will always send 4096, or a 
multiple of the page size, for the offset to be committed, even if fewer bytes 
than this have ever, or will ever, be written to the file. This will cause a 
write to a file from the AIX NFS client to hang on close unless the size of 
that file is a multiple of 4096.
# In the case of READDIR and READDIRPLUS, the AIX NFS client will send the same 
cookie verifier for a given directory seemingly forever after that directory is 
first accessed over NFS, instead of getting a new cookie verifier for every set 
of incremental readdir calls. This means that if a directory's mtime ever 
changes, the FS must be unmounted/remounted before readdir calls on that dir 
from AIX will ever succeed again.

From my interpretation of RFC-1813, the NFS Gateway is in fact doing the 
correct thing in both cases, but we can introduce simple changes on the NFS 
Gateway side to be able to optionally work around these incompatibilities.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: #Contributors on JIRA

[Aaron T. Myers]

Hey Pete,

The contributor list we're referring to is just the collection of folks
in JIRA that JIRAs can be assigned to. It doesn't impact anyone's ability
to follow JIRAs, just have new JIRAs assigned to them. It also doesn't do
anything to JIRAs that are already assigned.

--
Aaron T. Myers
Software Engineer, Cloudera


On Thu, May 15, 2014 at 6:39 AM, Pete of Pete peterbort...@gmail.comwrote:

 On behalf of those of us who keep an eye on posts for commercial reasons
 (but do not contribute) Is there the option of setting up a list that
 accesses the dev comms, but doesn't impact the contributor list?

 P


 On Wed, May 14, 2014 at 4:31 AM, Suresh Srinivas sur...@hortonworks.com
 wrote:

  Last time we cleaned up names of people who had not contributed in a long
  time. That could be an option.
 
 
  On Mon, May 12, 2014 at 12:03 PM, Karthik Kambatla ka...@cloudera.com
  wrote:
 
   Hi devs
  
   Looks like we ran over the max contributors allowed for a project,
  again. I
   don't remember what we did last time and can't find it in my email
  either.
  
   Can we bump up the number of contributors allowed? Otherwise, we might
  have
   to remove some of the currently inactive contributors from the list?
  
   Thanks
   Karthik
  
 
 
 
  --
  http://hortonworks.com/download/
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 

[jira] [Created] (HADOOP-10418) SaslRpcClient should not assume that remote principals are in the default_realm

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-10418:
---

 Summary: SaslRpcClient should not assume that remote principals 
are in the default_realm
 Key: HADOOP-10418
 URL: https://issues.apache.org/jira/browse/HADOOP-10418
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.4.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


We 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [VOTE] Release Apache Hadoop 2.3.0

[Aaron T. Myers]

I don't think any of what you describe below is a regression in behavior
from earlier releases. The fs.defaultFS has been set to file:/// for a long
time, and likewise you've similarly had to set up your YARN configs. Given
that, I don't think this warrants a new RC.

--
Aaron T. Myers
Software Engineer, Cloudera


On Wed, Feb 12, 2014 at 5:37 PM, Alejandro Abdelnur t...@cloudera.comwrote:

 Running pseudo cluster out of the box (expanding the binary tar, or
 building from source) does not work, you have to go an set the MR framework
 to yarn, the default FS URI to hdfs://localhost:8020, and so on.

 While I don't see this as showstopper (for the knowledgeable user), it will
 make may users to fail miserably.

 Plus, running a an example MR job out of the box uses the local runner. If
 the user does not pay attention to the output will think the job run in the
 cluster.

 Should we do a new RC fixing this?

 Thanks.



 On Wed, Feb 12, 2014 at 5:10 PM, Zhijie Shen zs...@hortonworks.com
 wrote:

  +1 (non-binding)
 
  I download the source tar ball, built from it, ran a number of MR jobs
 on a
  single-node cluster, checked the job history from job history server.
 
 
  On Wed, Feb 12, 2014 at 2:47 PM, Jian He j...@hortonworks.com wrote:
 
   +1 (non-binding)
  
   Built from source. Ran a few MR sample jobs on a pseudo cluster.
   Everything works fine.
  
   Jian
  
  
   On Wed, Feb 12, 2014 at 2:32 PM, Aaron T. Myers a...@cloudera.com
  wrote:
  
+1 (binding)
   
I downloaded the source tar ball, checked signatures, built from the
source, ran a few of the sample jobs on a pseudo cluster. Everything
  was
   as
expected.
   
--
Aaron T. Myers
Software Engineer, Cloudera
   
   
On Tue, Feb 11, 2014 at 6:49 AM, Arun C Murthy a...@hortonworks.com
wrote:
   
 Folks,

 I've created a release candidate (rc0) for hadoop-2.3.0 that I
 would
   like
 to get released.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.3.0-rc0
 The RC tag in svn is here:

  https://svn.apache.org/repos/asf/hadoop/common/tags/release-2.3.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7
   days.

 thanks,
 Arun

 PS: Thanks to Andrew, Vinod  Alejandro for all their help in
 various
 release activities.
 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or
   entity
to
 which it is addressed and may contain information that is
  confidential,
 privileged and exempt from disclosure under applicable law. If the
   reader
 of this message is not the intended recipient, you are hereby
  notified
that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you
 have
 received this communication in error, please contact the sender
immediately
 and delete it from your system. Thank You.

   
  
   --
   CONFIDENTIALITY NOTICE
   NOTICE: This message is intended for the use of the individual or
 entity
  to
   which it is addressed and may contain information that is confidential,
   privileged and exempt from disclosure under applicable law. If the
 reader
   of this message is not the intended recipient, you are hereby notified
  that
   any printing, copying, dissemination, distribution, disclosure or
   forwarding of this communication is strictly prohibited. If you have
   received this communication in error, please contact the sender
  immediately
   and delete it from your system. Thank You.
  
 
 
 
  --
  Zhijie Shen
  Hortonworks Inc.
  http://hortonworks.com/
 
  --
  CONFIDENTIALITY NOTICE
  NOTICE: This message is intended for the use of the individual or entity
 to
  which it is addressed and may contain information that is confidential,
  privileged and exempt from disclosure under applicable law. If the reader
  of this message is not the intended recipient, you are hereby notified
 that
  any printing, copying, dissemination, distribution, disclosure or
  forwarding of this communication is strictly prohibited. If you have
  received this communication in error, please contact the sender
 immediately
  and delete it from your system. Thank You.
 



 --
 Alejandro

Re: Re-swizzle 2.3

[Aaron T. Myers]

There's still ongoing discussion on HDFS-4858 and I don't think we should
hold up 2.3.0 for that. IMO we should target that for 2.3.1 or 2.4.0.

--
Aaron T. Myers
Software Engineer, Cloudera


On Mon, Feb 10, 2014 at 5:53 PM, Konstantin Shvachko
shv.had...@gmail.comwrote:

 Sorry for the last minute request.
 Can we add HDFS-4858 to the release, please?
 It solves pretty important bug related to failover.
 I can commit momentarily if there are no objections.

 Thanks,
 --Konstantin


 On Mon, Feb 10, 2014 at 4:50 PM, Aaron T. Myers a...@cloudera.com wrote:

  Just committed a fix for HDFS-5921 to branch-2.3.
 
  Fire away.
 
  --
  Aaron T. Myers
  Software Engineer, Cloudera
 
 
  On Mon, Feb 10, 2014 at 1:34 PM, Aaron T. Myers a...@cloudera.com
 wrote:
 
   OK. I think I should be able to get it in by 6pm PT, thanks to a quick
 +1
   from Andrew, but certainly don't let it hold up the train if for some
   reason it takes longer than that.
  
   --
   Aaron T. Myers
   Software Engineer, Cloudera
  
  
   On Mon, Feb 10, 2014 at 12:04 PM, Arun C Murthy a...@hortonworks.com
  wrote:
  
   Looks like we are down to 0 blockers; I'll create rc0 tonight.
  
   ATM - Your call, you have until 6pm tonight to get this in.
  
   thanks,
   Arun
  
   On Feb 10, 2014, at 11:44 AM, Aaron T. Myers a...@cloudera.com
  wrote:
  
I just filed an issue for the fact that browsing the FS from the NN
 is
broken if you have a directory with the sticky bit set:
   
https://issues.apache.org/jira/browse/HDFS-5921
   
I didn't set this to be targeted for 2.3 because it doesn't seem
 like
  a
_blocker_ to me, but if we're not going to get 2.3 out today anyway,
  I'd
like to put this in. It's a small fix, and since many people have
 the
sticky bit set on /tmp, they won't be able to browse any of the FS
hierarchy from the NN without this fix.
   
--
Aaron T. Myers
Software Engineer, Cloudera
   
   
On Fri, Feb 7, 2014 at 12:45 PM, Vinod Kumar Vavilapalli 
   vino...@apache.org
wrote:
   
Heres what I've done:
- Reverted YARN-1493,YARN-1490,YARN-1041,
YARN-1166,YARN-1566,YARN-1689,YARN-1661 from branch-2.3.
- Updated YARN's CHANGES.txt in trunk, branch-2 and branch-2.3.
- Updated these JIRAs to have 2.4 as the fix-version.
- Compiled branch-2.3.
   
Let me know if you run into any issues caused by this revert.
   
Thanks,
+Vinod
   
   
On Fri, Feb 7, 2014 at 11:41 AM, Vinod Kumar Vavilapalli 
vino...@apache.org
wrote:
   
Haven't heard back from Jian. Reverting the set from branch-2.3
   (only).
Tx
for the offline list.
   
+Vinod
   
   
On Fri, Feb 7, 2014 at 9:08 AM, Alejandro Abdelnur 
  t...@cloudera.com
wrote:
   
Vinod, I have the patches to revert most of the JIRAs, the first
   batch,
I'll send them off line to you.
   
Thanks.
   
   
On Thu, Feb 6, 2014 at 8:56 PM, Vinod Kumar Vavilapalli
vino...@apache.orgwrote:
   
   
Thanks. please post your findings, Jian wrote this part of the
  code
and
between him/me, we can take care of those issues.
   
+1 for going ahead with the revert on branch-2.3. I'll go do
 that
tomorrow
morning unless I hear otherwise from Jian.
   
Thanks,
+Vinod
   
   
On Feb 6, 2014, at 8:28 PM, Alejandro Abdelnur 
 t...@cloudera.com
  
wrote:
   
Hi Vinod,
   
Nothing confidential,
   
* With umanaged AMs I'm seeing the trace I've posted a couple
 of
days
ago
in YARN-1577 (
   
   
   
   
  
 
 https://issues.apache.org/jira/browse/YARN-1577?focusedCommentId=13891853page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13891853
).
   
* Also, Robert has been digging in Oozie testcases
  failing/getting
suck
with several token renewer threads, this failures happened
consistently
at
different places around the same testcases (like some file
descriptors
leaking out), reverting YARN-1490 fixes the problem. The
  potential
issue
with this is that a long running client (oozie) my run into
 this
situation
thus becoming unstable.
   
*Robert,* mind posting to YARN-1490 the jvm thread dump at the
  time
of
test
hanging?
   
After YARN-1493  YARN-1490 we have a couple of JIRAs trying to
  fix
issues
introduced by them, and we still didn't get them right.
   
Because this, the improvements driven by YARN-1493  YARN-1490
  seem
that
require more work before being stable.
   
IMO, being conservative, we should do 2.3 without them and roll
   them
with
2.4. If we want to do regular releases we will have to make
 this
kind
of
calls, else we will start dragging the releases.
   
Sounds like a plan?
   
Thanks.
   
   
   
On Thu, Feb 6, 2014 at 6:27 PM, Vinod Kumar Vavilapalli
vino...@apache.orgwrote:
   
Hey
   
I am not against removing them

Re: Re-swizzle 2.3

[Aaron T. Myers]

I just committed HADOOP-10310 to branch-2.3, so we're good to go there.
(Thanks to Andrew and Daryn for the prompt reviews.)

--
Aaron T. Myers
Software Engineer, Cloudera


On Wed, Jan 29, 2014 at 6:52 PM, Aaron T. Myers a...@cloudera.com wrote:

 I just filed this JIRA as a blocker for 2.3:
 https://issues.apache.org/jira/browse/HADOOP-10310

 The tl;dr is that JNs will not work with security enabled without this
 fix. If others don't think that supporting QJM with security enabled
 warrants a blocker for 2.3, then we can certainly lower the priority, but
 it seems pretty important to me.

 Best,
 Aaron

 --
 Aaron T. Myers
 Software Engineer, Cloudera


 On Wed, Jan 29, 2014 at 6:24 PM, Andrew Wang andrew.w...@cloudera.comwrote:

 I just finished tuning up branch-2.3 and fixing up the HDFS and Common
 CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a few
 JIRAs committed between the swizzle and now where the fix version was 2.3
 but weren't in branch-2.3.

 I think the only two HDFS and Common JIRAs that are marked for 2.4 are
 these:

 HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi and a
 doAs on a secure cluster
 HDFS-5781 Use an array to record the mapping between FSEditLogOpCode and
 the corresponding byte value

 Jing, these both look safe to me if you want to merge them back, or I can
 just do it.

 Thanks,
 Andrew

 On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting cutt...@apache.org wrote:
 
  On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe jl...@yahoo-inc.com
 wrote:
It is a bit concerning that the JIRA history showed that the target
 version
   was set at some point in the past but no record of it being cleared.
 
  Perhaps the version itself was renamed?
 
  Doug

[jira] [Created] (HADOOP-10310) SaslRpcServer should be initialized even when no secret manager present

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-10310:
---

 Summary: SaslRpcServer should be initialized even when no secret 
manager present
 Key: HADOOP-10310
 URL: https://issues.apache.org/jira/browse/HADOOP-10310
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.3.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
Priority: Blocker


HADOOP-8983 made a change which caused the SaslRpcServer not to be initialized 
if there is no secret manager present. This works fine for most Hadoop daemons 
because they need a secret manager to do their business, but JournalNodes do 
not. The result of this is that JournalNodes are broken and will not handle 
RPCs in a Kerberos-enabled environment, since the SaslRpcServer will not be 
initialized.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Re: Re-swizzle 2.3

[Aaron T. Myers]

I just filed this JIRA as a blocker for 2.3:
https://issues.apache.org/jira/browse/HADOOP-10310

The tl;dr is that JNs will not work with security enabled without this fix.
If others don't think that supporting QJM with security enabled warrants a
blocker for 2.3, then we can certainly lower the priority, but it seems
pretty important to me.

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera


On Wed, Jan 29, 2014 at 6:24 PM, Andrew Wang andrew.w...@cloudera.comwrote:

 I just finished tuning up branch-2.3 and fixing up the HDFS and Common
 CHANGES.txt in trunk, branch-2, and branch-2.3. I had to merge back a few
 JIRAs committed between the swizzle and now where the fix version was 2.3
 but weren't in branch-2.3.

 I think the only two HDFS and Common JIRAs that are marked for 2.4 are
 these:

 HDFS-5842 Cannot create hftp filesystem when using a proxy user ugi and a
 doAs on a secure cluster
 HDFS-5781 Use an array to record the mapping between FSEditLogOpCode and
 the corresponding byte value

 Jing, these both look safe to me if you want to merge them back, or I can
 just do it.

 Thanks,
 Andrew

 On Wed, Jan 29, 2014 at 1:21 PM, Doug Cutting cutt...@apache.org wrote:
 
  On Wed, Jan 29, 2014 at 12:30 PM, Jason Lowe jl...@yahoo-inc.com
 wrote:
It is a bit concerning that the JIRA history showed that the target
 version
   was set at some point in the past but no record of it being cleared.
 
  Perhaps the version itself was renamed?
 
  Doug

Re: [VOTE] Merge HDFS-4949 to trunk

[Aaron T. Myers]

I don't necessarily disagree with the general questions about the
procedural issues of merge votes. Thanks for bringing that up in the other
thread you mentioned. To some extent it seems like much of this has been
based on custom, and if folks feel that more precisely defining the merge
vote process is warranted, then I think we should take that up over on that
thread.

With regard to this particular merge vote, I've spoken with Chris offline
about his feelings on this. He said that he is not dead-set on restarting
the vote, though he suspects that others may be. It seems to me the
remaining unfinished asks (e.g. updating the design doc) can reasonably be
done either after this vote but before the merge to trunk proper, or could
even reasonably be done after merging to trunk.

Given that, I'll lend my +1 to this merge. I've been reviewing the branch
pretty consistently since work started on it, and have personally
run/tested several builds of it along the way. I've also reviewed the
design thoroughly. The implementation, overall design, and API seem to me
plenty stable enough to be merged into trunk. I know that there remains a
handful of javac warnings in the branch that aren't in trunk, but I trust
those will be taken care of before the merge.

If anyone out there does feel strongly that this merge vote should be
restarted, then please speak up soon. Again, we can restart the vote if
need be, but I honestly think we'll gain very little by doing so.

Best,
Aaron


On Fri, Oct 25, 2013 at 5:45 AM, Chris Nauroth cnaur...@hortonworks.comwrote:

 Hi Andrew,

 I've come to the conclusion that I'm very confused about merge votes.  :-)
  It's not just about HDFS-4949.  I'm confused about all merge votes.
  Rather than muddy the waters here, I've started a separate discussion on
 common-dev.

 I do agree with the general plan outlined here, and I will comment directly
 on the HDFS-4949 jira with a binding +1 when I see that we've completed
 that plan.

 Chris Nauroth
 Hortonworks
 http://hortonworks.com/



 On Wed, Oct 23, 2013 at 2:18 PM, Andrew Wang andrew.w...@cloudera.com
 wrote:

  Hey Chris,
 
  Right now we're on track to have all of those things done by tomorrow.
  Since the remaining issues are either not technical or do not involve
 major
  changes, I was hoping we could +1 this merge vote in the spirit of +1
  pending jenkins. We've gotten clean unit test runs on upstream Jenkins
 as
  well, so the only fixups we should need for test-patch.sh are findbugs
 and
  javac (which are normally pretty trivial to clean up). Of course, all of
  your listed prereqs and test-patch would be taken care of before actually
  merging to trunk.
 
  So, we can reset the vote if you feel strongly about this, but it seems
  like the only real result will be delaying the merge by a week.
 
  Thanks,
  Andrew
 
 
  On Wed, Oct 23, 2013 at 1:03 PM, Chris Nauroth cnaur...@hortonworks.com
  wrote:
 
   I've received some feedback that we haven't handled this merge vote the
   same as other comparable merge votes, and that the vote should be reset
   because of this.
  
   The recent custom is that we only call for the merge vote after all
   pre-requisites have been satisfied.  This would include committing to
 the
   feature branch all patches that the devs deem necessary before the code
   lands in trunk, posting a test plan, posting an updated design doc in
  case
   implementation choices diverged from the original design doc, and
  getting a
   good test-patch run from Jenkins on the merge patch.  This was the
  process
   followed for other recent major features like HDFS-2802 (snapshots),
   HDFS-347 (short-circuit reads via sharing file descriptors), and
   HADOOP-8562 (Windows compatibility).  In this thread, we've diverged
 from
   that process by calling for a vote on a branch that hasn't yet
 completed
   the pre-requisites and stating a plan for work to be done before the
  merge.
  
   I still support this work, but can we please restart the vote after the
   pre-requisites have landed in the branch?
  
   Chris Nauroth
   Hortonworks
   http://hortonworks.com/
  
  
  
   On Fri, Oct 18, 2013 at 1:37 PM, Chris Nauroth 
 cnaur...@hortonworks.com
   wrote:
  
+1
   
Sounds great!
   
Regarding testing caching+federation, this is another thing that I
 had
intended to pick up as part of HDFS-5149.  I'm not sure if I can get
  this
done in the next 7 days, so I'll keep you posted.
   
Chris Nauroth
Hortonworks
http://hortonworks.com/
   
   
   
On Fri, Oct 18, 2013 at 11:15 AM, Colin McCabe 
 cmcc...@alumni.cmu.edu
   wrote:
   
Hi Chris,
   
I think it's feasible to complete those tasks in the next 7 days.
Andrew is on HDFS-5386.
   
The test plan document is a great idea.  We'll try to get that up
early next week.  We have a lot of unit tests now, clearly, but some
manual testing is important too.
   
If we discover any issues during testing, then 

Re: [DISCUSS] What is the purpose of merge vote threads?

[Aaron T. Myers]

On Thu, Oct 24, 2013 at 3:46 PM, Doug Cutting cutt...@apache.org wrote:

 Here's my take, FWIW.  The entire project needs to determine whether
 it is willing to take on the maintenance of code developed in a
 branch.  This vote needs the widest audience.  On the other hand,
 discussion on the umbrella Jira for the branch concerns the precise
 details of the merge commit.  Even if the project is generally willing
 to accept maintenance of the code in a branch, committing it should
 not break the build that day.  So fine-tuning the precise details of
 the merge often happens in Jira rather than as a part of the formal
 merge vote.  Sometimes these are determined in parallel.

 Since a -1 must always be accompanied by a rationale, it should be
 clear whether such a vote concerns a fine point of the specific patch
 that's easily corrected or whether it's about a fundamental problem
 with the branch that will take longer to address.  Either sort might
 be cast in either place.  A fine-point objection shouldn't reset
 voting clocks and a fundamental objection concerns things that cannot
 be fixed in a voting window.  If something lies in the middle then
 should be discussion until consensus is reached about whether the
 merge date need be delayed, voting restarted later, etc.  I don't see
 a need for a more rigid policy here since we haven't seen things
 running amok around branch merges due to a lack of policy.

 Is that consistent with other folks understanding?


+1, I agree with all of this, and this is how I've always understood the
merge vote process myself.

Best,
Aaron

Re: [VOTE] Merge HDFS-4949 to trunk

[Aaron T. Myers]

On Thu, Oct 24, 2013 at 6:18 AM, Andrew Wang andrew.w...@cloudera.comwrote:

 Right now we're on track to have all of those things done by tomorrow.
 Since the remaining issues are either not technical or do not involve major
 changes, I was hoping we could +1 this merge vote in the spirit of +1
 pending jenkins. We've gotten clean unit test runs on upstream Jenkins as
 well, so the only fixups we should need for test-patch.sh are findbugs and
 javac (which are normally pretty trivial to clean up). Of course, all of
 your listed prereqs and test-patch would be taken care of before actually
 merging to trunk.

 So, we can reset the vote if you feel strongly about this, but it seems
 like the only real result will be delaying the merge by a week.


I agree with this. Chris raised some concerns 6 days ago, but seems like
these have all been addressed since then. Resetting the vote would seem to
serve little purpose except to delay the merge by another week. If the
merge vote were to be restarted, I'd expect that we'd quickly see the
requisite three +1s be cast, and then we'd wait around for 7 days.

Chris, does this make sense to you? Appreciate a prompt response since I
believe this vote is supposed to close at midnight tonight.

Thanks folks.

Best,
Aaron

[jira] [Created] (HADOOP-10070) RPC client doesn't use per-connection conf to determine server's expected Kerberos principal name

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-10070:
---

 Summary: RPC client doesn't use per-connection conf to determine 
server's expected Kerberos principal name
 Key: HADOOP-10070
 URL: https://issues.apache.org/jira/browse/HADOOP-10070
 Project: Hadoop Common
  Issue Type: Bug
  Components: security
Affects Versions: 2.2.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


Currently, RPC client caches the {{Configuration}} object that was passed in to 
its constructor and uses that same conf for every connection it sets up 
thereafter. This can cause problems when security is enabled if the 
{{Configuration}} object provided when the first RPC connection was made does 
not contain all possible entries for all server principals that will later be 
used by subsequent connections. When this happens, it will result in later RPC 
connections incorrectly failing with the error Failed to specify server's 
Kerberos principal name even though the principal name was specified in the 
{{Configuration}} object provided on later RPC connection attempts.

I believe this means that we've inadvertently reintroduced HADOOP-6907.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Re: [VOTE] Release Apache Hadoop 2.2.0

[Aaron T. Myers]

+1 (binding)

Downloaded the release, built from tarball, tested a single node cluster.
Everything worked as expected.

--
Aaron T. Myers
Software Engineer, Cloudera


On Mon, Oct 7, 2013 at 12:00 AM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

 I've created a release candidate (rc0) for hadoop-2.2.0 that I would like
 to get released - this release fixes a small number of bugs and some
 protocol/api issues which should ensure they are now stable and will not
 change in hadoop-2.x.

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.2.0-rc0
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.2.0-rc0

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 P.S.: Thanks to Colin, Andrew, Daryn, Chris and others for helping nail
 down the symlinks-related issues. I'll release note the fact that we have
 disabled it in 2.2. Also, thanks to Vinod for some heavy-lifting on the
 YARN side in the last couple of weeks.





 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Coverity Scan (MAPREDUCE-5032)

[Aaron T. Myers]

I strongly recommend that we take this conversation over to the
(committers-only) secur...@hadoop.apache.org mailing list. In general we
try to follow the Apache recommendations when it comes to addressing
security issues, which involves not publicly disclosing the vulnerability
until there are released version(s) with the issue(s) addressed.

Best,
Aaron


On Mon, Aug 26, 2013 at 8:24 PM, Jon Jarboe jjar...@coverity.com wrote:

 Thanks for the interest.  I'm in the process of building the 2.1.0 beta as
 suggested by Roman.

 Jon
 (214) 531-3496


  -Original Message-
  From: Ottenheimer, Davi [mailto:davi.ottenhei...@emc.com]
  Sent: Monday, August 26, 2013 1:11 PM
  To: common-dev@hadoop.apache.org
  Subject: RE: Coverity Scan (MAPREDUCE-5032)
 
  Perhaps open the JIRA with only a reference/link to the Coverity report,
 and
  limit access to only those working on the issues.
 
  Full disclosure, update the JIRA, after fix.
 
  --
  Davi Ottenheimer
  Senior Director of Trust
  EMC Corporation
  davi.ottenhei...@emc.com | @daviottenheimer | +1-415-271-6259
  blog: http://www.flyingpenguin.com/
 
 
   -Original Message-
   From: shaposh...@gmail.com [mailto:shaposh...@gmail.com] On Behalf
  Of
   Roman Shaposhnik
   Sent: Monday, August 26, 2013 10:50 AM
   To: common-dev@hadoop.apache.org
   Subject: Re: Coverity Scan (MAPREDUCE-5032)
  
   On Mon, Aug 26, 2013 at 10:43 AM, Vinod Kumar Vavilapalli
   vino...@apache.org wrote:
   
Can you file a JIRA and attach the report there? That is the best
way to
   move this forward.
  
   Last time I was involved in a Coverity scan was when they scanned
   another project I'm committer on (FFmpeg). The lesson there was that
   the value you get out of browsing on their site
   https://scan.coverity.com is immeasurably higher than from any static
  report that can be attached to a JIRA.
  
   Also, at least in FFmpeg's case, Coverity identified a few things that
   could've been used as potential exploits so it made perfect sense to
   have a white-list of project members who could get access to the
   initial report instead of going all public with it to begin with
   (which would happen if it just gets attached to a JIRA in its
 entirety).
  
   Just my 2c worth of working with them in the past.
  
   Thanks,
   Roman.
 

[jira] [Resolved] (HADOOP-9901) Cannot start Datanode,yarn, namenodemanager but able to start namenode

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-9901.


Resolution: Invalid

Apache JIRA is for reporting bugs or filing proposed enhancements or features. 
I recommend you email one of the user mailing lists with this question, e.g. 
u...@hadoop.apache.org or u...@bigtop.apache.org.

 Cannot start Datanode,yarn, namenodemanager but able to start namenode
 --

 Key: HADOOP-9901
 URL: https://issues.apache.org/jira/browse/HADOOP-9901
 Project: Hadoop Common
  Issue Type: Bug
 Environment: Opensuse 12.3/ i3 processor lenovo thinkpad edge
Reporter: Devendra Majhi

 Followed the steps provided in the link 
 https://cwiki.apache.org/confluence/display/BIGTOP/How+to+install+Hadoop+distribution+from+Bigtop+0.5.0;
  to set up hadoop. But failed in step Running Hadoop step 3 giving error as 
 Job for hadoop-yarn-nodemanager.service failed. See 'systemctl status 
 hadoop-yarn-nodemanager.service' and 'journalctl -n' for details
 same error for datanode also.
 Can anyone please throw some light on this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

CVE-2013-2192: Apache Hadoop Man in the Middle Vulnerability

[Aaron T. Myers]

Hello,

Please see below for the official announcement of a serious security
vulnerability which has been discovered and subsequently fixed in Apache
Hadoop releases.

Best,
Aaron

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

CVE-2013-2192: Apache Hadoop Man in the Middle Vulnerability

Severity: Severe

Vendor: The Apache Software Foundation

Versions Affected:
All versions of Hadoop 2.x prior to Hadoop 2.0.6-alpha.
All versions of Hadoop 0.23.x prior to Hadoop 0.23.9.
All versions of Hadoop 1.x prior to Hadoop 1.2.1.

Users affected: Users who have enabled Hadoop's Kerberos security features.

Impact: RPC traffic from clients, potentially including authentication
credentials, may be intercepted by a malicious user with access to run
tasks or containers on a cluster.

Description:
The Apache Hadoop RPC protocol is intended to provide bidirectional
authentication between clients and servers. However, a malicious server or
network attacker can unilaterally disable these authentication checks. This
allows for potential reduction in the configured quality of protection of
the RPC traffic, and privilege escalation if authentication credentials are
passed over RPC.

Mitigation:
Users of Hadoop 1.x versions should immediately upgrade to 1.2.1 or later.
Users of Hadoop 0.23.x versions should immediately upgrade to 0.23.9 or
later.
Users of Hadoop 2.x versions prior to 2.0.6-alpha should immediately
upgrade to 2.0.6-alpha or later.

Credit: This issue was discovered by Kyle Leckie of Microsoft and Aaron T.
Myers of Cloudera.
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJSF84CAAoJECEaGfB4kTjfI7kH/0v4JJ992vGV4esnAKgNnTmn
A7GCj2zT7KFgF7ii6G6+5Xny9AnISTZWfMII/Szs5qaFgiaByvsNR5FoN+o5BS8s
vPWU8v5f3/cayacQgl8vxUiTlkXYZWQX+3V+8RTqAR3fPsr9IUMse4hOEcXvAjHr
 gDeWKiQaXRRhVjfmTLll1OWuKT8PmVar3qcbsg3vo/tj/yjOoVEfhV3DMOdIi+ES
pWtTxs5/fB8t+wA4hdY1r6trE7X6fys9NYC11jp83ej+ecjnHy7kmKGl41WESD+G
GOhAPYCMS9D29KGs2c6q0xCqi22R0klTs9d3Z/f7F5htGfBSAfAOpC6xPJ66/ZY=
 =4+in
-END PGP SIGNATURE-

Re: [VOTE] Release Apache Hadoop 2.0.6-alpha (RC1)

[Aaron T. Myers]

+1 (binding)

I downloaded the bits, set up a 4-node cluster, and ran some example jobs.
Looks good to me.


--
Aaron T. Myers
Software Engineer, Cloudera


On Thu, Aug 15, 2013 at 10:29 PM, Konstantin Boudnik c...@apache.org wrote:

 All,

 I have created a release candidate (rc1) for hadoop-2.0.6-alpha that I
 would
 like to release.

 This is a stabilization release that includes fixed for a couple a of
 issues
 as outlined on the security list.

 The RC is available at:
 http://people.apache.org/~cos/hadoop-2.0.6-alpha-rc1/
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.6-alpha-rc1

 The maven artifacts are available via repository.apache.org.

 The only difference between rc0 and rc1 is ASL added to releasenotes.html
 and
 updated release dates in CHANGES.txt files.

 Please try the release bits and vote; the vote will run for the usual 7
 days.

 Thanks for your voting
   Cos

Re: [VOTE] Release Apache Hadoop 2.1.0-beta

[Aaron T. Myers]

I was evaluating the release bits when I noticed that the change done in
HDFS-4763 to add support for starting the HDFS NFSv3 gateway, which is
marked with a fix version of 2.1.0-beta and included in the release notes
of RC2, is not in fact included in the RC2 release bits. It looks to me
like the change is included in branch-2.1-beta, but not branch-2.1.0-beta.

Particularly since the release notes in RC2 are incorrect in claiming that
this change is in this release, it seems like a pretty serious
issue. Ordinarily I'd say that this issue should result in a new RC, and I
would vote -1 on RC2. But, given the previous discussion that folks are
interested in releasing 2.1.0-beta with several fairly substantial bugs
that we already know about, I'll withhold my vote. If RC2 ends up getting
released as-is, we should be sure to change the fix version field on that
JIRA to be correct.

--
Aaron T. Myers
Software Engineer, Cloudera


On Thu, Aug 15, 2013 at 2:15 PM, Arun C Murthy a...@hortonworks.com wrote:

 Folks,

 I've created a release candidate (rc2) for hadoop-2.1.0-beta that I would
 like to get released - this fixes the bugs we saw since the last go-around
 (rc1).

 The RC is available at:
 http://people.apache.org/~acmurthy/hadoop-2.1.0-beta-rc2/
 The RC tag in svn is here:
 http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.0-beta-rc2

 The maven artifacts are available via repository.apache.org.

 Please try the release and vote; the vote will run for the usual 7 days.

 thanks,
 Arun

 --
 Arun C. Murthy
 Hortonworks Inc.
 http://hortonworks.com/



 --
 CONFIDENTIALITY NOTICE
 NOTICE: This message is intended for the use of the individual or entity to
 which it is addressed and may contain information that is confidential,
 privileged and exempt from disclosure under applicable law. If the reader
 of this message is not the intended recipient, you are hereby notified that
 any printing, copying, dissemination, distribution, disclosure or
 forwarding of this communication is strictly prohibited. If you have
 received this communication in error, please contact the sender immediately
 and delete it from your system. Thank You.

Re: Fostering a Hadoop security dev community

[Aaron T. Myers]

On Thu, Jun 20, 2013 at 10:46 AM, Larry McCay lmc...@hortonworks.comwrote:

 I think that we could let the security vulnerability list know about it for
 one thing.


Small clarification - note that security@hadoop.a.o is ostensibly only
for Hadoop project security vulnerabilities - it's not really intended to
be for discussion broader than just the Hadoop project proper.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Fostering a Hadoop security dev community

[Aaron T. Myers]

I'm in favor of this in general, though I do think the proper way to do it
isn't obvious to me, given the cross-project nature of the goal.

On Thu, Jun 20, 2013 at 1:01 PM, Andrew Purtell apurt...@apache.org wrote:

 On Thu, Jun 20, 2013 at 10:31 AM, Alejandro Abdelnur t...@cloudera.com
 wrote:

  Is this restricted to the Hadoop project itself or the intention is to
  cover the whole Hadoop ecosystem? If the later, how are you planning to
  engage and sync up with the different projects?
 

 The intent is to cover the entire Hadoop ecosystem. How specifically to
 structure the work and engage different projects would depend on what facet
 of security is being addressed. I think it would be awesome if the Hadoop
 PMC is willing to lend resources for an ongoing virtual meetup on security
 concerns (a meetup ecosystem wide) that cross-cut everywhere, and that
 makes sense, at least to me, because in many cases we could build from the
 core outward and propose uptake of artifacts that solve a common problem on
 project specific JIRAs.


Sorry, what exactly do you mean by meetup ?

I think in general it makes sense for this effort to be hosted by the
Hadoop project proper, given that much of the security of the rest of the
system is built on top of the libraries in Hadoop Common. Note, however,
that certainly not all of what are generally considered the Hadoop
ecosystem projects build their security using only what's in Hadoop
Common, e.g. Hive makes extensive use of Thrift and Thrift's SASL
implementation.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [ANNOUNCE] New Hadoop PMC members

[Aaron T. Myers]

Welcome aboard, folks!

Best,
Aaron


On Mon, Jun 10, 2013 at 8:38 AM, Tom White t...@cloudera.com wrote:

 On behalf of the Apache Hadoop PMC, I'm pleased to announce the
 addition of the following new Hadoop PMC members:

 * Daryn Sharp
 * Hitesh Shah
 * Jonathan Eagles
 * Kihwal Lee
 * Luke Lu
 * Steve Loughran
 * Uma Maheswara Rao G

 Thank you for all of your work on the project! Please join me in welcoming
 them.

 Cheers,
 Tom

[jira] [Resolved] (HADOOP-9633) An incorrect data node might be added to the network topology, an exception is thrown though

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-9633.


Resolution: Fixed

Resolving as a duplicate of HDFS-4521. If we want to fix this in branch-1 as 
well, let's do it via that JIRA.

 An incorrect data node might be added to the network topology, an exception 
 is thrown though
 

 Key: HADOOP-9633
 URL: https://issues.apache.org/jira/browse/HADOOP-9633
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 1.3.0
Reporter: Xi Fang
Priority: Minor

 In NetworkTopology#add(Node node), an incorrect node may be added to the 
 cluster even if an exception is thrown.
 This is the original code:
 {code}
   if (clusterMap.add(node)) {
 LOG.info(Adding a new node: +NodeBase.getPath(node));
 if (rack == null) {
   numOfRacks++;
 }
 if (!(node instanceof InnerNode)) {
   if (depthOfAllLeaves == -1) {
 depthOfAllLeaves = node.getLevel();
   } else {
 if (depthOfAllLeaves != node.getLevel()) {
   LOG.error(Error: can't add leaf node at depth  +
   node.getLevel() +  to topology:\n + oldTopoStr);
   throw new InvalidTopologyException(Invalid network topology.  
 +
   You cannot have a rack and a non-rack node at the same  +
   level of the network topology.);
 }
   }
 }
 {code}
 This is a potential bug, because a wrong leaf node is already added to the 
 cluster before throwing the exception. However, we can't check this 
 (depthOfAllLeaves != node.getLevel()) before if (clusterMap.add(node)), 
 because node.getLevel() will work correctly only after clusterMap.add(node) 
 has been executed.
 A possible solution to this is checking the depthOfAllLeaves in 
 clusterMap.add(node). Note that this is a recursive call. A check should be 
 put at the bottom of this recursive call. If check fails, don't add this leaf 
 and all its upstream racks. 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9617) HA HDFS client is too strict with validating URI authorities

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-9617:
--

 Summary: HA HDFS client is too strict with validating URI 
authorities
 Key: HADOOP-9617
 URL: https://issues.apache.org/jira/browse/HADOOP-9617
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs, ha
Affects Versions: 2.0.5-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


HADOOP-9150 changed the way FS URIs are handled to prevent attempted DNS 
resolution of logical URIs. This has the side effect of changing the way Paths 
are verified when passed to a FileSystem instance created with an authority 
that differs from the authority of the Path. Previous to HADOOP-9150, a default 
port would be added to either authority in the event that either URI did not 
have a port. Post HADOOP-9150, no default port is added. This means that a 
FileSystem instance created using the URI hdfs://ha-logical-uri:8020 will no 
longer process paths containing just the authority hdfs://ha-logical-uri, and 
will throw an error like the following:

{noformat}
java.lang.IllegalArgumentException: Wrong FS: 
hdfs://ns1/user/hive/warehouse/sample_07/sample_07.csv, expected: 
hdfs://ns1:8020
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:625)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:173)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:249)
at 
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:82)
{noformat}

Though this is not necessarily incorrect behavior, it is a 
backward-incompatible change that at least breaks certain clients' ability to 
connect to an HA HDFS cluster.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[ANNOUNCE] New Hadoop Committers

[Aaron T. Myers]

On behalf of the Apache Hadoop PMC, I'd like to announce the addition of a
few new committers to the Apache Hadoop project:

* Brandon Li
* Chris Nauroth
* Colin Patrick McCabe
* Ivan Mitic
* Jing Zhao

We appreciate all of their contributions to date and look forward to many
more.

Please join me in welcoming them.

Best,
Aaron

Re: [PROPOSAL] change in bylaws to remove Release Plan vote

[Aaron T. Myers]

+1

I've always found the Release Plan votes a bit bizarre, and the fact that
we've gone through many releases that did not have a corresponding Release
Plan vote suggest to me that we should just scrap them.


--
Aaron T. Myers
Software Engineer, Cloudera


On Tue, May 21, 2013 at 5:37 PM, Chris Douglas cdoug...@apache.org wrote:

 +1

 Thanks for taking care of this, Matt. -C

 On Tue, May 21, 2013 at 2:10 PM, Matt Foley ma...@apache.org wrote:
  Hi all,
  This has been a side topic in several email threads recently.  Currently
 we
  have an ambiguity.  We have a tradition in the dev community that any
  committer can create a branch, and propose release candidates from it.
  Yet
  the Hadoop bylaws say that releases have to be planned in advance, the
 plan
  needs to be voted on, and presumably can be denied.
 
  Apache policies (primarily here http://www.apache.org/dev/release.html
   and here http://www.apache.org/foundation/voting.html, with
  non-normative commentary
  here
 http://incubator.apache.org/guides/releasemanagement.html#best-practice)
  are very clear on how Releases have to be approved, and our bylaws are
  consistent with those policies.  But Apache policies don't say anything
  I've found about Release Plans, nor about voting on Release Plans.
 
  I propose the following change, to remove Release Plan votes, and give a
  simple definition of Release Manager role.  I'm opening discussion with
  this proposal, and will put it to a vote if we seem to be getting
  consensus.  Here's the changes I suggest in the
  Bylawshttp://hadoop.apache.org/bylaws.html
   document:
 
  ===
 
  1. In the Decision Making : Actions section of the Bylaws, the
  following text is removed:
 
  ** Release Plan*
 
  Defines the timetable and actions for a release. The plan also nominates
 a
  Release Manager.
 
  Lazy majority of active committers
 
 
  2. In the Roles and Responsibilities section of the Bylaws, an
 additional
  role is defined:
 
  ** Release Manager*
 
  A Release Manager (RM) is a committer who volunteers to produce a Release
  Candidate according to
  HowToReleasehttps://wiki.apache.org/hadoop/HowToRelease.
   The RM shall publish a Release Plan on the *common-dev@* list stating
 the
  branch from which they intend to make a Release Candidate, at least one
  week before they do so. The RM is responsible for building consensus
 around
  the content of the Release Candidate, in order to achieve a successful
  Product Release vote.
 
  ===
 
  Please share your views.
  Best regards,
  --Matt (long-time release manager)

Re: Web UI for Active and Standby NN do not agree on safe mode setting

[Aaron T. Myers]

Hi Scott,

This is actually working as intended. NN safemode is not a system-wide
state - it can be set independently on both the active and standby NNs.
DistributedFileSystem is indeed doing the right thing in that it will
return the safemode value of the active NN, but on the web UI each NN
should display its own safemode state, which might differ from the other's.
You might take a look at the discussion/patch in
https://issues.apache.org/jira/browse/HDFS-3507 for some more background
here.

Best,
Aaron


--
Aaron T. Myers
Software Engineer, Cloudera


On Fri, May 10, 2013 at 8:53 AM, Scott Forman lsforman@gmail.comwrote:

 When safe mode is entered using the dfsadmin CLI command, only one of the
 Web UI's for the Namenodes in an HA environment seems to show that safe
 mode is on.  But retrieving the safe mode setting through the CLI (i.e.
 running hdfs dfsadmin -safemode get) seems to display the correct setting
 when run from any node in the cluster.

 I noticed this problem when upgrading from a non-HA environment to an HA
 environment, and in this case, the NN in the pre-upgraded system always
 shows the correct setting, regardless if it is active or standby, while the
 newly added NN never displays that safe mode is on.

 The dfsadmin command uses DistributedFileSystem to get the setting, while
 the Web UI seems to use FSNamesystem.

 Is this a bug, or do both UI's need to be consulted to determine the safe
 mode setting?

 Thanks,
 Scott

[jira] [Resolved] (HADOOP-9484) Genetic Algorithm Library for Hadoop

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9484?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-9484.


Resolution: Invalid

Hi Vaibhav,

While this is potentially very valuable work, this probably isn't something 
we'd want to include in core Hadoop. Per Colin's comment, I would take a look 
at Mahout and see if they would be interested in committing/releasing your 
contribution.

Best,
Aaron

 Genetic Algorithm Library for Hadoop
 

 Key: HADOOP-9484
 URL: https://issues.apache.org/jira/browse/HADOOP-9484
 Project: Hadoop Common
  Issue Type: New Feature
  Components: contrib/hod
 Environment: Linux Operating System
Reporter: Vaibhav Singh Rajput

 Developed a Genetic Algorithm Library for Hadoop. Using this library, 
 problems using Genetic Algrorithm can be solved based on Hadoop MapReduce.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Transferring a JIRA

[Aaron T. Myers]

On Tue, Apr 9, 2013 at 4:49 AM, Sandy Ryza sandy.r...@cloudera.com wrote:

 Weird, I didn't think I did, but maybe because I've contributed patches?


Just FYI - this is exactly right. There's a role in JIRA called
contributor which has a few extra privileges, e.g. you can't actually
have a JIRA assigned to you until you're marked as a contributor by one
of the Hadoop JIRA admins, which all the committers are.

--
Aaron T. Myers
Software Engineer, Cloudera

[jira] [Resolved] (HADOOP-9475) Distcp issue

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-9475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-9475.


Resolution: Invalid

Resolving as invalid per Steve's comment.

 Distcp issue
 

 Key: HADOOP-9475
 URL: https://issues.apache.org/jira/browse/HADOOP-9475
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Sambit Sahoo

 2013-04-13 05:11:43,327 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded 
 the native-hadoop library
 2013-04-13 05:11:43,439 INFO org.apache.hadoop.metrics.jvm.JvmMetrics: 
 Initializing JVM Metrics with processName=MAP, sessionId=
 2013-04-13 05:11:43,750 INFO org.apache.hadoop.mapred.MapTask: 
 numReduceTasks: 0
 2013-04-13 05:11:43,981 INFO org.apache.hadoop.io.compress.zlib.ZlibFactory: 
 Successfully loaded  initialized native-zlib library
 2013-04-13 05:31:08,282 INFO org.apache.hadoop.mapred.Task: 
 Task:attempt_201302011155_224614_m_09_0 is done. And is in the process of 
 commiting
 2013-04-13 05:31:09,359 INFO org.apache.hadoop.mapred.Task: Task 
 attempt_201302011155_224614_m_09_0 is allowed to commit now
 2013-04-13 05:31:09,937 INFO org.apache.hadoop.mapred.FileOutputCommitter: 
 Saved output of task 'attempt_201302011155_224614_m_09_0' to 
 /tmp/_distcp_logs_spti36
 2013-04-13 05:31:09,939 INFO org.apache.hadoop.mapred.Task: Task 
 'attempt_201302011155_224614_m_09_0' done.
 2013-04-13 05:31:09,942 INFO org.apache.hadoop.mapred.TaskLogsTruncater: 
 Initializing logs' truncater with mapRetainSize=-1 and reduceRetainSize=-1
 I am facing some delay during disctcp from one cluster to another.
 Here i am copying snappy compressed data.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9306) Refactor UserGroupInformation to reduce branching for multi-platform support

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-9306:
--

 Summary: Refactor UserGroupInformation to reduce branching for 
multi-platform support
 Key: HADOOP-9306
 URL: https://issues.apache.org/jira/browse/HADOOP-9306
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.0.3-alpha
Reporter: Aaron T. Myers


Per Tucu's comment on HADOOP-9305, we can refactor the code for conditionally 
loading classes based on the OS version, JRE version, and bitmode to use a map 
and struct. Seems like good cleanup.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-9283) Add support for running the Hadoop client on AIX

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-9283:
--

 Summary: Add support for running the Hadoop client on AIX
 Key: HADOOP-9283
 URL: https://issues.apache.org/jira/browse/HADOOP-9283
 Project: Hadoop Common
  Issue Type: New Feature
  Components: security
Affects Versions: 2.0.2-alpha
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


The UserGroupInformation class currently supports running with either Sun Java 
or IBM Java on Windows or Linux. This JIRA is to add support for using the 
Hadoop client on AIX. Since the is no Sun Java available for AIX, only IBM Java 
will be supported on AIX.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

.m2 repo messed up on hadoop6

[Aaron T. Myers]

A few pre-commit builds have been failing recently with compile errors
which I think are due to a bad jar in the /home/jenkins/.m2 repo on
hadoop6. For example, both of these builds:

https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-HDFS-Build/3878/
https://builds.apache.org/view/G-L/view/Hadoop/job/PreCommit-HDFS-Build/3879/

Failed with this error:

[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-compiler-plugin:2.5.1:compile
(default-compile) on project hadoop-yarn-api: Compilation failure:
Compilation failure:
[ERROR] error: error reading
/home/jenkins/.m2/repository/org/glassfish/grizzly/grizzly-framework/2.1.1/grizzly-framework-2.1.1.jar;
error in opening zip file
[ERROR] error: error reading
/home/jenkins/.m2/repository/org/glassfish/grizzly/grizzly-rcm/2.1.1/grizzly-rcm-2.1.1.jar;
error in opening zip file
[ERROR] error: error reading
/home/jenkins/.m2/repository/org/glassfish/grizzly/grizzly-framework/2.1.1/grizzly-framework-2.1.1-tests.jar;
error in opening zip file

Could someone with access to the build slaves please clear out
/home/jenkins/.m2 on hadoop6? Alternatively, could I be given access
to the build slave machines so I can fix issues like this in the
future myself?

Thanks a lot.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Hadoop QA for Common JIRAs

[Aaron T. Myers]

Hey Karthik,

Thanks a lot for bringing this up. It seems that all of the recent
precommit runs for Common were all triggered manually. My hunch is that
this is a repeat of the issue we had a while back where we crossed some
threshold of number of patch available JIRAs such that we're over some JIRA
query limit.

Todd and I are looking into what the problem is and should hopefully have
it cleared up soon.


--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Dec 6, 2012 at 11:32 AM, Karthik Kambatla ka...@cloudera.comwrote:

 Hello

 Looks like the Pre-commit builds for Hadoop (common) are stalled since Dec
 3. Can someone with permissions look into it.

 Many thanks,
 Karthik

Re: trailing whitespace

[Aaron T. Myers]

OK, if folks want to do something to get rid of trailing whitespace in the
project I won't object, but it doesn't seem like that big a deal to me. A
pre-commit hook makes sense to me. I just don't want to see the QA bot flag
patches containing trailing whitespace, thus requiring more round trips on
patches.

--
Aaron T. Myers
Software Engineer, Cloudera



On Mon, Nov 26, 2012 at 10:53 AM, Radim Kolar h...@filez.com wrote:

  I've never understood why folks get worked up over a little trailing
 whitespace here and there, since you can't see it and it doesn't affect
 correctness. Spurious whitespace changes that make a review harder - those
 are annoying. Trailing whitespace inadvertently left on lines where
 legitimate changes were made in a patch - doesn't seem too harmful to me.


 Trailing whitespace is annoying because:
 if you have editor to set killing it, it will produce large patch.
 if use use scroll up at end of line, then cursor will not jump to end
 of text but some space after it, it cost you more clicks for cursor
 movement and it is annoying if it ends of split line.
 its good and standard practise to avoid it, git and other tools
 highlight it in red.
 if you use ignore whitespace in git diff, it often produces patch
 failing to apply

 Trailing whitespace can be striped by pre-commit hook.

[jira] [Reopened] (HADOOP-8884) DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load native-hadoop with error: java.lang.UnsatisfiedLinkError

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers reopened HADOOP-8884:



 DEBUG should be WARN for DEBUG util.NativeCodeLoader: Failed to load 
 native-hadoop with error: java.lang.UnsatisfiedLinkError
 -

 Key: HADOOP-8884
 URL: https://issues.apache.org/jira/browse/HADOOP-8884
 Project: Hadoop Common
  Issue Type: Bug
  Components: util
Affects Versions: 2.0.1-alpha
Reporter: Anthony Rojas
Assignee: Anthony Rojas
 Fix For: 2.0.3-alpha

 Attachments: HADOOP-8884.patch, HADOOP-8884.patch, 
 HADOOP-8884-v2.patch


 Recommending to change the following debug message and promote it to a 
 warning instead:
 12/07/02 18:41:44 DEBUG util.NativeCodeLoader: Failed to load native-hadoop 
 with error: java.lang.UnsatisfiedLinkError: 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0: /lib64/libc.so.6: version 
 `GLIBC_2.6' not found (required by 
 /usr/lib/hadoop/lib/native/libhadoop.so.1.0.0)

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Hadoop-1.0.4-rc0

[Aaron T. Myers]

+1 (binding)

I verified the signatures and checksums of both the binary and source
artifacts. I built the source and ran some basic MR jobs from both the
build I made myself and the binary artifacts you provided. Everything
worked as expected.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Oct 4, 2012 at 1:59 PM, Matt Foley ma...@apache.org wrote:

 Hi,
 There has been a request from several PMC members for a maintenance release
 of hadoop-1.0.
 Please download and test this release candidate, Hadoop-1.0.4-rc0.

 It is available at http://people.apache.org/~mattf/hadoop-1.0.4-rc0/
 or from the Nexus maven repo.

 Release notes are at
 http://people.apache.org/~mattf/hadoop-1.0.4-rc0/releasenotes.html

 Vote will run for 1 week as usual, terminating at 2pm PDT, Thur 11 Oct
 2012.

 Thank you,
 --Matt Foley
 Release Manager

Re: [VOTE] Hadoop-1.1.0 release candidate 5

[Aaron T. Myers]

+1 (binding)

I downloaded the source artifact, verified the signatures and checksums,
built the source artifact, started HDFS and MR, and ran some simple jobs.
Everything worked as expected.

--
Aaron T. Myers
Software Engineer, Cloudera



On Fri, Oct 5, 2012 at 2:31 PM, Matt Foley ma...@apache.org wrote:

 Here at long last is a votable release candidate for Hadoop 1.1.0.
 Please download and test Hadoop-1.1.0-rc5.

 It is available at
 http://people.apache.org/~mattf/hadoop-1.1.0-rc5/
 http://people.apache.org/~mattf/hadoop-1.1.0-rc05/

 or from the Nexus maven repo.

 Release notes are at
 http://people.apache.org/~mattf/hadoop-1.1.0-rc5/releasenotes.html

 Vote will run for 1 week as usual, terminating at 2:30pm PDT, Fri 12 Oct
 2012.

 Thank you,
 --Matt Foley
 Release Manager

[jira] [Resolved] (HADOOP-8591) TestZKFailoverController tests time out

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8591?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8591.


Resolution: Invalid
  Assignee: Aaron T. Myers

I looked into this today and realized that it was a problem with a particular 
Jenkins slave. Whenever a pre-commit test or nightly build was run on hadoop1, 
it would fail. Whenever it was run anywhere else, it would pass. When I logged 
in to hadoop1, I noticed that there were a bunch of pre-commit processes and 
even a nightly build that had been running for weeks or months. After killing 
these zombie processes, TestZKFailoverController now passes reliably on hadoop1.

 TestZKFailoverController tests time out
 ---

 Key: HADOOP-8591
 URL: https://issues.apache.org/jira/browse/HADOOP-8591
 Project: Hadoop Common
  Issue Type: Bug
  Components: auto-failover, ha, test
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: Aaron T. Myers
  Labels: test-fail

 Looks like the TestZKFailoverController timeout needs to be bumped.
 {noformat}
 java.lang.Exception: test timed out after 3 milliseconds
   at java.lang.Object.wait(Native Method)
   at 
 org.apache.hadoop.ha.ZKFailoverController.waitForActiveAttempt(ZKFailoverController.java:460)
   at 
 org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:648)
   at 
 org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:58)
   at 
 org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:593)
   at 
 org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:590)
   at java.security.AccessController.doPrivileged(Native Method)
   at javax.security.auth.Subject.doAs(Subject.java:396)
   at 
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1334)
   at 
 org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:590)
   at 
 org.apache.hadoop.ha.TestZKFailoverController.testOneOfEverything(TestZKFailoverController.java:575)
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Test timeouts

[Aaron T. Myers]

Hello devs,

With the commit of HADOOP-8755, we now have support for printing a thread
dump whenever a test case times out. However, this will only happen for
test cases which are annotated with a JUnit timeout, i.e. those annotated
with something like @Test(timeout=X). Unfortunately, there doesn't seem
to be an easy way to add a default JUnit timeout for all of our tests, so
some tests may time out by reaching the Surefire fork timeout, in which
case the thread dump will not be printed. You can see more discussion about
this on HADOOP-8755.

So, if you see a test case fail by reaching the Surefire fork timeout,
please file a JIRA to add a JUnit timeout for that test. If when adding a
test case you think that it might time out, please add a JUnit timeout.

Thanks,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera

Re: 答复: regarding _HOST token replacement in security hadoop

[Aaron T. Myers]

What do you have set as the fs.defaultFS in your configuration? Make sure
that that is a fully-qualified domain name.

--
Aaron T. Myers
Software Engineer, Cloudera



On Fri, Jul 27, 2012 at 1:57 PM, Arpit Gupta ar...@hortonworks.com wrote:

 That does seem to be valid issue. Could you log a jira for it.

 Thanks


 On Thu, Jul 26, 2012 at 7:32 PM, Wangwenli wangwe...@huawei.com wrote:

  Could you spent one minute to check whether below code will cause issue
 or
  not?
 
  In org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser(),
  it use socAddr.getHostName() to get _HOST,
  But in org.apache.hadoop.security.SecurityUtil.replacePattern(), in
  getLocalHostName(), it use getCanonicalHostName() to get _HOST
 
  Meanwhile I will check what you said. Thank you~
 
 
  -邮件原件-
  发件人: Arpit Gupta [mailto:ar...@hortonworks.com]
  发送时间: 2012年7月27日 10:03
  收件人: common-dev@hadoop.apache.org
  主题: Re: regarding _HOST token replacement in security hadoop
 
  you need to use HTTP/_h...@site.com as that is the principal needed by
  spnego. So you would need create the HTTP/_HOST principal and add it to
 the
  same keytab (/home/hdfs/keytab/nn.service.keytab).
 
  --
  Arpit Gupta
  Hortonworks Inc.
  http://hortonworks.com/
 
  On Jul 26, 2012, at 6:54 PM, Wangwenli wangwe...@huawei.com wrote:
 
   Thank yours response.
   I am using hadoop-2.0.0-alpha from apache site.  In which version it
  should configure with HTTP/_h...@site.com?  I think not in
  hadoop-2.0.0-alpha. Because I login successful with other principal, pls
  refer below log:
  
   2012-07-23 22:48:17,303 INFO
 
 org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
  Login using keytab /home/hdfs/keytab/nn.service.keytab, for principal
  nn/167-52-0-56.site@site
   2012-07-23 22:48:17,310 INFO
 
 org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler:
  Initialized, principal [nn/167-52-0-56.site@site] from keytab
  [/home/hdfs/keytab/nn.service.keytab]
  
  
   -邮件原件-
   发件人: Arpit Gupta [mailto:ar...@hortonworks.com]
   发送时间: 2012年7月27日 9:22
   收件人: common-dev@hadoop.apache.org
   主题: Re: regarding _HOST token replacement in security hadoop
  
   what version of hadoop are you using?
  
   also
  
   dfs.web.authentication.kerberos.principal should be set to HTTP/_
  h...@site.com
  
   --
   Arpit Gupta
   Hortonworks Inc.
   http://hortonworks.com/
  
   On Jul 26, 2012, at 6:11 PM, Wangwenli wangwe...@huawei.com wrote:
  
   Hi all,
  
I configured like below in hdfs-site.xml:
  
   property
   namedfs.namenode.kerberos.principal/name
   valuenn/_HOST@site/value
   /property
  
  
   property
 namedfs.web.authentication.kerberos.principal/name
 valuenn/_HOST@site/value
   /property
  
  
When  start up namenode, I found, namenode will use principal :
  nn/167-52-0-56@site to login, but the http server will use
  nn/167-52-0-56.site@sitemailto:nn/167-52-0-56.site@site to lgin,  so
 it
  start failed.
  
   I checked the code,
  
   Namenode will use socAddr.getHostName() to get hostname in
  org.apache.hadoop.hdfs.server.namenode.NameNode.loginAsNameNodeUser.
  
  
   But httpserver 's default hostname is 0.0.0.0, so in
  org.apache.hadoop.security.SecurityUtil.replacePattern, it will get the
  hostname by invoking getLocalHostName,there it use
 getCanonicalHostName(),
  
   I think this inconsistent is wrong,  can someone confirm this? Need
  raise one bug ?
  
   Thanks
  
  
 
 

[CVE-2012-3376] Apache Hadoop HDFS information disclosure vulnerability

[Aaron T. Myers]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Hello,

Users of Apache Hadoop should be aware of a security vulnerability recently
discovered, as described by the following CVE. In particular, please note the
Users affected, Versions affected, and Mitigation sections.

The project team will be announcing a release vote shortly for Apache Hadoop
2.0.1-alpha, which will be comprised of the contents of Apache Hadoop
2.0.0-alpha, this security patch, and a few patches for YARN.

Best,
Aaron T. Myers
Software Engineer, Cloudera

CVE-2012-3376: Apache Hadoop HDFS information disclosure vulnerability

Severity: Critical

Vendor: The Apache Software Foundation

Versions Affected: Hadoop 2.0.0-alpha

Users affected:
Users who have enabled Hadoop's Kerberos/HDFS security features.

Impact:
Malicious clients may gain write access to data for which they have read-only
permission, or gain read access to any data blocks whose IDs they can
determine.

Description:
When Hadoop's security features are enabled, clients authenticate to DataNodes
using BlockTokens issued by the NameNode to the client. The DataNodes are able
to verify the validity of a BlockToken, and will reject BlockTokens that were
not issued by the NameNode. The DataNode determines whether or not it should
check for BlockTokens when it registers with the NameNode.

Due to a bug in the DataNode/NameNode registration process, a DataNode which
registers more than once for the same block pool will conclude that it
thereafter no longer needs to check for BlockTokens sent by clients. That is,
the client will continue to send BlockTokens as part of its communication with
DataNodes, but the DataNodes will not check the validity of the tokens. A
DataNode will register more than once for the same block pool whenever the
NameNode restarts, or when HA is enabled.

Mitigation:
Users of 2.0.0-alpha should immediately apply the patch provided below to their
systems. Users should upgrade to 2.0.1-alpha as soon as it becomes available.

Credit: This issue was discovered by Aaron T. Myers of Cloudera.

A signed patch against Apache Hadoop 2.0.0-alpha for this issue can be found
here: https://people.apache.org/~atm/cve-2012-3376/

-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.11 (GNU/Linux)

iQEcBAEBAgAGBQJP9xp7AAoJECEaGfB4kTjfGWMH/2fXnrngfpQL+d1QLG3wDOPn
OAJK3Tj/JrII1ETCguI6DOjpQaRrnzSvyCdWOHApbGG6LxwSvTlwEBPUR8SMZFxY
TA13eJPz21ZXtXZ9oGvg1BMw+wRwfmem0Sl508c8kJpSfHXD4W89wyG/5Z+1pz5d
s0aHUMVj5YT32yH45Tp192nB5d4XQ7gdUmCLB4HF8fxrrIH2jWU0NX63DT6dXE5w
DUqKq6nTFDHnuTA1IO0B8OAVGv2M/kq8P3Fi+pnVvVao+ttkWIK1z7Ts11gfL7d0
/rE4VgZ7Cwc2o1Fx8s1LCKKLIDrO15aULOSbEa9nl6yQywEEjn2h6cKXHv6RUHM=
=wrvr
-END PGP SIGNATURE-

[jira] [Resolved] (HADOOP-8463) hadoop.security.auth_to_local needs a key definition and doc

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8463.


Resolution: Unresolved

 hadoop.security.auth_to_local needs a key definition and doc 
 -

 Key: HADOOP-8463
 URL: https://issues.apache.org/jira/browse/HADOOP-8463
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: madhukara phatak
  Labels: newbie
 Attachments: HADOOP-8463.patch


 hadoop.security.auth_to_local should be defined in 
 CommonConfigurationKeysPublic.java, and update the uses of the raw string in 
 common and hdfs with the key. 
 It's definition in core-site.xml should also be updated with a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HADOOP-8463) hadoop.security.auth_to_local needs a key definition and doc

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers reopened HADOOP-8463:



 hadoop.security.auth_to_local needs a key definition and doc 
 -

 Key: HADOOP-8463
 URL: https://issues.apache.org/jira/browse/HADOOP-8463
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 2.0.0-alpha
Reporter: Eli Collins
Assignee: madhukara phatak
  Labels: newbie
 Attachments: HADOOP-8463.patch


 hadoop.security.auth_to_local should be defined in 
 CommonConfigurationKeysPublic.java, and update the uses of the raw string in 
 common and hdfs with the key. 
 It's definition in core-site.xml should also be updated with a description.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Reopened] (HADOOP-8408) MR doesn't work with a non-default ViewFS mount table and security enabled

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8408?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers reopened HADOOP-8408:



Re-opening to amend the patch per Daryn's feedback.

 MR doesn't work with a non-default ViewFS mount table and security enabled
 --

 Key: HADOOP-8408
 URL: https://issues.apache.org/jira/browse/HADOOP-8408
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Fix For: 2.0.1

 Attachments: HDFS-8408.patch


 With security enabled, if one sets up a ViewFS mount table using the default 
 mount table name, everything works as expected. However, if you try to create 
 a ViewFS mount table with a non-default name, you'll end up getting an error 
 like the following (in this case vfs-cluster was the name of the mount 
 table) when running an MR job:
 {noformat}
 java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster
 {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE - RELEASE PLAN] create branch-1.1 and make release candidate 1.1.0

[Aaron T. Myers]

Sounds good to me. Thanks a lot, Matt.

+1 (binding)

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, May 17, 2012 at 6:52 PM, Matt Foley ma...@apache.org wrote:

 Hi all,
 I would like to branch branch-1.1 from the current HEAD of branch-1,
 and proceed to make a release candidate for hadoop-1.1.0 from it.
 I'm also hereby offering to be Release Manager for branch-1.1, as I have
 been
 for branch-1.0.

 There's a LOT of stuff in branch-1, over 80 patches not in the branch-1.0
 line, so I anticipate
 the need for some stabilization.  So I'll float the RC0 for a week or so,
 before possibly calling
 a vote on it.  Sound okay?

 Somewhat to my surprise, I found that our bylaws require a seven-day formal
 vote
 on this proposal.  So, please vote.  Voting will close on Thursday 24 May,
 7pm PDT.

 Best regards,
 --Matt

[jira] [Created] (HADOOP-8408) MR doesn't work with a non-default ViewFS mount table and security enabled

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-8408:
--

 Summary: MR doesn't work with a non-default ViewFS mount table and 
security enabled
 Key: HADOOP-8408
 URL: https://issues.apache.org/jira/browse/HADOOP-8408
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


With security enabled, if one sets up a ViewFS mount table using the default 
mount table name, everything works as expected. However, if you try to create a 
ViewFS mount table with a non-default name, you'll end up getting an error like 
the following (in this case vfs-cluster was the name of the mount table) when 
running an MR job:

{noformat}
java.lang.IllegalArgumentException: java.net.UnknownHostException: vfs-cluster
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8349) ViewFS doesn't work when the root of a file system is mounted

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-8349:
--

 Summary: ViewFS doesn't work when the root of a file system is 
mounted
 Key: HADOOP-8349
 URL: https://issues.apache.org/jira/browse/HADOOP-8349
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


Viewing files under a ViewFS mount which mounts the root of a file system shows 
trimmed paths. Trying to perform operations on files or directories under the 
root-mounted file system doesn't work. More info in the first comment of this 
JIRA.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8320) FileSystem#checkPath and AbstractFileSystem#checkPath should share code

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-8320:
--

 Summary: FileSystem#checkPath and AbstractFileSystem#checkPath 
should share code
 Key: HADOOP-8320
 URL: https://issues.apache.org/jira/browse/HADOOP-8320
 Project: Hadoop Common
  Issue Type: Improvement
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Priority: Minor


Per the discussion on HADOOP-8310, these two methods can be refactored to share 
some code.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7416) Allow test-patch to work with cross sub-project changes

[Aaron T. Myers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7416?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-7416.


Resolution: Duplicate

Resolving as duplicate of HADOOP-8308.

 Allow test-patch to work with cross sub-project changes
 ---

 Key: HADOOP-7416
 URL: https://issues.apache.org/jira/browse/HADOOP-7416
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Attachments: HADOOP-7416.patch


 Now that the sub-projects are sub-directories in the same repo, we can make 
 {{test-patch.sh}} not barf on cross-project patches. We would need to make 
 {{smart-apply-patch.sh}} not fail hard when it detects this, and probably 
 make {{test-patch.sh}} run all the validation checks against all affected 
 sub-projects.
 This was inspired by HDFS-1900, which changes a config in Common that's 
 referenced in a bunch of places in HDFS.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8310) FileContext#checkPath should handle URIs with no port

[Aaron T. Myers (JIRA)]

Aaron T. Myers created HADOOP-8310:
--

 Summary: FileContext#checkPath should handle URIs with no port
 Key: HADOOP-8310
 URL: https://issues.apache.org/jira/browse/HADOOP-8310
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


AbstractFileSystem#checkPath is used to verify that a given path is for the 
same file system as represented by the AbstractFileSystem instance.

The original intent of the code was to allow for no port to be provided in the 
checked path, if the default port was being used by the AbstractFileSystem 
instance. However, before performing port handling, AFS#checkPath compares the 
full URI authorities for equality. Since the URI authority includes the port, 
the port handling code is never reached, and thus valid paths may be 
erroneously considered invalid.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Supporting cross-project Jenkins builds

[Aaron T. Myers]

On Tue, Apr 17, 2012 at 4:52 PM, Alejandro Abdelnur t...@cloudera.comwrote:

 * All patches must be at trunk/ level
 * All patches do a full clean TARBALL creation without running testcases
 * From the patch file we find out the maven modules and for those
 modules we do javac-warns/javadoc-warns/findbugs/testcases


+1, I like this proposal a lot. Seems to do a good job of balancing the
need to check for dependent sub-project breakage with the desire to not
unnecessarily inflate the run time of test-patch.

--
Aaron T. Myers
Software Engineer, Cloudera

[jira] [Reopened] (HADOOP-8280) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.

[Aaron T. Myers (Reopened) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers reopened HADOOP-8280:



Reopening for recommit using `svn mv ...'

  Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
 -

 Key: HADOOP-8280
 URL: https://issues.apache.org/jira/browse/HADOOP-8280
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test, util
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: HADOOP-8280.patch, HADOOP-8280_mv_script.sh, 
 HADOOP-8280_rev2.patch


  We need to use VersionUtil in MAPREDUCE-4150. Moving 
 VersionUtil/TestVersionUtil from HDFS into common will help this, especially 
 since test-patch doesn't seem to deal well with cross-sub-project patches.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8280) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8280.


Resolution: Fixed

I've just recommitted this using `svn mv ...'

  Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
 -

 Key: HADOOP-8280
 URL: https://issues.apache.org/jira/browse/HADOOP-8280
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test, util
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: HADOOP-8280.patch, HADOOP-8280_mv_script.sh, 
 HADOOP-8280_rev2.patch


  We need to use VersionUtil in MAPREDUCE-4150. Moving 
 VersionUtil/TestVersionUtil from HDFS into common will help this, especially 
 since test-patch doesn't seem to deal well with cross-sub-project patches.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8280) Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8280.


   Resolution: Fixed
Fix Version/s: 2.0.0
 Hadoop Flags: Reviewed

I've just committed this to trunk and branch-2.

Thanks a lot for the contribution, Ahmed.

  Move VersionUtil/TestVersionUtil and GenericTestUtils from HDFS into Common.
 -

 Key: HADOOP-8280
 URL: https://issues.apache.org/jira/browse/HADOOP-8280
 Project: Hadoop Common
  Issue Type: Improvement
  Components: test, util
Affects Versions: 0.23.1
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
 Fix For: 2.0.0

 Attachments: HADOOP-8280.patch


  We need to use VersionUtil in MAPREDUCE-4150. Moving 
 VersionUtil/TestVersionUtil from HDFS into common will help this, especially 
 since test-patch doesn't seem to deal well with cross-sub-project patches.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Supporting cross-project Jenkins builds

[Aaron T. Myers]

+1

This JIRA has been sitting patch available for a few weeks:
https://issues.apache.org/jira/browse/HADOOP-7416

I don't think it's quite ready for prime time (it doesn't implement all the
features you described) but it'd be a good starting point if someone wanted
to take it over the finish line.

--
Aaron T. Myers
Software Engineer, Cloudera



On Mon, Apr 16, 2012 at 1:51 PM, Tom White t...@cloudera.com wrote:

 Currently Jenkins QA builds don't support cross-project patches, since
 they try to apply them to the hadoop-{common,hdfs,mapreduce}-project
 tree, and reject any patches that are not strictly confined to that
 tree. For changes that span projects (e.g. moving code from HDFS or MR
 to Common, or build system changes) developers have to run test-patch
 and tests manually, which is cumbersome.

 I'd like to propose that we change the Common Jenkins build so that it
 runs from the top-level
 (http://svn.apache.org/repos/asf/hadoop/common/trunk). The HDFS and
 MapReduce builds would be unchanged. This would have two implications:
 i) all patches would have to be rooted at the top-level, and ii) the
 unit tests for all projects would be run for Common changes.

 I think i) is reasonable (and most patches are like this now) - and
 it's easy to regenerate a patch that is rooted at the wrong level. For
 ii), running all tests for a change in Common is probably a good thing
 to do in general.

 Thoughts?

 Tom

Re: Supporting cross-project Jenkins builds

[Aaron T. Myers]

On Mon, Apr 16, 2012 at 2:14 PM, Alejandro Abdelnur t...@cloudera.comwrote:

 * all testcases should always be run (else a change in hdfs could
 affect yarn/tools but not be detected, or one in yarn affect tools)


I'm -0 on this suggestion. Yes, it's a nice benefit to check all of the
dependent Hadoop sub-projects for every patch, but it will also
dramatically increase the time test-patch takes to run for any given patch.
In my experience, the vast majority of patches stand little chance of
breaking the dependent sub-projects, making this largely unnecessary and
thus a waste of time and Jenkins build slave resources.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: HADOOP-8268

[Aaron T. Myers]

2012/4/12 Radim Kolar h...@filez.com

 can somebody take a peek on patch attached to ticked and enlighten me why
 it does not apply?


I just commented on the JIRA.

Best,
Aaron

--
Aaron T. Myers
Software Engineer, Cloudera

[jira] [Created] (HADOOP-8261) Har file system doesn't deal with FS URIs with a host but no port

[Aaron T. Myers (Created) (JIRA)]

Har file system doesn't deal with FS URIs with a host but no port
-

 Key: HADOOP-8261
 URL: https://issues.apache.org/jira/browse/HADOOP-8261
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Affects Versions: 2.0.0
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


If you try to run an MR job with a Hadoop Archive as the input, but the URI you 
give it has no port specified (e.g. hdfs://simon) the job will fail with an 
error like the following:

{noformat}
java.io.IOException: Incomplete HDFS URI, no host: 
hdfs://simon:-1/user/atm/input.har/input
{noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Help a newbie on patch submission

[Aaron T. Myers]

I believe the issue is that the Hadoop QA bot runs from a subdirectory of
the project root, i.e. hadoop-common-project, hadoop-hdfs-project, etc.
Your patch only changes things under hadoop-dist, so I don't think it will
work no matter how you format the patch.

For this particular JIRA (HADOOP-8241) you should just comment saying
exactly what testing you did. A reviewer of the patch should probably
manually apply it and test it out before committing it.

--
Aaron T. Myers
Software Engineer, Cloudera



On Tue, Apr 3, 2012 at 10:03 AM, Mostafa Elhemali mosta...@microsoft.comwrote:

 Thanks Devaraj. I actually took care to convert the line endings to just
 LINE FEED (0A/^J) before submission, and I don't see ^M characters in
 either patch. I just opened the files in a binary editor to double-check
 and I don't see 0D anywhere.

 -Original Message-
 From: Devaraj k [mailto:devara...@huawei.com]
 Sent: Tuesday, April 03, 2012 9:51 AM
 To: common-dev@hadoop.apache.org
 Subject: RE: Help a newbie on patch submission

 I tried applying your patches in my environment, It works fine.

 The issue with Hadoop QA may be because of having ctl-M (^M) characters
 in the patch file. Can you try by removing the  ctl-M (^M) chars in the
 patch.

 Thanks
 Devaraj
 
 From: Mostafa Elhemali [mosta...@microsoft.com]
 Sent: Tuesday, April 03, 2012 9:42 PM
 To: common-dev@hadoop.apache.org
 Subject: Help a newbie on patch submission

 Hi,
 I'm trying to submit a patch to fix some build problems in Hadoop trunk on
 Windows, and I'm having a hard time pleasing Hadoop QA. This is for
 Hadoop-8241 (https://issues.apache.org/jira/browse/HADOOP-8241). I
 submitted the first patch and it said it couldn't apply it, so I figured it
 may have to do with the fact that one of the two files fixed is in
 hadoop-hdfs-https which isn't part of the Hadoop Common project. So I
 submitted a second patch just for hadoop-dist/pom.xml which I believe is
 part of the Common project, but that couldn't be applied as well.
 So I come to you: what am I doing horribly wrong and how can I appease the
 almighty Hadoop QA?

 Thanks,
 Mostafa Elhemali

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

Hi Milind,

On Tue, Apr 3, 2012 at 11:27 AM, milind.bhandar...@emc.com wrote:

 Here you say:


  Essentially 'trunk' is where incompatible changes *may* be committed (in
 future). We should allow for that.


What I believe Arun is alluding to here is that we expect for compatibility
to be maintained for the lifetime of a major release branch.



 On another thread, responding to Avner (re: MAPREDUCE-4049?) you say,

  We do expect 'new features' to make it to trunk before we can commit to
 either branch-1 or branch-2.


 Which one is it ?


These two statements aren't mutually exclusive. We require that all new
features go to trunk first so as to ensure that future releases are
supersets of the functionality of previous releases, except in the case of
explicit deprecation. Only once it's committed to trunk may it be
back-ported to an earlier branch.



 Do you expect that new features will always remain compatible ?


Not necessarily, but only if a feature is compatible may it be back-ported
to major release branches.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

On Tue, Apr 3, 2012 at 2:37 PM, milind.bhandar...@emc.com wrote:

 What would be guideline for a new feature, such as
 https://issues.apache.org/jira/browse/MAPREDUCE-4049, which maintains
 compatibility for 1.x, but is not relevant to trunk, because the codebases
 have completely diverged, so cannot be committed to trunk ?


Are you sure this isn't relevant to trunk? The target versions field of
that JIRA lists both 1.0.x and 0.24.0 (trunk.) In the latest comment on
that JIRA, the author appears to intend to do this work for both trunk and
1.0:

I want to have the described plugin-ability (desired with same interface)
for all future versions of Hadoop (as mentioned in the Target Version/s
field). snip On the first phase, I am focusing on the existing 1.0 branch
as I know it. In parallel, I'll try to learn what exists in 0.23

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

If that's the case then there doesn't seem to be any question here. The
feature is in trunk, and an implementation could be done for an older
release branch that would be compatible with that branch. Sure, the code to
implement the feature is quite different between the two branches, but
trunk will remain a superset of the functionality of the past release, so
no issue.

--
Aaron T. Myers
Software Engineer, Cloudera



On Tue, Apr 3, 2012 at 3:14 PM, milind.bhandar...@emc.com wrote:

 To my knowledge, shuffle is already pluggable in 0.23 onwards, as long as
 it is used only by mapreduce framework.

 That's why Avner says : In parallel, I'll try to *learn what exists* in
 0.23. (Emphasize my own.)

 That's why I was wondering about the insistence of committing to trunk
 first.

 - Milind

 ---
 Milind Bhandarkar
 Greenplum Labs, EMC
 (Disclaimer: Opinions expressed in this email are those of the author, and
 do not necessarily represent the views of any organization, past or
 present, the author might be affiliated with.)



 On 4/3/12 2:44 PM, Aaron T. Myers a...@cloudera.com wrote:

 On Tue, Apr 3, 2012 at 2:37 PM, milind.bhandar...@emc.com wrote:
 
  What would be guideline for a new feature, such as
  https://issues.apache.org/jira/browse/MAPREDUCE-4049, which maintains
  compatibility for 1.x, but is not relevant to trunk, because the
 codebases
  have completely diverged, so cannot be committed to trunk ?
 
 
 Are you sure this isn't relevant to trunk? The target versions field of
 that JIRA lists both 1.0.x and 0.24.0 (trunk.) In the latest comment on
 that JIRA, the author appears to intend to do this work for both trunk and
 1.0:
 
 I want to have the described plugin-ability (desired with same interface)
 for all future versions of Hadoop (as mentioned in the Target Version/s
 field). snip On the first phase, I am focusing on the existing 1.0
 branch
 as I know it. In parallel, I'll try to learn what exists in 0.23
 
 --
 Aaron T. Myers
 Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

Hey Arun,

On Wed, Mar 28, 2012 at 12:28 AM, Arun C Murthy a...@hortonworks.com wrote:

 Done, all clear; I've also created jira revisions. Please let me know if
 you find any issues.


Thanks a lot for making these changes. Two questions:

- Now that we've renamed branch-0.23 to branch-2, and since there is as yet
no branch-0.23.3, what should be done with JIRAs marked with the 0.23.3
version? Perhaps all of these should be updated to 2.0.0?

- We still have the JIRA version 0.24.0, which is presumably still
representative of trunk. Given that we will likely never release an 0.24.0,
should we rename this version in JIRA as well?

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

On Wed, Mar 28, 2012 at 11:53 AM, Todd Lipcon t...@cloudera.com wrote:

 Proposal: is it possible to call the JIRA fixVersion trunk, and then
 when we branch off trunk to make a release, rename it at that point to
 2.1 or 3.0 or whatever it gets called?


I like this idea. Just to be clear, I think the exact workflow would be:

1. Set the version fields to trunk if you're not committing the JIRA to
any current versioned branch.
2. When a new release branch is made off of trunk, rename the trunk JIRA
version to whatever the appropriate version number is.
3. At the same time as (2), create a new JIRA version also called trunk.
4. Go to 1.

Is this what you were thinking, Todd?

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

On Wed, Mar 28, 2012 at 1:57 PM, Arun C Murthy a...@hortonworks.com wrote:

 On on hand, we've historically bumped the version number for trunk (i.e.
 3.0.0-SNAPSHOT). This has the nice property that when we do cut a release
 branch off trunk we don't have to scramble to change fix-versions on all
 the jiras committed to trunk from 'trunk' to X.0.0. Since I've done my fair
 share of jira manipulation for releases as the RM, as recently as last
 night, it's nice to not force the burden on the RM for branch-3.


I don't think you'd have to change all the JIRAs. You'd just have to change
the name of the trunk JIRA version to whatever the right number is, and
then create a new version in JIRA also named trunk. This would serve the
same purpose, without having to update any individual JIRAs.


 OTOH, having a constant trunk-SNAPSHOT version string helps devs working
 on trunk since they don't have to deal with version bumps on trunk (albeit,
 once in a while). (Credit to Scott Carey for this idea.)


This is a nice benefit of Todd's (and Scott's?) proposal. Whenever we do
change the trunk version number, I have to update a bunch of scripts,
environment files, and configuration XML files. Not the end of the world,
but annoying nonetheless.

I'd also add to this benefit that users who are new to the project will
more easily be able to determine what version to put for a JIRA they want
to get committed to trunk. I've seen plenty of users who are confused by
having to set 0.24.0 as the version indicating trunk.

There's also a nice symmetry in having the branch in svn/git (named trunk)
have the same name as the JIRA version indicator.



 Given the above I'd stick with 3.0.0 since it means lesser confusion and
 lesser work for the RM on future major releases.


I honestly believe that this scheme is more confusing for devs and users,
and almost no different for RMs given what I described above with JIRA
version renaming. But, I don't feel super strongly about it. If this makes
sense to you, then I'll stop pushing.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [RESULT] - [VOTE] Rename hadoop branches post hadoop-1.x

[Aaron T. Myers]

Hi Owen,

On Wed, Mar 28, 2012 at 12:39 PM, Owen O'Malley omal...@apache.org wrote:

 Let's imagine that we already had a 2.0.0 release. Now we want to add
 features like HA. The only place to put that is in 2.1.0. On the other
 hand, you don't want to pull *ALL* of the changes from trunk. That is way
 too much scope. So the RM of the 2 branch needs to make the call of what
 should be 2.1 vs 3.0.


I don't think anyone disagrees with this line of reasoning. It's certainly
up to the RM what gets included in branch-2 and hence what gets put up for
release votes under the 2.y.z version numbers. I don't think Todd was
suggesting we rename the JIRA version 0.24.0 to 2.1.0.

But, the question still remains of how to refer to the branch trunk in
JIRA. I don't think it should be called 3.0.0, as that's not necessarily
the next release that will come off of it, and using a version number for
trunk that changes from time to time has other downsides as I described in
my response to Arun. Given this, do you object to renaming the JIRA fix
version that refers to the branch trunk to trunk ?

--
Aaron T. Myers
Software Engineer, Cloudera

Re: [VOTE] Hadoop-1.0.2-rc2

[Aaron T. Myers]

On Tue, Mar 27, 2012 at 11:52 AM, Matt Foley mfo...@hortonworks.com wrote:

 Under current rules, I think we have to restart the vote.  Later this
 afternoon I'll post a proposal for a change in these rules, because I agree
 with you we should try to accelerate the process.


Sounds good to me. Thanks, Matt.

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Help for Hadoop

[Aaron T. Myers]

Hi Bhushan,

Take a look at http://wiki.apache.org/hadoop/HowToContribute. Some of those
instructions are specifically about how to contribute patches back to the
project (which you are certainly encouraged to do!) but much of it is about
where to check out the code, how to compile the different branches, how to
set up your IDE, etc.

--
Aaron T. Myers
Software Engineer, Cloudera



On Sun, Mar 25, 2012 at 9:33 PM, Bhushan Kandalkar 
kandalkarbhus...@gmail.com wrote:

 Hello Sir,

 I want to make changes in hadoop source code for job tracker and task
 tracker program.
 How should i make change to source code of hadoop?How to test and compile
 it?

 Waiting for your reply.

 Bhushan Kandalkar,
 Pune,India

[jira] [Resolved] (HADOOP-8213) TestGetBlocks is asserting on a wrong string

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8213.


Resolution: Duplicate

Resolving as a duplicate of HDFS-3143.

 TestGetBlocks is asserting on a wrong string
 

 Key: HADOOP-8213
 URL: https://issues.apache.org/jira/browse/HADOOP-8213
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 0.24.0
Reporter: Hari Mankude
Assignee: Hari Mankude
Priority: Trivial

 TestGetBlocks is asserting on a wrong string in getBlocksWithException() 
 resulting in a test failure.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: [VOTE] Hadoop-1.0.2-rc1

[Aaron T. Myers]

On Thu, Mar 22, 2012 at 11:12 PM, Matt Foley mfo...@hortonworks.com wrote:

 I believe
 this bug caused Snappy to not be built correctly, which in turn causes MR
 jobs to fail if Snappy compression is configured on.


Ah, I see. Thanks for the clarification.

Given that this fix is quite small, and given that no one has so far voted
on RC-1, I'd personally be in favor of not restarting the voting period for
RC-2. Thoughts?

--
Aaron T. Myers
Software Engineer, Cloudera

Re: Add user to assignee list?

[Aaron T. Myers]

Done.

--
Aaron T. Myers
Software Engineer, Cloudera



On Mon, Mar 19, 2012 at 8:32 AM, Ravi Prakash ravihad...@gmail.com wrote:

 Hi folks,

 Can someone with the proper authorization please add Anty to the list of
 users who can assign themselves JIRAs?

 Cheers
 Ravi.


 -- Forwarded message --
 From: Schubert Zhang (Commented) (JIRA) j...@apache.org
 Date: Mon, Mar 19, 2012 at 10:00 AM
 Subject: [jira] [Commented] (MAPREDUCE-3685) There are some bugs in
 implementation of MergeManager
 To: ravihad...@gmail.com



   [

 https://issues.apache.org/jira/browse/MAPREDUCE-3685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13232654#comment-13232654
 ]

 Schubert Zhang commented on MAPREDUCE-3685:
 ---

 @Ravi
 Thank you very much. Is someone can and Anty Rao into the assignee list?

 @Anty
 Seems we should add junit code, and make patch for a right version.

  There are some bugs in implementation of MergeManager
  -
 
  Key: MAPREDUCE-3685
  URL:
 https://issues.apache.org/jira/browse/MAPREDUCE-3685
  Project: Hadoop Map/Reduce
   Issue Type: Bug
   Components: mrv2
 Affects Versions: 0.23.1
 Reporter: anty.rao
 Assignee: Zhihong Yu
 Priority: Minor
  Fix For: 0.24.0
 
  Attachments: MAPREDUCE-3685-branch-0.23.1.patch,
 MAPREDUCE-3685-branch-0.23.1.patch, MAPREDUCE-3685.patch
 
 


 --
 This message is automatically generated by JIRA.
 If you think it was sent incorrectly, please contact your JIRA
 administrators:
 https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
 For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Debugging with kerberos enabled

[Aaron T. Myers]

+ common-dev
bcc: general

Moving this to the more appropriate common-dev@

I personally like to run a KDC on my dev box and build/deploy a
pseudo-distributed cluster right there. This lets me make principals at
will, adjust krb5 settings as I please, regenerate keytabs, etc. Setting up
a local MIT KDC is actually quite easy.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Mar 8, 2012 at 4:05 PM, Alejandro Abdelnur t...@cloudera.comwrote:

 Why not just attach a debugger from an IDE (with the right hadoop sources)
 to DN JVM?

 On Thu, Mar 8, 2012 at 1:16 AM, Evert Lammerts evert.lamme...@sara.nl
 wrote:

  Hi list,
 
  I've spent the whole of yesterday debugging a DN to find and fix a
 trivial
  bug (HDFS-3059). Difficulty is that we have Kerberos enabled and I didn't
  know how to emulate that environment locally. So what I did was editing
 the
  source, building the jars, deploying them on my NN and a single DN,
  starting deamons, and looking at my debug statements.
 
  Obviously not the most optimal way. How do you debug deamons with
 Kerberos
  enabled? Do you add your development machine to the cluster, give it a
  principal, and configure the network? Any tips / best practices? I expect
  we're going to need this more often since we're just bringing up a fair
  sized multi-tenant production environment (~70 machines) and I guess this
  won't be the only time we'll want to see what exactly happens.
 
  Thanks,
  Evert
 

[jira] [Created] (HADOOP-8152) Expand public APIs for security library classes

[Aaron T. Myers (Created) (JIRA)]

Expand public APIs for security library classes
---

 Key: HADOOP-8152
 URL: https://issues.apache.org/jira/browse/HADOOP-8152
 Project: Hadoop Common
  Issue Type: Improvement
  Components: security
Affects Versions: 0.23.3
Reporter: Aaron T. Myers


Currently projects like Hive and HBase use UserGroupInformation and 
SecurityUtil methods. Both of these classes are marked LimitedPrivate(HDFS,MR) 
but should probably be marked more generally public.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7454) Common side of High Availability Framework (HDFS-1623)

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-7454.


   Resolution: Fixed
Fix Version/s: 0.24.0
 Assignee: (was: Aaron T. Myers)
 Hadoop Flags: Reviewed

I just merged this to trunk.

 Common side of High Availability Framework (HDFS-1623)
 --

 Key: HADOOP-7454
 URL: https://issues.apache.org/jira/browse/HADOOP-7454
 Project: Hadoop Common
  Issue Type: New Feature
Reporter: Aaron T. Myers
 Fix For: 0.24.0


 There will likely need to be a few changes to Hadoop Common (e.g. HDFS-7380) 
 to complete HDFS-1623 (High Availability Framework for HDFS NN). This JIRA is 
 an umbrella for those Common changes.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-8116) HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-8116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-8116.


  Resolution: Fixed
Hadoop Flags: Reviewed

I've committed this to the HA branch and filed the following JIRA: 
MAPREDUCE-3934

Thanks a lot for the review, Eli.

 HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896
 ---

 Key: HADOOP-8116
 URL: https://issues.apache.org/jira/browse/HADOOP-8116
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Affects Versions: HA Branch (HDFS-1623)
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
 Attachments: HADOOP-8116-HDFS-1623.patch, HADOOP-8116-HDFS-1623.patch


 HADOOP-7896 (on the HA branch) refactored RetryAction from an enum to a 
 class, and also moved the act of sleeping to delay retries from the 
 RetryPolicy implementations into RetryInvocationHandler. RetriableCommand, in 
 the rewritten distcp tool, uses RetryPolicy and associated classes from 
 o.a.h.io.retry. When MAPREDUCE-2765 was merged into the HA branch, 
 RetriableCommand wasn't adjusted accordingly to make use of the new structure 
 of the o.a.h.io.retry classes.
 It's probably generally not kosher for RetriableCommand to be using the 
 RetryPolicy classes at all, since they're not really intended to be used 
 except by RetryInvocationHandler. But, regardless, this JIRA aims to make 
 distcp's use of the o.a.h.io.retry classes functional again.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8116) HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896

[Aaron T. Myers (Created) (JIRA)]

HA: RetriableCommand is using RetryPolicy incorrectly after HADOOP-7896
---

 Key: HADOOP-8116
 URL: https://issues.apache.org/jira/browse/HADOOP-8116
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: util
Affects Versions: HA Branch (HDFS-1623)
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers


HADOOP-7896 (on the HA branch) refactored RetryAction from an enum to a class, 
and also moved the act of sleeping to delay retries from the RetryPolicy 
implementations into RetryInvocationHandler. RetriableCommand, in the rewritten 
distcp tool, uses RetryPolicy and associated classes from o.a.h.io.retry. When 
MAPREDUCE-2765 was merged into the HA branch, RetriableCommand wasn't adjusted 
accordingly to make use of the new structure of the o.a.h.io.retry classes.

It's probably generally not kosher for RetriableCommand to be using the 
RetryPolicy classes at all, since they're not really intended to be used except 
by RetryInvocationHandler. But, regardless, this JIRA aims to make distcp's use 
of the o.a.h.io.retry classes functional again.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Created] (HADOOP-8066) The full docs build intermittently fails

[Aaron T. Myers (Created) (JIRA)]

The full docs build intermittently fails


 Key: HADOOP-8066
 URL: https://issues.apache.org/jira/browse/HADOOP-8066
 Project: Hadoop Common
  Issue Type: Bug
  Components: build
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Assignee: Andrew Bayer


See for example:

https://builds.apache.org/job/Hadoop-Hdfs-trunk/954/
https://builds.apache.org/job/Hadoop-Common-trunk/317/

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7928) HA: Client failover policy is incorrectly trying to fail over all IOExceptions

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-7928.


   Resolution: Fixed
Fix Version/s: HA Branch (HDFS-1623)
 Hadoop Flags: Reviewed

Thanks a lot for the quick review, Todd. I've just committed this to the HA 
branch.

 HA: Client failover policy is incorrectly trying to fail over all IOExceptions
 --

 Key: HADOOP-7928
 URL: https://issues.apache.org/jira/browse/HADOOP-7928
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ha, ipc
Affects Versions: HA Branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Blocker
 Fix For: HA Branch (HDFS-1623)

 Attachments: HADOOP-7928-HDFS-1623.patch


 In the {{FailoverOnNetworkExceptionRetry}} implementation, we're returning 
 FAILOVER_AND_RETRY for any IOException, so long as the call is idempotent. 
 So, for example, {{getFileInfo}} on a file you don't have access to throws 
 AccessControlException which incorrectly triggers a failover. We should not 
 failover on RemoteExceptions unless they are StandbyException.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Resolved] (HADOOP-7896) HA: if both NNs are in Standby mode, client needs to try failing back and forth several times with sleeps

[Aaron T. Myers (Resolved) (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-7896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aaron T. Myers resolved HADOOP-7896.


   Resolution: Fixed
Fix Version/s: HA Branch (HDFS-1623)
 Hadoop Flags: Reviewed

Thanks a lot for the reviews, Eli and Todd. I've just committed this.

 HA: if both NNs are in Standby mode, client needs to try failing back and 
 forth several times with sleeps
 -

 Key: HADOOP-7896
 URL: https://issues.apache.org/jira/browse/HADOOP-7896
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: ha, ipc
Affects Versions: HA Branch (HDFS-1623)
Reporter: Todd Lipcon
Assignee: Aaron T. Myers
Priority: Critical
 Fix For: HA Branch (HDFS-1623)

 Attachments: HADOOP-7896-HDFS-1623.patch, 
 HADOOP-7896-HDFS-1623.patch, HADOOP-7896-HDFS-1623.patch


 For a manual failover, there may be an intermediate state for a non-trivial 
 amount of time where both NNs are in standby mode. Currently, the failover 
 proxy will immediately failover on receiving this exception from the first 
 NN, and when it hits the same exception on the second NN, it immediately 
 fails. It should probably fail back and forth nearly indefinitely if both NNs 
 are in Standby mode.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

Re: Is it possible to use AF_UINX sockets in hadoop?

[Aaron T. Myers]

Hi Raghu,

You might want to check out the work that's been done over at
https://issues.apache.org/jira/browse/HDFS-347.

--
Aaron T. Myers
Software Engineer, Cloudera



On Thu, Nov 24, 2011 at 12:56 PM, Raghu Reddy rre...@psc.edu wrote:

 Currently I'm experimenting with version 0.20.

 I would like to modify it so that it uses Unix Domain Sockets instead of
 INET sockets because we will be using this on a single node with a large
 number of cores.

 I would like to use junixsockets 1.3 implementation of AFUNIXSocket in
 place of INET sockets.

 The first question is is this possible?

 For the clients, it appears that one can provide a socket factory that
 returns AFUNIXSocket.  The problem seems to be to get the servers to use a
 different implementation because the servers are using
 ServerSocketChannels, and according to:

 http://docs.oracle.com/**javase/6/docs/api/java/nio/**
 channels/ServerSocketChannel.**htmlhttp://docs.oracle.com/javase/6/docs/api/java/nio/channels/ServerSocketChannel.html
 

 it is not possible to specify SocketImpl.

 I have tried to use ServerSocket.setSocketFactory(**) to try and set a
 different implementation, but I have not yet figured out if this is working
 or not.

 Does anyone have any idea if this is supposed to work?  Is there a better
 way of accomplishing this?

 Thanks!

 --rr