Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

FYI that we got our last blocker in today, so I'm currently rolling RC1.
Stay tuned!

On Thu, Nov 30, 2017 at 8:32 AM, Allen Wittenauer 
wrote:

>
> > On Nov 30, 2017, at 1:07 AM, Rohith Sharma K S <
> rohithsharm...@apache.org> wrote:
> >
> >
> > >. If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated?
> > Ideally it should not be. Can you point out where it is marked as
> deprecated? If it is in historyserver daemon start, that change made very
> long back when timeline server added.
>
>
> Ahh, I see where all the problems lie.  No one is paying attention to the
> deprecation message because it’s kind of oddly worded:
>
> * It really means “don’t use ‘yarn historyserver’ use ‘yarn
> timelineserver’ ”
> * ‘yarn historyserver’ was removed from the documentation in 2.7.0
> * ‘yarn historyserver’ doesn’t appear in the yarn usage output
> * ‘yarn timelineserver’ runs the exact same class
>
> There’s no reason for ‘yarn historyserver’ to exist in 3.x.  Just run
> ‘yarn timelineserver’ instead.
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Allen Wittenauer]

> On Nov 21, 2017, at 2:16 PM, Vinod Kumar Vavilapalli  
> wrote:
> 
>>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>>> work. Not just deprecated in favor of timelineserver as was advertised.
>> 
>>  This works for me in trunk and the bash code doesn’t appear to have 
>> changed in a very long time.  Probably something local to your install.  (I 
>> do notice that the deprecation message says “starting” which is awkward when 
>> the stop command is given though.)  Also: is the deprecation message even 
>> true at this point?
> 
> 
> Sorry, I mischaracterized the problem.
> 
> The real issue is that I cannot use this command line when the MapReduce 
> JobHistoryServer is already started on the same machine.

The specific string is:

hadoop-${HADOOP_IDENT_STRING}-${HADOOP_SUBCMD}.pid

More specifically, the pid handling code will conflict if the following 
are true:

* same machine (obviously)
* same subcommand name
* same HADOOP_IDENT_USER: which by default is the user name of 
whatever starts it… but was designed to be overridden way back in hadoop 0.X.

… which means for most production setups, this is probably not real a 
problem.


> So, it looks like in shell-scripts, there can ever be only one daemon of a 
> given name, irrespective of which daemon scripts are invoked.

Correct.  Naming multiple, different daemons the same thing is 
extremely anti-user.   In fact, I thought this was originally about the “other” 
history server.

> 
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the 
> MapReduce JHS - only one of them can be started on the same node.

… by the same user, by default.  Started by a different user or 
different HADOOP_IDENT_USER, it will come up just fine.

>  (b) We need to figure out if this V1 TimelineService should even be support 
> given ATSv2.

If ATSv1 isn’t replaced by ATSv2, then why is it marked deprecated?

> On Nov 22, 2017, at 9:45 AM, Brahma Reddy Battula  wrote:
> 
> 1) Change the name
> 2) Create PID based on the CLASS Name, here applicationhistoryserver and 
> jobhistoryserver
> 3) Use same as branch-2.9..i.e suffixing with mapred or yarn
> 
> 
> @allen, any thoughts on this..?

Using the classname works in this instance, but just as we saw with the 
router daemons, people tend to use the same class names when building different 
components. It also means that if different daemons can be started in different 
ways from the same class dependent upon options, this conflict will still 
exist.  Also, with dynamic commands, it is very possible to run the same daemon 
from multiple start points.

As part of this discussion, I think it’s important to recognize:

a) This is likely to be primarily impacting developers.
b) We’re talking about two daemons where one has been deprecated.
c) Calling two different daemons “history server” is just awful from an end 
user perspective.
d) There is already a work around in place if one absolutely needs to run both 
on the same node as the same user, just as people do with datanode and 
nodemanager today.



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Brahma Reddy Battula]

bq. (a) The behavior of this command. Clearly, it will conflict with the
MapReduce JHS - only one of them can be started on the same node.

Yes,Since the PID  file will be same.

This problem will not present in branch-2.9 + as pid file will be suffix
with mapred(for JHS)
and
yarn (for AHS)
..i.e
PID is exported on each daemon script.

May be,
1) Change the name
2) Create PID based on the CLASS Name, here applicationhistoryserver and
jobhistoryserver
3) Use same as branch-2.9..i.e suffixing with mapred or yarn


@allen, any thoughts on this..?




On Wed, Nov 22, 2017 at 2:46 AM, Vinod Kumar Vavilapalli  wrote:

> >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> even work. Not just deprecated in favor of timelineserver as was advertised.
> >
> >   This works for me in trunk and the bash code doesn’t appear to
> have changed in a very long time.  Probably something local to your
> install.  (I do notice that the deprecation message says “starting” which
> is awkward when the stop command is given though.)  Also: is the
> deprecation message even true at this point?
>
>
> Sorry, I mischaracterized the problem.
>
> The real issue is that I cannot use this command line when the MapReduce
> JobHistoryServer is already started on the same machine.
>
> ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> WARNING: Use of this script to start YARN daemons is deprecated.
> WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> DEPRECATED: Use of this command to start the timeline server is deprecated.
> Instead use the timelineserver command for it.
> Starting the History Server anyway...
> historyserver is running as process 86156.  Stop it first.
>
> So, it looks like in shell-scripts, there can ever be only one daemon of a
> given name, irrespective of which daemon scripts are invoked.
>
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the
> MapReduce JHS - only one of them can be started on the same node.
>  (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> comment on (b).
>
> Thanks
> +Vinod




-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Hi folks,

Thanks again for the testing help with the RC. Here's our dashboard for the
3.0.0 release:

https://issues.apache.org/jira/secure/Dashboard.jspa?selectPageId=12329849

Right now we're tracking three blockers:

* HADOOP-15058 is the create-release fix, I just put up a patch which needs
reviews. It's the worst timing, but I'm hoping Allen could give it a quick
sanity check.
* HADOOP-15059 is the MR rolling upgrade issue that Junping found, needs
triage and an assignee. I asked Ray to look at what we've done with our
existing rolling upgrade testing, since it does run an MR job.
* HDFS-12480 is an EC issue that Eddy would like to get in if we're rolling
another RC, looks close.

Is there anything else from this thread that needs to be addressed? I rely
on the dashboard to track blockers, so please file a JIRA and prioritize if
so.

Best,
Andrew



On Tue, Nov 21, 2017 at 2:08 PM, Vrushali C  wrote:

> Hi Vinod,
>
> bq. (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> Yes, I am following this discussion. Let me chat with Rohith and Varun
> about this and we will respond on this thread. As such, my preliminary
> thoughts are that we should continue to support Timeline Service V1 till we
> have the detailed entity level ACLs in V2 and perhaps also a proposal
> around upgrade/migration paths from TSv1 to TSv2.
>
> But in any case, we do need to work towards phasing out Timeline Service
> V1.
>
> thanks
> Vrushali
>
>
> On Tue, Nov 21, 2017 at 1:16 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org
> > wrote:
>
> > >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> > even work. Not just deprecated in favor of timelineserver as was
> advertised.
> > >
> > >   This works for me in trunk and the bash code doesn’t appear to
> > have changed in a very long time.  Probably something local to your
> > install.  (I do notice that the deprecation message says “starting” which
> > is awkward when the stop command is given though.)  Also: is the
> > deprecation message even true at this point?
> >
> >
> > Sorry, I mischaracterized the problem.
> >
> > The real issue is that I cannot use this command line when the MapReduce
> > JobHistoryServer is already started on the same machine.
> >
> > ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> > WARNING: Use of this script to start YARN daemons is deprecated.
> > WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> > DEPRECATED: Use of this command to start the timeline server is
> deprecated.
> > Instead use the timelineserver command for it.
> > Starting the History Server anyway...
> > historyserver is running as process 86156.  Stop it first.
> >
> > So, it looks like in shell-scripts, there can ever be only one daemon of
> a
> > given name, irrespective of which daemon scripts are invoked.
> >
> > We need to figure out two things here
> >  (a) The behavior of this command. Clearly, it will conflict with the
> > MapReduce JHS - only one of them can be started on the same node.
> >  (b) We need to figure out if this V1 TimelineService should even be
> > support given ATSv2.
> >
> > @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> > comment on (b).
> >
> > Thanks
> > +Vinod
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vrushali C]

Hi Vinod,

bq. (b) We need to figure out if this V1 TimelineService should even be
support given ATSv2.

Yes, I am following this discussion. Let me chat with Rohith and Varun
about this and we will respond on this thread. As such, my preliminary
thoughts are that we should continue to support Timeline Service V1 till we
have the detailed entity level ACLs in V2 and perhaps also a proposal
around upgrade/migration paths from TSv1 to TSv2.

But in any case, we do need to work towards phasing out Timeline Service V1.

thanks
Vrushali


On Tue, Nov 21, 2017 at 1:16 PM, Vinod Kumar Vavilapalli  wrote:

> >> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
> even work. Not just deprecated in favor of timelineserver as was advertised.
> >
> >   This works for me in trunk and the bash code doesn’t appear to
> have changed in a very long time.  Probably something local to your
> install.  (I do notice that the deprecation message says “starting” which
> is awkward when the stop command is given though.)  Also: is the
> deprecation message even true at this point?
>
>
> Sorry, I mischaracterized the problem.
>
> The real issue is that I cannot use this command line when the MapReduce
> JobHistoryServer is already started on the same machine.
>
> ~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
> WARNING: Use of this script to start YARN daemons is deprecated.
> WARNING: Attempting to execute replacement "yarn --daemon start" instead.
> DEPRECATED: Use of this command to start the timeline server is deprecated.
> Instead use the timelineserver command for it.
> Starting the History Server anyway...
> historyserver is running as process 86156.  Stop it first.
>
> So, it looks like in shell-scripts, there can ever be only one daemon of a
> given name, irrespective of which daemon scripts are invoked.
>
> We need to figure out two things here
>  (a) The behavior of this command. Clearly, it will conflict with the
> MapReduce JHS - only one of them can be started on the same node.
>  (b) We need to figure out if this V1 TimelineService should even be
> support given ATSv2.
>
> @Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please
> comment on (b).
>
> Thanks
> +Vinod

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
>> work. Not just deprecated in favor of timelineserver as was advertised.
> 
>   This works for me in trunk and the bash code doesn’t appear to have 
> changed in a very long time.  Probably something local to your install.  (I 
> do notice that the deprecation message says “starting” which is awkward when 
> the stop command is given though.)  Also: is the deprecation message even 
> true at this point?


Sorry, I mischaracterized the problem.

The real issue is that I cannot use this command line when the MapReduce 
JobHistoryServer is already started on the same machine.

~/tmp/yarn$ $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver
WARNING: Use of this script to start YARN daemons is deprecated.
WARNING: Attempting to execute replacement "yarn --daemon start" instead.
DEPRECATED: Use of this command to start the timeline server is deprecated.
Instead use the timelineserver command for it.
Starting the History Server anyway...
historyserver is running as process 86156.  Stop it first.

So, it looks like in shell-scripts, there can ever be only one daemon of a 
given name, irrespective of which daemon scripts are invoked.

We need to figure out two things here
 (a) The behavior of this command. Clearly, it will conflict with the MapReduce 
JHS - only one of them can be started on the same node.
 (b) We need to figure out if this V1 TimelineService should even be support 
given ATSv2.

@Vrushani / @Rohith / @Varun Saxena et.al, if you are watching, please comment 
on (b).

Thanks
+Vinod

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Junping Du]

Another issue I just found is we have token format compatibility issue between 
2.x and 3.0. I tried to run a simple MR job on 3.0 RC0 against with 2.9.0 
tarball which is failed. This incompatibility change should also break other 
applications which will break rolling upgrade.
I think it is another blocker for 3.0.0 release.

Thanks,

Junping

From: Vinod Kumar Vavilapalli <vino...@apache.org>
Sent: Tuesday, November 21, 2017 12:59 PM
To: Allen Wittenauer
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>
>>
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>>
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
>
>   -1 on making yarn-ui always build.
>
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>>> - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>> 
>> 
>> The yarn-ui profile has always been off by default, AFAIK. It's documented
>> to turn it on in BUILDING.txt for release builds, and we do it in
>> create-release.
>> 
>> IMO not a blocker. I think it's also more of a dev question (do we want to
>> do this on every YARN build?) than a release one.
> 
>   -1 on making yarn-ui always build.
> 
>   For what is effectively an optional component (the old UI is still 
> there), it’s heavy dependency requirements make it a special burden outside 
> of the Docker container.


Thanks for the background on this.

I got schooled offline too about the heaviness of the dependencies.


> If it can be changed such that it either always downloads the necessary bits 
> (regardless of the OS/chipset!) and/or doesn’t kill the maven build if those 
> bits can’t be found  (i.e., truly optional), then I’d be less opposed.  (and, 
> actually, quite pleased because then the docker image build would be 
> significantly faster.)


This is a good idea (to the extend I understand your proposal). Long term, we'd 
like YARN UI2 to replace current UI, so it shouldn't be optionally build and 
hence the build should be fast. Let me track this separately as a non-blocker.

Thanks
+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

>> - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:
>> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
>> not in the CLI node -list
>> 
> 
> Is this a blocker? Could we get a JIRA?


I just cross verified this with others - looks like an issue with my 
environment - not a blocker, ignore.

+Vinod
-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Allen Wittenauer]

The original release script and instructions broke the build up into 
three or so steps. When I rewrote it, I kept that same model. It’s probably 
time to re-think that.  In particular, it should probably be one big step that 
even does the maven deploy.  There’s really no harm in doing that given that 
there is still a manual step to release the deployed jars into the production 
area.

We just need need to:

a) add an option to do deploy instead of just install.  if c-r is in asf mode, 
always activate deploy
b) pull the maven settings.xml file (and only the maven settings file… we don’t 
want the repo!) into the docker build environment
c) consolidate the mvn steps

This has the added benefit of greatly speeding up the build by removing 
several passes.

Probably not a small change, but I’d have to look at the code.  I’m on 
a plane tomorrow morning though.

Also:

>> 
>> Major
>> - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>> 
> 
> Is this something people use? I figured that the sub-tarballs were a relic
> from the project split, and nowadays Hadoop is one project with one release
> tarball. I actually thought about getting rid of these extra tarballs since
> they add extra overhead to a full build.

I’m guessing no one noticed the tar errors when running mvn -Pdist.  
Not sure when they started happening.

> >   - When did we stop putting CHANGES files into the source artifacts?
> 
> CHANGES files were removed by 
> https://issues.apache.org/jira/browse/HADOOP-11792

To be a bit more specific about it, the maven assembly for source only 
includes things (more or less) that are part of the git repo.  When CHANGES.txt 
was removed from the source tree, it also went away from the tar ball.  This 
isn’t too much of an issue in practice though given the notes are put up on the 
web, part of the binary tar ball, and can be generated by following the 
directions in BUILDING.txt.  I don’t remember if Hadoop uploads them into the 
dist area, but if not probably should.

> - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
> work. Not just deprecated in favor of timelineserver as was advertised.

This works for me in trunk and the bash code doesn’t appear to have 
changed in a very long time.  Probably something local to your install.  (I do 
notice that the deprecation message says “starting” which is awkward when the 
stop command is given though.)  Also: is the deprecation message even true at 
this point?

>> - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>> 
> 
> The yarn-ui profile has always been off by default, AFAIK. It's documented
> to turn it on in BUILDING.txt for release builds, and we do it in
> create-release.
> 
> IMO not a blocker. I think it's also more of a dev question (do we want to
> do this on every YARN build?) than a release one.

-1 on making yarn-ui always build.

For what is effectively an optional component (the old UI is still 
there), it’s heavy dependency requirements make it a special burden outside of 
the Docker container.  If it can be changed such that it either always 
downloads the necessary bits (regardless of the OS/chipset!) and/or doesn’t 
kill the maven build if those bits can’t be found  (i.e., truly optional), then 
I’d be less opposed.  (and, actually, quite pleased because then the docker 
image build would be significantly faster.)



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

On Mon, Nov 20, 2017 at 9:59 PM, Sangjin Lee  wrote:

>
> On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang 
> wrote:
>
>> Thanks for the spot Sangjin. I think this bug introduced in
>> create-release by HADOOP-14835. The multi-pass maven build generates these
>> dummy client jars during the site build since skipShade is specified.
>>
>> This might be enough to cancel the RC. Thoughts?
>>
>
> IMO yes. This was one of the key features mentioned in the 3.0 release
> notes. I appreciate your effort for the release Andrew!
>
>
Yea, I was leaning that way too. Let's cancel this RC. I hope to have a new
RC up tomorrow. With the upcoming holidays, we'll probably have to extend
the vote until mid-next week.

I'm also worried about the "mvn deploy" step since I thought it was safe to
specify skipShade there too. I'll check that as well.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the thorough review Vinod, some inline responses:

*Issues found during testing*
>
> Major
>  - The previously supported way of being able to use different tar-balls
> for different sub-modules is completely broken - common and HDFS tar.gz are
> completely empty.
>

Is this something people use? I figured that the sub-tarballs were a relic
from the project split, and nowadays Hadoop is one project with one release
tarball. I actually thought about getting rid of these extra tarballs since
they add extra overhead to a full build.


>  - Cannot enable new UI in YARN because it is under a non-default
> compilation flag. It should be on by default.
>

The yarn-ui profile has always been off by default, AFAIK. It's documented
to turn it on in BUILDING.txt for release builds, and we do it in
create-release.

IMO not a blocker. I think it's also more of a dev question (do we want to
do this on every YARN build?) than a release one.


>  - One decommissioned node in YARN ResourceManager UI always appears to
> start with, even when there are no NodeManagers that are started yet:
> Info :-1, DECOMMISSIONED, null rack. It shows up only in the UI though,
> not in the CLI node -list
>

Is this a blocker? Could we get a JIRA?

Thanks,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 9:46 PM, Andrew Wang 
wrote:

> Thanks for the spot Sangjin. I think this bug introduced in create-release
> by HADOOP-14835. The multi-pass maven build generates these dummy client
> jars during the site build since skipShade is specified.
>
> This might be enough to cancel the RC. Thoughts?
>

IMO yes. This was one of the key features mentioned in the 3.0 release
notes. I appreciate your effort for the release Andrew!


>
> Best,
> Andrew
>
> On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee  wrote:
>
>> I checked the client jars that are supposed to contain shaded
>> dependencies, and they don't look quite right:
>>
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
>> -rw-r--r--  0 andrew andrew44531 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
>> -rw-r--r--  0 andrew andrew45533 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
>> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.
>> 0.jar
>> -rw-r--r--  0 andrew andrew47015 Nov 14 11:53
>> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>>
>> When I look at what's inside those jar, they only seem to include
>> pom-related files with no class files. Am I missing something?
>>
>> When I build from the source with -Pdist, I do get much bigger jars:
>> total 113760
>> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
>> hadoop-client-api-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
>> hadoop-client-minicluster-3.0.0.jar
>> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
>> hadoop-client-runtime-3.0.0.jar
>>
>> Sangjin
>>
>> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:
>>
>>>
>>>
>>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>>> vino...@apache.org> wrote:
>>>
 Thanks for all the push, Andrew!

 Looking at the RC. Went through my usual check-list. Here's my summary.
 Will cast my final vote after comparing and validating my findings with
 others.

 Verification

  - [Check] Successful recompilation from source tar-ball
  - [Check] Signature verification
  - [Check] Generating dist tarballs from source tar-ball
  - [Check] Testing
 -- Start NN, DN, RM, NM, JHS, Timeline Service
 -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
 grep, pi
 -- Tested CLIs to print nodes, apps etc and also navigated UIs

 Issues found during testing

 Major
  - The previously supported way of being able to use different
 tar-balls for different sub-modules is completely broken - common and HDFS
 tar.gz are completely empty.
  - Cannot enable new UI in YARN because it is under a non-default
 compilation flag. It should be on by default.
  - One decommissioned node in YARN ResourceManager UI always appears to
 start with, even when there are no NodeManagers that are started yet:  Info
 :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
 in the CLI node -list

 Minor
  - resourcemanager-metrics.out is going into current directory instead
 of log directory
  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
 even work. Not just deprecated in favor of timelineserver as was 
 advertised.
  - Spurious warnings on CLI
 17/11/20 17:07:34 INFO conf.Configuration:
 resource-types.xml not found
 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
 find 'resource-types.xml'.

 Side notes

  - When did we stop putting CHANGES files into the source artifacts?
  - Even after "mvn install"ing once, shading is repeated again and
 again for every new 'mvn install' even though there are no source changes -
 we should see how this can be avoided.
  - Compatibility notes
 -- NM's env list is curtailed unlike in 2.x (For e.g,
 HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
 -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
 into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

>>>
>>> Sleep has always been in the jobclient test jar as long as I can
>>> remember, so it's not new for 3.0.
>>>
>>>

 Thanks
 +Vinod

 > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
 wrote:
 >
 > Hi folks,
 >
 > Thanks as always to the many, many contributors who helped with this
 > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
 > available here:
 >
 > http://people.apache.org/~wang/3.0.0-RC0/
 >
 > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
 >
 > 3.0.0 GA contains 291 fixed 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the spot Sangjin. I think this bug introduced in create-release
by HADOOP-14835. The multi-pass maven build generates these dummy client
jars during the site build since skipShade is specified.

This might be enough to cancel the RC. Thoughts?

Best,
Andrew

On Mon, Nov 20, 2017 at 7:51 PM, Sangjin Lee  wrote:

> I checked the client jars that are supposed to contain shaded
> dependencies, and they don't look quite right:
>
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
> -rw-r--r--  0 andrew andrew44531 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
> -rw-r--r--  0 andrew andrew45533 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
> $ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
> -rw-r--r--  0 andrew andrew47015 Nov 14 11:53
> hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar
>
> When I look at what's inside those jar, they only seem to include
> pom-related files with no class files. Am I missing something?
>
> When I build from the source with -Pdist, I do get much bigger jars:
> total 113760
> -rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
> hadoop-client-api-3.0.0.jar
> -rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
> hadoop-client-minicluster-3.0.0.jar
> -rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
> hadoop-client-runtime-3.0.0.jar
>
> Sangjin
>
> On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:
>
>>
>>
>> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
>> vino...@apache.org> wrote:
>>
>>> Thanks for all the push, Andrew!
>>>
>>> Looking at the RC. Went through my usual check-list. Here's my summary.
>>> Will cast my final vote after comparing and validating my findings with
>>> others.
>>>
>>> Verification
>>>
>>>  - [Check] Successful recompilation from source tar-ball
>>>  - [Check] Signature verification
>>>  - [Check] Generating dist tarballs from source tar-ball
>>>  - [Check] Testing
>>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>>> grep, pi
>>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>>
>>> Issues found during testing
>>>
>>> Major
>>>  - The previously supported way of being able to use different tar-balls
>>> for different sub-modules is completely broken - common and HDFS tar.gz are
>>> completely empty.
>>>  - Cannot enable new UI in YARN because it is under a non-default
>>> compilation flag. It should be on by default.
>>>  - One decommissioned node in YARN ResourceManager UI always appears to
>>> start with, even when there are no NodeManagers that are started yet:  Info
>>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>>> in the CLI node -list
>>>
>>> Minor
>>>  - resourcemanager-metrics.out is going into current directory instead
>>> of log directory
>>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>>  - Spurious warnings on CLI
>>> 17/11/20 17:07:34 INFO conf.Configuration:
>>> resource-types.xml not found
>>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to
>>> find 'resource-types.xml'.
>>>
>>> Side notes
>>>
>>>  - When did we stop putting CHANGES files into the source artifacts?
>>>  - Even after "mvn install"ing once, shading is repeated again and again
>>> for every new 'mvn install' even though there are no source changes - we
>>> should see how this can be avoided.
>>>  - Compatibility notes
>>> -- NM's env list is curtailed unlike in 2.x (For e.g,
>>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>>
>>
>> Sleep has always been in the jobclient test jar as long as I can
>> remember, so it's not new for 3.0.
>>
>>
>>>
>>> Thanks
>>> +Vinod
>>>
>>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>>> wrote:
>>> >
>>> > Hi folks,
>>> >
>>> > Thanks as always to the many, many contributors who helped with this
>>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>> > available here:
>>> >
>>> > http://people.apache.org/~wang/3.0.0-RC0/
>>> >
>>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>> >
>>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>> > additions include the merge of YARN resource types, API-based
>>> configuration
>>> > of the CapacityScheduler, and HDFS router-based federation.
>>> >
>>> > I've done my traditional testing with a pseudo cluster and a Pi job.
>>> My +1
>>> > to start.
>>> >
>>> > Best,
>>> > Andrew
>>>
>>>
>>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Akira Ajisaka]

>   - When did we stop putting CHANGES files into the source artifacts?

CHANGES files were removed by https://issues.apache.org/jira/browse/HADOOP-11792

>   - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see how this can 
be avoided.

Shading can be avoided by -DskipShade option and the option is documented in 
BUILDING.txt.

Regards,
Akira

On 2017/11/21 10:26, Vinod Kumar Vavilapalli wrote:

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

  - [Check] Successful recompilation from source tar-ball
  - [Check] Signature verification
  - [Check] Generating dist tarballs from source tar-ball
  - [Check] Testing
 -- Start NN, DN, RM, NM, JHS, Timeline Service
 -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, 
pi
 -- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
  - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
  - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
  - One decommissioned node in YARN ResourceManager UI always appears to start with, 
even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI node 
-list

Minor
  - resourcemanager-metrics.out is going into current directory instead of log 
directory
  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even 
work. Not just deprecated in favor of timelineserver as was advertised.
  - Spurious warnings on CLI
 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

  - When did we stop putting CHANGES files into the source artifacts?
  - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see how this 
can be avoided.
  - Compatibility notes
 -- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME 
is not automatically inherited. Correct behavior)
 -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod


On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew





-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

I checked the client jars that are supposed to contain shaded dependencies,
and they don't look quite right:

$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-api-3.0.0.jar
-rw-r--r--  0 andrew andrew44531 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-api-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-runtime-3.0.0.jar
-rw-r--r--  0 andrew andrew45533 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-runtime-3.0.0.jar
$ tar -tzvf hadoop-3.0.0.tar.gz | grep hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  0 andrew andrew47015 Nov 14 11:53
hadoop-3.0.0/share/hadoop/client/hadoop-client-minicluster-3.0.0.jar

When I look at what's inside those jar, they only seem to include
pom-related files with no class files. Am I missing something?

When I build from the source with -Pdist, I do get much bigger jars:
total 113760
-rw-r--r--  1 sangjinlee  120039211  17055399 Nov 20 17:17
hadoop-client-api-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20451447 Nov 20 17:19
hadoop-client-minicluster-3.0.0.jar
-rw-r--r--  1 sangjinlee  120039211  20730866 Nov 20 17:18
hadoop-client-runtime-3.0.0.jar

Sangjin

On Mon, Nov 20, 2017 at 5:52 PM, Sangjin Lee  wrote:

>
>
> On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
>
>> Thanks for all the push, Andrew!
>>
>> Looking at the RC. Went through my usual check-list. Here's my summary.
>> Will cast my final vote after comparing and validating my findings with
>> others.
>>
>> Verification
>>
>>  - [Check] Successful recompilation from source tar-ball
>>  - [Check] Signature verification
>>  - [Check] Generating dist tarballs from source tar-ball
>>  - [Check] Testing
>> -- Start NN, DN, RM, NM, JHS, Timeline Service
>> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
>> grep, pi
>> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>>
>> Issues found during testing
>>
>> Major
>>  - The previously supported way of being able to use different tar-balls
>> for different sub-modules is completely broken - common and HDFS tar.gz are
>> completely empty.
>>  - Cannot enable new UI in YARN because it is under a non-default
>> compilation flag. It should be on by default.
>>  - One decommissioned node in YARN ResourceManager UI always appears to
>> start with, even when there are no NodeManagers that are started yet:  Info
>> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
>> in the CLI node -list
>>
>> Minor
>>  - resourcemanager-metrics.out is going into current directory instead of
>> log directory
>>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't
>> even work. Not just deprecated in favor of timelineserver as was advertised.
>>  - Spurious warnings on CLI
>> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
>> not found
>> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
>> 'resource-types.xml'.
>>
>> Side notes
>>
>>  - When did we stop putting CHANGES files into the source artifacts?
>>  - Even after "mvn install"ing once, shading is repeated again and again
>> for every new 'mvn install' even though there are no source changes - we
>> should see how this can be avoided.
>>  - Compatibility notes
>> -- NM's env list is curtailed unlike in 2.x (For e.g,
>> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
>> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
>> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>>
>
> Sleep has always been in the jobclient test jar as long as I can remember,
> so it's not new for 3.0.
>
>
>>
>> Thanks
>> +Vinod
>>
>> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>> wrote:
>> >
>> > Hi folks,
>> >
>> > Thanks as always to the many, many contributors who helped with this
>> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> > available here:
>> >
>> > http://people.apache.org/~wang/3.0.0-RC0/
>> >
>> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> >
>> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> > additions include the merge of YARN resource types, API-based
>> configuration
>> > of the CapacityScheduler, and HDFS router-based federation.
>> >
>> > I've done my traditional testing with a pseudo cluster and a Pi job. My
>> +1
>> > to start.
>> >
>> > Best,
>> > Andrew
>>
>>
>

RE: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Chen, Sammi]

+1 (binding)

1) Compile with native from source code
2) Set up a one-node cluster, enabling short-circuit read
3) Tried some basic HDFS commands, ls, put etc. 
4) MapReduce workloads: TestDFSIO, TeraGen and TeraSort
5) Test storage policy and mover successfully

Sorry for the late response. 

-Original Message-
From: John Zhuge [mailto:john.zh...@gmail.com]
Sent: Friday, November 17, 2017 6:56 AM
To: Andrew Wang <andrew.w...@cloudera.com>
Cc: common-...@hadoop.apache.org; yarn-...@hadoop.apache.org; 
mapreduce-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

+1 (binding)

   - Verified checksums of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Passed all S3A and ADL integration tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (skipped in SSL+Kerberos mode)
  - KMS and HttpFS basic
  - Balancer start/stop


On Tue, Nov 14, 2017 at 1:34 PM, Andrew Wang <andrew.w...@cloudera.com>
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this 
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are 
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable 
> additions include the merge of YARN resource types, API-based 
> configuration of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. 
> My +1 to start.
>
> Best,
> Andrew
>



--
John

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Sangjin Lee]

On Mon, Nov 20, 2017 at 5:26 PM, Vinod Kumar Vavilapalli  wrote:

> Thanks for all the push, Andrew!
>
> Looking at the RC. Went through my usual check-list. Here's my summary.
> Will cast my final vote after comparing and validating my findings with
> others.
>
> Verification
>
>  - [Check] Successful recompilation from source tar-ball
>  - [Check] Signature verification
>  - [Check] Generating dist tarballs from source tar-ball
>  - [Check] Testing
> -- Start NN, DN, RM, NM, JHS, Timeline Service
> -- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort,
> grep, pi
> -- Tested CLIs to print nodes, apps etc and also navigated UIs
>
> Issues found during testing
>
> Major
>  - The previously supported way of being able to use different tar-balls
> for different sub-modules is completely broken - common and HDFS tar.gz are
> completely empty.
>  - Cannot enable new UI in YARN because it is under a non-default
> compilation flag. It should be on by default.
>  - One decommissioned node in YARN ResourceManager UI always appears to
> start with, even when there are no NodeManagers that are started yet:  Info
> :-1, DECOMMISSIONED, null rack. It shows up only in the UI though, not
> in the CLI node -list
>
> Minor
>  - resourcemanager-metrics.out is going into current directory instead of
> log directory
>  - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even
> work. Not just deprecated in favor of timelineserver as was advertised.
>  - Spurious warnings on CLI
> 17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml
> not found
> 17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find
> 'resource-types.xml'.
>
> Side notes
>
>  - When did we stop putting CHANGES files into the source artifacts?
>  - Even after "mvn install"ing once, shading is repeated again and again
> for every new 'mvn install' even though there are no source changes - we
> should see how this can be avoided.
>  - Compatibility notes
> -- NM's env list is curtailed unlike in 2.x (For e.g,
> HADOOP_MAPRED_HOME is not automatically inherited. Correct behavior)
> -- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar
> into hadoop-mapreduce-client-jobclient-3.0.0-tests.jar
>

Sleep has always been in the jobclient test jar as long as I can remember,
so it's not new for 3.0.


>
> Thanks
> +Vinod
>
> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Thanks for all the push, Andrew!

Looking at the RC. Went through my usual check-list. Here's my summary. Will 
cast my final vote after comparing and validating my findings with others.

Verification

 - [Check] Successful recompilation from source tar-ball
 - [Check] Signature verification
 - [Check] Generating dist tarballs from source tar-ball
 - [Check] Testing
-- Start NN, DN, RM, NM, JHS, Timeline Service
-- Ran dist-shell example, MR sleep, wordcount, randomwriter, sort, grep, pi
-- Tested CLIs to print nodes, apps etc and also navigated UIs

Issues found during testing

Major
 - The previously supported way of being able to use different tar-balls for 
different sub-modules is completely broken - common and HDFS tar.gz are 
completely empty.
 - Cannot enable new UI in YARN because it is under a non-default compilation 
flag. It should be on by default.
 - One decommissioned node in YARN ResourceManager UI always appears to start 
with, even when there are no NodeManagers that are started yet:  Info :-1, 
DECOMMISSIONED, null rack. It shows up only in the UI though, not in the CLI 
node -list

Minor
 - resourcemanager-metrics.out is going into current directory instead of log 
directory
 - $HADOOP_YARN_HOME/sbin/yarn-daemon.sh start historyserver doesn't even work. 
Not just deprecated in favor of timelineserver as was advertised.
 - Spurious warnings on CLI
17/11/20 17:07:34 INFO conf.Configuration: resource-types.xml not 
found
17/11/20 17:07:34 INFO resource.ResourceUtils: Unable to find 
'resource-types.xml'.

Side notes

 - When did we stop putting CHANGES files into the source artifacts?
 - Even after "mvn install"ing once, shading is repeated again and again for 
every new 'mvn install' even though there are no source changes - we should see 
how this can be avoided.
 - Compatibility notes
-- NM's env list is curtailed unlike in 2.x (For e.g, HADOOP_MAPRED_HOME is 
not automatically inherited. Correct behavior)
-- Sleep is moved from hadoop-mapreduce-client-jobclient-3.0.0.jar into 
hadoop-mapreduce-client-jobclient-3.0.0-tests.jar

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Compilation passed for me. Using jdk1.8.0_40.jdk.

+Vinod

> On Nov 20, 2017, at 4:16 PM, Wei-Chiu Chuang  wrote:
> 
> @vinod
> I followed your command but I could not reproduce your problem.
> 
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
> ommon/target/hadoop-common-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
> hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
> [weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
> s/target/hadoop-hdfs-3.0.0.tar.gz
> -rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
> hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
> 
> During compilation I found the following error with a Java 1.8.0_5 JDK:
> 
> [ERROR] Failed to execute goal org.apache.maven.plugins:maven
> -compiler-plugin:3.1:testCompile (default-testCompile) on project
> hadoop-aws: Compilation failure: Compilation failure:
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> [ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
> test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
> reference to intercept is ambiguous
> [ERROR]   both method intercept(java.lang.Class> ,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
> org.apache.hadoop.test.LambdaTestUtils and method
> intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
> in org.apache.hadoop.test.LambdaTestUtils match
> 
> And then I realized Ray filed HADOOP-14900
>  for the same
> issue. This problem is not reproducible with a more recent JDK 8, such as
> 1.8.0_151
> Maybe it would be a good idea to name a list of JDKs that are known to be
> buggy. Can we get this documented somewhere? I don't consider it a blocker
> so a release note in a later release or a wiki entry should be good enough.
> 
> On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
> vino...@apache.org> wrote:
> 
>> Quick question.
>> 
>> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
>> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
>> voted on (hadoop-3.0.0-src.tar.gz).
>> 
>> The idea is to install HDFS, YARN, MR separately in separate
>> root-directories from the generated individual dist tarballs.
>> 
>> But now I see that HDFS and common dist tarballs are empty
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
>> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
>> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
>> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>> 
>> But YARN and MR are fine
>> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
>> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
>> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
>> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>> 
>> Is it just me? Or is this broken?
>> 
>> Thanks
>> +Vinod
>> 
>>> On Nov 14, 2017, at 1:34 PM, Andrew Wang 
>> wrote:
>>> 
>>> Hi folks,
>>> 
>>> Thanks as always to the many, many contributors who helped with this
>>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>> available here:
>>> 
>>> http://people.apache.org/~wang/3.0.0-RC0/
>>> 
>>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>> 
>>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Wei-Chiu Chuang]

@vinod
I followed your command but I could not reproduce your problem.

[weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-common-project/hadoop-c
ommon/target/hadoop-common-3.0.0.tar.gz
-rw-rw-r-- 1 weichiu weichiu 37052439 Nov 20 21:59
hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz
[weichiu@storage-1 hadoop-3.0.0-src]$ ls -al hadoop-hdfs-project/hadoop-hdf
s/target/hadoop-hdfs-3.0.0.tar.gz
-rw-rw-r-- 1 weichiu weichiu 29044067 Nov 20 22:00
hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

During compilation I found the following error with a Java 1.8.0_5 JDK:

[ERROR] Failed to execute goal org.apache.maven.plugins:maven
-compiler-plugin:3.1:testCompile (default-testCompile) on project
hadoop-aws: Compilation failure: Compilation failure:
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[45,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[69,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[94,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match
[ERROR] /home/weichiu/hadoop-3.0.0-src/hadoop-tools/hadoop-aws/src/
test/java/org/apache/hadoop/fs/s3a/ITestS3AEncryptionAlgorithmValidation.java:[120,5]
reference to intercept is ambiguous
[ERROR]   both method intercept(java.lang.Class,java.lang.String,org.apache.hadoop.test.LambdaTestUtils.VoidCallable) in
org.apache.hadoop.test.LambdaTestUtils and method
intercept(java.lang.Class,java.lang.String,java.util.concurrent.Callable)
in org.apache.hadoop.test.LambdaTestUtils match

And then I realized Ray filed HADOOP-14900
 for the same
issue. This problem is not reproducible with a more recent JDK 8, such as
1.8.0_151
Maybe it would be a good idea to name a list of JDKs that are known to be
buggy. Can we get this documented somewhere? I don't consider it a blocker
so a release note in a later release or a wiki entry should be good enough.

On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli <
vino...@apache.org> wrote:

> Quick question.
>
> I used to be able (in 2.x line) to create dist tarballs (mvn clean install
> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being
> voted on (hadoop-3.0.0-src.tar.gz).
>
> The idea is to install HDFS, YARN, MR separately in separate
> root-directories from the generated individual dist tarballs.
>
> But now I see that HDFS and common dist tarballs are empty
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39
> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40
> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>
> But YARN and MR are fine
> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41
> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41
> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>
> Is it just me? Or is this broken?
>
> Thanks
> +Vinod
>
> > On Nov 14, 2017, at 1:34 PM, Andrew Wang 
> wrote:
> >
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
>
>
> 

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Xiao Chen]

+1 (binding)

Thanks Andrew!


   - Verified md5 and built from source
   - Started a pseudo distributed cluster with KMS,
   - Performed basic hdfs operations plus encryption related operations
   - Verified logs and webui
   - Confidence from CDH testings (will let Andrew answer officially, but
   we have smokes and nightlies for various downstream projects)


-Xiao

On Mon, Nov 20, 2017 at 3:38 PM, Shane Kumpf 
wrote:

> Thanks, Andrew!
>
> +1 (non-binding)
>
> - Verified checksums and signatures
> - Deployed a single node cluster on CentOS 7.4 using the binary and source
> release
> - Ran hdfs commands
> - Ran pi and distributed shell using the default and docker runtimes
> - Verified the UIs
> - Verified the change log
>
> -Shane
>
>
> On Tue, Nov 14, 2017 at 2:34 PM, Andrew Wang 
> wrote:
>
> > Hi folks,
> >
> > Thanks as always to the many, many contributors who helped with this
> > release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> > available here:
> >
> > http://people.apache.org/~wang/3.0.0-RC0/
> >
> > This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> >
> > 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> > additions include the merge of YARN resource types, API-based
> configuration
> > of the CapacityScheduler, and HDFS router-based federation.
> >
> > I've done my traditional testing with a pseudo cluster and a Pi job. My
> +1
> > to start.
> >
> > Best,
> > Andrew
> >
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Shane Kumpf]

Thanks, Andrew!

+1 (non-binding)

- Verified checksums and signatures
- Deployed a single node cluster on CentOS 7.4 using the binary and source
release
- Ran hdfs commands
- Ran pi and distributed shell using the default and docker runtimes
- Verified the UIs
- Verified the change log

-Shane


On Tue, Nov 14, 2017 at 2:34 PM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Lei Xu]

+1 binding

Run the following steps:

* Check md5 of sources and package.
* Run a YARN + HDFS pseudo cluster.
* Run terasuite on YARN.
* Run HDFS CLIs (ls , rm , etc)


On Mon, Nov 20, 2017 at 12:58 PM, Vinod Kumar Vavilapalli
 wrote:
> Quick question.
>
> I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
> -Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being 
> voted on (hadoop-3.0.0-src.tar.gz).
>
> The idea is to install HDFS, YARN, MR separately in separate root-directories 
> from the generated individual dist tarballs.
>
> But now I see that HDFS and common dist tarballs are empty
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
> ./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz -
> -rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
> ./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz
>
> But YARN and MR are fine
> -rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
> ./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
> -rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
> ./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz
>
> Is it just me? Or is this broken?
>
> Thanks
> +Vinod
>
>> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
>>
>> Hi folks,
>>
>> Thanks as always to the many, many contributors who helped with this
>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> available here:
>>
>> http://people.apache.org/~wang/3.0.0-RC0/
>>
>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>
>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> additions include the merge of YARN resource types, API-based configuration
>> of the CapacityScheduler, and HDFS router-based federation.
>>
>> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
>> to start.
>>
>> Best,
>> Andrew
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>



-- 
Lei (Eddy) Xu
Software Engineer, Cloudera

-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

Quick question.

I used to be able (in 2.x line) to create dist tarballs (mvn clean install 
-Pdist -Dtar -DskipTests -Dmaven.javadoc.skip=true) from the source being voted 
on (hadoop-3.0.0-src.tar.gz).

The idea is to install HDFS, YARN, MR separately in separate root-directories 
from the generated individual dist tarballs.

But now I see that HDFS and common dist tarballs are empty
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:39 
./hadoop-common-project/hadoop-common/target/hadoop-common-3.0.0.tar.gz - 
-rw-r--r--  1 vinodkv  staff 45 Nov 20 12:40 
./hadoop-hdfs-project/hadoop-hdfs/target/hadoop-hdfs-3.0.0.tar.gz

But YARN and MR are fine
-rw-r--r--  1 vinodkv  staff   64474187 Nov 20 12:41 
./hadoop-yarn-project/target/hadoop-yarn-project-3.0.0.tar.gz
-rw-r--r--  1 vinodkv  staff   21674457 Nov 20 12:41 
./hadoop-mapreduce-project/target/hadoop-mapreduce-3.0.0.tar.gz

Is it just me? Or is this broken?

Thanks
+Vinod

> On Nov 14, 2017, at 1:34 PM, Andrew Wang  wrote:
> 
> Hi folks,
> 
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
> 
> http://people.apache.org/~wang/3.0.0-RC0/
> 
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
> 
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
> 
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
> 
> Best,
> Andrew


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Thanks for that proposal Andrew, and for not wrapping up the vote yesterday.

> In terms of downstream testing, we've done extensive
> integration testing with downstreams via the alphas
> and betas, and we have continuous integration running
> at Cloudera against branch-3.0.

Could you please share what kind of downstream testing you have performed and 
with which downstream projects?

Regards,
Arpit


On 11/17/17, 3:27 PM, "Andrew Wang"  wrote:

Hi Arpit,

I agree the timing is not great here, but extending it to meaningfully
avoid the holidays would mean extending it an extra week (e.g. to the
29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
really like to get the RC out before then.

In terms of downstream testing, we've done extensive integration testing
with downstreams via the alphas and betas, and we have continuous
integration running at Cloudera against branch-3.0. Because of this, I have
more confidence in our integration for 3.0.0 than most Hadoop releases.

Is it meaningful to extend to say, the 21st, which provides for a full week
of voting?

Best,
Andrew

On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
wrote:

> Hi Andrew,
>
> Thank you for your hard work in getting us to this step. This is our first
> major GA release in many years.
>
> I feel a 5-day vote window ending over the weekend before thanksgiving may
> not provide sufficient time to evaluate this RC especially for downstream
> components.
>
> Would you please consider extending the voting deadline until a few days
> after the thanksgiving holiday? It would be a courtesy to our broader
> community and I see no harm in giving everyone a few days to evaluate it
> more thoroughly.
>
> On a lighter note, your deadline is also 4 minutes short of the required 5
> days. :)
>
> Regards,
> Arpit
>
>
>
> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>
> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based
> configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job.
> My +1
> to start.
>
> Best,
> Andrew
>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Vinod Kumar Vavilapalli]

I'd definitely extend it for a few more days. I only see 3 binding +1s so far - 
not a great number to brag about on our first major release in years.

Also going to nudge folks into voting.

+Vinod

> On Nov 17, 2017, at 3:26 PM, Andrew Wang  wrote:
> 
> Hi Arpit,
> 
> I agree the timing is not great here, but extending it to meaningfully
> avoid the holidays would mean extending it an extra week (e.g. to the
> 29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
> really like to get the RC out before then.
> 
> In terms of downstream testing, we've done extensive integration testing
> with downstreams via the alphas and betas, and we have continuous
> integration running at Cloudera against branch-3.0. Because of this, I have
> more confidence in our integration for 3.0.0 than most Hadoop releases.
> 
> Is it meaningful to extend to say, the 21st, which provides for a full week
> of voting?
> 
> Best,
> Andrew
> 
> On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
> wrote:
> 
>> Hi Andrew,
>> 
>> Thank you for your hard work in getting us to this step. This is our first
>> major GA release in many years.
>> 
>> I feel a 5-day vote window ending over the weekend before thanksgiving may
>> not provide sufficient time to evaluate this RC especially for downstream
>> components.
>> 
>> Would you please consider extending the voting deadline until a few days
>> after the thanksgiving holiday? It would be a courtesy to our broader
>> community and I see no harm in giving everyone a few days to evaluate it
>> more thoroughly.
>> 
>> On a lighter note, your deadline is also 4 minutes short of the required 5
>> days. :)
>> 
>> Regards,
>> Arpit
>> 
>> 
>> 
>> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>> 
>>Hi folks,
>> 
>>Thanks as always to the many, many contributors who helped with this
>>release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>>available here:
>> 
>>http://people.apache.org/~wang/3.0.0-RC0/
>> 
>>This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>> 
>>3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>>additions include the merge of YARN resource types, API-based
>> configuration
>>of the CapacityScheduler, and HDFS router-based federation.
>> 
>>I've done my traditional testing with a pseudo cluster and a Pi job.
>> My +1
>>to start.
>> 
>>Best,
>>Andrew
>> 
>> 
>> 


-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Rakesh Radhakrishnan]

Thanks Andrew for getting this out !

+1 (non-binding)

* Built from source on CentOS 7.3.1611, jdk1.8.0_111
* Deployed non-ha cluster and tested few EC file operations.
* Ran basic shell commands(ls, mkdir, put, get, ec, dfsadmin).
* Ran some sample jobs.
* HDFS Namenode UI looks good.

Thanks,
Rakesh

On Wed, Nov 15, 2017 at 3:04 AM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Brahma Reddy Battula]

+1 (non-binding).

-Built from the source
-Installed the HA Cluster
-Ran basic shell commands
-Ran sample jobs like pi,Slive

Thanks Andrew for driving this.

On Wed, Nov 15, 2017 at 3:04 AM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>



-- 



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Hi Arpit,

I agree the timing is not great here, but extending it to meaningfully
avoid the holidays would mean extending it an extra week (e.g. to the
29th). We've been coordinating with ASF PR for that Tuesday, so I'd really,
really like to get the RC out before then.

In terms of downstream testing, we've done extensive integration testing
with downstreams via the alphas and betas, and we have continuous
integration running at Cloudera against branch-3.0. Because of this, I have
more confidence in our integration for 3.0.0 than most Hadoop releases.

Is it meaningful to extend to say, the 21st, which provides for a full week
of voting?

Best,
Andrew

On Fri, Nov 17, 2017 at 1:27 PM, Arpit Agarwal 
wrote:

> Hi Andrew,
>
> Thank you for your hard work in getting us to this step. This is our first
> major GA release in many years.
>
> I feel a 5-day vote window ending over the weekend before thanksgiving may
> not provide sufficient time to evaluate this RC especially for downstream
> components.
>
> Would you please consider extending the voting deadline until a few days
> after the thanksgiving holiday? It would be a courtesy to our broader
> community and I see no harm in giving everyone a few days to evaluate it
> more thoroughly.
>
> On a lighter note, your deadline is also 4 minutes short of the required 5
> days. :)
>
> Regards,
> Arpit
>
>
>
> On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:
>
> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based
> configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job.
> My +1
> to start.
>
> Best,
> Andrew
>
>
>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Eric Payne]

Thanks for the all of the work it took to finally get htere.
+1 (binding)
Built from source and stood up a pseudo cluster with 4 NMs
Tested the following:
o Cross queue preemption
o Restarting the RM preserves work
o User limits are honored during In-queue preemption 

o Priorities are honored during In-queue preemption
o Users with different weights are assigned resources proportional to their 
weights.
o User weights are refreshable, and in-queue preemption works to honor the 
post-refresh weights 

Thanks,
-Eric Payne

  From: Andrew Wang 
 To: "common-...@hadoop.apache.org" ; 
"yarn-...@hadoop.apache.org" ; 
"mapreduce-dev@hadoop.apache.org" ; 
"hdfs-...@hadoop.apache.org"  
 Sent: Tuesday, November 14, 2017 3:34 PM
 Subject: [VOTE] Release Apache Hadoop 3.0.0 RC0
   
Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew


   

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Arpit Agarwal]

Hi Andrew,

Thank you for your hard work in getting us to this step. This is our first 
major GA release in many years.

I feel a 5-day vote window ending over the weekend before thanksgiving may not 
provide sufficient time to evaluate this RC especially for downstream 
components.

Would you please consider extending the voting deadline until a few days after 
the thanksgiving holiday? It would be a courtesy to our broader community and I 
see no harm in giving everyone a few days to evaluate it more thoroughly.

On a lighter note, your deadline is also 4 minutes short of the required 5 
days. :)

Regards,
Arpit



On 11/14/17, 1:34 PM, "Andrew Wang"  wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Andrew Wang]

Thanks for the spot, normally create-release spits those out. I uploaded
asc and mds for the release artifacts.

Best,
Andrew

On Thu, Nov 16, 2017 at 11:33 PM, Akira Ajisaka  wrote:

> Hi Andrew,
>
> Signatures are missing. Would you upload them?
>
> Thanks,
> Akira
>
>
> On 2017/11/15 6:34, Andrew Wang wrote:
>
>> Hi folks,
>>
>> Thanks as always to the many, many contributors who helped with this
>> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
>> available here:
>>
>> http://people.apache.org/~wang/3.0.0-RC0/
>>
>> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>>
>> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
>> additions include the merge of YARN resource types, API-based
>> configuration
>> of the CapacityScheduler, and HDFS router-based federation.
>>
>> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
>> to start.
>>
>> Best,
>> Andrew
>>
>>

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[Akira Ajisaka]

Hi Andrew,

Signatures are missing. Would you upload them?

Thanks,
Akira

On 2017/11/15 6:34, Andrew Wang wrote:

Hi folks,

Thanks as always to the many, many contributors who helped with this
release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
available here:

http://people.apache.org/~wang/3.0.0-RC0/

This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.

3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
additions include the merge of YARN resource types, API-based configuration
of the CapacityScheduler, and HDFS router-based federation.

I've done my traditional testing with a pseudo cluster and a Pi job. My +1
to start.

Best,
Andrew



-
To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.0.0 RC0

[John Zhuge]

+1 (binding)

   - Verified checksums of all tarballs
   - Built source with native, Java 1.8.0_131-b11 on Mac OS X 10.12.6
   - Passed all S3A and ADL integration tests
   - Deployed both binary and built source to a pseudo cluster, passed the
   following sanity tests in insecure, SSL, and SSL+Kerberos mode:
  - HDFS basic and ACL
  - DistCp basic
  - MapReduce wordcount (skipped in SSL+Kerberos mode)
  - KMS and HttpFS basic
  - Balancer start/stop


On Tue, Nov 14, 2017 at 1:34 PM, Andrew Wang 
wrote:

> Hi folks,
>
> Thanks as always to the many, many contributors who helped with this
> release. I've created RC0 for Apache Hadoop 3.0.0. The artifacts are
> available here:
>
> http://people.apache.org/~wang/3.0.0-RC0/
>
> This vote will run 5 days, ending on Nov 19th at 1:30pm Pacific.
>
> 3.0.0 GA contains 291 fixed JIRA issues since 3.0.0-beta1. Notable
> additions include the merge of YARN resource types, API-based configuration
> of the CapacityScheduler, and HDFS router-based federation.
>
> I've done my traditional testing with a pseudo cluster and a Pi job. My +1
> to start.
>
> Best,
> Andrew
>



-- 
John