Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-05 Thread Akira Ajisaka 
+1

-Akira

On Wed, Feb 6, 2019 at 9:13 AM Wangda Tan  wrote:
>
> +1, make sense to me.
>
> On Tue, Feb 5, 2019 at 3:29 PM Konstantin Shvachko 
> wrote:
>
> > +1 Makes sense to me.
> >
> > Thanks,
> > --Konst
> >
> > On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:
> >
> > > Hello,
> > >
> > > Starting a vote based on the discuss thread [1] for moving branch-2
> > > precommit/nightly test builds to openjdk8. After this change, the test
> > > phase for precommit builds [2] and branch-2 nightly build [3] will run on
> > > openjdk8. To maintain source compatibility, these builds will still run
> > > their compile phase for branch-2 on openjdk7 as they do now (in addition
> > to
> > > compiling on openjdk8).
> > >
> > > Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
> > >
> > > [1]
> > >
> > >
> > https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> > >
> > > [2]
> > >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> > >
> > >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> > >
> > > [3]
> > >
> > >
> > https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> > >
> > > Jonathan Hung
> > >
> >

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

2019-02-05 Thread Vinayakumar B 
Sorry for being late.

Thanks everyone for the Release efforts.

+1 (Binding)

-Built from tag on Ubuntu-18.04
-Successful Native build.
-Deployed a 3-Node docker cluster with kerberos Security Enabled on Centos7
images.
-Verified basic HDFS Commands
-Ran basic wordcount Job.

-Vinay

On Wed, Feb 6, 2019 at 2:38 AM Wangda Tan  wrote:

> Thanks everyone for voting, we have 6 binding votes and 6 non-binding
> votes, no vetos. The vote passed. We will go ahead to stage this release.
>
> - Wangda and Sunil.
>
> On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
> wrote:
>
> > Thanks Sunil, the new source tarball matches the RC tag and its checksum
> > and signature look good.
> >
> > Billie
> >
> > On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
> >
> > > Thanks Billie for pointing out.
> > > I have updated source by removing patchprocess and extra line create
> > > release.
> > >
> > > Also updated checksum as well.
> > >
> > > @bil...@apache.org   @Wangda Tan  >
> > > please help to verify this changed bit once.
> > >
> > > Thanks
> > > Sunil
> > >
> > > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi <
> billie.rina...@gmail.com>
> > > wrote:
> > >
> > >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> > >> patchprocess directory with some yetus code in it. Also, the file
> > >> dev-support/bin/create-release file has the following line added:
> > >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> > >> gpg-agent):1"
> > >>
> > >> I think we are probably due for an overall review of LICENSE and
> NOTICE.
> > >> I saw some idiosyncrasies there but nothing that looked like a
> blocker.
> > >>
> > >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> > >>>
> > >>> The artifacts are available here:
> > >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> > >>>
> > >>> The RC tag in git is release-3.1.2-RC1:
> > >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> > >>>
> > >>> The maven artifacts are available via repository.apache.org at
> > >>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1215
> > >>>
> > >>> This vote will run 5 days from now.
> > >>>
> > >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> > >>>
> > >>> We have done testing with a pseudo cluster and distributed shell job.
> > >>>
> > >>> My +1 to start.
> > >>>
> > >>> Best,
> > >>> Wangda Tan and Sunil Govindan
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> > (3.1.2)
> > >>> ORDER BY priority DESC
> > >>>
> > >>
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-05 Thread Wangda Tan 
+1, make sense to me.

On Tue, Feb 5, 2019 at 3:29 PM Konstantin Shvachko 
wrote:

> +1 Makes sense to me.
>
> Thanks,
> --Konst
>
> On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:
>
> > Hello,
> >
> > Starting a vote based on the discuss thread [1] for moving branch-2
> > precommit/nightly test builds to openjdk8. After this change, the test
> > phase for precommit builds [2] and branch-2 nightly build [3] will run on
> > openjdk8. To maintain source compatibility, these builds will still run
> > their compile phase for branch-2 on openjdk7 as they do now (in addition
> to
> > compiling on openjdk8).
> >
> > Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
> >
> > [1]
> >
> >
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> >
> > [2]
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> >
> > [3]
> >
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> >
> > Jonathan Hung
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-05 Thread Konstantin Shvachko 
+1 Makes sense to me.

Thanks,
--Konst

On Mon, Feb 4, 2019 at 6:14 PM Jonathan Hung  wrote:

> Hello,
>
> Starting a vote based on the discuss thread [1] for moving branch-2
> precommit/nightly test builds to openjdk8. After this change, the test
> phase for precommit builds [2] and branch-2 nightly build [3] will run on
> openjdk8. To maintain source compatibility, these builds will still run
> their compile phase for branch-2 on openjdk7 as they do now (in addition to
> compiling on openjdk8).
>
> Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
>
> [1]
>
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
>
> [2]
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
>
> [3]
>
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
>
> Jonathan Hung
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

2019-02-05 Thread Wangda Tan 
Thanks everyone for voting, we have 6 binding votes and 6 non-binding
votes, no vetos. The vote passed. We will go ahead to stage this release.

- Wangda and Sunil.

On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
wrote:

> Thanks Sunil, the new source tarball matches the RC tag and its checksum
> and signature look good.
>
> Billie
>
> On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
>
> > Thanks Billie for pointing out.
> > I have updated source by removing patchprocess and extra line create
> > release.
> >
> > Also updated checksum as well.
> >
> > @bil...@apache.org   @Wangda Tan 
> > please help to verify this changed bit once.
> >
> > Thanks
> > Sunil
> >
> > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
> > wrote:
> >
> >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> >> patchprocess directory with some yetus code in it. Also, the file
> >> dev-support/bin/create-release file has the following line added:
> >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> >> gpg-agent):1"
> >>
> >> I think we are probably due for an overall review of LICENSE and NOTICE.
> >> I saw some idiosyncrasies there but nothing that looked like a blocker.
> >>
> >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> >>>
> >>> The artifacts are available here:
> >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> >>>
> >>> The RC tag in git is release-3.1.2-RC1:
> >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> >>>
> >>> This vote will run 5 days from now.
> >>>
> >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> >>>
> >>> We have done testing with a pseudo cluster and distributed shell job.
> >>>
> >>> My +1 to start.
> >>>
> >>> Best,
> >>> Wangda Tan and Sunil Govindan
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.2)
> >>> ORDER BY priority DESC
> >>>
> >>
>

[jira] [Created] (HADOOP-16094) AuthenticationFilter can trigger NullPointerException in KerberosName class

2019-02-05 Thread Eric Yang (JIRA) 
Eric Yang created HADOOP-16094:
--

 Summary: AuthenticationFilter can trigger NullPointerException in 
KerberosName class
 Key: HADOOP-16094
 URL: https://issues.apache.org/jira/browse/HADOOP-16094
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Eric Yang


Hadoop AuthenticationFilter example can fail with NullPointerException if 
auth_to_local rules has not been parsed from Configuration object.  This can 
happen if the web application does not have any initialization code that leads 
to triggering: UserGroupInformation.initialize(conf, boolean);

Stacktrace:
{code}
2019-02-05 20:08:05,668 [http-bio-8080-exec-11] DEBUG 
org.apache.hadoop.security.authentication.server.AuthenticationFilter- 
Authentication exception: java.lang.NullPointerException
org.apache.hadoop.security.authentication.client.AuthenticationException: 
java.lang.NullPointerException
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:315)
at 
org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:536)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:506)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1087)
at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at 
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at 
org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.NullPointerException
at 
org.apache.hadoop.security.authentication.util.KerberosName.getShortName(KerberosName.java:422)
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.runWithPrincipal(KerberosAuthenticationHandler.java:352)
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.access$000(KerberosAuthenticationHandler.java:64)
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:304)
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler$2.run(KerberosAuthenticationHandler.java:301)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.authentication.server.KerberosAuthenticationHandler.authenticate(KerberosAuthenticationHandler.java:300)
... 18 more
{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

2019-02-05 Thread Ayush Saxena 
Thanks Sunil and Wangda for driving the release.

+1 (non binding)

-Built from source on Ubuntu-18.04
-Successful Native build.
-Verified basic HDFS Commands.
-Verified basic Erasure Coding Commands.
-Verified basic RBF commands.
-Browsed HDFS and RBF UI.

Thanks
-Ayush

On Wed, 6 Feb 2019 at 01:07, Arpit Agarwal 
wrote:

> +1 binding for updated source package.
>
>   - Rechecked signatures and checksums
>   - Source matches release git tag
>   - Built from source
>
>
> > On Feb 5, 2019, at 10:50 AM, Sunil G  wrote:
> >
> > Thanks Billie for pointing out.
> > I have updated source by removing patchprocess and extra line create
> > release.
> >
> > Also updated checksum as well.
> >
> > @bil...@apache.org   @Wangda Tan 
> > please help to verify this changed bit once.
> >
> > Thanks
> > Sunil
> >
> > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
> > wrote:
> >
> >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> >> patchprocess directory with some yetus code in it. Also, the file
> >> dev-support/bin/create-release file has the following line added:
> >>  export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> >> gpg-agent):1"
> >>
> >> I think we are probably due for an overall review of LICENSE and
> NOTICE. I
> >> saw some idiosyncrasies there but nothing that looked like a blocker.
> >>
> >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> >>
> >>> Hi Folks,
> >>>
> >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> >>>
> >>> The artifacts are available here:
> >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> >>>
> >>> The RC tag in git is release-3.1.2-RC1:
> >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> >>>
> >>> The maven artifacts are available via repository.apache.org at
> >>>
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> >>>
> >>> This vote will run 5 days from now.
> >>>
> >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> >>>
> >>> We have done testing with a pseudo cluster and distributed shell job.
> >>>
> >>> My +1 to start.
> >>>
> >>> Best,
> >>> Wangda Tan and Sunil Govindan
> >>>
> >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.2)
> >>> ORDER BY priority DESC
> >>>
> >>
>
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

2019-02-05 Thread Sunil G 
Thanks Billie for pointing out.
I have updated source by removing patchprocess and extra line create
release.

Also updated checksum as well.

@bil...@apache.org   @Wangda Tan 
please help to verify this changed bit once.

Thanks
Sunil

On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi 
wrote:

> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> patchprocess directory with some yetus code in it. Also, the file
> dev-support/bin/create-release file has the following line added:
>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> gpg-agent):1"
>
> I think we are probably due for an overall review of LICENSE and NOTICE. I
> saw some idiosyncrasies there but nothing that looked like a blocker.
>
> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
>
>> Hi Folks,
>>
>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
>>
>> The artifacts are available here:
>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
>>
>> The RC tag in git is release-3.1.2-RC1:
>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
>>
>> The maven artifacts are available via repository.apache.org at
>> https://repository.apache.org/content/repositories/orgapachehadoop-1215
>>
>> This vote will run 5 days from now.
>>
>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
>>
>> We have done testing with a pseudo cluster and distributed shell job.
>>
>> My +1 to start.
>>
>> Best,
>> Wangda Tan and Sunil Govindan
>>
>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in (3.1.2)
>> ORDER BY priority DESC
>>
>

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

2019-02-05 Thread Gabor Bota 
  Thanks for working on this Sunil and Wangda!

  +1 (non-binding)

  checked out git tag release-3.1.2-RC1
  hadoop-aws integration test run on eu-west-1 with a known test issue
(AbstractS3GuardToolTestBase.testDestroyNoBucket)
  built from source on Mac OS X 10.14.2, java version 8.0.181-oracle
  deployed on a 3 node cluster
  verified pi job, teragen, terasort and teravalidate

  Regards,
  Gabor Bota

On Tue, Feb 5, 2019 at 6:03 AM Suma Shivaprasad <
sumasai.shivapra...@gmail.com> wrote:

> +1
>
> - Built from source
> - Deployed a single node cluster
> - Ran few MR and Distributed Shell Jobs
> - Check YARN UI2 flow with Yarn services and jobs
>
> Thanks
> Suma
>
> On Mon, Feb 4, 2019 at 7:51 PM Wilfred Spiegelenburg
>  wrote:
>
> > +1 (non binding)
> >
> > - build from source on MacOSX 10.14.2, 1.8.0u181
> > - successful native build on Ubuntu 16.04.3
> > - deployed a single node cluster  (openjdk 1.8u191 / centos 7.5)
> > - uploaded the MR framework
> > - configured YARN with the FS
> > - ran multiple MR jobs
> >
> > > On 29 Jan 2019, at 17:19, Sunil G  wrote:
> > >
> > > Hi Folks,
> > >
> > > On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> > >
> > > The artifacts are available here:
> > > http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> > >
> > > The RC tag in git is release-3.1.2-RC1:
> > > https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> > >
> > > The maven artifacts are available via repository.apache.org at
> > >
> https://repository.apache.org/content/repositories/orgapachehadoop-1215
> > >
> > > This vote will run 5 days from now.
> > >
> > > 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> > >
> > > We have done testing with a pseudo cluster and distributed shell job.
> > >
> > > My +1 to start.
> > >
> > > Best,
> > > Wangda Tan and Sunil Govindan
> > >
> > > [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.1.2)
> > > ORDER BY priority DESC
> >
> >
> > Wilfred Spiegelenburg | Software Engineer
> > cloudera.com 
> >
> >
> >
> >
> >
> >
> >
> >
>

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-05 Thread Iñigo Goiri 
+1

On Tue, Feb 5, 2019 at 8:22 AM Masatake Iwasaki 
wrote:

> +1
>
> Masatake Iwasaki
>
> On 2/4/19 18:13, Jonathan Hung wrote:
> > Hello,
> >
> > Starting a vote based on the discuss thread [1] for moving branch-2
> > precommit/nightly test builds to openjdk8. After this change, the test
> > phase for precommit builds [2] and branch-2 nightly build [3] will run on
> > openjdk8. To maintain source compatibility, these builds will still run
> > their compile phase for branch-2 on openjdk7 as they do now (in addition
> to
> > compiling on openjdk8).
> >
> > Vote will run for three business days until Thursday Feb 7 6:00PM PDT.
> >
> > [1]
> >
> https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E
> >
> > [2]
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
> > https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/
> >
> > [3]
> >
> https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/
> >
> > Jonathan Hung
> >
>
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-16093) Move DurationInfo from hadoop-aws to hadoop-common org.apache.fs.impl

2019-02-05 Thread Steve Loughran (JIRA) 
Steve Loughran created HADOOP-16093:
---

 Summary: Move DurationInfo from hadoop-aws to hadoop-common 
org.apache.fs.impl
 Key: HADOOP-16093
 URL: https://issues.apache.org/jira/browse/HADOOP-16093
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: fs/s3, util
Reporter: Steve Loughran


It'd be useful to have DurationInfo usable in other places (e.g. distcp, abfs, 
...). But as it is in hadoop-aws under 
{{org.apache.hadoop.fs.s3a.commit.DurationInfo
org.apache.hadoop.fs.s3a.commit.DurationInfo}} we can't do that

Move it.

We'll have to rename the Duration class in the process, as java 8 time has a 
class of that name too. Maybe "OperationDuration", with DurationInfo a subclass 
of that

Probably need a test too, won't it?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Moving branch-2 precommit/nightly test builds to java 8

2019-02-05 Thread Masatake Iwasaki 

+1

Masatake Iwasaki

On 2/4/19 18:13, Jonathan Hung wrote:

Hello,

Starting a vote based on the discuss thread [1] for moving branch-2
precommit/nightly test builds to openjdk8. After this change, the test
phase for precommit builds [2] and branch-2 nightly build [3] will run on
openjdk8. To maintain source compatibility, these builds will still run
their compile phase for branch-2 on openjdk7 as they do now (in addition to
compiling on openjdk8).

Vote will run for three business days until Thursday Feb 7 6:00PM PDT.

[1]
https://lists.apache.org/thread.html/7e6fb28fc67560f83a2eb62752df35a8d58d86b2a3df4cacb5d738ca@%3Ccommon-dev.hadoop.apache.org%3E

[2]
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HADOOP-Build/
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-HDFS-Build/
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-YARN-Build/
https://builds.apache.org/view/H-L/view/Hadoop/job/PreCommit-MAPREDUCE-Build/

[3]
https://builds.apache.org/view/H-L/view/Hadoop/job/hadoop-qbt-branch2-java7-linux-x86/

Jonathan Hung




-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Apache Hadoop qbt Report: trunk+JDK8 on Linux/x86

2019-02-05 Thread Apache Jenkins Server 
For more details, see 
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/

[Feb 4, 2019 4:51:07 AM] (aajisaka) HDFS-14158. Checkpointer ignores configured 
time period > 5 minutes
[Feb 4, 2019 5:03:34 AM] (aajisaka) MAPREDUCE-7177. Disable speculative 
execution in TestDFSIO. Contributed
[Feb 4, 2019 5:46:35 AM] (aajisaka) HADOOP-16089. AliyunOSS: update oss-sdk 
version to 3.4.1. Contributed by
[Feb 4, 2019 7:18:28 AM] (sunilg) YARN-9206. RMServerUtils does not count 
SHUTDOWN as an accepted state.
[Feb 4, 2019 5:38:40 PM] (stevel) HADOOP-16076. SPNEGO+SSL Client Connections 
with HttpClient Broken.
[Feb 4, 2019 7:59:48 PM] (inigoiri) HDFS-14202. 
dfs.disk.balancer.max.disk.throughputInMBperSec property is
[Feb 4, 2019 10:52:07 PM] (weichiu) HDFS-14125. Use parameterized log format in 
ECTopologyVerifier.
[Feb 5, 2019 12:30:07 AM] (ajay) HDDS-1039. OzoneManager fails to connect with 
secure SCM. Contributed by
[Feb 5, 2019 12:42:44 AM] (aajisaka) HADOOP-15938. [JDK 11] Remove 
animal-sniffer-maven-plugin to fix build.
[Feb 5, 2019 2:51:52 AM] (yqlin) HDDS-1029. Allow option for force in 
DeleteContainerCommand. Contributed




-1 overall


The following subsystems voted -1:
asflicense findbugs hadolint pathlen unit


The following subsystems voted -1 but
were configured to be filtered/ignored:
cc checkstyle javac javadoc pylint shellcheck shelldocs whitespace


The following subsystems are considered long running:
(runtime bigger than 1h  0m  0s)
unit


Specific tests:

Failed junit tests :

   hadoop.hdfs.web.TestWebHdfsTimeouts 
   hadoop.yarn.client.api.impl.TestAMRMClient 
  

   cc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-compile-cc-root.txt
  [4.0K]

   javac:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-compile-javac-root.txt
  [336K]

   checkstyle:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-checkstyle-root.txt
  [17M]

   hadolint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-patch-hadolint.txt
  [8.0K]

   pathlen:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/pathlen.txt
  [12K]

   pylint:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-patch-pylint.txt
  [104K]

   shellcheck:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-patch-shellcheck.txt
  [20K]

   shelldocs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-patch-shelldocs.txt
  [12K]

   whitespace:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/whitespace-eol.txt
  [9.3M]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/whitespace-tabs.txt
  [1.1M]

   findbugs:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-hdds_client.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-hdds_container-service.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-hdds_framework.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-hdds_tools.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_client.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_common.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_objectstore-service.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_ozone-manager.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_ozonefs.txt
  [8.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_s3gateway.txt
  [4.0K]
   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/branch-findbugs-hadoop-ozone_tools.txt
  [8.0K]

   javadoc:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/diff-javadoc-javadoc-root.txt
  [752K]

   unit:

   
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1038/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
  [324K]

Re: proposed new repository for hadoop/ozone docker images (+update on docker works)

2019-02-05 Thread Elek, Marton 
Thanks Eric the answers.


If I understood well, these are two proposals (use the same repository,
use inline build). I created separated jiras for both of them where we
can discuss the technical details:

https://issues.apache.org/jira/browse/HADOOP-16092

https://issues.apache.org/jira/browse/HADOOP-16091


Until the implementation of the jiras we can use the existing approach,
but (again) I am fine with switching to any newer approach anytime. The
only thing what we need is the availability of the images during any
transition.


I started to document the current state in the wiki to make the
discussion easier.

https://cwiki.apache.org/confluence/display/HADOOP/Container+support

https://cwiki.apache.org/confluence/display/HADOOP/Ozone+Container+support

Marton




On 1/31/19 8:00 PM, Eric Yang wrote:
> 1, 3. There are 38 Apache projects hosting docker images on Docker hub using 
> Apache Organization.  By browsing Apache github mirror.  There are only 7 
> projects using a separate repository for docker image build.  Popular 
> projects official images are not from Apache organization, such as zookeeper, 
> tomcat, httpd.  We may not disrupt what other Apache projects are doing, but 
> it looks like inline build process is widely employed by majority of projects 
> such as Nifi, Brooklyn, thrift, karaf, syncope and others.  The situation 
> seems a bit chaotic for Apache as a whole.  However, Hadoop community can 
> decide what is best for Hadoop.  My preference is to remove ozone from source 
> tree naming, if Ozone is intended to be subproject of Hadoop for long period 
> of time.  This enables Hadoop community to host docker images for various 
> subproject without having to check out several source tree to trigger a grand 
> build.  However, inline build process seems more popular than separated 
> process.  Hence, I highly recommend making docker build inline if possible.
> 
> 2. I think open an INFRA ticket, and there are Jenkins users who can 
> configure the job to run on nodes that have Apache repo credential.
> 
> 4. The docker image name maps to maven project name.  Hence, if it is 
> Hadoop-ozone as project name.  The convention automatically follows the maven 
> artifact name with option to customize.  I think it is reasonable and it 
> automatically tagged with the same maven project version, which minimize 
> version number management between maven and docker.
> 
> Regards,
> Eric
> 
> On 1/31/19, 8:59 AM, "Elek, Marton"  wrote:
> 
> 
> Hi Eric,
> 
> Thanks for the answers
> 
> 1.
> 
> > Hadoop-docker-ozone.git source tree naming seems to create a unique
> process for Ozone.
> 
> Not at all. We would like to follow the existing practice which is
> established in HADOOP-14898. In HDDS-851 we discussed why we need two
> separated repositories for hadoop/ozone: because the limitation of the
> dockerhub branch/tag mapping.
> 
> I am 100% open to switch to use an other approach. I would suggest to
> create a JIRA for that as it requires code modification in the
> docker-hadoop-* branches.
> 
> 
> 2.
> 
> > Flagging automated build on dockerhub seems conflicts with Apache
> release policy.
> 
> Honestly I don't know. It was discussed in HADOOP-14989 and the
> connected INFRA ticket and there was no arguments against it. Especially
> as we just followed the existing practice and we just followed the
> practice which is started by other projects.
> 
> Now I checked again the docker related INFRA tickets it seems that we
> have two other practice since than:
> 
>  1) build docker image on the jenkins (is it compliant?)
>  2) get permission to push to the apache/... from local.
> 
> You suggested to the the second one. Do you have more information how is
> it possible? How and who can request permission to push the
> apache/hadoop for example?
> 
> 
> 3.
> 
> From one point of view, publishing existing, voted releases in docker
> images is something like to repackage it. But you may have right and
> this is wrong because it should be handled as separated releases.
> 
> Do you know any official ASF wiki/doc/mail discussion about managing
> docker images? If not, I would suggest to create a new wiki/doc as it
> seems that we have no clear answer which is the most compliant way to do 
> it.
> 
> 4.
> 
> Thank you the suggestions to use dockerhub/own namespace to stage docker
> images during the build. Sounds good to me. But I also wrote some other
> problems in my previous mail (3 b,c,d), this is is just one (3/a). Do
> you have any suggestion to solve the other problems?
> 
>  * Updating existing images (for example in case of an ssl bug, rebuild
> all the existing images with exactly the same payload but updated base
> image/os environment)
> 
>  *

[jira] [Created] (HADOOP-16092) Move the source of hadoop/ozone containers to the same repository

2019-02-05 Thread Elek, Marton (JIRA) 
Elek, Marton created HADOOP-16092:
-

 Summary: Move the source of hadoop/ozone containers to the same 
repository
 Key: HADOOP-16092
 URL: https://issues.apache.org/jira/browse/HADOOP-16092
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Elek, Marton


This is proposed by [~eyang] in 
[this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
 mailing thread.

bq. Hadoop community can decide what is best for Hadoop.  My preference is to 
remove ozone from source tree naming, if Ozone is intended to be subproject of 
Hadoop for long period of time.  This enables Hadoop community to host docker 
images for various subproject without having to check out several source tree 
to trigger a grand build

As of now the source of  hadoop docker images are stored in the hadoop git 
repository (docker-* branches) for hadoop and in hadoop-docker-ozone git 
repository for ozone (all branches).

As it's discussed in HDDS-851 the biggest challenge to solve here is the 
mapping between git branches and dockerhub tags. It's not possible to use the 
captured part of a github branch.

For example it's not possible to define a rule to build all the ozone-(.*) 
branches and use a tag $1 for it. Without this support we need to create a new 
mapping for all the releases manually (with the help of the INFRA).

Note: HADOOP-16091 can solve this problem as it doesn't require branch mapping 
any more.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16091) Create hadoop/ozone docker images with inline build process

2019-02-05 Thread Elek, Marton (JIRA) 
Elek, Marton created HADOOP-16091:
-

 Summary: Create hadoop/ozone docker images with inline build 
process
 Key: HADOOP-16091
 URL: https://issues.apache.org/jira/browse/HADOOP-16091
 Project: Hadoop Common
  Issue Type: New Feature
Reporter: Elek, Marton


This is proposed by [~eyang] in 
[this|https://lists.apache.org/thread.html/33ac54bdeacb4beb023ebd452464603aaffa095bd104cb43c22f484e@%3Chdfs-dev.hadoop.apache.org%3E]
 mailing thread.

bq. 1, 3. There are 38 Apache projects hosting docker images on Docker hub 
using Apache Organization.  By browsing Apache github mirror.  There are only 7 
projects using a separate repository for docker image build.  Popular projects 
official images are not from Apache organization, such as zookeeper, tomcat, 
httpd.  We may not disrupt what other Apache projects are doing, but it looks 
like inline build process is widely employed by majority of projects such as 
Nifi, Brooklyn, thrift, karaf, syncope and others.  The situation seems a bit 
chaotic for Apache as a whole.  However, Hadoop community can decide what is 
best for Hadoop.  My preference is to remove ozone from source tree naming, if 
Ozone is intended to be subproject of Hadoop for long period of time.  This 
enables Hadoop community to host docker images for various subproject without 
having to check out several source tree to trigger a grand build.  However, 
inline build process seems more popular than separated process.  Hence, I 
highly recommend making docker build inline if possible.

The main challenges are also discussed in the thread:

{code}
3. Technically it would be possible to add the Dockerfile to the source
tree and publish the docker image together with the release by the
release manager but it's also problematic:

{code}
  a) there is no easy way to stage the images for the vote
  c) it couldn't be flagged as automated on dockerhub
  d) It couldn't support the critical updates.


 * Updating existing images (for example in case of an ssl bug, rebuild
all the existing images with exactly the same payload but updated base
image/os environment)

 * Creating image for older releases (We would like to provide images,
for hadoop 2.6/2.7/2.7/2.8/2.9. Especially for doing automatic testing
with different versions).


{code}

The a) can be solved (as [~eyang] suggested) with using a personal docker image 
during the vote and publish it to the dockerhub after the vote (in case the 
permission can be set by the INFRA)

Note: based on LEGAL-270 and linked discussion both approaches (inline build 
process / external build process) are compatible with the apache release.

Note: HDDS-851 and HADOOP-14898 contains more information about these problems.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org