[jira] [Resolved] (HADOOP-19010) NullPointerException in Hadoop Credential Check CLI Command

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-19010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-19010.

Fix Version/s: 3.4.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

> NullPointerException in Hadoop Credential Check CLI Command
> ---
>
> Key: HADOOP-19010
> URL: https://issues.apache.org/jira/browse/HADOOP-19010
> Project: Hadoop Common
>  Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Anika Kelhanka
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.4.0
>
>
> *Description*: Hadoop's credential check throws {{NullPointerException}} when 
> alias not found.
> {code:bash}
> hadoop credential check "fs.gs.proxy.username" -provider 
> "jceks://file/usr/lib/hive/conf/hive.jceks" {code}
> Checking aliases for CredentialProvider: 
> jceks://file/usr/lib/hive/conf/hive.jceks
> Enter alias password: 
> java.lang.NullPointerException
> at
> org.apache.hadoop.security.alias.CredentialShell$CheckCommand.execute(CredentialShell.java:369)
> at org.apache.hadoop.tools.CommandShell.run(CommandShell.java:73)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:82)
> at 
> org.apache.hadoop.security.alias.CredentialShell.main(CredentialShell.java:529)}}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[ANNOUNCE] New Hadoop PMC member - Mukund Thakur

[Vinayakumar B]

On behalf of the Apache Hadoop PMC, I am pleased to announce that Mukund
Thakur (mthakur) has accepted the PMC's invitation to become a PMC member on
the project. We appreciate all of Mukund’s generous contributions thus far
and look forward to his continued involvement.

Congratulations and welcome, Mukund !

-Vinayakumar B
-- 
-Vinay

Re: [DISCUSS] cull unused modules from under hadoop-tools

[Vinayakumar B]

Can move unused ones to separate repo under hadoop, and introduce a
separate independent release cycle in case required.

Any thoughts on this?
-Vinay

On Fri, 1 Apr 2022 at 1:31 PM, Ayush Saxena  wrote:

> Distcp (I have a use case for it)
> Hadoop-federation-balance(RBF uses it, we will take care in case it
> bothers)
> Hadoop-Dynamometer(This I feel is being used from jira activies)
>
> So, these three we should let stay as is.
> Others the object store ones are active. You know which all are needed.
> For rest we can call for a vote and drop them if everyone agrees.
>
> -Ayush
>
> > On 31-Mar-2022, at 4:40 PM, Steve Loughran 
> wrote:
> >
> >  how many of the modules under hadoop-tools get used/maintained?
> >
> > hadoop-aliyun
> > hadoop-ant
> > hadoop-archive-logs
> > hadoop-archives
> > hadoop-aws
> > hadoop-azure
> > hadoop-azure-datalake
> > hadoop-datajoin
> > hadoop-distcp
> > hadoop-dynamometer
> > hadoop-extras
> > hadoop-federation-balance
> > hadoop-fs2img
> > hadoop-ftp
> > hadoop-gridmix
> > hadoop-kafka
> > hadoop-openstack
> > hadoop-ozone
> > hadoop-pipes
> > hadoop-resourceestimator
> > hadoop-rumen
> > hadoop-sls
> > hadoop-streaming
> >
> > I know distcp is universal, and the aws. azure, aliyun modules are
> > active. hadoop-azure-datalake doesn't get maintenance, but it should stay
> > around until microsoft remove the gen1 ADLS service
> >
> > But what about all the others? the hadoop-openstack one hasn't been
> touched
> > or tested for a few years, and IMO could be cut immediately. what about
> > others? does hadoop-streaming or hadoop-pipes get use any more?
> >
> > Existing code may use these, but having them in the codebase only creates
> > maintenance work, especially if security fixes need to go in on the code
> or
> > are caused by dependencies.
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> --
-Vinay

Re: [DISCUSS] Race condition in ProtobufRpcEngine2

[Vinayakumar B]

Call.toString() was used only while sending the response, in the response
processor thread. At this point, header would be already read and
processed. So no chance of race conditions.

This race condition was not introduced by the HADOOP-17046
, instead existing code
was never intended to be used for logging purposes by multiple threads.
Though the locking mechanism proposed in HADOOP-18143
 solves the problems,
it adds overhead in both memory as well as performance.

This is a critical path for RPC processing, and adding a
locking/synchronization in this place would affect the overall performance
(may not be huge).

I would advise to remove logging and locking both altogether and leave the
code as is in Reader thread.

If logging is required, then add it in the response processor thread or the
handler threads.

-Vinay


On Mon, Feb 28, 2022 at 10:02 PM Ayush Saxena  wrote:

> Hey Andras,
> I had a quick look at HADOOP-18143, the methods in question in
> ProtobufRpcEngine2 are identical to the ones in ProtobufRpcEngine. So, I am
> not very sure how the  race condition doesn't happen in  ProtobufRpcEngine.
> I have to debug and spend some more time, considering that I have
> reverted HADOOP-18082 for now to unblock YARN. Though the issue would still
> be there as you said, but will give us some time to analyse.
>
> Thanks
> -Ayush
>
> On Mon, 28 Feb 2022 at 21:26, Gyori Andras 
> wrote:
>
>> Hey everyone!
>>
>> We have started seeing test failures in YARN PRs for a while. We have
>> identified the problematic commit, which is HADOOP-18082
>> , however, this
>> change
>> just revealed the race condition lying in ProtobufRpcEngine2 introduced in
>> HADOOP-17046 . We
>> have
>> also fixed the underlying issue via a locking mechanism, presented in
>> HADOOP-18143 , but
>> since it is out of our area of expertise, we can neither verify nor
>> guarantee that it will not cause some subtle issues in the RPC system.
>> As we think it is a core part of Hadoop, we would use feedback from
>> someone
>> who is proficient in this part.
>>
>> Regards:
>> Andras
>>
>

Re: [VOTE] Release Apache Hadoop 3.3.1 RC3

[Vinayakumar B]

+1 (Binding)

1. Built from Tag.
2. Successful Native Build on Ubuntu 20.04
3. Verified Checksums
4. Deployed the docker cluster with 3 nodes
5. Ran sample MR Jobs

-Vinay


On Sat, Jun 12, 2021 at 6:40 PM Ayush Saxena  wrote:

> +1,
> Built from Source.
> Successful Native Build on Ubuntu 20.04
> Verified Checksums
> Ran basic hdfs shell commands.
> Ran simple MR jobs.
> Browsed NN,DN,RM and NM UI.
>
> Thanx Wei-Chiu for driving the release.
>
> -Ayush
>
>
> > On 12-Jun-2021, at 1:45 AM, epa...@apache.org wrote:
> >
> > +1 (binding)
> > Eric
> >
> >
> > On Tuesday, June 1, 2021, 5:29:49 AM CDT, Wei-Chiu Chuang <
> weic...@apache.org> wrote:
> >
> > Hi community,
> >
> > This is the release candidate RC3 of Apache Hadoop 3.3.1 line. All
> blocker
> > issues have been resolved [1] again.
> >
> > There are 2 additional issues resolved for RC3:
> > * Revert "MAPREDUCE-7303. Fix TestJobResourceUploader failures after
> > HADOOP-16878
> > * Revert "HADOOP-16878. FileUtil.copy() to throw IOException if the
> source
> > and destination are the same
> >
> > There are 4 issues resolved for RC2:
> > * HADOOP-17666. Update LICENSE for 3.3.1
> > * MAPREDUCE-7348. TestFrameworkUploader#testNativeIO fails. (#3053)
> > * Revert "HADOOP-17563. Update Bouncy Castle to 1.68. (#2740)" (#3055)
> > * HADOOP-17739. Use hadoop-thirdparty 1.1.1. (#3064)
> >
> > The Hadoop-thirdparty 1.1.1, as previously mentioned, contains two extra
> > fixes compared to hadoop-thirdparty 1.1.0:
> > * HADOOP-17707. Remove jaeger document from site index.
> > * HADOOP-17730. Add back error_prone
> >
> > *RC tag is release-3.3.1-RC3
> > https://github.com/apache/hadoop/releases/tag/release-3.3.1-RC3
> >
> > *The RC3 artifacts are at*:
> > https://home.apache.org/~weichiu/hadoop-3.3.1-RC3/
> > ARM artifacts: https://home.apache.org/~weichiu/hadoop-3.3.1-RC3-arm/
> >
> > *The maven artifacts are hosted here:*
> > https://repository.apache.org/content/repositories/orgapachehadoop-1320/
> >
> > *My public key is available here:*
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> >
> > Things I've verified:
> > * all blocker issues targeting 3.3.1 have been resolved.
> > * stable/evolving API changes between 3.3.0 and 3.3.1 are compatible.
> > * LICENSE and NOTICE files checked
> > * RELEASENOTES and CHANGELOG
> > * rat check passed.
> > * Built HBase master branch on top of Hadoop 3.3.1 RC2, ran unit tests.
> > * Built Ozone master on top fo Hadoop 3.3.1 RC2, ran unit tests.
> > * Extra: built 50 other open source projects on top of Hadoop 3.3.1 RC2.
> > Had to patch some of them due to commons-lang migration (Hadoop 3.2.0)
> and
> > dependency divergence. Issues are being identified but so far nothing
> > blocker for Hadoop itself.
> >
> > Please try the release and vote. The vote will run for 5 days.
> >
> > My +1 to start,
> >
> > [1] https://issues.apache.org/jira/issues/?filter=12350491
> > [2]
> >
> https://github.com/apache/hadoop/compare/release-3.3.1-RC1...release-3.3.1-RC3
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: [DISCUSS] Hadoop 3.3.1 release

[Vinayakumar B]

Hi David,

>> Still hoping for help here:

>> https://issues.apache.org/jira/browse/HDFS-15790

I will raise a PR for the said solution soon (in a day or two).

-Vinay

On Thu, 11 Mar 2021 at 5:39 AM, David  wrote:

> Hello,
>
> Still hoping for help here:
>
> https://issues.apache.org/jira/browse/HDFS-15790
>
> Looks like it has been worked on, not sure how to best move it forward.
>
> On Wed, Mar 10, 2021, 12:21 PM Steve Loughran  >
> wrote:
>
> > I'm going to argue its too late to do IPv6 support close to a release, as
> > it's best if its on developer machines for some time to let all quirks
> > surface. It's not so much IPv6 itself, but do we cause any regressions on
> > IPv4?
> >
> > But: it can/should go into trunk and stabilize there
> >
> > On Thu, 4 Mar 2021 at 03:52, Muralikrishna Dmmkr <
> > muralikrishna.dm...@gmail.com> wrote:
> >
> > > Hi Brahma,
> > >
> > > I have missed out mentioning about the IPV6 feature in the last mail,
> > > Support for IPV6 has been in development since 2015 and We have done a
> > good
> > > amount of testing at our organisation, the feature is stable and used
> by
> > > our customers extensively in the last one year. I think it is a good
> time
> > > to add the IPV6 support to 3.3.1.
> > >
> > > https://issues.apache.org/jira/browse/HADOOP-11890
> > >
> > > Thanks
> > > D M Murali Krishna Reddy
> > >
> > > On Wed, Feb 24, 2021 at 9:13 AM Muralikrishna Dmmkr <
> > > muralikrishna.dm...@gmail.com> wrote:
> > >
> > > > Hi Brahma,
> > > >
> > > > Can we have this new feature "YARN Registry based AM discovery with
> > retry
> > > > and in-flight task persistent via JHS" in the upcoming 3.1.1
> release. I
> > > > have also attached a test-report in the below jira.
> > > >
> > > > https://issues.apache.org/jira/browse/MAPREDUCE-6726
> > > >
> > > >
> > > > Thanks,
> > > > D M Murali Krishna Reddy
> > > >
> > > > On Tue, Feb 23, 2021 at 10:11 AM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > > wrote:
> > > >
> > > >> Hi Bilwa,
> > > >>
> > > >> I have commented on the jira's you mentioned. Based on the stability
> > we
> > > >> can
> > > >> plan this.But needs to be merged ASAP.
> > > >>
> > > >>
> > > >>
> > > >> On Fri, Feb 19, 2021 at 5:20 PM bilwa st  wrote:
> > > >>
> > > >> > Hi Brahma,
> > > >> >
> > > >> > Can we have below features in 3.3.1 release? We have been using
> > these
> > > >> > features for a long time. They are stable and tested in bigger
> > > clusters.
> > > >> >
> > > >> > 1. Container reuse -
> > > >> https://issues.apache.org/jira/browse/MAPREDUCE-6749
> > > >> > 2. Speculative attempts should not run on the same node -
> > > >> > https://issues.apache.org/jira/browse/MAPREDUCE-7169
> > > >> >
> > > >> > Thanks,
> > > >> > Bilwa
> > > >> >
> > > >> > On Thu, Feb 18, 2021, 1:49 PM Brahma Reddy Battula <
> > bra...@apache.org
> > > >
> > > >> > wrote:
> > > >> >
> > > >> >> Sorry for the late reply..
> > > >> >>
> > > >> >> I will come up with a plan.. Please let me know if anybody has
> some
> > > >> >> features/improvements/bugs that need to be included.
> > > >> >>
> > > >> >> On Mon, Feb 15, 2021 at 9:39 PM Sunil Govindan <
> sun...@apache.org>
> > > >> wrote:
> > > >> >>
> > > >> >> > Hi Wei-Chiu,
> > > >> >> >
> > > >> >> > What will be the next steps here for 3.3.1 planning?
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Sunil
> > > >> >> >
> > > >> >> > On Mon, Feb 8, 2021 at 11:56 PM Stack 
> wrote:
> > > >> >> >
> > > >> >> > > On Wed, Feb 3, 2021 at 6:41 AM Steve Loughran
> > > >> >> >  > > >> >> > > >
> > > >> >> > > wrote:
> > > >> >> > >
> > > >> >> > > >
> > > >> >> > > > Regarding blockers : how about we have a little hackathon
> > > >> where we
> > > >> >> > try
> > > >> >> > > > and get things in. This means a promise of review time from
> > the
> > > >> >> people
> > > >> >> > > with
> > > >> >> > > > commit rights and other people who understand the code
> > (Stack?)
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> > > I'm up for helping get 3.3.1 out (reviewing, hackathon,
> > testing).
> > > >> >> > > Thanks,
> > > >> >> > > S
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > > -steve
> > > >> >> > > >
> > > >> >> > > > On Thu, 28 Jan 2021 at 06:48, Ayush Saxena <
> > ayush...@gmail.com
> > > >
> > > >> >> wrote:
> > > >> >> > > >
> > > >> >> > > > > +1
> > > >> >> > > > > Just to mention we would need to release
> hadoop-thirdparty
> > > too
> > > >> >> > before.
> > > >> >> > > > > Presently we are using the snapshot version of it.
> > > >> >> > > > >
> > > >> >> > > > > -Ayush
> > > >> >> > > > >
> > > >> >> > > > > > On 28-Jan-2021, at 6:59 AM, Wei-Chiu Chuang <
> > > >> weic...@apache.org
> > > >> >> >
> > > >> >> > > > wrote:
> > > >> >> > > > > >
> > > >> >> > > > > > Hi all,
> > > >> >> > > > > >
> > > >> >> > > > > > Hadoop 3.3.0 was released half a year ago, and as of
> now
> > > >> we've
> > > >> >> > > > > accumulated
> > > >> >> > > > > > more than 400 

[jira] [Created] (HADOOP-17505) public interface GroupMappingServiceProvider needs default impl for getGroupsSet()

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-17505:
--

 Summary: public interface GroupMappingServiceProvider needs 
default impl for getGroupsSet() 
 Key: HADOOP-17505
 URL: https://issues.apache.org/jira/browse/HADOOP-17505
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinayakumar B


HADOOP-17079 added "GroupMappingServiceProvider#getGroupsSet()" interface.

But since this is a public interface, it will break compilation of existing 
implementations in downstreams.

Consider adding a default implementation in the interface to avoid such 
failures.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Wire compatibility between Hadoop 3.x client and 2.x server

[Vinayakumar B]

Thanks Jianliang,

I saw jira assigned to you.  Are you planning to provide a fix as well for
this?

If not, would you mind assigning to me?

-Vinay

On Wed, 28 Oct 2020 at 3:46 PM, Wu,Jianliang(vip.com) <
jianliang...@vipshop.com> wrote:

> Hi  VinayaKumar and Wei-Chiu
>
> I filed jira https://issues.apache.org/jira/browse/HDFS-15660 for details
>
>
> 2020年10月28日 17:08，Vinayakumar B  写道：
>
> Hi Wu,Jianliang,
> Have you created the Jira for the issue you mentioned due to
> getContentSummary?
>
> I might have a fix for this. Of-course it needs to be applied both client
> and server side.
>
> Let me know.
>
> -Vinay
>
>
> On Wed, Oct 14, 2020 at 12:26 PM Wu,Jianliang(vip.com) <
> jianliang...@vipshop.com> wrote:
>
> Ok, I will file a HDFS jira to report this issue.
>
>
> 2020年10月13日 20:43，Wei-Chiu Chuang  写道：
>
> Thanks Jialiang for reporting the issue.
> That sounds bad and should've not happened. Could you file a HDFS jira
>
> and
>
> fill in more details?
>
> On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> jianliang...@vipshop.com> wrote:
>
> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> 2.6,  we found hive to call getContentSummary method , the client and
> server was not compatible  because of hadoop3 added new PROVIDED storage
> type.
>
> 2020年10月13日 06:41，Chao Sun 
> sunc...@apache.org>>
>
> 写道：
>
>
>
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
>
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an
>
> authorized
>
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without
>
> making
>
> or otherwise retaining a copy.
>
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> 
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
> 
>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
-- 
-Vinay

Re: Wire compatibility between Hadoop 3.x client and 2.x server

[Vinayakumar B]

Hi Wu,Jianliang,
Have you created the Jira for the issue you mentioned due to
getContentSummary?

I might have a fix for this. Of-course it needs to be applied both client
and server side.

Let me know.

-Vinay


On Wed, Oct 14, 2020 at 12:26 PM Wu,Jianliang(vip.com) <
jianliang...@vipshop.com> wrote:

> Ok, I will file a HDFS jira to report this issue.
>
>
> > 2020年10月13日 20:43，Wei-Chiu Chuang  写道：
> >
> > Thanks Jialiang for reporting the issue.
> > That sounds bad and should've not happened. Could you file a HDFS jira
> and
> > fill in more details?
> >
> > On Mon, Oct 12, 2020 at 8:59 PM Wu,Jianliang(vip.com) <
> > jianliang...@vipshop.com> wrote:
> >
> >> In our case, when nn has upgraded to 3.1.3 and dn’s version was still
> >> 2.6,  we found hive to call getContentSummary method , the client and
> >> server was not compatible  because of hadoop3 added new PROVIDED storage
> >> type.
> >>
> >> 2020年10月13日 06:41，Chao Sun  sunc...@apache.org>>
> >> 写道：
> >>
> >>
> >>
> >>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> >> This communication is intended only for the addressee(s) and may contain
> >> information that is privileged and confidential. You are hereby notified
> >> that, if you are not an intended recipient listed above, or an
> authorized
> >> employee or agent of an addressee of this communication responsible for
> >> delivering e-mail messages to an intended recipient, any dissemination,
> >> distribution or reproduction of this communication (including any
> >> attachments hereto) is strictly prohibited. If you have received this
> >> communication in error, please notify us immediately by a reply e-mail
> >> addressed to the sender and permanently delete the original e-mail
> >> communication and any attachments from all storage devices without
> making
> >> or otherwise retaining a copy.
> >>
>
> 本电子邮件可能为保密文件。如果阁下非电子邮件所指定之收件人，谨请立即通知本人。敬请阁下不要使用、保存、复印、打印、散布本电子邮件及其内容，或将其用于其他任何目的或向任何人披露。谢谢您的合作！
> This communication is intended only for the addressee(s) and may contain
> information that is privileged and confidential. You are hereby notified
> that, if you are not an intended recipient listed above, or an authorized
> employee or agent of an addressee of this communication responsible for
> delivering e-mail messages to an intended recipient, any dissemination,
> distribution or reproduction of this communication (including any
> attachments hereto) is strictly prohibited. If you have received this
> communication in error, please notify us immediately by a reply e-mail
> addressed to the sender and permanently delete the original e-mail
> communication and any attachments from all storage devices without making
> or otherwise retaining a copy.
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>

[jira] [Resolved] (HADOOP-17306) RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17306?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-17306.

Fix Version/s: 3.4.0
   3.3.1
   3.2.2
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to trunk, branch-3.3 and branch-3.2

 

Thanks [~aajisaka] and [~ayushsaxena] for reviews.

> RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10.b09
> 
>
> Key: HADOOP-17306
> URL: https://issues.apache.org/jira/browse/HADOOP-17306
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: fs
>        Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.2.2, 3.3.1, 3.4.0
>
>  Time Spent: 1.5h
>  Remaining Estimate: 0h
>
> RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.
> This api looses milliseconds due to JDK bug.
> [https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]
> This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still 
> being used in many productions.
> Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns 
> correct time.
> Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as 
> workaround. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-17306) RawLocalFileSystem's lastModifiedTime() looses milli seconds in JDK < 10 b09

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-17306:
--

 Summary: RawLocalFileSystem's lastModifiedTime() looses milli 
seconds in JDK < 10 b09
 Key: HADOOP-17306
 URL: https://issues.apache.org/jira/browse/HADOOP-17306
 Project: Hadoop Common
  Issue Type: Bug
  Components: fs
Reporter: Vinayakumar B


RawLocalFileSystem's FileStatus uses {{File.lastModified()}} api from JDK.

This api looses milliseconds due to JDK bug.

[https://bugs.java.com/bugdatabase/view_bug.do?bug_id=8177809]

This bug fixed in JDK 10 b09 onwards and still exists in JDK 8 which is still 
being used in many productions.

Apparently, {{Files.getLastModifiedTime()}} from java's nio package returns 
correct time.

Use {{Files.getLastModifiedTime()}} instead of {{File.lastModified}} as 
workaround. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-17278) Shade guava 29.0-jre in hadoop thirdparty

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-17278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-17278.

Fix Version/s: thirdparty-1.1.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to trunk of hadoop-thirdparty

> Shade guava 29.0-jre in hadoop thirdparty
> -
>
> Key: HADOOP-17278
> URL: https://issues.apache.org/jira/browse/HADOOP-17278
> Project: Hadoop Common
>  Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
>  Labels: pull-request-available
> Fix For: thirdparty-1.1.0
>
>
> Shade guava 27.0-jre in hadoop-thirdparty



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Moving Ozone to a separated Apache project

[Vinayakumar B]

+1

-Vinay

On Fri, 25 Sep 2020 at 6:22 PM, Lin, Yiqun  wrote:

> +1.
>
>
>
> Thanks,
>
> Yiqun
>
>
>
> On 2020/9/25, 2:08 PM, "Elek, Marton"  wrote:
>
>
>
> External Email
>
>
>
> Hi all,
>
>
>
> Thank you for all the feedback and requests,
>
>
>
> As we discussed in the previous thread(s) [1], Ozone is proposed to be
> a
>
> separated Apache Top Level Project (TLP)
>
>
>
> The proposal with all the details, motivation and history is here:
>
>
>
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FHADOOP%2FOzone%2BHadoop%2Bsubproject%2Bto%2BApache%2BTLP%2Bproposaldata=02%7C01%7Cyiqlin%40ebay.com%7Cca782c249c314e43864208d861195a0e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637366108801790752sdata=Hr7LEWorhjBMOJ0ZUYm5vUPrgW2O2ZWfuVO%2FWY1VQ7Q%3Dreserved=0
>
>
>
> This voting runs for 7 days and will be concluded at 2nd of October,
> 6AM
>
> GMT.
>
>
>
> Thanks,
>
> Marton Elek
>
>
>
> [1]:
>
>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread.html%2Frc6c79463330b3e993e24a564c6817aca1d290f186a1206c43ff0436a%2540%253Chdfs-dev.hadoop.apache.org%253Edata=02%7C01%7Cyiqlin%40ebay.com%7Cca782c249c314e43864208d861195a0e%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637366108801790752sdata=nyZrG7hS8BMwbEmD866kR%2BBukY1vK%2FBfENrU5AUhqtw%3Dreserved=0
>
>
>
> -
>
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
>
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>
>
>
>
>
>
> -
>
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
>
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
> --
-Vinay

Re: [VOTE] Release Apache Hadoop 2.10.1 (RC0)

[Vinayakumar B]

Apologies for delayed voting.

+1 (Binding)

1. Verified the signature. Signatures are good.
2. Verified the sha512 checksum
   (I tried to use the sha512sum tool to verify. But ".sha512"
files doesn't have the expected format. I guess these files are regenerated
using "gpg --print-md", not the one used by createrelease scripts,
'sha512sum --tag'. It would be easy to verify using tool if the .sha512
have the proper format)
3. src tar have the "patchprocess" directory, may be need to backport the
fix in trunk to branch-2.10 to avoid this in future.
4. Deployed 3 node docker cluster and ran basic jobs. All-Ok.

-Vinay


On Mon, Sep 21, 2020 at 5:24 AM Wei-Chiu Chuang  wrote:

> +1 (binding)
>
> I did a security scan for the 2.10.1 RC0 and it looks fine to me.
>
> Checked recent critical/blocker HDFS issues that are not in 2.10.1. It
> looks mostly fine. Most of them are Hadoop 3.x features (EC, ... etc) but
> there is one worth attention:
>
>
>1. HDFS-14674  [SBN
>read] Got an unexpected txid when tail editlog.
>2.
>   1. But looking at the jira, it doesn't apply to 2.x so I think we are
>   good there.
>   2.
>   3.
>   4. I wanted to do an API compat check but didn't finish it yet. If
>   someone can do it quickly that would be great. (Does anyone know
> of a cloud
>   service that we can quickly do a Java API compat check?)
>
> Cheers,
> Wei-Chiu
>
> On Sun, Sep 20, 2020 at 9:25 AM Sunil Govindan  wrote:
>
> > +1 (binding)
> >
> > - verified checksum and sign. Shows as a Good signature from "Masatake
> > Iwasaki (CODE SIGNING KEY) "
> > - built from source
> > - ran basic MR job and looks good
> > - UI also seems fine
> >
> > Thanks,
> > Sunil
> >
> > On Sun, Sep 20, 2020 at 11:38 AM Masatake Iwasaki <
> > iwasak...@oss.nttdata.co.jp> wrote:
> >
> > > The RC0 got 2 binding +1's and 2 non-binging +1's [1].
> > >
> > > Based on the discussion about release vote [2],
> > > bylaws[3] defines the periods in minimum terms.
> > > We can extend it if there is not enough activity.
> > >
> > > I would like to extend the period to 7 days,
> > > until Monday September 21 at 10:00 am PDT.
> > >
> > > I will appreciate additional votes.
> > >
> > > Thanks,
> > > Masatake Iwasaki
> > >
> > > [1]
> > >
> >
> https://lists.apache.org/thread.html/r16a7f36315a0673c7d522c41065e7ef9c9ee15c76ffcb5db80931002%40%3Ccommon-dev.hadoop.apache.org%3E
> > > [2]
> > >
> >
> https://lists.apache.org/thread.html/e392b902273ee0c14ba34d72c44630e05f54cb3976109af510592ea2%401403330080%40%3Ccommon-dev.hadoop.apache.org%3E
> > > [3] https://hadoop.apache.org/bylaws.html
> > >
> > > On 2020/09/15 2:59, Masatake Iwasaki wrote:
> > > > Hi folks,
> > > >
> > > > This is the first release candidate for the second release of Apache
> > > Hadoop 2.10.
> > > > It contains 218 fixes/improvements since 2.10.0 [1].
> > > >
> > > > The RC0 artifacts are at:
> > > > http://home.apache.org/~iwasakims/hadoop-2.10.1-RC0/
> > > >
> > > > RC tag is release-2.10.1-RC0:
> > > > https://github.com/apache/hadoop/tree/release-2.10.1-RC0
> > > >
> > > > The maven artifacts are hosted here:
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1279/
> > > >
> > > > My public key is available here:
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > The vote will run for 5 days, until Saturday, September 19 at 10:00
> am
> > > PDT.
> > > >
> > > > [1]
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20in%20(HDFS%2C%20YARN%2C%20HADOOP%2C%20MAPREDUCE)%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.10.1
> > > >
> > > > Thanks,
> > > > Masatake Iwasaki
> > > >
> > > > -
> > > > To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: common-dev-h...@hadoop.apache.org
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: mapreduce-dev-unsubscr...@hadoop.apache.org
> > > For additional commands, e-mail: mapreduce-dev-h...@hadoop.apache.org
> > >
> > >
> >
>

[ANNOUNCE] New Apache Hadoop PMC member : Ayush Saxena

[Vinayakumar B]

Hi all,

I am very glad to announce that Ayush Saxena was voted to join Apache
Hadoop PMC.

Congratulations Ayush! Well deserved and thank you for your dedication to
the project. Please keep up the good work.

Thanks,
-Vinay

[ANNOUNCE] New Apache Hadoop PMC member : Ayush Saxena

[Vinayakumar B]

Hi all,

I am very glad to announce that Ayush Saxena was voted to join Apache
Hadoop PMC.

Congratulations Ayush! Well deserved and thank you for your dedication to
the project. Please keep up the good work.

Thanks,
-Vinay

Re: [VOTE] Release Apache Hadoop 3.3.0 - RC0

[Vinayakumar B]

+1 (Binding)

-Verified all checksums and Signatures.
-Verified site, Release notes and Change logs
  + May be changelog and release notes could be grouped based on the
project at second level for better look (this needs to be supported from
yetus)
-Tested in x86 local 3-node docker cluster.
  + Built from source with OpenJdk 8 and Ubuntu 18.04
  + Deployed 3 node docker cluster
  + Ran various Jobs (wordcount, Terasort, Pi, etc)

No Issues reported.

-Vinay

On Fri, Jul 10, 2020 at 1:19 PM Sheng Liu  wrote:

> +1 (non-binding)
>
> - checkout the "3.3.0-aarch64-RC0" binaries packages
>
> - started a clusters with 3 nodes VMs of Ubuntu 18.04 ARM/aarch64,
> openjdk-11-jdk
>
> - checked some web UIs (NN, DN, RM, NM)
>
> - Executed a wordcount, TeraGen, TeraSort and TeraValidate
>
> - Executed a TestDFSIO job
>
> - Executed a Pi job
>
> BR,
> Liusheng
>
> Zhenyu Zheng  于2020年7月10日周五 下午3:45写道：
>
> > +1 (non-binding)
> >
> > - Verified all hashes and checksums
> > - Tested on ARM platform for the following actions:
> >   + Built from source on Ubuntu 18.04, OpenJDK 8
> >   + Deployed a pseudo cluster
> >   + Ran some example jobs(grep, wordcount, pi)
> >   + Ran teragen/terasort/teravalidate
> >   + Ran TestDFSIO job
> >
> > BR,
> >
> > Zhenyu
> >
> > On Fri, Jul 10, 2020 at 2:40 PM Akira Ajisaka 
> wrote:
> >
> > > +1 (binding)
> > >
> > > - Verified checksums and signatures.
> > > - Built from the source with CentOS 7 and OpenJDK 8.
> > > - Successfully upgraded HDFS to 3.3.0-RC0 in our development cluster
> > (with
> > > RBF, security, and OpenJDK 11) for end-users. No issues reported.
> > > - The document looks good.
> > > - Deployed pseudo cluster and ran some MapReduce jobs.
> > >
> > > Thanks,
> > > Akira
> > >
> > >
> > > On Tue, Jul 7, 2020 at 7:27 AM Brahma Reddy Battula  >
> > > wrote:
> > >
> > > > Hi folks,
> > > >
> > > > This is the first release candidate for the first release of Apache
> > > > Hadoop 3.3.0
> > > > line.
> > > >
> > > > It contains *1644[1]* fixed jira issues since 3.2.1 which include a
> lot
> > > of
> > > > features and improvements(read the full set of release notes).
> > > >
> > > > Below feature additions are the highlights of the release.
> > > >
> > > > - ARM Support
> > > > - Enhancements and new features on S3a,S3Guard,ABFS
> > > > - Java 11 Runtime support and TLS 1.3.
> > > > - Support Tencent Cloud COS File System.
> > > > - Added security to HDFS Router.
> > > > - Support non-volatile storage class memory(SCM) in HDFS cache
> > directives
> > > > - Support Interactive Docker Shell for running Containers.
> > > > - Scheduling of opportunistic containers
> > > > - A pluggable device plugin framework to ease vendor plugin
> development
> > > >
> > > > *The RC0 artifacts are at*:
> > > > http://home.apache.org/~brahma/Hadoop-3.3.0-RC0/
> > > >
> > > > *First release to include ARM binary, Have a check.*
> > > > *RC tag is *release-3.3.0-RC0.
> > > >
> > > >
> > > > *The maven artifacts are hosted here:*
> > > >
> > https://repository.apache.org/content/repositories/orgapachehadoop-1271/
> > > >
> > > > *My public key is available here:*
> > > > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> > > >
> > > > The vote will run for 5 weekdays, until Tuesday, July 13 at 3:50 AM
> > IST.
> > > >
> > > >
> > > > I have done a few testing with my pseudo cluster. My +1 to start.
> > > >
> > > >
> > > >
> > > > Regards,
> > > > Brahma Reddy Battula
> > > >
> > > >
> > > > 1. project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> (3.3.0)
> > > AND
> > > > fixVersion not in (3.2.0, 3.2.1, 3.1.3) AND status = Resolved ORDER
> BY
> > > > fixVersion ASC
> > > >
> > >
> >
>

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Vinayakumar B]

lass file for
> > org.apache.hadoop.thirdparty.protobuf.Descriptors not found
> >   [ERROR]
> >
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/io/asyncfs/FanOutOneBlockAsyncDFSOutput.java:[323,64]
> > incompatible types:
> > org.apache.hadoop.hdfs.protocol.proto.DataTransferProtos.PipelineAckProto
> > cannot be converted to com.google.protobuf.MessageLite
> >   [ERROR]
> >
> /Users/weichiu/sandbox/hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/wal/SyncReplicationWALProvider.java:[209,68]
> > invalid method reference
> >   [ERROR]   non-static method get() cannot be referenced from a
> > static context
> >
> >
> > Ozone:
> >
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR] COMPILATION ERROR :
> > 17:01:19 2020/04/16 00:01:19 INFO: [INFO]
> > -
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR]
> >
> >
> /grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java:[110,47]
> > incompatible types: com.google.protobuf.ServiceException cannot be
> > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > 17:01:19 2020/04/16 00:01:19 INFO: [ERROR]
> >
> >
> /grid/0/jenkins/workspace/workspace/CDH-CANARY-parallel-centos7/SOURCES/ozone/hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/StorageContainerLocationProtocolClientSideTranslatorPB.java:[116,47]
> > incompatible types: com.google.protobuf.ServiceException cannot be
> > converted to org.apache.hadoop.thirdparty.protobuf.ServiceException
> > 17:01:19 2020/04/16 00:01:19 INFO: [INFO] 2 errors
> >
> >
> > There's another error where Ozone uses the Hadoop RPC framework which
> uses
> > the hadoop.thirdparty protobuf.
> >
> > [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.1:testCompile
> > (default-testCompile) on project hadoop-hdds-container-service:
> Compilation
> > failure
> > [ERROR]
> >
> >
> /Users/weichiu/sandbox/ozone/hadoop-hdds/container-service/src/test/java/org/apache/hadoop/ozone/container/common/SCMTestUtils.java:[103,41]
> > incompatible types: com.google.protobuf.BlockingService cannot be
> converted
> > to org.apache.hadoop.thirdparty.protobuf.BlockingService
> >
> > BlockingService scmDatanodeService =
> > StorageContainerDatanodeProtocolService.
> > newReflectiveBlockingService(
> > new StorageContainerDatanodeProtocolServerSideTranslatorPB(
> > server, Mockito.mock(ProtocolMessageMetrics.class)));
> >
> >
> >
> > Ratis probably breaks as well since it depends on the Hadoop RPC
> framework
> > too.
> >
> > On Tue, Apr 28, 2020 at 10:58 PM Vinayakumar B 
> > wrote:
> >
> > > hi Wei-Chiu,
> > >
> > > Can you elaborate on what failures you are facing related to relocated
> > > protobuf classes.. ?
> > >
> > > IFAIK, if the issue with location of protobuf classes, still old jar
> > > protobuf-2.5.0.jar will be available in classpath. So downstream
> > depending
> > > on 2.5.0 version of protobuf still be able to access them.
> > >
> > > -vinay
> > >
> > > On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang, 
> > > wrote:
> > >
> > >> I'm sorry for coming to this late. I missed this message. It should
> have
> > >> been a DISCUSS thread rather than NOTICE.
> > >>
> > >> Looks like this is inevitable. But we should make the downstream
> > >> developers aware & make the update easier. As long as it is stated
> > clearly
> > >> how to update the code to support Hadoop 3.3, I am okay with that.
> > >>
> > >> Here's what I suggest:
> > >> (1) label the jira incompatible (just updated the jira) and updated
> the
> > >> release note to tell app developer how to update.
> > >> (2) declare ProtobufHelper a public API HADOOP-17019
> > >> <https://issues.apache.org/jira/browse/HADOOP-17019>
> > >>
> > >> Tez doesn't use the removed Token API, but there's code that breaks
> with
> > >> the relocated protobuf class. The ProtobufHelper API will make this
> > >> transition much easier.
> > >>
> > >> Other downstreamers that break with the relocated prot

[jira] [Created] (HADOOP-17046) Support downstreams' existing Hadoop-rpc implementations using non-shaded protobuf classes.

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-17046:
--

 Summary: Support downstreams' existing Hadoop-rpc implementations 
using non-shaded protobuf classes.
 Key: HADOOP-17046
 URL: https://issues.apache.org/jira/browse/HADOOP-17046
 Project: Hadoop Common
  Issue Type: Improvement
  Components: rpc-server
Affects Versions: 3.3.0
Reporter: Vinayakumar B


After upgrade/shade of protobuf to 3.7 version, existing Hadoop-Rpc 
client-server implementations using ProtobufRpcEngine will not work.

So, this Jira proposes to keep existing ProtobuRpcEngine as-is (without shading 
and with protobuf-2.5.0 implementation) to support downstream implementations.

Use new ProtobufRpcEngine2 to use shaded protobuf classes within Hadoop and 
later projects who wish to upgrade their protobufs to 3.x.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] making Ozone a separate Apache project

[Vinayakumar B]

+1

-Vinay

On Thu, 14 May 2020, 12:20 pm Surendra Singh Lilhore, <
surendralilh...@gmail.com> wrote:

> +1
>
> -Surendra
>
>
> On Thu, 14 May, 2020, 9:30 am yisheng lien,  wrote:
>
> > +1
> >
> > Thank you Marton for the detailed description.
> >
> >
> > Wanqiang Ji  於 2020年5月14日 週四 上午11:27 寫道：
> >
> > > +1
> > >
> > > On Wed, May 13, 2020 at 3:53 PM Elek, Marton  wrote:
> > >
> > > >
> > > >
> > > > I would like to start a discussion to make a separate Apache project
> > for
> > > > Ozone
> > > >
> > > >
> > > >
> > > > ### HISTORY [1]
> > > >
> > > >   * Apache Hadoop Ozone development started on a feature branch of
> > > > Hadoop repository (HDFS-7240)
> > > >
> > > >   * In the October of 2017 a discussion has been started to merge it
> to
> > > > the Hadoop main branch
> > > >
> > > >   * After a long discussion it's merged to Hadoop trunk at the March
> of
> > > > 2018
> > > >
> > > >   * During the discussion of the merge, it was suggested multiple
> times
> > > > to create a separated project for the Ozone. But at that time:
> > > >  1). Ozone was tightly integrated with Hadoop/HDFS
> > > >  2). There was an active plan to use Block layer of Ozone (HDDS
> or
> > > > HDSL at that time) as the block level of HDFS
> > > >  3). The community of Ozone was a subset of the HDFS community
> > > >
> > > >   * The first beta release of Ozone was just released. Seems to be a
> > > > good time before the first GA to make a decision about the future.
> > > >
> > > >
> > > >
> > > > ### WHAT HAS BEEN CHANGED
> > > >
> > > >   During the last years Ozone became more and more independent both
> at
> > > > the community and code side. The separation has been suggested again
> > and
> > > > again (for example by Owen [2] and Vinod [3])
> > > >
> > > >
> > > >
> > > >   From COMMUNITY point of view:
> > > >
> > > >
> > > >* Fortunately more and more new contributors are helping Ozone.
> > > > Originally the Ozone community was a subset of HDFS project. But now
> a
> > > > bigger and bigger part of the community is related to Ozone only.
> > > >
> > > >* It seems to be easier to _build_ the community as a separated
> > > project.
> > > >
> > > >* A new, younger project might have different practices
> > > > (communication, commiter criteria, development style) compared to
> old,
> > > > mature project
> > > >
> > > >* It's easier to communicate (and improve) these standards in a
> > > > separated projects with clean boundaries
> > > >
> > > >* Separated project/brand can help to increase the adoption rate
> and
> > > > attract more individual contributor (AFAIK it has been seen in
> > Submarine
> > > > after a similar move)
> > > >
> > > >   * Contribution process can be communicated more easily, we can make
> > > > first time contribution more easy
> > > >
> > > >
> > > >
> > > >   From CODE point of view Ozone became more and more independent:
> > > >
> > > >
> > > >   * Ozone has different release cycle
> > > >
> > > >   * Code is already separated from Hadoop code base
> > > > (apache/hadoop-ozone.git)
> > > >
> > > >   * It has separated CI (github actions)
> > > >
> > > >   * Ozone uses different (more strict) coding style (zero toleration
> of
> > > > unit test / checkstyle errors)
> > > >
> > > >   * The code itself became more and more independent from Hadoop on
> > > > Maven level. Originally it was compiled together with the in-tree
> > latest
> > > > Hadoop snapshot. Now it depends on released Hadoop artifacts (RPC,
> > > > Configuration...)
> > > >
> > > >   * It starts to use multiple version of Hadoop (on client side)
> > > >
> > > >   * Volume of resolved issues are already very high on Ozone side
> > (Ozone
> > > > had slightly more resolved issues than HDFS/YARN/MAPREDUCE/COMMON all
> > > > together in the last 2-3 months)
> > > >
> > > >
> > > > Summary: Before the first Ozone GA release, It seems to be a good
> time
> > > > to discuss the long-term future of Ozone. Managing it as a separated
> > TLP
> > > > project seems to have more benefits.
> > > >
> > > >
> > > > Please let me know what your opinion is...
> > > >
> > > > Thanks a lot,
> > > > Marton
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > [1]: For more details, see:
> > > > https://github.com/apache/hadoop-ozone/blob/master/HISTORY.md
> > > >
> > > > [2]:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/0d0253f6e5fa4f609bd9b917df8e1e4d8848e2b7fdb3099b730095e6%40%3Cprivate.hadoop.apache.org%3E
> > > >
> > > > [3]:
> > > >
> > > >
> > >
> >
> https://lists.apache.org/thread.html/8be74421ea495a62e159f2b15d74627c63ea1f67a2464fa02c85d4aa%40%3Chdfs-dev.hadoop.apache.org%3E
> > > >
> > > > -
> > > > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > > > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> > > >
> > > >
> > >
> >
>

Re: [NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Vinayakumar B]

hi Wei-Chiu,

Can you elaborate on what failures you are facing related to relocated
protobuf classes.. ?

IFAIK, if the issue with location of protobuf classes, still old jar
protobuf-2.5.0.jar will be available in classpath. So downstream depending
on 2.5.0 version of protobuf still be able to access them.

-vinay

On Wed, 29 Apr 2020, 11:17 am Wei-Chiu Chuang,  wrote:

> I'm sorry for coming to this late. I missed this message. It should have
> been a DISCUSS thread rather than NOTICE.
>
> Looks like this is inevitable. But we should make the downstream
> developers aware & make the update easier. As long as it is stated clearly
> how to update the code to support Hadoop 3.3, I am okay with that.
>
> Here's what I suggest:
> (1) label the jira incompatible (just updated the jira) and updated the
> release note to tell app developer how to update.
> (2) declare ProtobufHelper a public API HADOOP-17019
> <https://issues.apache.org/jira/browse/HADOOP-17019>
>
> Tez doesn't use the removed Token API, but there's code that breaks with
> the relocated protobuf class. The ProtobufHelper API will make this
> transition much easier.
>
> Other downstreamers that break with the relocated protobuf include: Ozone
> and HBase. but neither of them use the removed Token API.
>
>
> On Wed, Jan 8, 2020 at 4:40 AM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>>This mail is to notify about the Removal of following public APIs from
>> Hadoop Common.
>>
>>  ClassName: org.apache.hadoop.security.token.Token
>>  APIs:
>>  public Token(TokenProto tokenPB);
>>  public TokenProto toTokenProto();
>>
>>Reason: These APIs are having Generated protobuf classes in the
>> signature. Right now due to protobuf upgrade in trunk (soon to be 3.3.0
>> release) these APIs are breaking the downstream builds, even though
>> downstreams dont use these APIs (just Loading Token class). Downstreams
>> are
>> still referencing having older version (2.5.0) of protobuf, hence build is
>> being broken.
>>
>> These APIs were added for the internal purpose(HADOOP-12563), to
>> support serializing tokens using protobuf in UGI Credentials.
>> Same purpose can be achieved using the Helper classes without introducing
>> protobuf classes in API signatures.
>>
>> Token.java is marked as Evolving, so I believe APIs can be changed
>> whenever
>> absolute necessary.
>>
>> Jira https://issues.apache.org/jira/browse/HADOOP-16621 has been
>> reported to solve downstream build failure.
>>
>> So since this API was added for internal purpose easy approach to solve
>> this is to remove APIs and use helper classes. Otherwise, as mentioned in
>> HADOOP-16621, workaround will add unnecessary codes to be maintained.
>>
>> If anyone using these APIs outside hadoop project accidentally, please
>> reply to this mail immediately.
>>
>> If no objection by next week, will go ahead with removal of above said
>> APIs
>> in HADOOP-16621.
>>
>> -Vinay
>>
>

[jira] [Resolved] (HADOOP-16985) Handle release package related issues

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16985.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to branch-3.3 and trunk

> Handle release package related issues
> -
>
> Key: HADOOP-16985
> URL: https://issues.apache.org/jira/browse/HADOOP-16985
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: 3.3.0
>
>
> Same issue as mentioned in HADOOP-16919 is present in hadoop distribution 
> generation as well.
> Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread 
> here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
> {quote}3. Yetus seems to be included in the source package. I am not sure if
>  it's intentional but I would remove the patchprocess directory from the
>  tar file.
> 7. Minor nit: I would suggest to use only the filename in the sha512
>  files (instead of having the /build/source/target prefix). It would help
>  to use `sha512 -c` command to validate the checksum.
> {quote}
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16985) Handle release package related issues

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16985:
--

 Summary: Handle release package related issues
 Key: HADOOP-16985
 URL: https://issues.apache.org/jira/browse/HADOOP-16985
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Vinayakumar B


Same issue as mentioned in HADOOP-16919 is present in hadoop distribution 
generation as well.

Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread 
here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
{quote}3. Yetus seems to be included in the source package. I am not sure if
 it's intentional but I would remove the patchprocess directory from the
 tar file.

7. Minor nit: I would suggest to use only the filename in the sha512
 files (instead of having the /build/source/target prefix). It would help
 to use `sha512 -c` command to validate the checksum.
{quote}
 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16927) Update hadoop-thirdparty dependency version to 1.0.0

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16927.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to trunk.

> Update hadoop-thirdparty dependency version to 1.0.0
> 
>
> Key: HADOOP-16927
> URL: https://issues.apache.org/jira/browse/HADOOP-16927
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
>  Labels: release-blocker
> Fix For: 3.3.0
>
>
> Now hadoop-thirdparty 1.0.0 is released, its time to upgrade to released 
> version in hadoop



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Thanks brahma.
So total 5 +1s (4 binding, 1 non-binding).

Done with pushing bits. Should get mirrored in next 24 hours.

-Vinay


On Thu, Mar 19, 2020 at 12:55 AM Brahma Reddy Battula 
wrote:

>
> My late +1.. I missed this..
>
>
> On Thu, 19 Mar 2020 at 12:43 AM, Vinayakumar B 
> wrote:
>
>> Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s
>>
>> Thanks everyone for verification.
>> I will push out bits soon.
>>
>> -Vinay
>>
>>
>> On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
>> surendralilh...@gmail.com> wrote:
>>
>> > +1
>> >
>> > -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
>> > -Verified artifacts available in repo.
>> >
>> >
>> > Thanks
>> > Surendra
>> >
>> >
>> > On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka 
>> wrote:
>> >
>> >> +1
>> >>
>> >> - Verified signatures and checksums
>> >> - Built jars and docs from source
>> >> - Built hadoop trunk with hadoop-thirdparty 1.0.0
>> >> - Checked rat files and documents
>> >> - Checked LICENSE and NOTICE files
>> >>
>> >> Thanks,
>> >> Akira
>> >>
>> >> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B > >
>> >> wrote:
>> >>
>> >> > Hi folks,
>> >> >
>> >> > Thanks to everyone's help on this release.
>> >> >
>> >> > I have re-created a release candidate (RC1) for Apache Hadoop
>> Thirdparty
>> >> > 1.0.0.
>> >> >
>> >> > RC Release artifacts are available at :
>> >> >
>> >> >
>> >>
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
>> >> >
>> >> > Maven artifacts are available in staging repo:
>> >> >
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >> >
>> >> > The RC tag in git is here:
>> >> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
>> >> >
>> >> > And my public key is at:
>> >> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >> >
>> >> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
>> >> IST.*
>> >> >
>> >> > For the testing, I have verified Hadoop trunk compilation with
>> >> >"-DdistMgmtSnapshotsUrl=
>> >> >
>> >>
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >> >
>> >> > My +1 to start.
>> >> >
>> >> > -Vinay
>> >> >
>> >>
>> >
>>
> --
>
>
>
> --Brahma Reddy Battula
>

[jira] [Created] (HADOOP-16927) Update hadoop-thirdparty dependency version to 1.0.0

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16927:
--

 Summary: Update hadoop-thirdparty dependency version to 1.0.0
 Key: HADOOP-16927
 URL: https://issues.apache.org/jira/browse/HADOOP-16927
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Vinayakumar B






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[RESULT] [VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Vote passes with 4 +1s (3 binding, 1 non-binding) and no -1s

Thanks everyone for verification.
I will push out bits soon.

-Vinay


On Sun, Mar 15, 2020 at 5:25 PM Surendra Singh Lilhore <
surendralilh...@gmail.com> wrote:

> +1
>
> -Built trunk with -Dhadoop-thirdparty-protobuf.version=1.0.0
> -Verified artifacts available in repo.
>
>
> Thanks
> Surendra
>
>
> On Fri, Mar 13, 2020 at 9:51 AM Akira Ajisaka  wrote:
>
>> +1
>>
>> - Verified signatures and checksums
>> - Built jars and docs from source
>> - Built hadoop trunk with hadoop-thirdparty 1.0.0
>> - Checked rat files and documents
>> - Checked LICENSE and NOTICE files
>>
>> Thanks,
>> Akira
>>
>> On Thu, Mar 12, 2020 at 5:26 AM Vinayakumar B 
>> wrote:
>>
>> > Hi folks,
>> >
>> > Thanks to everyone's help on this release.
>> >
>> > I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty
>> > 1.0.0.
>> >
>> > RC Release artifacts are available at :
>> >
>> >
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/
>> >
>> > Maven artifacts are available in staging repo:
>> >
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >
>> > The RC tag in git is here:
>> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1
>> >
>> > And my public key is at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > *This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm
>> IST.*
>> >
>> > For the testing, I have verified Hadoop trunk compilation with
>> >"-DdistMgmtSnapshotsUrl=
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1261/
>> >  -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >
>> > My +1 to start.
>> >
>> > -Vinay
>> >
>>
>

Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary

[Vinayakumar B]

Making ARM artifact optional, makes the release process simpler for RM  and
unblocks release process (if there is unavailability of ARM resources).

Still there are possible options to collaborate with RM ( as brahma
mentioned earlier) and provide ARM artifact may be before or after vote.
If feasible RM can decide to add ARM artifact by collaborating with @Brahma
Reddy Battula  or me to get the ARM artifact.

-Vinay

On Tue, Mar 17, 2020 at 11:39 PM Arpit Agarwal
 wrote:

> Thanks for the clarification Brahma. Can you update the proposal to state
> that it is optional (it may help to put the proposal on cwiki)?
>
> Also if we go ahead then the RM documentation should be clear this is an
> optional step.
>
>
> > On Mar 17, 2020, at 11:06 AM, Brahma Reddy Battula 
> wrote:
> >
> > Sure, we can't make mandatory while voting and we can upload to downloads
> > once release vote is passed.
> >
> > On Tue, 17 Mar 2020 at 11:24 PM, Arpit Agarwal
> >  wrote:
> >
> >>> Sorry,didn't get you...do you mean, once release voting is
> >>> processed and upload by RM..?
> >>
> >> Yes, that is what I meant. I don’t want us to make more mandatory work
> for
> >> the release manager because the job is hard enough already.
> >>
> >>
> >>> On Mar 17, 2020, at 10:46 AM, Brahma Reddy Battula 
> >> wrote:
> >>>
> >>> Sorry,didn't get you...do you mean, once release voting is processed
> and
> >>> upload by RM..?
> >>>
> >>> FYI. There is docker image for ARM also which support all scripts
> >>> (createrelease, start-build-env.sh, etc ).
> >>>
> >>> https://issues.apache.org/jira/browse/HADOOP-16797
> >>>
> >>> On Tue, Mar 17, 2020 at 10:59 PM Arpit Agarwal
> >>>  wrote:
> >>>
>  Can ARM binaries be provided after the fact? We cannot increase the
> RM’s
>  burden by asking them to generate an extra set of binaries.
> 
> 
> > On Mar 17, 2020, at 10:23 AM, Brahma Reddy Battula <
> bra...@apache.org>
>  wrote:
> >
> > + Dev mailing list.
> >
> > -- Forwarded message -
> > From: Brahma Reddy Battula 
> > Date: Tue, Mar 17, 2020 at 10:31 PM
> > Subject: Re: [DISCUSS] Hadoop 3.3.0 Release include ARM binary
> > To: junping_du 
> >
> >
> > thanks junping for your reply.
> >
> > bq.  I think most of us in Hadoop community doesn't want to have
>  biased
> > on ARM or any other platforms.
> >
> > Yes, release voting will be based on the source code.AFAIK,Binary we
> >> are
> > providing for user to easy to download and verify.
> >
> > bq. The only thing I try to understand is how much complexity get
> > involved for our RM work. Does that potentially become a blocker for
>  future
> > releases? And how we can get rid of this risk.
> >
> > As I mentioned earlier, RM need to access the ARM machine(it will be
> > donated and current qbt also using one ARM machine) and build tar
> using
>  the
> > keys. As it can be common machine, RM can delete his keys once
> release
> > approved.
> > Can be sorted out as I mentioned earlier.(For accessing the ARM
> >> machine)
> >
> > bq.   If you can list the concrete work that RM need to do extra
> >> for
> > ARM release, that would help us to better understand.
> >
> > I can write and update for future reference.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > On Tue, Mar 17, 2020 at 10:41 AM 俊平堵  wrote:
> >
> >> Hi Brahma,
> >>   I think most of us in Hadoop community doesn't want to have biased
>  on
> >> ARM or any other platforms.
> >>   The only thing I try to understand is how much complexity get
> >> involved for our RM work. Does that potentially become a blocker for
>  future
> >> releases? And how we can get rid of this risk.
> >>If you can list the concrete work that RM need to do extra for
> ARM
> >> release, that would help us to better understand.
> >>
> >> Thanks,
> >>
> >> Junping
> >>
> >> Akira Ajisaka  于2020年3月13日周五 上午12:34写道：
> >>
> >>> If you can provide ARM release for future releases, I'm fine with
> >> that.
> >>>
> >>> Thanks,
> >>> Akira
> >>>
> >>> On Thu, Mar 12, 2020 at 9:41 PM Brahma Reddy Battula <
>  bra...@apache.org>
> >>> wrote:
> >>>
>  thanks Akira.
> 
>  Currently only problem is dedicated ARM for future RM.This i want
> to
> >>> sort
>  out like below,if you've some other,please let me know.
> 
>  i) Single machine and share cred to future RM ( as we can delete
> >> keys
> >>> once
>  release is over).
>  ii) Creating the jenkins project ( may be we need to discuss in
> the
>  board..)
>  iii) I can provide ARM release for future releases.
> 
> 
> 
> 
> 
> 
> 
>  On Thu, Mar 12, 2020 at 

[VOTE] Release Apache Hadoop Thirdparty 1.0.0 - RC1

[Vinayakumar B]

Hi folks,

Thanks to everyone's help on this release.

I have re-created a release candidate (RC1) for Apache Hadoop Thirdparty
1.0.0.

RC Release artifacts are available at :
  http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC1/

Maven artifacts are available in staging repo:
https://repository.apache.org/content/repositories/orgapachehadoop-1261/

The RC tag in git is here:
https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC1

And my public key is at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

*This vote will run for 5 days, ending on March 18th 2020 at 11:59 pm IST.*

For the testing, I have verified Hadoop trunk compilation with
   "-DdistMgmtSnapshotsUrl=
https://repository.apache.org/content/repositories/orgapachehadoop-1261/
 -Dhadoop-thirdparty-protobuf.version=1.0.0"

My +1 to start.

-Vinay

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Hello everyone.

RC0 has been canceled.

Since issues mentioned above are already fixed now, soon I will create RC1
and re-create VOTE thread.

Thanks for trying out RC,
-Vinay


On Tue, Mar 3, 2020 at 11:41 AM Ayush Saxena  wrote:

> Hi Vinay
> Thanx for driving the release.
> Verified checksums and tried building from source.
> Everything seems to be working fine.
> But I feel the concerns regarding licences are valid.
> IMO we should fix them and include HADOOP-16895 too in the release
>
> -Ayush
>
> > On 29-Feb-2020, at 1:45 AM, Vinayakumar B 
> wrote:
> >
> > https://issues.apache.org/jira/browse/HADOOP-16895 jira created for
> > handling LICENCE and NOTICEs
> > PR also has been raised for a proposal. Please validate
> > https://github.com/apache/hadoop-thirdparty/pull/6
> >
> > -Vinay
> >
> >
> >> On Fri, Feb 28, 2020 at 11:48 PM Vinayakumar B  >
> >> wrote:
> >>
> >> Thanks Elek for detailed verification.
> >>
> >> Please find inline replies.
> >>
> >> -Vinay
> >>
> >>
> >>> On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:
> >>>
> >>>
> >>> Thank you very much to work on this release Vinay, 1.0.0 is always a
> >>> hard work...
> >>>
> >>>
> >>> 1. I downloaded it and I can build it from the source
> >>>
> >>> 2. Checked the signature and the sha512 of the src package and they are
> >>> fine
> >>>
> >>> 3. Yetus seems to be included in the source package. I am not sure if
> >>> it's intentional but I would remove the patchprocess directory from the
> >>> tar file.
> >>>
> >>> Since dev-support/create-release script and assembly file is copied
> from
> >> hadoop-repo,  I can find this issue exits in hadoop source release
> packages
> >> as well. ex: I checked 3.1.2 and 2.10 src packages.
> >> I will raise a Jira and fix this for both hadoop and thirdparty.
> >>
> >> 4. NOTICE.txt seems to be outdated (I am not sure, but I think the
> >>> Export Notice is unnecessary, especially for the source release, also
> >>> the note about the bouncycastle and Yarn server is unnecessary).
> >>>
> >>> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create
> a
> >> jira to decide about NOTICE and LICENSEs
> >>
> >> 5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
> >>> unrelated entries, especially the NOTICE). IMHO
> >>>
> >>> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to
> be
> >> used or not.
> >>
> >> 6. As far as I understand the binary release in this case is the maven
> >>> artifact. IANAL but the original protobuf license seems to be missing
> >>> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar
> META-INF/LICENSE.txt"
> >>>
> >>
> >> I observed that there is one more file "META-INF/DEPENDENCIES" generated
> >> by shade plugin, which have reference to shaded artifacts and poniting
> to
> >> link of the original artifact LICENSE. I think this should be sufficient
> >> about protobuf's original license.
> >> IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
> >> which in-turn can have contents/pointers of dependents' licenses.
> Siimilar
> >> approach followed in hadoop-shaded-client jars.
> >>
> >> hadoop's artifacts also will be uploaded to maven repo during release,
> >> which doesnot carry all LICENSE files in artifacts. It just says "See
> >> licenses/ for text of these licenses" which doesnot exist in artifact.
> May
> >> be we need to fix this too.
> >>
> >> 7. Minor nit: I would suggest to use only the filename in the sha512
> >>> files (instead of having the /build/source/target prefix). It would
> help
> >>> to use `sha512 -c` command to validate the checksum.
> >>>
> >>>
> >> Again, this is from create-release  script. will update the script.
> >>
> >> Thanks again to work on this,
> >>> Marton
> >>>
> >>> ps: I am not experienced with licensing enough to judge which one of
> >>> these are blocking and I might be wrong.
> >>>
> >>>
> >> IMO, none of these should be blocking and can be handled before next
> >> release. Still if someone feels this shou

[jira] [Resolved] (HADOOP-16919) [thirdparty] Handle release package related issues

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16919.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to trunk,branch-1.0 of hadoop-thirdparty.

Thanks [~ayushtkn] for reviews

> [thirdparty] Handle release package related issues
> --
>
> Key: HADOOP-16919
> URL: https://issues.apache.org/jira/browse/HADOOP-16919
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: hadoop-thirdparty
>    Reporter: Vinayakumar B
>        Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread 
> here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
> {quote}3. Yetus seems to be included in the source package. I am not sure if
>  it's intentional but I would remove the patchprocess directory from the
>  tar file.
> 7. Minor nit: I would suggest to use only the filename in the sha512
>  files (instead of having the /build/source/target prefix). It would help
>  to use `sha512 -c` command to validate the checksum.
> {quote}
> Also, update available artifacts in docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16895) [thirdparty] Revisit LICENSEs and NOTICEs

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16895.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Committed to brach-1.0, trunk of hadoop-thirdparty

Thanks [~aajisaka] and [~elek] for reviews.

> [thirdparty] Revisit LICENSEs and NOTICEs
> -
>
> Key: HADOOP-16895
> URL: https://issues.apache.org/jira/browse/HADOOP-16895
> Project: Hadoop Common
>  Issue Type: Improvement
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> LICENSE.txt and NOTICE.txt have many entries which are unrelated to 
> thirdparty,
> Revisit and cleanup such entries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16919) [thirdparty] Handle release package related issues

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16919:
--

 Summary: [thirdparty] Handle release package related issues
 Key: HADOOP-16919
 URL: https://issues.apache.org/jira/browse/HADOOP-16919
 Project: Hadoop Common
  Issue Type: Bug
  Components: hadoop-thirdparty
Reporter: Vinayakumar B


Handle following comments from [~elek] in 1.0.0-RC0 voting mail thread 
here[[https://lists.apache.org/thread.html/r1f2e8325ecef239f0d713c683a16336e2a22431a9f6bfbde3c763816%40%3Ccommon-dev.hadoop.apache.org%3E]]
{quote}3. Yetus seems to be included in the source package. I am not sure if
 it's intentional but I would remove the patchprocess directory from the
 tar file.

7. Minor nit: I would suggest to use only the filename in the sha512
 files (instead of having the /build/source/target prefix). It would help
 to use `sha512 -c` command to validate the checksum.
{quote}
Also, update available artifacts in docs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

https://issues.apache.org/jira/browse/HADOOP-16895 jira created for
handling LICENCE and NOTICEs
PR also has been raised for a proposal. Please validate
https://github.com/apache/hadoop-thirdparty/pull/6

-Vinay


On Fri, Feb 28, 2020 at 11:48 PM Vinayakumar B 
wrote:

> Thanks Elek for detailed verification.
>
> Please find inline replies.
>
> -Vinay
>
>
> On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:
>
>>
>> Thank you very much to work on this release Vinay, 1.0.0 is always a
>> hard work...
>>
>>
>> 1. I downloaded it and I can build it from the source
>>
>> 2. Checked the signature and the sha512 of the src package and they are
>> fine
>>
>> 3. Yetus seems to be included in the source package. I am not sure if
>> it's intentional but I would remove the patchprocess directory from the
>> tar file.
>>
>> Since dev-support/create-release script and assembly file is copied from
> hadoop-repo,  I can find this issue exits in hadoop source release packages
> as well. ex: I checked 3.1.2 and 2.10 src packages.
> I will raise a Jira and fix this for both hadoop and thirdparty.
>
> 4. NOTICE.txt seems to be outdated (I am not sure, but I think the
>> Export Notice is unnecessary, especially for the source release, also
>> the note about the bouncycastle and Yarn server is unnecessary).
>>
>> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create a
> jira to decide about NOTICE and LICENSEs
>
> 5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
>> unrelated entries, especially the NOTICE). IMHO
>>
>> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to be
> used or not.
>
> 6. As far as I understand the binary release in this case is the maven
>> artifact. IANAL but the original protobuf license seems to be missing
>> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar META-INF/LICENSE.txt"
>>
>
> I observed that there is one more file "META-INF/DEPENDENCIES" generated
> by shade plugin, which have reference to shaded artifacts and poniting to
> link of the original artifact LICENSE. I think this should be sufficient
> about protobuf's original license.
> IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
> which in-turn can have contents/pointers of dependents' licenses. Siimilar
> approach followed in hadoop-shaded-client jars.
>
> hadoop's artifacts also will be uploaded to maven repo during release,
> which doesnot carry all LICENSE files in artifacts. It just says "See
> licenses/ for text of these licenses" which doesnot exist in artifact. May
> be we need to fix this too.
>
> 7. Minor nit: I would suggest to use only the filename in the sha512
>> files (instead of having the /build/source/target prefix). It would help
>> to use `sha512 -c` command to validate the checksum.
>>
>>
> Again, this is from create-release  script. will update the script.
>
> Thanks again to work on this,
>> Marton
>>
>> ps: I am not experienced with licensing enough to judge which one of
>> these are blocking and I might be wrong.
>>
>>
> IMO, none of these should be blocking and can be handled before next
> release. Still if someone feels this should be fixed and RC should be cut
> again, I am open to it.
>
> Thanks.
>
>>
>>
>> On 2/25/20 8:17 PM, Vinayakumar B wrote:
>> > Hi folks,
>> >
>> > Thanks to everyone's help on this release.
>> >
>> > I have created a release candidate (RC0) for Apache Hadoop Thirdparty
>> 1.0.0.
>> >
>> > RC Release artifacts are available at :
>> >
>> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/
>> >
>> > Maven artifacts are available in staging repo:
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
>> >
>> > The RC tag in git is here:
>> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0
>> >
>> > And my public key is at:
>> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
>> >
>> > *This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm
>> IST.*
>> >
>> > For the testing, I have verified Hadoop trunk compilation with
>> > "-DdistMgmtSnapshotsUrl=
>> >
>> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
>> > -Dhadoop-thirdparty-protobuf.version=1.0.0"
>> >
>> > My +1 to start.
>> >
>> > -Vinay
>> >
>>
>

[jira] [Created] (HADOOP-16895) [thirdparty] Revisit LICENSEs and NOTICEs

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16895:
--

 Summary: [thirdparty] Revisit LICENSEs and NOTICEs
 Key: HADOOP-16895
 URL: https://issues.apache.org/jira/browse/HADOOP-16895
 Project: Hadoop Common
  Issue Type: Improvement
Reporter: Vinayakumar B


LICENSE.txt and NOTICE.txt have many entries which are unrelated to thirdparty,
Revisit and cleanup such entries.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Thanks Elek for detailed verification.

Please find inline replies.

-Vinay


On Fri, Feb 28, 2020 at 7:49 PM Elek, Marton  wrote:

>
> Thank you very much to work on this release Vinay, 1.0.0 is always a
> hard work...
>
>
> 1. I downloaded it and I can build it from the source
>
> 2. Checked the signature and the sha512 of the src package and they are
> fine
>
> 3. Yetus seems to be included in the source package. I am not sure if
> it's intentional but I would remove the patchprocess directory from the
> tar file.
>
> Since dev-support/create-release script and assembly file is copied from
hadoop-repo,  I can find this issue exits in hadoop source release packages
as well. ex: I checked 3.1.2 and 2.10 src packages.
I will raise a Jira and fix this for both hadoop and thirdparty.

4. NOTICE.txt seems to be outdated (I am not sure, but I think the
> Export Notice is unnecessary, especially for the source release, also
> the note about the bouncycastle and Yarn server is unnecessary).
>
> Again, NOTICE.txt was copied from Hadoop and kept as is. I will create a
jira to decide about NOTICE and LICENSEs

5. NOTICE-binary and LICENSE-binary seems to be unused (and they contain
> unrelated entries, especially the NOTICE). IMHO
>
> We can decide in the Jira whether NOTICE-binary and LICENSE-binary to be
used or not.

6. As far as I understand the binary release in this case is the maven
> artifact. IANAL but the original protobuf license seems to be missing
> from "unzip -p hadoop-shaded-protobuf_3_7-1.0.0.jar META-INF/LICENSE.txt"
>

I observed that there is one more file "META-INF/DEPENDENCIES" generated by
shade plugin, which have reference to shaded artifacts and poniting to link
of the original artifact LICENSE. I think this should be sufficient about
protobuf's original license.
IMO, "META-INF/LICENSE.txt" should point to current project's LICENSE,
which in-turn can have contents/pointers of dependents' licenses. Siimilar
approach followed in hadoop-shaded-client jars.

hadoop's artifacts also will be uploaded to maven repo during release,
which doesnot carry all LICENSE files in artifacts. It just says "See
licenses/ for text of these licenses" which doesnot exist in artifact. May
be we need to fix this too.

7. Minor nit: I would suggest to use only the filename in the sha512
> files (instead of having the /build/source/target prefix). It would help
> to use `sha512 -c` command to validate the checksum.
>
>
Again, this is from create-release  script. will update the script.

Thanks again to work on this,
> Marton
>
> ps: I am not experienced with licensing enough to judge which one of
> these are blocking and I might be wrong.
>
>
IMO, none of these should be blocking and can be handled before next
release. Still if someone feels this should be fixed and RC should be cut
again, I am open to it.

Thanks.

>
>
> On 2/25/20 8:17 PM, Vinayakumar B wrote:
> > Hi folks,
> >
> > Thanks to everyone's help on this release.
> >
> > I have created a release candidate (RC0) for Apache Hadoop Thirdparty
> 1.0.0.
> >
> > RC Release artifacts are available at :
> >
> http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/
> >
> > Maven artifacts are available in staging repo:
> >
> https://repository.apache.org/content/repositories/orgapachehadoop-1258/
> >
> > The RC tag in git is here:
> > https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0
> >
> > And my public key is at:
> > https://dist.apache.org/repos/dist/release/hadoop/common/KEYS
> >
> > *This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm
> IST.*
> >
> > For the testing, I have verified Hadoop trunk compilation with
> > "-DdistMgmtSnapshotsUrl=
> > https://repository.apache.org/content/repositories/orgapachehadoop-1258/
> > -Dhadoop-thirdparty-protobuf.version=1.0.0"
> >
> > My +1 to start.
> >
> > -Vinay
> >
>

[VOTE] Release Apache Hadoop Thirdparty 1.0.0

[Vinayakumar B]

Hi folks,

Thanks to everyone's help on this release.

I have created a release candidate (RC0) for Apache Hadoop Thirdparty 1.0.0.

RC Release artifacts are available at :
  http://home.apache.org/~vinayakumarb/release/hadoop-thirdparty-1.0.0-RC0/

Maven artifacts are available in staging repo:
https://repository.apache.org/content/repositories/orgapachehadoop-1258/

The RC tag in git is here:
https://github.com/apache/hadoop-thirdparty/tree/release-1.0.0-RC0

And my public key is at:
https://dist.apache.org/repos/dist/release/hadoop/common/KEYS

*This vote will run for 5 days, ending on March 1st 2020 at 11:59 pm IST.*

For the testing, I have verified Hadoop trunk compilation with
   "-DdistMgmtSnapshotsUrl=
https://repository.apache.org/content/repositories/orgapachehadoop-1258/
-Dhadoop-thirdparty-protobuf.version=1.0.0"

My +1 to start.

-Vinay

[DISCUSS] hadoop-thirdparty 1.0.0 release

[Vinayakumar B]

Hi All,

Since Hadoop-3.3.0 release is around the corner, its time to release
hadoop-thirdparty's first ever release, without which hadoop-3.3.0 cannot
proceed for release.

Below are the tentative date for RC and release. Since there is no much
activity in this repo (other than the opentracing related one, which I just
merged), Keeping the plan little aggressive.
Please let me know any concerns regarding the same.

 RC-0 : 25-Feb-2020
Release : 03-Mar-2020 (after 7 days Voting)

-Vinay

[jira] [Resolved] (HADOOP-16596) [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16596.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
 Release Note: All protobuf classes will be used from 
hadooop-shaded-protobuf_3_7 artifact with package prefix as 
'org.apache.hadoop.thirdparty.protobuf' instead of 'com.google.protobuf'
   Resolution: Fixed

Merged to trunk. Thanks everyone for reviews

> [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency
> --
>
> Key: HADOOP-16596
> URL: https://issues.apache.org/jira/browse/HADOOP-16596
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: 3.3.0
>
>
> Use the shaded protobuf classes from "hadoop-thirdparty" in hadoop codebase.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16824) [thirdparty] port HADOOP-16754 (Fix docker failed to build yetus/hadoop) to thirdparty Dockerfile

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16824.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged PR.
Thanks [~aajisaka] for review.

> [thirdparty] port HADOOP-16754 (Fix docker failed to build yetus/hadoop) to 
> thirdparty Dockerfile
> -
>
> Key: HADOOP-16824
> URL: https://issues.apache.org/jira/browse/HADOOP-16824
> Project: Hadoop Common
>  Issue Type: Bug
>    Reporter: Vinayakumar B
>        Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> port HADOOP-16754 to avoid Docker build failure



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16824) [thirdparty] port HADOOP-16754 (Fix docker failed to build yetus/hadoop) to thirdparty Dockerfile

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16824:
--

 Summary: [thirdparty] port HADOOP-16754 (Fix docker failed to 
build yetus/hadoop) to thirdparty Dockerfile
 Key: HADOOP-16824
 URL: https://issues.apache.org/jira/browse/HADOOP-16824
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinayakumar B


port HADOOP-16754 to avoid Docker build failure



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16820) [thirdparty] ChangeLog and ReleaseNote are not packaged by createrelease script

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16820?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16820.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
 Assignee: Vinayakumar B
   Resolution: Fixed

Merged PR.
Thanks [~ayushtkn] for review.

> [thirdparty] ChangeLog and ReleaseNote are not packaged by createrelease 
> script
> ---
>
> Key: HADOOP-16820
> URL: https://issues.apache.org/jira/browse/HADOOP-16820
> Project: Hadoop Common
>  Issue Type: Bug
>  Components: hadoop-thirdparty
>Affects Versions: thirdparty-1.0.0
>    Reporter: Vinayakumar B
>Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> createrelease script is not packaging CHANGELOGS and RELEASENOTES during 
> generation of site package for hadoop-thirdparty module.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16821) [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded prefix instead of 'protobuf_3_7'

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16821?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16821.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
 Assignee: Vinayakumar B
   Resolution: Fixed

Committed to trunk.
Thanks [~ste...@apache.org] for review.

> [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded prefix instead of 
> 'protobuf_3_7'
> 
>
> Key: HADOOP-16821
> URL: https://issues.apache.org/jira/browse/HADOOP-16821
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: hadoop-thirdparty
>        Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> As per discussion  
> [here|https://github.com/apache/hadoop/pull/1635#issuecomment-576247014], 
> versioned package name may make upgrade of library to a non-trivial task. 
> package name needs to be updated in all usages in all modules. 
> So common package name is preferred.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16821) [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded prefix instead of 'protobuf_3_7'

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16821:
--

 Summary: [pb-upgrade] Use 'o.a.h.thirdparty.protobuf' shaded 
prefix instead of 'protobuf_3_7'
 Key: HADOOP-16821
 URL: https://issues.apache.org/jira/browse/HADOOP-16821
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


As per discussion  
[here|https://github.com/apache/hadoop/pull/1635#issuecomment-576247014], 
versioned package name may make upgrade of library to a non-trivial task. 
package name needs to be updated in all usages in all modules. 

So common package name is preferred.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16820) [thirdparty] ChangeLog and ReleaseNote are not packaged by createrelease script

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16820:
--

 Summary: [thirdparty] ChangeLog and ReleaseNote are not packaged 
by createrelease script
 Key: HADOOP-16820
 URL: https://issues.apache.org/jira/browse/HADOOP-16820
 Project: Hadoop Common
  Issue Type: Bug
  Components: hadoop-thirdparty
Affects Versions: thirdparty-1.0.0
Reporter: Vinayakumar B


createrelease script is not packaging CHANGELOGS and RELEASENOTES during 
generation of site package for hadoop-thirdparty module.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16621) [pb-upgrade] Remove Protobuf classes from signatures of Public APIs

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16621?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16621.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
 Release Note: 
Following APIs have been removed from Token.java to avoid protobuf classes in 
signature.
1.   o.a.h.security.token.Token(TokenProto tokenPB)
2.   o.a.h.security.token.Token.toTokenProto()
   Resolution: Fixed

Merged PR to trunk.
Thanks [~ste...@apache.org] and [~ayushtkn] for reviews. 

> [pb-upgrade] Remove Protobuf classes from signatures of Public APIs
> ---
>
> Key: HADOOP-16621
> URL: https://issues.apache.org/jira/browse/HADOOP-16621
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: common
>Affects Versions: 3.3.0
>Reporter: Steve Loughran
>    Assignee: Vinayakumar B
>Priority: Critical
> Fix For: 3.3.0
>
>
> the move to protobuf 3.x stops spark building because Token has a method 
> which returns a protobuf, and now its returning some v3 types.
> if we want to isolate downstream code from protobuf changes, we need to move 
> that marshalling method from token and put in a helper class.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

PR has been merged.

Thanks everyone for discussions.

-Vinay


On Thu, Jan 9, 2020 at 4:47 PM Ayush Saxena  wrote:

> Hi All,
> FYI :
> We will be going ahead with the present approach, will merge by tomorrow
> EOD. Considering no one has objections.
> Thanx Everyone!!!
>
> -Ayush
>
> > On 07-Jan-2020, at 9:22 PM, Brahma Reddy Battula 
> wrote:
> >
> > Hi Sree vaddi,Owen,stack,Duo Zhang,
> >
> > We can move forward based on your comments, just waiting for your
> > reply.Hope all of your comments answered..(unification we can think
> > parallel thread as Vinay mentioned).
> >
> >
> >
> > On Mon, 6 Jan 2020 at 6:21 PM, Vinayakumar B 
> > wrote:
> >
> >> Hi Sree,
> >>
> >>> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> >> Project ? Or as a TLP ?
> >>> Or as a new project definition ?
> >> As already mentioned by Ayush, this will be a subproject of Hadoop.
> >> Releases will be voted by Hadoop PMC as per ASF process.
> >>
> >>
> >>> The effort to streamline and put in an accepted standard for the
> >> dependencies that require shading,
> >>> seems beyond the siloed efforts of hadoop, hbase, etc
> >>
> >>> I propose, we bring all the decision makers from all these artifacts in
> >> one room and decide best course of action.
> >>> I am looking at, no projects should ever had to shade any artifacts
> >> except as an absolute necessary alternative.
> >>
> >> This is the ideal proposal for any project. But unfortunately some
> projects
> >> takes their own course based on need.
> >>
> >> In the current case of protobuf in Hadoop,
> >>Protobuf upgrade from 2.5.0 (which is already EOL) was not taken up
> to
> >> avoid downstream failures. Since Hadoop is a platform, its dependencies
> >> will get added to downstream projects' classpath. So any change in
> Hadoop's
> >> dependencies will directly affect downstreams. Hadoop strictly follows
> >> backward compatibility as far as possible.
> >>Though protobuf provides wire compatibility b/w versions, it doesnt
> >> provide compatibility for generated sources.
> >>Now, to support ARM protobuf upgrade is mandatory. Using shading
> >> technique, In Hadoop internally can upgrade to shaded protobuf 3.x and
> >> still have 2.5.0 protobuf (deprecated) for downstreams.
> >>
> >> This shading is necessary to have both versions of protobuf supported.
> >> (2.5.0 (non-shaded) for downstream's classpath and 3.x (shaded) for
> >> hadoop's internal usage).
> >> And this entire work to be done before 3.3.0 release.
> >>
> >> So, though its ideal to make a common approach for all projects, I
> suggest
> >> for Hadoop we can go ahead as per current approach.
> >> We can also start the parallel effort to address these problems in a
> >> separate discussion/proposal. Once the solution is available we can
> revisit
> >> and adopt new solution accordingly in all such projects (ex: HBase,
> Hadoop,
> >> Ratis).
> >>
> >> -Vinay
> >>
> >>> On Mon, Jan 6, 2020 at 12:39 AM Ayush Saxena 
> wrote:
> >>>
> >>> Hey Sree
> >>>
> >>>> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> >>>> Project ? Or as a TLP ?
> >>>> Or as a new project definition ?
> >>>>
> >>> A sub project of Apache Hadoop, having its own independent release
> >> cycles.
> >>> May be you can put this into the same column as ozone or as
> >>> submarine(couple of months ago).
> >>>
> >>> Unifying for all, seems interesting but each project is independent and
> >> has
> >>> its own limitations and way of thinking, I don't think it would be an
> >> easy
> >>> task to bring all on the same table and get them agree to a common
> stuff.
> >>>
> >>> I guess this has been into discussion since quite long, and there
> hasn't
> >>> been any other alternative suggested. Still we can hold up for a week,
> if
> >>> someone comes up with a better solution, else we can continue in the
> >>> present direction.
> >>>
> >>> -Ayush
> >>>
> >>>
> >>>
> >>> On Sun, 5 Jan 2020 at 05:03, Sree Vaddi  >> .invalid>
> >>> wrote:
> >>>
> >>

[jira] [Resolved] (HADOOP-16595) [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16595.

Fix Version/s: thirdparty-1.0.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged  PR.
Thanks everyone.

> [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf
> --
>
> Key: HADOOP-16595
> URL: https://issues.apache.org/jira/browse/HADOOP-16595
> Project: Hadoop Common
>  Issue Type: Sub-task
>  Components: hadoop-thirdparty
>    Reporter: Vinayakumar B
>        Assignee: Vinayakumar B
>Priority: Major
> Fix For: thirdparty-1.0.0
>
>
> Create a separate repo "hadoop-thirdparty" to have shaded dependencies.
> starting with protobuf-java:3.7.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16797) Add dockerfile for ARM builds

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16797?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16797.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged PR to trunk. 
Thanks [~ayushtkn] and [~aajisaka] for reviews.

> Add dockerfile for ARM builds
> -
>
> Key: HADOOP-16797
> URL: https://issues.apache.org/jira/browse/HADOOP-16797
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: 3.3.0
>
>
> Similar to x86 docker image in {{dev-support/docker/Dockerfile}},
> add one more Dockerfile to support aarch64 builds.
> And support all scripts (createrelease, start-build-env.sh, etc ) to make use 
> of it in ARM platform.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16797) Add dockerfile for ARM builds

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16797:
--

 Summary: Add dockerfile for ARM builds
 Key: HADOOP-16797
 URL: https://issues.apache.org/jira/browse/HADOOP-16797
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Similar to x86 docker image in {{dev-support/docker/Dockerfile}},
add one more Dockerfile to support aarch64 builds.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[NOTICE] Removal of protobuf classes from Hadoop Token's public APIs' signature

[Vinayakumar B]

Hi All,

   This mail is to notify about the Removal of following public APIs from
Hadoop Common.

 ClassName: org.apache.hadoop.security.token.Token
 APIs:
 public Token(TokenProto tokenPB);
 public TokenProto toTokenProto();

   Reason: These APIs are having Generated protobuf classes in the
signature. Right now due to protobuf upgrade in trunk (soon to be 3.3.0
release) these APIs are breaking the downstream builds, even though
downstreams dont use these APIs (just Loading Token class). Downstreams are
still referencing having older version (2.5.0) of protobuf, hence build is
being broken.

These APIs were added for the internal purpose(HADOOP-12563), to
support serializing tokens using protobuf in UGI Credentials.
Same purpose can be achieved using the Helper classes without introducing
protobuf classes in API signatures.

Token.java is marked as Evolving, so I believe APIs can be changed whenever
absolute necessary.

Jira https://issues.apache.org/jira/browse/HADOOP-16621 has been
reported to solve downstream build failure.

So since this API was added for internal purpose easy approach to solve
this is to remove APIs and use helper classes. Otherwise, as mentioned in
HADOOP-16621, workaround will add unnecessary codes to be maintained.

If anyone using these APIs outside hadoop project accidentally, please
reply to this mail immediately.

If no objection by next week, will go ahead with removal of above said APIs
in HADOOP-16621.

-Vinay

[jira] [Resolved] (HADOOP-16358) Add an ARM CI for Hadoop

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16358.

Fix Version/s: 3.3.0
   Resolution: Fixed

A Jenkins Job has been created to run nightly tests on aarch64

[https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-qbt-linux-ARM-trunk/]

> Add an ARM CI for Hadoop
> 
>
> Key: HADOOP-16358
> URL: https://issues.apache.org/jira/browse/HADOOP-16358
> Project: Hadoop Common
>  Issue Type: Task
>  Components: build
>Reporter: Zhenyu Zheng
>Priority: Major
> Fix For: 3.3.0
>
>
> Now the CI of Hadoop is handled by jenkins. While the tests are running under 
> x86 ARCH, the ARM arch has not being considered. This leads an problem that 
> we don't have a way to test every pull request that if it'll break the Hadoop 
> deployment on ARM or not.
> We should add a CI system that support ARM arch. Using it, Hadoop can 
> officially support arm release in the future. Here I'd like to introduce 
> OpenLab to the community. [OpenLab|https://openlabtesting.org/] is a open 
> source CI system that can test any open source software on either x86 or arm 
> ARCH, it's mainly used by github projects. Now some 
> [projects|https://github.com/theopenlab/openlab-zuul-jobs/blob/master/zuul.d/jobs.yaml]
>  has integrated it already. Such as containerd (a graduated CNCF project, the 
> arm build will be triggerd in every PR, 
> [https://github.com/containerd/containerd/pulls]), terraform and so on.
> OpenLab uses the open source CI software [Zuul 
> |https://github.com/openstack-infra/zuul] for CI system. Zuul is used by 
> OpenStack community as well. integrating with OpneLab is quite easy using its 
> github app. All config info is open source as well.
> If apache Hadoop community has interested with it, we can help for the 
> integration.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi Sree,

> apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
Project ? Or as a TLP ?
> Or as a new project definition ?
As already mentioned by Ayush, this will be a subproject of Hadoop.
Releases will be voted by Hadoop PMC as per ASF process.


> The effort to streamline and put in an accepted standard for the
dependencies that require shading,
> seems beyond the siloed efforts of hadoop, hbase, etc

>I propose, we bring all the decision makers from all these artifacts in
one room and decide best course of action.
> I am looking at, no projects should ever had to shade any artifacts
except as an absolute necessary alternative.

This is the ideal proposal for any project. But unfortunately some projects
takes their own course based on need.

In the current case of protobuf in Hadoop,
Protobuf upgrade from 2.5.0 (which is already EOL) was not taken up to
avoid downstream failures. Since Hadoop is a platform, its dependencies
will get added to downstream projects' classpath. So any change in Hadoop's
dependencies will directly affect downstreams. Hadoop strictly follows
backward compatibility as far as possible.
Though protobuf provides wire compatibility b/w versions, it doesnt
provide compatibility for generated sources.
Now, to support ARM protobuf upgrade is mandatory. Using shading
technique, In Hadoop internally can upgrade to shaded protobuf 3.x and
still have 2.5.0 protobuf (deprecated) for downstreams.

This shading is necessary to have both versions of protobuf supported.
(2.5.0 (non-shaded) for downstream's classpath and 3.x (shaded) for
hadoop's internal usage).
And this entire work to be done before 3.3.0 release.

So, though its ideal to make a common approach for all projects, I suggest
for Hadoop we can go ahead as per current approach.
We can also start the parallel effort to address these problems in a
separate discussion/proposal. Once the solution is available we can revisit
and adopt new solution accordingly in all such projects (ex: HBase, Hadoop,
Ratis).

-Vinay

On Mon, Jan 6, 2020 at 12:39 AM Ayush Saxena  wrote:

> Hey Sree
>
> > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> > Project ? Or as a TLP ?
> > Or as a new project definition ?
> >
> A sub project of Apache Hadoop, having its own independent release cycles.
> May be you can put this into the same column as ozone or as
> submarine(couple of months ago).
>
> Unifying for all, seems interesting but each project is independent and has
> its own limitations and way of thinking, I don't think it would be an easy
> task to bring all on the same table and get them agree to a common stuff.
>
> I guess this has been into discussion since quite long, and there hasn't
> been any other alternative suggested. Still we can hold up for a week, if
> someone comes up with a better solution, else we can continue in the
> present direction.
>
> -Ayush
>
>
>
> On Sun, 5 Jan 2020 at 05:03, Sree Vaddi 
> wrote:
>
> > apache/hadoop-thirdparty, How would it fit into ASF ? As an Incubating
> > Project ? Or as a TLP ?
> > Or as a new project definition ?
> >
> > The effort to streamline and put in an accepted standard for the
> > dependencies that require shading,seems beyond the siloed efforts of
> > hadoop, hbase, etc
> >
> > I propose, we bring all the decision makers from all these artifacts in
> > one room and decide best course of action.I am looking at, no projects
> > should ever had to shade any artifacts except as an absolute necessary
> > alternative.
> >
> >
> > Thank you./Sree
> >
> >
> >
> > On Saturday, January 4, 2020, 7:49:18 AM PST, Vinayakumar B <
> > vinayakum...@apache.org> wrote:
> >
> >  Hi,
> > Sorry for the late reply,.
> > >>> To be exact, how can we better use the thirdparty repo? Looking at
> > HBase as an example, it looks like everything that are known to break a
> lot
> > after an update get shaded into the hbase-thirdparty artifact: guava,
> > netty, ... etc.
> > Is it the purpose to isolate these naughty dependencies?
> > Yes, shading is to isolate these naughty dependencies from downstream
> > classpath and have independent control on these upgrades without breaking
> > downstreams.
> >
> > First PR https://github.com/apache/hadoop-thirdparty/pull/1 to create
> the
> > protobuf shaded jar is ready to merge.
> >
> > Please take a look if anyone interested, will be merged may be after two
> > days if no objections.
> >
> > -Vinay
> >
> >
> > On Thu, Oct 10, 2019 at 3:30 AM Wei-Chiu Chuang 
> > wrote:
> >
> > > Hi I am late to this but I am keen to un

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi,
Sorry for the late reply,.
>>> To be exact, how can we better use the thirdparty repo? Looking at
HBase as an example, it looks like everything that are known to break a lot
after an update get shaded into the hbase-thirdparty artifact: guava,
netty, ... etc.
Is it the purpose to isolate these naughty dependencies?
Yes, shading is to isolate these naughty dependencies from downstream
classpath and have independent control on these upgrades without breaking
downstreams.

First PR https://github.com/apache/hadoop-thirdparty/pull/1 to create the
protobuf shaded jar is ready to merge.

Please take a look if anyone interested, will be merged may be after two
days if no objections.

-Vinay


On Thu, Oct 10, 2019 at 3:30 AM Wei-Chiu Chuang  wrote:

> Hi I am late to this but I am keen to understand more.
>
> To be exact, how can we better use the thirdparty repo? Looking at HBase
> as an example, it looks like everything that are known to break a lot after
> an update get shaded into the hbase-thirdparty artifact: guava, netty, ...
> etc.
> Is it the purpose to isolate these naughty dependencies?
>
> On Wed, Oct 9, 2019 at 12:38 PM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>> I have updated the PR as per @Owen O'Malley 
>> 's suggestions.
>>
>> i. Renamed the module to 'hadoop-shaded-protobuf37'
>> ii. Kept the shaded package to 'o.a.h.thirdparty.protobuf37'
>>
>> Please review!!
>>
>> Thanks,
>> -Vinay
>>
>>
>> On Sat, Sep 28, 2019 at 10:29 AM 张铎(Duo Zhang) 
>> wrote:
>>
>> > For HBase we have a separated repo for hbase-thirdparty
>> >
>> > https://github.com/apache/hbase-thirdparty
>> >
>> > We will publish the artifacts to nexus so we do not need to include
>> > binaries in our git repo, just add a dependency in the pom.
>> >
>> >
>> >
>> https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-protobuf
>> >
>> >
>> > And it has its own release cycles, only when there are special
>> requirements
>> > or we want to upgrade some of the dependencies. This is the vote thread
>> for
>> > the newest release, where we want to provide a shaded gson for jdk7.
>> >
>> >
>> >
>> https://lists.apache.org/thread.html/f12c589baabbc79c7fb2843422d4590bea982cd102e2bd9d21e9884b@%3Cdev.hbase.apache.org%3E
>> >
>> >
>> > Thanks.
>> >
>> > Vinayakumar B  于2019年9月28日周六 上午1:28写道：
>> >
>> > > Please find replies inline.
>> > >
>> > > -Vinay
>> > >
>> > > On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley <
>> owen.omal...@gmail.com>
>> > > wrote:
>> > >
>> > > > I'm very unhappy with this direction. In particular, I don't think
>> git
>> > is
>> > > > a good place for distribution of binary artifacts. Furthermore, the
>> PMC
>> > > > shouldn't be releasing anything without a release vote.
>> > > >
>> > > >
>> > > Proposed solution doesnt release any binaries in git. Its actually a
>> > > complete sub-project which follows entire release process, including
>> VOTE
>> > > in public. I have mentioned already that release process is similar to
>> > > hadoop.
>> > > To be specific, using the (almost) same script used in hadoop to
>> generate
>> > > artifacts, sign and deploy to staging repository. Please let me know
>> If I
>> > > am conveying anything wrong.
>> > >
>> > >
>> > > > I'd propose that we make a third party module that contains the
>> > *source*
>> > > > of the pom files to build the relocated jars. This should
>> absolutely be
>> > > > treated as a last resort for the mostly Google projects that
>> regularly
>> > > > break binary compatibility (eg. Protobuf & Guava).
>> > > >
>> > > >
>> > > Same has been implemented in the PR
>> > > https://github.com/apache/hadoop-thirdparty/pull/1. Please check and
>> let
>> > > me
>> > > know If I misunderstood. Yes, this is the last option we have AFAIK.
>> > >
>> > >
>> > > > In terms of naming, I'd propose something like:
>> > > >
>> > > > org.apache.hadoop.thirdparty.protobuf2_5
>> > > > org.apache.hadoop.thirdparty.guava28
>> > > >
>> > > > In particular, I think we absolutely need to include the ver

[jira] [Created] (HADOOP-16774) TestDiskChecker and TestReadWriteDiskValidator fails when run with -Pparallel-tests

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16774:
--

 Summary: TestDiskChecker and TestReadWriteDiskValidator fails when 
run with -Pparallel-tests
 Key: HADOOP-16774
 URL: https://issues.apache.org/jira/browse/HADOOP-16774
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinayakumar B


{noformat}
$  mvn test -Pparallel-tests -Dtest=TestReadWriteDiskValidator,TestDiskChecker 
-Pnative
 {noformat}
{noformat}
[INFO] Results:
[INFO] 
[ERROR] Errors: 
[ERROR]   
TestDiskChecker.testCheckDir_normal:111->_checkDirs:158->createTempDir:153 » 
NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_normal_local:180->checkDirs:205->createTempDir:153 
» NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notDir:116->_checkDirs:158->createTempFile:142 » 
NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notDir_local:185->checkDirs:205->createTempFile:142
 » NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notListable:131->_checkDirs:158->createTempDir:153 
» NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notListable_local:200->checkDirs:205->createTempDir:153
 » NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notReadable:121->_checkDirs:158->createTempDir:153 
» NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notReadable_local:190->checkDirs:205->createTempDir:153
 » NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notWritable:126->_checkDirs:158->createTempDir:153 
» NoSuchFile
[ERROR]   
TestDiskChecker.testCheckDir_notWritable_local:195->checkDirs:205->createTempDir:153
 » NoSuchFile
[ERROR]   TestReadWriteDiskValidator.testCheckFailures:114 » NoSuchFile 
/usr1/code/hadoo...
[ERROR]   TestReadWriteDiskValidator.testReadWriteDiskValidator:62 » DiskError 
Disk Chec...
[INFO] 
[ERROR] Tests run: 16, Failures: 0, Errors: 12, Skipped: 0

{noformat}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Some updates for Hadoop on ARM and next steps

[Vinayakumar B]

Also note that.. Nothing gets deployed back to maven central repo from this
job. So no interference with other jobs and nodes as well.

-Vinay

On Thu, 28 Nov 2019, 10:55 pm Vinayakumar B, 
wrote:

> Hi all,
>
> As a starter..
> Created a simple mvn based job (not yetus and docker as current qbt on
> trunk on x86) in
>
> https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/7/console
>
> Right now using the manual tools installed and the previous workarounds
> mentioned related to various third-party dependencies related ARM
> architecture. This will be triggered daily once automatically.
>
> Going forward we can make sure that yetus with docker works fine in this
> node as well and configure similar to x86 qbt run.
>
> -Vinay
>
> On Thu, 28 Nov 2019, 7:30 am Zhenyu Zheng, 
> wrote:
>
>> Thanks for the reply Chris, And really appriaciated about all the things
>> you have done to made our node work. I'm sending this ML to send out info
>> about the node is ready. And hope someone from Hadoop project could help
>> us
>> set some new jobs/builds, I totally understand your role and opinion, I'm
>> not asking you to add jobs for Hadoop, I'm just trying to make clear about
>> what we are looking for.
>>
>> As Chris mentioned in previous email interactions, there are 3 kinds of CI
>> nodes available in the CI system, the 1st and 2nd type have to use the
>> current infra management tools to install tools and software required for
>> the system, which the infra management tool is currently not ready for ARM
>> platform. And the 3rd kind of CI nodes is what we are ready now - we
>> manually install all the required tools and software and maintain them
>> according to infra's other nodes.  And we will try to make the infra
>> management tools usable for ARM platform to make the nodes type2 or type1.
>>
>> As for jobs/builds, seems a periodic job/builds like
>> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
>> seems
>> to be the most suitable for what we are looking for the current step.
>> Since
>> we are still having some errors and failures(15 erros in Hadoop-YARN, 4
>> Failures and 2 Errors in Hadoop-HDFS, 23 Failures in Hadoop-MapReduce,
>> which is a quite small number comparing to the total number of tests, and
>> failures/errors in same sub-project seems to be caused by same problem)
>> that our team will work on, so we want to propose 4 different jobs similar
>> to the mechanism used in Hadoop-trunk-Commit, a SCM triggered periodic job
>> test out building and UT for each sub-project:
>> Hadoop-YARN-trunk-Commit-Aarch64, Hadoop-HDFS-trunk-Commit-Aarch64,
>> Hadoop-MapReducer-trunk-Commit-Aarch64 and Hadoop-Common-trunk-Commit
>> Aarch64 to be more tracked for each project. We can also start one by one,
>> of cause.
>>
>> Hope this could clear all the misunderstanding.
>>
>> BR,
>>
>> On Wed, Nov 27, 2019 at 10:28 PM Chris Thistlethwaite 
>> wrote:
>>
>> > If anyone would like to follow along in JIRA, here's the ticket
>> > https://issues.apache.org/jira/browse/INFRA-19369. I've been updating
>> > that ticket with any issues. arm-poc has been moved to a node in
>> Singapore
>> > and will need to be tested again with builds.
>> >
>> > I'm going to mention again that someone from Hadoop should be changing
>> > these builds in order to run against arm-poc. In my reply below, I
>> thought
>> > that the project knew about the ARM nodes and was involved with setting
>> up
>> > new builds, which is why I said I'd be willing to make simple changes
>> for
>> > testing. However I don't want to change things without the knowledge of
>> the
>> > project. The builds themselves are created by the project, not Infra,
>> which
>> > means I have no idea which build should run against ARM vs any other
>> CPU.
>> >
>> > -Chris T.
>> > #asfinfra
>> >
>> > On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
>> >
>> > In order to run builds against arm-poc, someone (me included) will need
>> to
>> > change a build config to only use that label. The node itself isn't
>> fully
>> > built out like our other ASF nodes, due to the fact that it's ARM and we
>> > don't have all the packaged tools built for that architecture, it will
>> > likely take some time to fix issues.
>> >
>> >
>> > -Chris T.
>> > #asfinfra
>> >
>> > On 11/22/19 3:46 AM, bo zhaobo wrote:
>> >
>> > Thanks. That 

Re: Some updates for Hadoop on ARM and next steps

[Vinayakumar B]

Hi all,

As a starter..
Created a simple mvn based job (not yetus and docker as current qbt on
trunk on x86) in

https://builds.apache.org/job/Hadoop-qbt-linux-ARM-trunk/7/console

Right now using the manual tools installed and the previous workarounds
mentioned related to various third-party dependencies related ARM
architecture. This will be triggered daily once automatically.

Going forward we can make sure that yetus with docker works fine in this
node as well and configure similar to x86 qbt run.

-Vinay

On Thu, 28 Nov 2019, 7:30 am Zhenyu Zheng, 
wrote:

> Thanks for the reply Chris, And really appriaciated about all the things
> you have done to made our node work. I'm sending this ML to send out info
> about the node is ready. And hope someone from Hadoop project could help us
> set some new jobs/builds, I totally understand your role and opinion, I'm
> not asking you to add jobs for Hadoop, I'm just trying to make clear about
> what we are looking for.
>
> As Chris mentioned in previous email interactions, there are 3 kinds of CI
> nodes available in the CI system, the 1st and 2nd type have to use the
> current infra management tools to install tools and software required for
> the system, which the infra management tool is currently not ready for ARM
> platform. And the 3rd kind of CI nodes is what we are ready now - we
> manually install all the required tools and software and maintain them
> according to infra's other nodes.  And we will try to make the infra
> management tools usable for ARM platform to make the nodes type2 or type1.
>
> As for jobs/builds, seems a periodic job/builds like
> https://builds.apache.org/view/H-L/view/Hadoop/job/Hadoop-trunk-Commit/
> seems
> to be the most suitable for what we are looking for the current step. Since
> we are still having some errors and failures(15 erros in Hadoop-YARN, 4
> Failures and 2 Errors in Hadoop-HDFS, 23 Failures in Hadoop-MapReduce,
> which is a quite small number comparing to the total number of tests, and
> failures/errors in same sub-project seems to be caused by same problem)
> that our team will work on, so we want to propose 4 different jobs similar
> to the mechanism used in Hadoop-trunk-Commit, a SCM triggered periodic job
> test out building and UT for each sub-project:
> Hadoop-YARN-trunk-Commit-Aarch64, Hadoop-HDFS-trunk-Commit-Aarch64,
> Hadoop-MapReducer-trunk-Commit-Aarch64 and Hadoop-Common-trunk-Commit
> Aarch64 to be more tracked for each project. We can also start one by one,
> of cause.
>
> Hope this could clear all the misunderstanding.
>
> BR,
>
> On Wed, Nov 27, 2019 at 10:28 PM Chris Thistlethwaite 
> wrote:
>
> > If anyone would like to follow along in JIRA, here's the ticket
> > https://issues.apache.org/jira/browse/INFRA-19369. I've been updating
> > that ticket with any issues. arm-poc has been moved to a node in
> Singapore
> > and will need to be tested again with builds.
> >
> > I'm going to mention again that someone from Hadoop should be changing
> > these builds in order to run against arm-poc. In my reply below, I
> thought
> > that the project knew about the ARM nodes and was involved with setting
> up
> > new builds, which is why I said I'd be willing to make simple changes for
> > testing. However I don't want to change things without the knowledge of
> the
> > project. The builds themselves are created by the project, not Infra,
> which
> > means I have no idea which build should run against ARM vs any other CPU.
> >
> > -Chris T.
> > #asfinfra
> >
> > On 11/22/19 9:28 AM, Chris Thistlethwaite wrote:
> >
> > In order to run builds against arm-poc, someone (me included) will need
> to
> > change a build config to only use that label. The node itself isn't fully
> > built out like our other ASF nodes, due to the fact that it's ARM and we
> > don't have all the packaged tools built for that architecture, it will
> > likely take some time to fix issues.
> >
> >
> > -Chris T.
> > #asfinfra
> >
> > On 11/22/19 3:46 AM, bo zhaobo wrote:
> >
> > Thanks. That would be great if a project can use the ARM test worker to
> do
> > the specific testing on ARM.
> >
> > Also I think it's better to make @Chris Thistlethwaite <
> chr...@apache.org> know
> > this email.  Could you please give some kind advices? Thank you.
> >
> > BR
> >
> > ZhaoBo
> >
> >
> >
> > [image: Mailtrack]
> > <
> https://mailtrack.io?utm_source=gmail_medium=signature_campaign=signaturevirality5;>
> Sender
> > notified by
> > Mailtrack
> > <
> https://mailtrack.io?utm_source=gmail_medium=signature_campaign=signaturevirality5;>
> 19/11/22
> > 下午04:42:30
> >
> > Zhenyu Zheng  于2019年11月22日周五 下午4:32写道：
> >
> >> Hi Hadoop,
> >>
> >>
> >>
> >> First off, I want to thanks to Wei-Chiu for having me on the last week's
> >> Hadoop community sync to introduce our ideas of ARM support on Hadoop.
> And
> >> also for all the attendees for listening and providing suggestions.
> >>
> >>
> >>
> >> I want to provide some update on the status:
> >>
> >> 1. 

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi All,

I have updated the PR as per @Owen O'Malley 
's suggestions.

i. Renamed the module to 'hadoop-shaded-protobuf37'
ii. Kept the shaded package to 'o.a.h.thirdparty.protobuf37'

Please review!!

Thanks,
-Vinay


On Sat, Sep 28, 2019 at 10:29 AM 张铎(Duo Zhang) 
wrote:

> For HBase we have a separated repo for hbase-thirdparty
>
> https://github.com/apache/hbase-thirdparty
>
> We will publish the artifacts to nexus so we do not need to include
> binaries in our git repo, just add a dependency in the pom.
>
>
> https://mvnrepository.com/artifact/org.apache.hbase.thirdparty/hbase-shaded-protobuf
>
>
> And it has its own release cycles, only when there are special requirements
> or we want to upgrade some of the dependencies. This is the vote thread for
> the newest release, where we want to provide a shaded gson for jdk7.
>
>
> https://lists.apache.org/thread.html/f12c589baabbc79c7fb2843422d4590bea982cd102e2bd9d21e9884b@%3Cdev.hbase.apache.org%3E
>
>
> Thanks.
>
> Vinayakumar B  于2019年9月28日周六 上午1:28写道：
>
> > Please find replies inline.
> >
> > -Vinay
> >
> > On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley 
> > wrote:
> >
> > > I'm very unhappy with this direction. In particular, I don't think git
> is
> > > a good place for distribution of binary artifacts. Furthermore, the PMC
> > > shouldn't be releasing anything without a release vote.
> > >
> > >
> > Proposed solution doesnt release any binaries in git. Its actually a
> > complete sub-project which follows entire release process, including VOTE
> > in public. I have mentioned already that release process is similar to
> > hadoop.
> > To be specific, using the (almost) same script used in hadoop to generate
> > artifacts, sign and deploy to staging repository. Please let me know If I
> > am conveying anything wrong.
> >
> >
> > > I'd propose that we make a third party module that contains the
> *source*
> > > of the pom files to build the relocated jars. This should absolutely be
> > > treated as a last resort for the mostly Google projects that regularly
> > > break binary compatibility (eg. Protobuf & Guava).
> > >
> > >
> > Same has been implemented in the PR
> > https://github.com/apache/hadoop-thirdparty/pull/1. Please check and let
> > me
> > know If I misunderstood. Yes, this is the last option we have AFAIK.
> >
> >
> > > In terms of naming, I'd propose something like:
> > >
> > > org.apache.hadoop.thirdparty.protobuf2_5
> > > org.apache.hadoop.thirdparty.guava28
> > >
> > > In particular, I think we absolutely need to include the version of the
> > > underlying project. On the other hand, since we should not be shading
> > > *everything* we can drop the leading com.google.
> > >
> > >
> > IMO, This naming convention is easy for identifying the underlying
> project,
> > but  it will be difficult to maintain going forward if underlying project
> > versions changes. Since thirdparty module have its own releases, each of
> > those release can be mapped to specific version of underlying project.
> Even
> > the binary artifact can include a MANIFEST with underlying project
> details
> > as per Steve's suggestion on HADOOP-13363.
> > That said, if you still prefer to have project number in artifact id, it
> > can be done.
> >
> > The Hadoop project can make releases of  the thirdparty module:
> > >
> > > 
> > >   org.apache.hadoop
> > >   hadoop-thirdparty-protobuf25
> > >   1.0
> > > 
> > >
> > >
> > Note that the version has to be the hadoop thirdparty release number,
> which
> > > is part of why you need to have the underlying version in the artifact
> > > name. These we can push to maven central as new releases from Hadoop.
> > >
> > >
> > Exactly, same has been implemented in the PR. hadoop-thirdparty module
> have
> > its own releases. But in HADOOP Jira, thirdparty versions can be
> > differentiated using prefix "thirdparty-".
> >
> > Same solution is being followed in HBase. May be people involved in HBase
> > can add some points here.
> >
> > Thoughts?
> > >
> > > .. Owen
> > >
> > > On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B  >
> > > wrote:
> > >
> > >> Hi All,
> > >>
> > >>I wanted to discuss about the separate repo for thirdparty
> > dependencies
> > >> which we need to shaded an

Re: Daily Builds Getting Aborted Due To Timeout

[Vinayakumar B]

Thanks Ayush,

I think possible other ways are improving the test runtime.

1. try "parallel-tests" profile if applicable on all modules whichever is
okay with tests running in parallel. This may reduce the total unit tests
runtime drastically. ex: I can see hadoop-hdfs-rbf takes around ~22 min.
Here parallel-tests can be implemented (Also need to run and verify that
all tests are using local filesystem path which doesnot collide with other
tests running in parallel.

2. Right now all modules of HDDS and OZone also running as part of
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/
, But
I can also see there was a QBT Job created for Ozone alone. Can we make
that work and exclude HDDS and OZone related executions from
https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1271/ ?
  In above aborted builds, it was running unit tests in HDDS/Ozone
module. That means all other modules have finished.

3. If above options are not possible or not resulting in any improvements
then we can definitely go for timeout increase.

-Vinay


On Fri, Sep 27, 2019 at 8:54 PM Ayush Saxena  wrote:

> Hi All,
> Just to bring into notice the hadoop daily builds are getting aborted due
> to timeout( Configured to be 900 Minutes).
>
> > Build timed out (after 900 minutes). Marking the build as aborted.
> > Build was aborted
> > [CHECKSTYLE] Skipping publisher since build result is ABORTED
> > [FINDBUGS] Skipping publisher since build result is ABORTED
> >
> > Recording test results
> > No emails were triggered.
> > Finished: ABORTED
> >
> >
> Reference :
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/1271/
>
> Checked with the infra team, The only resolution told was to increase the
> configured time of 900 mins or make the build take less time.
>
> Someone with access to change the config can probably increase the time.
> (Probably people in https://whimsy.apache.org/roster/group/hudson-jobadmin
> have access)
> *Link To Change Configured Time* :
> https://builds.apache.org/job/hadoop-qbt-trunk-java8-linux-x86/configure
> <
> https://slack-redir.net/link?url=https%3A%2F%2Fbuilds.apache.org%2Fjob%2Fhadoop-qbt-trunk-java8-linux-x86%2Fconfigure=3
> >
>
>
> -Ayush
>

Re: [DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Please find replies inline.

-Vinay

On Fri, Sep 27, 2019 at 10:21 PM Owen O'Malley 
wrote:

> I'm very unhappy with this direction. In particular, I don't think git is
> a good place for distribution of binary artifacts. Furthermore, the PMC
> shouldn't be releasing anything without a release vote.
>
>
Proposed solution doesnt release any binaries in git. Its actually a
complete sub-project which follows entire release process, including VOTE
in public. I have mentioned already that release process is similar to
hadoop.
To be specific, using the (almost) same script used in hadoop to generate
artifacts, sign and deploy to staging repository. Please let me know If I
am conveying anything wrong.


> I'd propose that we make a third party module that contains the *source*
> of the pom files to build the relocated jars. This should absolutely be
> treated as a last resort for the mostly Google projects that regularly
> break binary compatibility (eg. Protobuf & Guava).
>
>
Same has been implemented in the PR
https://github.com/apache/hadoop-thirdparty/pull/1. Please check and let me
know If I misunderstood. Yes, this is the last option we have AFAIK.


> In terms of naming, I'd propose something like:
>
> org.apache.hadoop.thirdparty.protobuf2_5
> org.apache.hadoop.thirdparty.guava28
>
> In particular, I think we absolutely need to include the version of the
> underlying project. On the other hand, since we should not be shading
> *everything* we can drop the leading com.google.
>
>
IMO, This naming convention is easy for identifying the underlying project,
but  it will be difficult to maintain going forward if underlying project
versions changes. Since thirdparty module have its own releases, each of
those release can be mapped to specific version of underlying project. Even
the binary artifact can include a MANIFEST with underlying project details
as per Steve's suggestion on HADOOP-13363.
That said, if you still prefer to have project number in artifact id, it
can be done.

The Hadoop project can make releases of  the thirdparty module:
>
> 
>   org.apache.hadoop
>   hadoop-thirdparty-protobuf25
>   1.0
> 
>
>
Note that the version has to be the hadoop thirdparty release number, which
> is part of why you need to have the underlying version in the artifact
> name. These we can push to maven central as new releases from Hadoop.
>
>
Exactly, same has been implemented in the PR. hadoop-thirdparty module have
its own releases. But in HADOOP Jira, thirdparty versions can be
differentiated using prefix "thirdparty-".

Same solution is being followed in HBase. May be people involved in HBase
can add some points here.

Thoughts?
>
> .. Owen
>
> On Fri, Sep 27, 2019 at 8:38 AM Vinayakumar B 
> wrote:
>
>> Hi All,
>>
>>I wanted to discuss about the separate repo for thirdparty dependencies
>> which we need to shaded and include in Hadoop component's jars.
>>
>>Apologies for the big text ahead, but this needs clear explanation!!
>>
>>Right now most needed such dependency is protobuf. Protobuf dependency
>> was not upgraded from 2.5.0 onwards with the fear that downstream builds,
>> which depends on transitive dependency protobuf coming from hadoop's jars,
>> may fail with the upgrade. Apparently protobuf does not guarantee source
>> compatibility, though it guarantees wire compatibility between versions.
>> Because of this behavior, version upgrade may cause breakage in known and
>> unknown (private?) downstreams.
>>
>>So to tackle this, we came up the following proposal in HADOOP-13363.
>>
>>Luckily, As far as I know, no APIs, either public to user or between
>> Hadoop processes, is not directly using protobuf classes in signatures.
>> (If
>> any exist, please let us know).
>>
>>Proposal:
>>
>>
>>1. Create a artifact(s) which contains shaded dependencies. All such
>> shading/relocation will be with known prefix
>> **org.apache.hadoop.thirdparty.**.
>>2. Right now protobuf jar (ex: o.a.h.thirdparty:hadoop-shaded-protobuf)
>> to start with, all **com.google.protobuf** classes will be relocated as
>> **org.apache.hadoop.thirdparty.com.google.protobuf**.
>>3. Hadoop modules, which needs protobuf as dependency, will add this
>> shaded artifact as dependency (ex:
>> o.a.h.thirdparty:hadoop-shaded-protobuf).
>>4. All previous usages of "com.google.protobuf" will be relocated to
>> "org.apache.hadoop.thirdparty.com.google.protobuf" in the code and will be
>> committed. Please note, this replacement is One-Time directly in source
>> code, NOT during compile and package.
>>5. Once all us

[DISCUSS] About creation of Hadoop Thirdparty repository for shaded artifacts

[Vinayakumar B]

Hi All,

   I wanted to discuss about the separate repo for thirdparty dependencies
which we need to shaded and include in Hadoop component's jars.

   Apologies for the big text ahead, but this needs clear explanation!!

   Right now most needed such dependency is protobuf. Protobuf dependency
was not upgraded from 2.5.0 onwards with the fear that downstream builds,
which depends on transitive dependency protobuf coming from hadoop's jars,
may fail with the upgrade. Apparently protobuf does not guarantee source
compatibility, though it guarantees wire compatibility between versions.
Because of this behavior, version upgrade may cause breakage in known and
unknown (private?) downstreams.

   So to tackle this, we came up the following proposal in HADOOP-13363.

   Luckily, As far as I know, no APIs, either public to user or between
Hadoop processes, is not directly using protobuf classes in signatures. (If
any exist, please let us know).

   Proposal:
   

   1. Create a artifact(s) which contains shaded dependencies. All such
shading/relocation will be with known prefix
**org.apache.hadoop.thirdparty.**.
   2. Right now protobuf jar (ex: o.a.h.thirdparty:hadoop-shaded-protobuf)
to start with, all **com.google.protobuf** classes will be relocated as
**org.apache.hadoop.thirdparty.com.google.protobuf**.
   3. Hadoop modules, which needs protobuf as dependency, will add this
shaded artifact as dependency (ex: o.a.h.thirdparty:hadoop-shaded-protobuf).
   4. All previous usages of "com.google.protobuf" will be relocated to
"org.apache.hadoop.thirdparty.com.google.protobuf" in the code and will be
committed. Please note, this replacement is One-Time directly in source
code, NOT during compile and package.
   5. Once all usages of "com.google.protobuf" is relocated, then hadoop
dont care about which version of original  "protobuf-java" is in dependency.
   6. Just keep "protobuf-java:2.5.0" in dependency tree not to break the
downstreams. But hadoop will be originally using the latest protobuf
present in "o.a.h.thirdparty:hadoop-shaded-protobuf".

   7. Coming back to separate repo, Following are most appropriate reasons
of keeping shaded dependency artifact in separate repo instead of submodule.

  7a. These artifacts need not be built all the time. It needs to be
built only when there is a change in the dependency version or the build
process.
  7b. If added as "submodule in Hadoop repo", maven-shade-plugin:shade
will execute only in package phase. That means, "mvn compile" or "mvn
test-compile" will not be failed as this artifact will not have relocated
classes, instead it will have original classes, resulting in compilation
failure. Workaround, build thirdparty submodule first and exclude
"thirdparty" submodule in other executions. This will be a complex process
compared to keeping in a separate repo.

  7c. Separate repo, will be a subproject of Hadoop, using the same
HADOOP jira project, with different versioning prefixed with "thirdparty-"
(ex: thirdparty-1.0.0).
  7d. Separate will have same release process as Hadoop.


HADOOP-13363 (https://issues.apache.org/jira/browse/HADOOP-13363) is an
umbrella jira tracking the changes to protobuf upgrade.

PR (https://github.com/apache/hadoop-thirdparty/pull/1) has been raised
for separate repo creation in (HADOOP-16595 (
https://issues.apache.org/jira/browse/HADOOP-16595)

Please provide your inputs for the proposal and review the PR to
proceed with the proposal.


   -Thanks,
Vinay

On Fri, Sep 27, 2019 at 11:54 AM Vinod Kumar Vavilapalli 
wrote:

> Moving the thread to the dev lists.
>
> Thanks
> +Vinod
>
> > On Sep 23, 2019, at 11:43 PM, Vinayakumar B 
> wrote:
> >
> > Thanks Marton,
> >
> > Current created 'hadoop-thirdparty' repo is empty right now.
> > Whether to use that repo  for shaded artifact or not will be monitored in
> > HADOOP-13363 umbrella jira. Please feel free to join the discussion.
> >
> > There is no existing codebase is being moved out of hadoop repo. So I
> think
> > right now we are good to go.
> >
> > -Vinay
> >
> > On Mon, Sep 23, 2019 at 11:38 PM Marton Elek  wrote:
> >
> >>
> >> I am not sure if it's defined when is a vote required.
> >>
> >> https://www.apache.org/foundation/voting.html
> >>
> >> Personally I think it's a big enough change to send a notification to
> the
> >> dev lists with a 'lazy consensus'  closure
> >>
> >> Marton
> >>
> >> On 2019/09/23 17:46:37, Vinayakumar B  wrote:
> >>> Hi,
> >>>
> >>> As discussed in HADOOP-13363, protobuf 3.x jar (and may be more in
> >> future)
> >>> will be kept as

[jira] [Created] (HADOOP-16596) [pb-upgrade] Use shaded protobuf classes from hadoop-thirdparty dependency

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16596:
--

 Summary: [pb-upgrade] Use shaded protobuf classes from 
hadoop-thirdparty dependency
 Key: HADOOP-16596
 URL: https://issues.apache.org/jira/browse/HADOOP-16596
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Use the shaded protobuf classes from "hadoop-thirdparty" in hadoop codebase.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16595) [pb-upgrade] Create hadoop-thirdparty artifact to have shaded protobuf

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16595:
--

 Summary: [pb-upgrade] Create hadoop-thirdparty artifact to have 
shaded protobuf
 Key: HADOOP-16595
 URL: https://issues.apache.org/jira/browse/HADOOP-16595
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: hadoop-thirdparty
Reporter: Vinayakumar B


Create a separate repo "hadoop-thirdparty" to have shaded dependencies.

starting with protobuf-java:3.7.1



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16561) [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16561.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to trunk.

> [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes
> --
>
> Key: HADOOP-16561
> URL: https://issues.apache.org/jira/browse/HADOOP-16561
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Vinayakumar B
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.3.0
>
>
> Use "protoc-maven-plugin" to dynamically download protobuf executable to 
> generate protobuf classes from proto file



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Resolved] (HADOOP-16560) [YARN] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16560.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged already to trunk

> [YARN] use protobuf-maven-plugin to generate protobuf classes
> -
>
> Key: HADOOP-16560
> URL: https://issues.apache.org/jira/browse/HADOOP-16560
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Vinayakumar B
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.3.0
>
>
> Use "protoc-maven-plugin" to dynamically download protobuf executable to 
> generate protobuf classes from proto file



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

Thanks Steve.

Idea is not to shade all artifacts.
Instead maintain one artifact ( hadoop-thirdparty) which have all such
dependencies ( com.google.* may be), add  this artifact as dependency in
hadoop modules. Use shaded classes directly in the code of hadoop modules
instead of shading at package phase.

Hbase, ozone and ratis already following this way. The artifact (
hadoop-thirdparty) with shaded dependencies can be maintained in a separate
repo as suggested by stack on HADOOP-13363 or could be maintained as a
separate module in Hadoop repo. If maintained in separate repo, need to
build this only when there are changes related to shaded dependencies.


-Vinay

On Sun, 22 Sep 2019, 10:11 pm Steve Loughran,  wrote:

>
>
> On Sun, Sep 22, 2019 at 3:22 PM Vinayakumar B 
> wrote:
>
>>Protobuf provides Wire compatibility between releases.. but not
>> guarantees the source compatibility in generated sources. There will be a
>> problem in compatibility if anyone uses generated protobuf message outside
>> of Hadoop modules. Which ideally shouldn't be as generated sources are not
>> public APIs.
>>
>>There should not be any compatibility problems between releases in
>> terms
>> of communication provided both uses same syntax (proto2) of proto message.
>> This I have verified by communication between protobuf 2.5.0 client with
>> protobuf 3.7.1 server.
>>
>>To avoid the downstream transitive dependency classpath problem, who
>> might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes
>> and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop
>> classpath.
>>
>> Hope I have answered your question.
>>
>> -Vinay
>>
>>
> While I support the move and CP isolation, this is going to (finally)
> force us to make shaded versions of all artifacts which we publish with the
> intent of them being loaded on the classpath of other applications
>

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

   Protobuf provides Wire compatibility between releases.. but not
guarantees the source compatibility in generated sources. There will be a
problem in compatibility if anyone uses generated protobuf message outside
of Hadoop modules. Which ideally shouldn't be as generated sources are not
public APIs.

   There should not be any compatibility problems between releases in terms
of communication provided both uses same syntax (proto2) of proto message.
This I have verified by communication between protobuf 2.5.0 client with
protobuf 3.7.1 server.

   To avoid the downstream transitive dependency classpath problem, who
might be using protobuf 2.5.0 classes, planning to shade the 3.7.1 classes
and its usages in all hadoop modules.. and keep 2.5.0 jar back in hadoop
classpath.

Hope I have answered your question.

-Vinay

On Sun, 22 Sep 2019, 7:38 pm Vinod Kumar Vavilapalli, 
wrote:

> Quick question, being lazy here, lots of JIRA updates on HADOOP-13363 over
> the years not helping either.
>
> Does anyone know what this upgrade will mean w.r.t compatibility for the
> Hadoop releases themselves? Remember that trunk is still 3.x.
>
> Thanks
> +Vinod
>
> > On Sep 21, 2019, at 9:55 AM, Vinayakumar B 
> wrote:
> >
> > @Wangda Tan  ,
> > Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
> > stages of protobuf upgrade in subtasks. (jar upgrade, Docker update,
> plugin
> > upgrade, shading, etc).
> > Right now, first task of jar upgrade is done. So need to update the
> protoc
> > executable in the in build environments.
> >
> > @张铎(Duo Zhang)  ,
> > Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
> > was possible. Sorry I missed it.
> >
> > Plugin update needed to be done for whole project, for which precommit
> > jenkins will need more time complete end-to-end runs.
> > So plugin update is planned in stages in further subtasks. It could be
> done
> > in 2-3 days.
> >
> > -Vinay
> >
> > On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang), 
> wrote:
> >
> >> I think this one is alread in place so we have to upgrade...
> >>
> >> https://issues.apache.org/jira/browse/HADOOP-16557
> >>
> >> Wangda Tan  于2019年9月21日周六 上午7:19写道：
> >>
> >>> Hi Vinay,
> >>>
> >>> A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
> >>> upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is
> completed?
> >>>
> >>> Thanks,
> >>> Wangda
> >>>
> >>> On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B  >
> >>> wrote:
> >>>
> >>>> Hi All,
> >>>>
> >>>> A very long pending task, protobuf upgrade is happening in
> >> HADOOP-13363.
> >>> As
> >>>> part of that protobuf version is upgraded to 3.7.1.
> >>>>
> >>>> Please update your build environments to have 3.7.1 protobuf version.
> >>>>
> >>>> BUILIDING.txt has been updated with latest instructions.
> >>>>
> >>>> This pre-requisite to update protoc dependecy manually is required
> >> until
> >>>> 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
> >>>> dynamically resolve required protoc exe.
> >>>>
> >>>> Dockerfile is being updated to have latest 3.7.1 as default protoc for
> >>> test
> >>>> environments.
> >>>>
> >>>> Thanks,
> >>>> -Vinay
> >>>>
> >>>
> >>
>
>

[jira] [Resolved] (HADOOP-16589) [pb-upgrade] Update docker image to make 3.7.1 protoc as default

[Vinayakumar B (Jira)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-16589?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B resolved HADOOP-16589.

Fix Version/s: 3.3.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to trunk.

> [pb-upgrade] Update docker image to make 3.7.1 protoc as default
> 
>
> Key: HADOOP-16589
> URL: https://issues.apache.org/jira/browse/HADOOP-16589
> Project: Hadoop Common
>  Issue Type: Sub-task
>    Reporter: Vinayakumar B
>    Assignee: Vinayakumar B
>Priority: Major
> Fix For: 3.3.0
>
>
> Right now, docker image contains both 2.5.0 protoc and 3.7.1 protoc.
> 2.5.0 is default protoc in PATH.
> After HADOOP-16557, protoc version expected in PATH is 3.7.1. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

@Wangda Tan  ,
Sorry for the confusion. HADOOP-13363 is umbrella jira to track multiple
stages of protobuf upgrade in subtasks. (jar upgrade, Docker update, plugin
upgrade, shading, etc).
Right now, first task of jar upgrade is done. So need to update the protoc
executable in the in build environments.

@张铎(Duo Zhang)  ,
Sorry for the inconvenience. Yes, indeed plugin update before jar upgrde
was possible. Sorry I missed it.

Plugin update needed to be done for whole project, for which precommit
jenkins will need more time complete end-to-end runs.
So plugin update is planned in stages in further subtasks. It could be done
in 2-3 days.

-Vinay

On Sat, 21 Sep 2019, 5:55 am 张铎(Duo Zhang),  wrote:

> I think this one is alread in place so we have to upgrade...
>
> https://issues.apache.org/jira/browse/HADOOP-16557
>
> Wangda Tan  于2019年9月21日周六 上午7:19写道：
>
> > Hi Vinay,
> >
> > A bit confused, I saw the HADOOP-13363 is still pending. Do we need to
> > upgrade protobuf version to 3.7.1 NOW or once HADOOP-13363 is completed?
> >
> > Thanks,
> > Wangda
> >
> > On Fri, Sep 20, 2019 at 8:11 AM Vinayakumar B 
> > wrote:
> >
> > > Hi All,
> > >
> > > A very long pending task, protobuf upgrade is happening in
> HADOOP-13363.
> > As
> > > part of that protobuf version is upgraded to 3.7.1.
> > >
> > > Please update your build environments to have 3.7.1 protobuf version.
> > >
> > > BUILIDING.txt has been updated with latest instructions.
> > >
> > > This pre-requisite to update protoc dependecy manually is required
> until
> > > 'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
> > > dynamically resolve required protoc exe.
> > >
> > > Dockerfile is being updated to have latest 3.7.1 as default protoc for
> > test
> > > environments.
> > >
> > > Thanks,
> > > -Vinay
> > >
> >
>

[NOTICE] Building trunk needs protoc 3.7.1

[Vinayakumar B]

Hi All,

A very long pending task, protobuf upgrade is happening in HADOOP-13363. As
part of that protobuf version is upgraded to 3.7.1.

Please update your build environments to have 3.7.1 protobuf version.

BUILIDING.txt has been updated with latest instructions.

This pre-requisite to update protoc dependecy manually is required until
'hadoop-maven-plugin' is replaced with 'protobuf-mavem-plugin' to
dynamically resolve required protoc exe.

Dockerfile is being updated to have latest 3.7.1 as default protoc for test
environments.

Thanks,
-Vinay

[jira] [Created] (HADOOP-16589) [pb-upgrade] Update docker image to make 3.7.1 protoc as default

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16589:
--

 Summary: [pb-upgrade] Update docker image to make 3.7.1 protoc as 
default
 Key: HADOOP-16589
 URL: https://issues.apache.org/jira/browse/HADOOP-16589
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Right now, docker image contains both 2.5.0 protoc and 3.7.1 protoc.

2.5.0 is default protoc in PATH.

After HADOOP-16557, protoc version expected in PATH is 3.7.1. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16562) Update docker image to have 3.7.1 protoc executable

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16562:
--

 Summary: Update docker image to have 3.7.1 protoc executable
 Key: HADOOP-16562
 URL: https://issues.apache.org/jira/browse/HADOOP-16562
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Current docker image is installed with 2.5.0 protobuf executable.

During the process of upgrading protobuf to 3.7.1, docker needs to have both 
versions for yetus to verify.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16561) [MAPREDUCE] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16561:
--

 Summary: [MAPREDUCE] use protobuf-maven-plugin to generate 
protobuf classes
 Key: HADOOP-16561
 URL: https://issues.apache.org/jira/browse/HADOOP-16561
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Use "protoc-maven-plugin" to dynamically download protobuf executable to 
generate protobuf classes from proto file



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16560) [YARN] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16560:
--

 Summary: [YARN] use protobuf-maven-plugin to generate protobuf 
classes
 Key: HADOOP-16560
 URL: https://issues.apache.org/jira/browse/HADOOP-16560
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Use "protoc-maven-plugin" to dynamically download protobuf executable to 
generate protobuf classes from proto file



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16559) [HDFS] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16559:
--

 Summary: [HDFS] use protobuf-maven-plugin to generate protobuf 
classes
 Key: HADOOP-16559
 URL: https://issues.apache.org/jira/browse/HADOOP-16559
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Use "protoc-maven-plugin" to dynamically download protobuf executable to 
generate protobuf classes from proto file



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16558) [COMMON] use protobuf-maven-plugin to generate protobuf classes

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16558:
--

 Summary: [COMMON] use protobuf-maven-plugin to generate protobuf 
classes
 Key: HADOOP-16558
 URL: https://issues.apache.org/jira/browse/HADOOP-16558
 Project: Hadoop Common
  Issue Type: Sub-task
  Components: common
Reporter: Vinayakumar B


Use "protoc-maven-plugin" to dynamically download protobuf executable to 
generate protobuf classes from proto files.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-16557) Upgrade protobuf.version to 3.7.1

[Vinayakumar B (Jira)]

Vinayakumar B created HADOOP-16557:
--

 Summary: Upgrade protobuf.version to 3.7.1 
 Key: HADOOP-16557
 URL: https://issues.apache.org/jira/browse/HADOOP-16557
 Project: Hadoop Common
  Issue Type: Sub-task
Reporter: Vinayakumar B


Bump up the "protobuf.version" to 3.7.1 and ensure all compile is successful.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

Thanks @Anu

I understand the concern. I took it in different manner.

Anyway, since protobuf upgrade looks huge, and need everyone's eyes on
changes as early as possible, its better to do it trunk itself.

I was able to come to successfull attempt of upgrading protobuf as per
suggestion of stack in Jira
https://issues.apache.org/jira/browse/HADOOP-13363?focusedCommentId=15958253=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15958253
 .

Have created the PR. Please review.  Changes looks huge, because all
references of "com.google.protobuf" relocated to
"o.a.h.shaded.com.google.protobuf".
Otherwise changes are reasonable.

This change is with still keeping the current 2.5.0 dependency, for
downstream builds. So essentially nothing should be changed for downstreams.

-Vinay


On Thu, Sep 5, 2019 at 9:56 PM Anu Engineer  wrote:

> Yes, I think that is what Sunil and I are trying to suggest; the complex
> dependencies like Protobuf, if you do it in the trunk you have a better
> change of getting it done. Otherwise, at merge point random downstream
> applications which you have never heard of will object, and Hadoop
> compatibility rules are very clear so you cannot fix it.
>
> With that said, even doing this in the trunk is complex; It might be good
> for you to host a meeting and get some feedback. I have openly said it is a
> great idea like "belling the cat", but the effort is in getting the
> community to agree and align. Solve that, most of your technical problems
> will be easier to solve.
>
> If you go into a branch, it might be that the community might forget about
> your work; and when you come in to merge you will see issues which you did
> not think about.
>
> So, Here is what would be great if you can make this happen; for ARM work,
> get a list of dependencies that needed to be upgraded; see if you can get
> the community aligned with this goal; since ARM might not be in the target
> for many users. You need to convince users that even without ARM, this is a
> great idea.
>
> If you like we can get together during one of the HDFS meetups hosted by
> Wei-chiu.
>
> Thanks
> Anu
>
>
>
> On Thu, Sep 5, 2019 at 3:19 AM Vinayakumar B 
> wrote:
>
>> Hi all,
>>
>> Thanks for the response.
>> As I see, protobuf upgrade is long pending and most awaited one.
>>
>> @Sunil
>> Protobuf upgrade looks to be a non-trivial task.
>> Thanks @Duo Zhang for the suggestion of
>> 'org.xolstice.maven.plugins:protobuf-maven-plugin'. This solves the
>> problem
>> of dependency on build environment.
>> However more problem lies in upgrade protobuf without breaking the
>> downstream builds.
>> Reason is many downstream projects directly refers server specific jars
>> and
>> they expect protobuf-2.5.0 jar to get added to classpath by transitive
>> dependency.
>>
>> There are some historical discussions and suggestions on
>> https://issues.apache.org/jira/browse/HADOOP-13363 related to protobuf
>> upgrade.
>> Recommended path for solution is, try to upgrade protobuf using shading of
>> latest protobuf for hadoop, and still keep protobuf-2.5.0 jar in
>> dependencies for downstreams.
>> I am trying out ideas on this and if it gets completed within time, may be
>> we can target trunk itself for the protobuf upgrade.
>>
>> separate branch idea is suggested for the overall ARM support including
>> protobuf upgrade and other problems mentioned in the discussion above.
>>
>> I dont expect separate branch to have a huge changes, apart from bug
>> fixes,
>> since there are no separate features specific to ARM is being planned.
>> So timely rebase of separate branch would reduce the overhead on branch
>> review/merge task.
>>
>> Still, if the solution to protobuf upgrade winds up early, without any
>> side
>> effects, I am more than happy to land it in trunk itself.
>>
>> Thanks,
>> Vinay
>> On Thu, Sep 5, 2019 at 2:27 PM Sunil Govindan  wrote:
>>
>> > Thanks Vinay for starting the thread.
>> >
>> > I agree to Anu's view point related to protobuf. And with the suggestion
>> > pointed out by Duo Zhang, if we can make use
>> > of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to
>> 3.0.0
>> > of protobuf will also be more easier.
>> >
>> > However i think its better to do this effort in trunk itself.
>> > In offline talks, few members were interested to start 3.3.0 release.
>> And
>> > given that happens soon, I feel its better
>> > we do this task in trunk itself as branch diverge is very much possibl

Re: [DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

s also willing to solve the problem.
>> >> >
>> >> > Anu Engineer  于2019年9月3日周二 上午1:23写道：
>> >> >
>> >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving
>> that
>> >> >> Hadoop and the downstream projects work correctly after you upgrade
>> >> core
>> >> >> components like Protobuf.
>> >> >> So while branching and working on a branch is easy, merging back
>> after
>> >> you
>> >> >> upgrade some of these core components is insanely hard. You might
>> want
>> >> to
>> >> >> make sure that community buys into upgrading these components in the
>> >> trunk.
>> >> >> That way we will get testing and downstream components will notice
>> when
>> >> >> things break.
>> >> >>
>> >> >> That said, I have lobbied for the upgrade of Protobuf for a really
>> long
>> >> >> time; I have argued that 2.5 is out of support and we cannot stay on
>> >> that
>> >> >> branch forever; or we need to take ownership of the Protobuf 2.5
>> code
>> >> base.
>> >> >> It has been rightly pointed to me that while all the arguments I
>> make
>> >> is
>> >> >> correct; it is a very complicated task to upgrade Protobuf, and the
>> >> worst
>> >> >> part is we will not even know what breaks until downstream projects
>> >> pick up
>> >> >> these changes and work against us.
>> >> >>
>> >> >> If we work off the Hadoop version 3 — and assume that we have
>> >> "shading" in
>> >> >> place for all deployments; it might be possible to get there; still
>> a
>> >> >> daunting task.
>> >> >>
>> >> >> So best of luck with the branch approach — But please remember,
>> Merging
>> >> >> back will be hard, Just my 2 cents.
>> >> >>
>> >> >> — Anu
>> >> >>
>> >> >>
>> >> >>
>> >> >>
>> >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng <
>> zhengzhenyul...@gmail.com
>> >> >
>> >> >> wrote:
>> >> >>
>> >> >>> Hi,
>> >> >>>
>> >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A
>> >> separate
>> >> >>> branch with it's own ARM CI seems a really good idea.
>> >> >>> By doing this we won't break any of the undergoing development in
>> >> trunk
>> >> >> and
>> >> >>> a CI can be a very good way to show what are the
>> >> >>> current problems and what have been fixed, it will also provide a
>> very
>> >> >> good
>> >> >>> view for contributors that are intrested to working on
>> >> >>> this. We can finally merge back the branch to trunk until the
>> >> community
>> >> >>> thinks it is good enough and stable enough. We can donate
>> >> >>> ARM machines to the existing CI system for the job.
>> >> >>>
>> >> >>> I wonder if this approch possible?
>> >> >>>
>> >> >>> BR,
>> >> >>>
>> >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <
>> liusheng2...@gmail.com>
>> >> >>> wrote:
>> >> >>>
>> >> >>>> Hi,
>> >> >>>>
>> >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab"
>> community
>> >> >>>> mentioned by Vinay. I am working on building and
>> >> >>>> testing Hadoop components on aarch64 server these days, besides
>> the
>> >> >>> missing
>> >> >>>> dependices of ARM platform issues #1 #2 #3
>> >> >>>> mentioned by Vinay, other similar issue has also be found, such as
>> >> the
>> >> >>>> "PhantomJS" dependent package also missing for aarch64.
>> >> >>>>
>> >> >>>> To promote the ARM support for Hadoop, we have discussed and
>> hoped to
>> >> >> add
>> >> >>>> an ARM spec

Re: [VOTE] Moving Submarine to a separate Apache project proposal

[Vinayakumar B]

+1

Thanks for the proposal.
Its very interesting project and looks very promising one by looking at the
participations from various companies and the speed of development.

I would also like to participate in the project.
Please add me as well.

Thanks,
-Vinay

On Tue, 3 Sep 2019, 12:38 pm Rakesh Radhakrishnan, 
wrote:

> +1, Thanks for the proposal.
>
> I am interested to participate in this project. Please include me as well
> in the project.
>
> Thanks,
> Rakesh
>
> On Tue, Sep 3, 2019 at 11:59 AM zhankun tang 
> wrote:
>
> > +1
> >
> > Thanks for Wangda's proposal.
> >
> > The submarine project is born within Hadoop, but not limited to Hadoop.
> It
> > began with a trainer on YARN but it quickly realized that only a trainer
> is
> > not enough to meet the AI platform requirements. But now there's no
> > user-friendly open-source solution covers the whole AI pipeline like data
> > engineering, training, and serving. And the underlying data
> infrastructure
> > itself is also evolving, for instance, many people love k8s. Not
> mentioning
> > there're many AI domain problems in this area to be solved.
> > It's almost for sure that building such an ML platform would utilize
> > various other open-source components taking ML into consideration
> > initially.
> >
> > I see submarine grows rapidly towards an enterprise-grade ML platform
> which
> > could potentially enable AI ability for data engineer and scientist. This
> > is an exciting thing for both the community and the industry.
> >
> > BR,
> > Zhankun
> >
> >
> > On Tue, 3 Sep 2019 at 13:34, Xun Liu  wrote:
> >
> > > +1
> > >
> > > Hello everyone, I am a member of the submarine development team.
> > > I have been contributing to submarine for more than a year.
> > > I have seen the progress of submarine development very fast.
> > > In more than a year, there are 9 long-term developers of different
> > > companies. Contributing,
> > > submarine cumulative code has more than 200,000 lines of code, is
> growing
> > > very fast,
> > > and is used in the production environment of multiple companies.
> > >
> > > In the submarine development group, there are 5 PMCs and 7committer
> > members
> > > from Hadoop, spark, zeppelin projects.
> > > They are very familiar with the development process and specifications
> of
> > > the apache community,
> > > and can well grasp the project development progress and project
> quality.
> > > So I recommend submarine to be a TLP project directly.
> > >
> > > We will continue to contribute to the submarine project. :-)
> > >
> > > Xun Liu
> > > Regards
> > >
> > > On Tue, 3 Sep 2019 at 12:01, Devaraj K  wrote:
> > >
> > > > +1
> > > >
> > > > Thanks Wangda for the proposal.
> > > > I would like to participate in this project, Please add me also to
> the
> > > > project.
> > > >
> > > > Regards
> > > > Devaraj K
> > > >
> > > > On Mon, Sep 2, 2019 at 8:50 PM zac yuan 
> wrote:
> > > >
> > > > > +1
> > > > >
> > > > > Submarine will be a complete solution for AI service development.
> It
> > > can
> > > > > take advantage of two best cluster systems: yarn and k8s, which
> will
> > > help
> > > > > more and more people get AI ability. To be a separate Apache
> project,
> > > > will
> > > > > accelerate the procedure of development apparently.
> > > > >
> > > > > Look forward to a big success in submarine project~
> > > > >
> > > > > 朱林浩  于2019年9月3日周二 上午10:38写道：
> > > > >
> > > > > > +1,
> > > > > > Hopefully, that will become the top project,
> > > > > >
> > > > > > I also hope to make more contributions to this project.
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > At 2019-09-03 09:26:53, "Naganarasimha Garla" <
> > > > > naganarasimha...@apache.org>
> > > > > > wrote:
> > > > > > >+ 1,
> > > > > > >I would also like start participate in this project, hope to
> > get
> > > > > > myself
> > > > > > >added to the project.
> > > > > > >
> > > > > > >Thanks and Regards,
> > > > > > >+ Naga
> > > > > > >
> > > > > > >On Tue, Sep 3, 2019 at 8:35 AM Wangda Tan 
> > > > wrote:
> > > > > > >
> > > > > > >> Hi Sree,
> > > > > > >>
> > > > > > >> I put it to the proposal, please let me know what you think:
> > > > > > >>
> > > > > > >> The traditional path at Apache would have been to create an
> > > > incubator
> > > > > > >> > project, but the code is already being released by Apache
> and
> > > most
> > > > > of
> > > > > > the
> > > > > > >> > developers are familiar with Apache rules and guidelines. In
> > > > > > particular,
> > > > > > >> > the proposed PMC has 2 Apache TLP PMCs and proposed initial
> > > > > committers
> > > > > > >> > have 4 Apache TLP PMCs (from 3 different companies). They
> will
> > > > > provide
> > > > > > >> > oversight and guidance for the developers that are less
> > > > experienced
> > > > > in
> > > > > > >> the
> > > > > > >> > Apache Way. Therefore, the Submarine project would like to
> > > propose
> > > > > > >> becoming
> > > > > 

[DISCUSS] ARM/aarch64 support for Hadoop

[Vinayakumar B]

Hi Folks,

ARM is becoming famous lately in its processing capability and has got the
potential to run Bigdata workloads.
Many users have been moving to ARM machines due to its low cost.

In the past there were attempts to compile Hadoop on ARM (Rasberry PI) for
experimental purposes. Today ARM architecture is taking some of the
serverside processing as well. So there will be/is a real need of Hadoop to
support ARM architecture as well.

There are bunch of users who are trying out building Hadoop on ARM, trying
to add ARM CI to hadoop and facing issues[1]. Also some

As of today, Hadoop does not compile on ARM due to below issues, found from
testing done in openlab in [2].

1. Protobuf :
---
 Hadoop project (also some downstream projects) stuck to protobuf 2.5.0
version, due to backward compatibility reasons. Protobuf-2.5.0 is not being
maintained in the community. While protobuf 3.x is being actively adopted
widely, still protobuf 3.x provides wire compatibility for proto2 messages.
Due to some compilation issues in the generated java code, which can induce
problems in downstream. Due to this reason protobuf upgrade from 2.5.0 was
not taken up.
In 3.0.0 onwards, hadoop supports shading of libraries to avoid classpath
problem in downstream projects.
There are patches available to fix compilation in Hadoop. But need to
find a way to upgrade protobuf to latest version and still maintain the
downstream's classpath using shading feature of Hadoop build.

 There is a Jira for protobuf upgrade[3] created even before shade
support was added to Hadoop. Now need to revisit the Jira and continue
explore possibilities.

2. leveldbjni:
---
Current leveldbjni used in YARN doesnot support ARM architecture, need
to check whether any of the future versions support ARM and can hadoop
upgrade to that version.


3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1'
-
'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by default in
the maven repository. Workaround is to build it locally and keep in local
maven repository.
Need to check whether any future versions of 'protoc-gen-grpc-java' is
having ARM executable and whether hadoop-yarn-csi can upgrade it?


Once the compilation issues are solved, then there might be many native
code related issues due to different architectures.
So to explore everything, need to join hands together and proceed.


Let us discuss and check, whether any body else out there who also need the
support of Hadoop on ARM architectures and ready to lend their hands and
time in this work.


[1] https://issues.apache.org/jira/browse/HADOOP-16358
[2]
https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887
[3] https://issues.apache.org/jira/browse/HADOOP-13363

-Vinay

[ANNOUNCE] New Apache Hadoop Committer - Ayush Saxena

[Vinayakumar B]

In bcc: general@, please bcc: (and not cc:) general@ if you want to include

It's my pleasure to announce that Ayush Saxena has been elected as
committer
on the Apache Hadoop project recognising his continued contributions to the
project.

Please join me in congratulating him.

Hearty Congratulations & Welcome aboard Ayush!


Vinayakumar B
(On behalf of the Hadoop PMC)

[ANNOUNCE] New Apache Hadoop Committer - Ayush Saxena

[Vinayakumar B]

In bcc: general@, please bcc: (and not cc:) general@ if you want to include

It's my pleasure to announce that Ayush Saxena has been elected as
committer
on the Apache Hadoop project recognising his continued contributions to the
project.

Please join me in congratulating him.

Hearty Congratulations & Welcome aboard Ayush!


Vinayakumar B
(On behalf of the Hadoop PMC)

Re: [VOTE]: Support for RBF data locality Solution

[Vinayakumar B]

+1 for approach A.

On Thu, 11 Apr 2019, 12:23 pm Akira Ajisaka,  wrote:

> The Approach A looks good to me.
>
> Thanks,
> Akira
>
> On Thu, Apr 11, 2019 at 2:30 PM Xiaoqiao He  wrote:
> >
> > Hi forks,
> >
> > The current implementation of RBF is not sensitive about data locality,
> > since NameNode could not get real client hostname by invoke
> > Server#getRemoteAddress when RPC request forward by Router to NameNode.
> > Therefore, it will lead to several challenges, for instance,
> >
> >- a. Client could have to go for remote read instead of local read,
> >Short-Circuit could not be used in most cases.
> >- b. Block placement policy could not run as except based on defined
> >rack aware. Thus it will loss local node write.
> >
> > There are some different solutions to solve data locality issue after
> > discussion, some of them will change RPC protocol, so we look forward to
> > furthermore suggestions and votes. HDFS-13248 is tracking the issue.
> >
> >- Approach A: Changing IPC/RPC layer protocol
> (IpcConnectionContextProto
> >or RpcHeader#RpcRequestHeaderProto) and add extra field about client
> >hostname. Of course the new field is optional, only input by Router
> and
> >parse by Namenode in generally. This approach is compatibility and
> Client
> >should do nothing after changing.
> >- Approach B: Changing ClientProtocol and add extra interface
> >create/append/getBlockLocations with additional parameter about client
> >hostname. As approach A, it is input by Router and parse by Namenode,
> and
> >also is compatibility.
> >- Approach C: Solve write and read locality separately based on
> current
> >interface and no changes, for write, hack client hostname as one of
> favor
> >nodes for addBlocks, for read, reorder targets at Router after
> Namenode
> >returns result to Router.
> >
> > As discussion and evaluation in HDFS-13248, we prefer to change IPC/RPC
> > layer protocol to support RPC data locality. We welcome more suggestions,
> > votes or just give us feedback to push forward this feature. Thanks.
> >
> > Best Regards,
> > Hexiaoqiao
> >
> > reference
> > [1] https://issues.apache.org/jira/browse/HDFS-13248
> > [2] https://issues.apache.org/jira/browse/HDFS-10467
> >
> > [3] https://issues.apache.org/jira/browse/HDFS-12615
>
> -
> To unsubscribe, e-mail: hdfs-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: hdfs-dev-h...@hadoop.apache.org
>
>

Re: Including Original Author in git commits.

[Vinayakumar B]

So.. if we started doing that already.. we can encourage contributors to
attach formatted patch.. or create PRs.

And update wiki to follow exact steps to contribute and commit.

-Vinay


On Thu, 14 Feb 2019, 4:54 pm Steve Loughran  I've been trying to do that recently, though as it forces me to go to the
> command line rather than using Atlassian Sourcetree, I've been getting
> other things wrong. To those people who have been dealing with commits I've
> managed to mess up: apologies.
>
> 1. Once someone is down as an author you don't need to add their email
> address; the first time you will need to get their email address
> 2. Akira, Aaron and I also use the -S option to GPG sign the commits. We
> should all be doing that, as it is the way to show who really committed the
> patch. Add --show-signature to the end of any git log to command to see
> those.
> 3. note that if you cherry-pick a patch into a different branch, you have
> to use -S in the git cherry-pick command to resign it.
>
> we should all have our GPG keys in the KEYS file, and co-sign the others in
> there, so that we have that mutual trust.
>
> -Steve
>
> ps: one flaw in the GPG process: if you ever revoke the key then all
> existing commits are considered untrusted
>
> http://steveloughran.blogspot.com/2017/10/roca-breaks-my-commit-process.html
>
>
>
>
> On Thu, Feb 14, 2019 at 9:12 AM Akira Ajisaka  wrote:
>
> > Hi Vinay,
> >
> > I'm already doing this if I can get the original author name and the
> > email address in some way.
> > If the patch is created by git format-patch command, smart-apply-patch
> > --committer option can do this automatically.
> >
>
> Never knew that
>

Including Original Author in git commits.

[Vinayakumar B]

Hi all,

Right now we are committing with commitrer name as author and adding
contributor name in commit message. This way of committing will not reflect
in contributors git profile.

I believe current way of crediting.. is based on SVN days..

Since now all apache repos are following git.. we can use git
functionalities (already some are using PRs) and use original author name
in commits and use signed of message for committers.


Any thoughts?

-Vinay

Re: [VOTE] Release Apache Hadoop 3.1.2 - RC1

[Vinayakumar B]

Sorry for being late.

Thanks everyone for the Release efforts.

+1 (Binding)

-Built from tag on Ubuntu-18.04
-Successful Native build.
-Deployed a 3-Node docker cluster with kerberos Security Enabled on Centos7
images.
-Verified basic HDFS Commands
-Ran basic wordcount Job.

-Vinay

On Wed, Feb 6, 2019 at 2:38 AM Wangda Tan  wrote:

> Thanks everyone for voting, we have 6 binding votes and 6 non-binding
> votes, no vetos. The vote passed. We will go ahead to stage this release.
>
> - Wangda and Sunil.
>
> On Tue, Feb 5, 2019 at 12:05 PM Billie Rinaldi 
> wrote:
>
> > Thanks Sunil, the new source tarball matches the RC tag and its checksum
> > and signature look good.
> >
> > Billie
> >
> > On Tue, Feb 5, 2019 at 10:50 AM Sunil G  wrote:
> >
> > > Thanks Billie for pointing out.
> > > I have updated source by removing patchprocess and extra line create
> > > release.
> > >
> > > Also updated checksum as well.
> > >
> > > @bil...@apache.org   @Wangda Tan  >
> > > please help to verify this changed bit once.
> > >
> > > Thanks
> > > Sunil
> > >
> > > On Tue, Feb 5, 2019 at 5:23 AM Billie Rinaldi <
> billie.rina...@gmail.com>
> > > wrote:
> > >
> > >> Hey Sunil and Wangda, thanks for the RC. The source tarball has a
> > >> patchprocess directory with some yetus code in it. Also, the file
> > >> dev-support/bin/create-release file has the following line added:
> > >>   export GPG_AGENT_INFO="/home/sunilg/.gnupg/S.gpg-agent:$(pgrep
> > >> gpg-agent):1"
> > >>
> > >> I think we are probably due for an overall review of LICENSE and
> NOTICE.
> > >> I saw some idiosyncrasies there but nothing that looked like a
> blocker.
> > >>
> > >> On Mon, Jan 28, 2019 at 10:20 PM Sunil G  wrote:
> > >>
> > >>> Hi Folks,
> > >>>
> > >>> On behalf of Wangda, we have an RC1 for Apache Hadoop 3.1.2.
> > >>>
> > >>> The artifacts are available here:
> > >>> http://home.apache.org/~sunilg/hadoop-3.1.2-RC1/
> > >>>
> > >>> The RC tag in git is release-3.1.2-RC1:
> > >>> https://github.com/apache/hadoop/commits/release-3.1.2-RC1
> > >>>
> > >>> The maven artifacts are available via repository.apache.org at
> > >>>
> > https://repository.apache.org/content/repositories/orgapachehadoop-1215
> > >>>
> > >>> This vote will run 5 days from now.
> > >>>
> > >>> 3.1.2 contains 325 [1] fixed JIRA issues since 3.1.1.
> > >>>
> > >>> We have done testing with a pseudo cluster and distributed shell job.
> > >>>
> > >>> My +1 to start.
> > >>>
> > >>> Best,
> > >>> Wangda Tan and Sunil Govindan
> > >>>
> > >>> [1] project in (YARN, HADOOP, MAPREDUCE, HDFS) AND fixVersion in
> > (3.1.2)
> > >>> ORDER BY priority DESC
> > >>>
> > >>
> >
>

Re: [DISCUSS] Move to gitbox

[Vinayakumar B]

+1

-Vinay

On Mon, 10 Dec 2018, 1:22 pm Elek, Marton 
> Thanks Akira,
>
> +1 (non-binding)
>
> I think it's better to do it now at a planned date.
>
> If I understood well the only bigger task here is to update all the
> jenkins jobs. (I am happy to help/contribute what I can do)
>
>
> Marton
>
> On 12/8/18 6:25 AM, Akira Ajisaka wrote:
> > Hi all,
> >
> > Apache Hadoop git repository is in git-wip-us server and it will be
> > decommissioned.
> > If there are no objection, I'll file a JIRA ticket with INFRA to
> > migrate to https://gitbox.apache.org/ and update documentation.
> >
> > According to ASF infra team, the timeframe is as follows:
> >
> >> - December 9th 2018 -> January 9th 2019: Voluntary (coordinated)
> relocation
> >> - January 9th -> February 6th: Mandated (coordinated) relocation
> >> - February 7th: All remaining repositories are mass migrated.
> >> This timeline may change to accommodate various scenarios.
> >
> > If we got consensus by January 9th, I can file a ticket with INFRA and
> > migrate it.
> > Even if we cannot got consensus, the repository will be migrated by
> > February 7th.
> >
> > Regards,
> > Akira
> >
> > -
> > To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> > For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
> >
>
> -
> To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: common-dev-h...@hadoop.apache.org
>
>

[jira] [Created] (HADOOP-15901) IPC Client and Server should use Time.monotonicNow() for elapsed times.

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-15901:
--

 Summary: IPC Client and Server should use Time.monotonicNow() for 
elapsed times.
 Key: HADOOP-15901
 URL: https://issues.apache.org/jira/browse/HADOOP-15901
 Project: Hadoop Common
  Issue Type: Bug
  Components: ipc, metrics
Reporter: Vinayakumar B


Client.java and Server.java  uses {{Time.now()}} to calculate the elapsed 
times/timeouts. This could result in undesired results when system clock's time 
changes.

{{Time.monotonicNow()}} should be used for elapsed time calculations within 
same JVM.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-15856) Trunk build fails to compile native on Windows

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-15856:
--

 Summary: Trunk build fails to compile native on Windows
 Key: HADOOP-15856
 URL: https://issues.apache.org/jira/browse/HADOOP-15856
 Project: Hadoop Common
  Issue Type: Bug
  Components: native
Reporter: Vinayakumar B


After removal of {{javah}} dependency in HADOOP-15767
Trunk build fails with unable to find JNI headers.

HADOOP-15767 fixed javah isssue with JDK10 only for linux builds.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: HADOOP-14163 proposal for new hadoop.apache.org

[Vinayakumar B]

+1, New site looks great.

Just one nit in README
'--refresh' flag for hugo server no longer available.

-Vinay

On Mon, 3 Sep 2018, 10:21 am Shashikant Banerjee, 
wrote:

> +1
>
> Thanks
> Shashi
>
> On 9/3/18, 9:23 AM, "Mukul Kumar Singh"  wrote:
>
> +1, Thanks for working on this Marton.
>
> -Mukul
>
> On 03/09/18, 9:02 AM, "John Zhuge"  wrote:
>
> +1 Like the new site.
>
> On Sun, Sep 2, 2018 at 7:02 PM Weiwei Yang 
> wrote:
>
> > That's really nice, +1.
> >
> > --
> > Weiwei
> >
> > On Sat, Sep 1, 2018 at 4:36 AM Wangda Tan 
> wrote:
> >
> > > +1, thanks for working on this, Marton!
> > >
> > > Best,
> > > Wangda
> > >
> > > On Fri, Aug 31, 2018 at 11:24 AM Arpit Agarwal <
> aagar...@hortonworks.com
> > >
> > > wrote:
> > >
> > > > +1
> > > >
> > > > Thanks for initiating this Marton.
> > > >
> > > >
> > > > On 8/31/18, 1:07 AM, "Elek, Marton"  wrote:
> > > >
> > > > Bumping this thread at last time.
> > > >
> > > > I have the following proposal:
> > > >
> > > > 1. I will request a new git repository hadoop-site.git
> and import
> > the
> > > > new site to there (which has exactly the same content as
> the
> > existing
> > > > site).
> > > >
> > > > 2. I will ask infra to use the new repository as the
> source of
> > > > hadoop.apache.org
> > > >
> > > > 3. I will sync manually all of the changes in the next
> two months
> > > back
> > > > to the svn site from the git (release announcements, new
> > committers)
> > > >
> > > > IN CASE OF ANY PROBLEM we can switch back to the svn
> without any
> > > > problem.
> > > >
> > > > If no-one objects within three days, I'll assume lazy
> consensus and
> > > > start with this plan. Please comment if you have
> objections.
> > > >
> > > > Again: it allows immediate fallback at any time as svn
> repo will be
> > > > kept
> > > > as is (+ I will keep it up-to-date in the next 2 months)
> > > >
> > > > Thanks,
> > > > Marton
> > > >
> > > >
> > > > On 06/21/2018 09:00 PM, Elek, Marton wrote:
> > > > >
> > > > > Thank you very much to bump up this thread.
> > > > >
> > > > >
> > > > > About [2]: (Just for the clarification) the content of
> the
> > proposed
> > > > > website is exactly the same as the old one.
> > > > >
> > > > > About [1]. I believe that the "mvn site" is perfect
> for the
> > > > > documentation but for website creation there are more
> simple and
> > > > > powerful tools.
> > > > >
> > > > > Hugo has more simple compared to jekyll. Just one
> binary, without
> > > > > dependencies, works everywhere (mac, linux, windows)
> > > > >
> > > > > Hugo has much more powerful compared to "mvn site".
> Easier to
> > > > create/use
> > > > > more modern layout/theme, and easier to handle the
> content (for
> > > > example
> > > > > new release announcements could be generated as part
> of the
> > release
> > > > > process)
> > > > >
> > > > > I think it's very low risk to try out a new approach
> for the site
> > > > (and
> > > > > easy to rollback in case of problems)
> > > > >
> > > > > Marton
> > > > >
> > > > > ps: I just updated the patch/preview site with the
> recent
> > releases:
> > > > >
> > > > > ***
> > > > > * http://hadoop.anzix.net *
> > > > > ***
> > > > >
> > > > > On 06/21/2018 01:27 AM, Vinod Kumar Vavilapalli wrote:
> > > > >> Got pinged about this offline.
> > > > >>
> > > > >> Thanks for keeping at it, Marton!
> > > > >>
> > > > >> I think there are two road-blocks here
> > > > >>   (1) Is the mechanism using which the website is
> built good
> > > enough
> > > > -
> > > > >> mvn-site / hugo etc?
> > > > >>   (2) Is the new website good enough?
> > > > >>
> > > > >> For (1), I just think we need more committer
> attention and get
> > > > >> feedback rapidly and get it in.
> > > > >>
> > > > >> For (2), how about we do it in a different way in the
> interest
> > of
>   

[jira] [Created] (HADOOP-15602) Support SASL Rpc request handling in separate Handlers

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-15602:
--

 Summary: Support SASL Rpc request handling in separate Handlers 
 Key: HADOOP-15602
 URL: https://issues.apache.org/jira/browse/HADOOP-15602
 Project: Hadoop Common
  Issue Type: Improvement
  Components: ipc
Reporter: Vinayakumar B


Right now, during RPC Connection establishment, all SASL requests are 
considered as OutOfBand requests and handled within the same Reader thread.

SASL handling involves authentication with Kerberos and SecretManagers(for 
Token validation). During this time, Reader thread would be blocked, hence 
blocking all the incoming RPC requests on other established connections. Some 
secretManager impls require to communicate to external systems (ex: ZK) for 
verification.

SASL RPC handling in separate dedicated handlers, would enable Reader threads 
to read RPC requests from established connections without blocking.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Re: Can't pull the trunk code in windows- >File name too long

[Vinayakumar B]

this is happening  in Windows, Please somebody from yarn look into this.

-Vinay

On 13 Mar 2018 11:30 am, "Brahma Reddy Battula" <
brahmareddy.batt...@huawei.com> wrote:

Hi

Not able to pull/clone the trunk code. Does it possible make following
shorten.


fatal: cannot create directory at 'hadoop-yarn-project/hadoop-
yarn/hadoop-yarn-server/hadoop-yarn-server-timelineservice-hbase/hadoop-
yarn-server-timelineservice-hbase-server/hadoop-yarn-
server-timelineservice-hbase-server-1/src/main/java/org/
apache/hadoop/yarn/server/timelineservice': Filename too long



--Brahma Reddy Battula

Re: [VOTE] Release Apache Hadoop 3.0.1 (RC0)

[Vinayakumar B]

+1 Binding,

1. Checked out code from tag.
2. Built and deployed a 3-node docker cluster in secure mode (HDFS HA,
Non-HA YARN). Docker script available in [1], if anyone need,
3. Ran wordcount job successfully.
4. Verified basic webhdfs commands using curl.
5. Verified basic HDFS commands


[1] https://github.com/vinayakumarb/docker-cluster-deployer

Thanks,
-Vinay

On Wed, Feb 21, 2018 at 5:41 AM, Lei Xu  wrote:

> Dear PMC and committers
>
> Please help to verify and vote this release. I will extend the voting
> period to Friday (Feb 23th) of this week.
>
> Thanks!
>
> On Thu, Feb 15, 2018 at 3:36 PM, Lei Xu  wrote:
> > Hi, all
> >
> > I've created release candidate 0 for Apache Hadoop 3.0.1
> >
> > Apache Hadoop 3.0.1 will be the first bug fix release for Apache
> > Hadoop 3.0 release. It includes 49 bug fixes, which include 10
> > blockers and 8 are critical.
> >
> > Please note:
> > * HDFS-12990. Change default NameNode RPC port back to 8020. It makes
> > incompatible changes to Hadoop 3.0.0.  After 3.0.1 releases, Apache
> > Hadoop 3.0.0 will be deprecated due to this change.
> >
> > The release page is:
> > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0+Release
> >
> > New RC is available at: http://home.apache.org/~lei/hadoop-3.0.1-RC0/
> >
> > The git tag is release-3.0.1-RC0, and the latest commit is
> > 494d075055b52b0cc922bc25237e231bb3771c90
> >
> > The maven artifacts are available:
> > https://repository.apache.org/content/repositories/orgapachehadoop-1078/
> >
> > Please try the release and vote; the vote will run for the usual 5
> > days, ending on 2/20/2017 6pm PST time.
> >
> > Thanks!
> >
> > --
> > Lei (Eddy) Xu
>
>
>
> --
> Lei (Eddy) Xu
> Software Engineer, Cloudera
>
> -
> To unsubscribe, e-mail: yarn-dev-unsubscr...@hadoop.apache.org
> For additional commands, e-mail: yarn-dev-h...@hadoop.apache.org
>
>

[jira] [Reopened] (HADOOP-13738) DiskChecker should perform some disk IO

[Vinayakumar B (JIRA)]

 [ 
https://issues.apache.org/jira/browse/HADOOP-13738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vinayakumar B reopened HADOOP-13738:


Re-opening to cherry-pick to branch-2.8

> DiskChecker should perform some disk IO
> ---
>
> Key: HADOOP-13738
> URL: https://issues.apache.org/jira/browse/HADOOP-13738
> Project: Hadoop Common
>  Issue Type: Improvement
>Reporter: Arpit Agarwal
>Assignee: Arpit Agarwal
> Fix For: 2.9.0, 3.0.0-alpha2
>
> Attachments: HADOOP-13738.01.patch, HADOOP-13738.02.patch, 
> HADOOP-13738.03.patch, HADOOP-13738.04.patch, HADOOP-13738.05.patch
>
>
> DiskChecker can fail to detect total disk/controller failures indefinitely. 
> We have seen this in real clusters. DiskChecker performs simple 
> permissions-based checks on directories which do not guarantee that any disk 
> IO will be attempted.
> A simple improvement is to write some data and flush it to the disk.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14910) Upgrade netty-all jar to 4.0.37.Final

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-14910:
--

 Summary: Upgrade netty-all jar to 4.0.37.Final
 Key: HADOOP-14910
 URL: https://issues.apache.org/jira/browse/HADOOP-14910
 Project: Hadoop Common
  Issue Type: Bug
Reporter: Vinayakumar B
Priority: Critical


Upgrade netty-all jar to 4.0.37.Final version to fix latest vulnerabilities 
reported.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

Hadoop 2.8.1 docs not available at http://hadoop.apache.org/docs/r2.8.1/

[Vinayakumar B]

There are no webpages available in the below docs directory. All it lists
is empty directories.

  http://hadoop.apache.org/docs/r2.8.1/

-Vinay

[jira] [Created] (HADOOP-14427) Avoid reloading of Configuration in ViewFileSystem creation.

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-14427:
--

 Summary: Avoid reloading of Configuration in ViewFileSystem 
creation.
 Key: HADOOP-14427
 URL: https://issues.apache.org/jira/browse/HADOOP-14427
 Project: Hadoop Common
  Issue Type: Bug
  Components: viewfs
Reporter: Vinayakumar B
Assignee: Vinayakumar B


Avoid {{new Configuration()}} in below code. during viewfilesystem creation
{code}public InternalDirOfViewFs(final InodeTree.INodeDir dir,
final long cTime, final UserGroupInformation ugi, URI uri)
  throws URISyntaxException {
  myUri = uri;
  try {
initialize(myUri, new Configuration());
  } catch (IOException e) {
throw new RuntimeException("Cannot occur");
  }
  theInternalDir = dir;
  creationTime = cTime;
  this.ugi = ugi;
}{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

[jira] [Created] (HADOOP-14100) Upgrade Jsch jar to latest version

[Vinayakumar B (JIRA)]

Vinayakumar B created HADOOP-14100:
--

 Summary: Upgrade Jsch jar to latest version
 Key: HADOOP-14100
 URL: https://issues.apache.org/jira/browse/HADOOP-14100
 Project: Hadoop Common
  Issue Type: Bug
Affects Versions: 2.6.5, 2.7.3
Reporter: Vinayakumar B
Assignee: Vinayakumar B
Priority: Critical


Recently there was on vulnerability reported on jsch library. Its fixed in 
latest 0.1.54 version before CVE was made public.
https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-5725

So, need to  upgrade jsch to latest 0.1.54 version.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

-
To unsubscribe, e-mail: common-dev-unsubscr...@hadoop.apache.org
For additional commands, e-mail: common-dev-h...@hadoop.apache.org

RE: [VOTE] Release Apache Hadoop 2.7.3 RC0

[Vinayakumar B]

+1 (binding)

1. Downloaded and Built from branch-2.7.3
2. Started up HDFS and YARN in Single Node cluster.
3. Ran WordCount job multiple times and Success.
4. Verified the "Release Notes" available at the URL mentioned by Vinod.


Apart from that, 
Faced same issues as Andrew wang, while running the WordCount job first time in 
my new Ubuntu installation, without 'configuring the shuffle handler properly'. 
Whole session got logged by closing all other applications open. After 
configuring the shuffle handler properly, job was successful though.

-Vinay

-Original Message-
From: Andrew Wang [mailto:andrew.w...@cloudera.com] 
Sent: 26 July 2016 00:22
To: Karthik Kambatla 
Cc: larry mccay ; Vinod Kumar Vavilapalli 
; common-dev@hadoop.apache.org; hdfs-...@hadoop.apache.org; 
yarn-...@hadoop.apache.org; mapreduce-...@hadoop.apache.org
Subject: Re: [VOTE] Release Apache Hadoop 2.7.3 RC0

I'll also add that, as a YARN newbie, I did hit two usability issues. These are 
very unlikely to be regressions, and I can file JIRAs if they seem fixable.

* I didn't have SSH to localhost set up (new laptop), and when I tried to run 
the Pi job, it'd exit my window manager session. I feel there must be a more 
developer-friendly solution here.
* If you start the NodeManager and not the RM, the NM has a handler for SIGTERM 
and SIGINT that blocked my Ctrl-C and kill attempts during startup.
I had to kill -9 it.

On Mon, Jul 25, 2016 at 11:44 AM, Andrew Wang 
wrote:

> I got asked this off-list, so as a reminder, only PMC votes are 
> binding on releases. Everyone is encouraged to vote on releases though!
>
> +1 (binding)
>
> * Downloaded source, built
> * Started up HDFS and YARN
> * Ran Pi job which as usual returned 4, and a little teragen
>
> On Mon, Jul 25, 2016 at 11:08 AM, Karthik Kambatla 
> 
> wrote:
>
>> +1 (binding)
>>
>> * Downloaded and build from source
>> * Checked LICENSE and NOTICE
>> * Pseudo-distributed cluster with FairScheduler
>> * Ran MR and HDFS tests
>> * Verified basic UI
>>
>> On Sun, Jul 24, 2016 at 1:07 PM, larry mccay  wrote:
>>
>> > +1 binding
>> >
>> > * downloaded and built from source
>> > * checked LICENSE and NOTICE files
>> > * verified signatures
>> > * ran standalone tests
>> > * installed pseudo-distributed instance on my mac
>> > * ran through HDFS and mapreduce tests
>> > * tested credential command
>> > * tested webhdfs access through Apache Knox
>> >
>> >
>> > On Fri, Jul 22, 2016 at 10:15 PM, Vinod Kumar Vavilapalli < 
>> > vino...@apache.org> wrote:
>> >
>> > > Hi all,
>> > >
>> > > I've created a release candidate RC0 for Apache Hadoop 2.7.3.
>> > >
>> > > As discussed before, this is the next maintenance release to 
>> > > follow up 2.7.2.
>> > >
>> > > The RC is available for validation at:
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/ < 
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/>
>> > >
>> > > The RC tag in git is: release-2.7.3-RC0
>> > >
>> > > The maven artifacts are available via repository.apache.org < 
>> > > http://repository.apache.org/> at
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-10
>> 40/
>> > <
>> > >
>> https://repository.apache.org/content/repositories/orgapachehadoop-10
>> 40/
>> > >
>> > >
>> > > The release-notes are inside the tar-balls at location 
>> > > hadoop-common-project/hadoop-common/src/main/docs/releasenotes.ht
>> > > ml. I hosted this at 
>> > > http://home.apache.org/~vinodkv/hadoop-2.7.3-RC0/releasenotes.htm
>> > > l < 
>> > > http://people.apache.org/~vinodkv/hadoop-2.7.2-RC1/releasenotes.h
>> > > tml>
>> > for
>> > > your quick perusal.
>> > >
>> > > As you may have noted, a very long fix-cycle for the License & 
>> > > Notice issues (HADOOP-12893) caused 2.7.3 (along with every other 
>> > > Hadoop
>> > release)
>> > > to slip by quite a bit. This release's related discussion thread 
>> > > is
>> > linked
>> > > below: [1].
>> > >
>> > > Please try the release and vote; the vote will run for the usual 
>> > > 5
>> days.
>> > >
>> > > Thanks,
>> > > Vinod
>> > >
>> > > [1]: 2.7.3 release plan:
>> > >
>> https://www.mail-archive.com/hdfs-dev%40hadoop.apache.org/msg24439.ht
>> ml
>> > <
>> > > http://markmail.org/thread/6yv2fyrs4jlepmmr>
>> >
>>
>
>