Thanks Vinay for starting the thread. I agree to Anu's view point related to protobuf. And with the suggestion pointed out by Duo Zhang, if we can make use of org.xolstice.maven.plugins:protobuf-maven-plugin, our upgrade to 3.0.0 of protobuf will also be more easier.
However i think its better to do this effort in trunk itself. In offline talks, few members were interested to start 3.3.0 release. And given that happens soon, I feel its better we do this task in trunk itself as branch diverge is very much possible. And to bring to call a merge on such a big branch will be even more tough task. my 2 cents. Thanks Sunil On Thu, Sep 5, 2019 at 6:04 AM 张铎(Duo Zhang) <palomino...@gmail.com> wrote: > Suggest to use org.xolstice.maven.plugins:protobuf-maven-plugin to generate > the protobuf code. It will download the protoc binary from the maven > central so we do not need to install protoc on the build machine any more. > > Zhenyu Zheng <zhengzhenyul...@gmail.com> 于2019年9月4日周三 下午5:27写道： > > > BTW, I also noticed that the Hadoop-trunk-Commit job has been failling > for > > over 2 month related to the Protobuf problem . > > According to the latest successful build log: > > > https://builds.apache.org/job/Hadoop-trunk-Commit/lastSuccessfulBuild/consoleFull > > the > > os version was ubuntu 14.04 and for the jobs that are failling now such > > as: https://builds.apache.org/job/Hadoop-trunk-Commit/17222/console, > > the os version is 18.04. I'm not very familiar with the version changing > > for the jobs but I did a little search, according to: > > > > > https://packages.ubuntu.com/search?keywords=protobuf-compiler&searchon=names > > & > > > > > https://packages.ubuntu.com/search?suite=default§ion=all&arch=any&keywords=libprotoc-dev&searchon=names > > it both said that the version of libprotc-dev and protobuf-compiler > > available for ubuntu 18.04 is 3.0.0 > > > > > > On Wed, Sep 4, 2019 at 4:39 PM Ayush Saxena <ayush...@gmail.com> wrote: > > > >> Thanx Vinay for the initiative, Makes sense to add support for different > >> architectures. > >> > >> +1, for the branch idea. > >> Good Luck!!! > >> > >> -Ayush > >> > >> > On 03-Sep-2019, at 6:19 AM, 张铎(Duo Zhang) <palomino...@gmail.com> > >> wrote: > >> > > >> > For HBase, we purged all the protobuf related things from the public > >> API, > >> > and then upgraded to a shaded and relocated version of protobuf. We > have > >> > created a repo for this: > >> > > >> > https://github.com/apache/hbase-thirdparty > >> > > >> > But since the hadoop dependencies still pull in the protobuf 2.5 jars, > >> our > >> > coprocessors are still on protobuf 2.5. Recently we have opened a > >> discuss > >> > on how to deal with the upgrading of coprocessor. Glad to see that the > >> > hadoop community is also willing to solve the problem. > >> > > >> > Anu Engineer <aengin...@cloudera.com.invalid> 于2019年9月3日周二 上午1:23写道： > >> > > >> >> +1, for the branch idea. Just FYI, Your biggest problem is proving > that > >> >> Hadoop and the downstream projects work correctly after you upgrade > >> core > >> >> components like Protobuf. > >> >> So while branching and working on a branch is easy, merging back > after > >> you > >> >> upgrade some of these core components is insanely hard. You might > want > >> to > >> >> make sure that community buys into upgrading these components in the > >> trunk. > >> >> That way we will get testing and downstream components will notice > when > >> >> things break. > >> >> > >> >> That said, I have lobbied for the upgrade of Protobuf for a really > long > >> >> time; I have argued that 2.5 is out of support and we cannot stay on > >> that > >> >> branch forever; or we need to take ownership of the Protobuf 2.5 code > >> base. > >> >> It has been rightly pointed to me that while all the arguments I make > >> is > >> >> correct; it is a very complicated task to upgrade Protobuf, and the > >> worst > >> >> part is we will not even know what breaks until downstream projects > >> pick up > >> >> these changes and work against us. > >> >> > >> >> If we work off the Hadoop version 3 — and assume that we have > >> "shading" in > >> >> place for all deployments; it might be possible to get there; still a > >> >> daunting task. > >> >> > >> >> So best of luck with the branch approach — But please remember, > Merging > >> >> back will be hard, Just my 2 cents. > >> >> > >> >> — Anu > >> >> > >> >> > >> >> > >> >> > >> >> On Sun, Sep 1, 2019 at 7:40 PM Zhenyu Zheng < > zhengzhenyul...@gmail.com > >> > > >> >> wrote: > >> >> > >> >>> Hi, > >> >>> > >> >>> Thanks Vinaya for bring this up and thanks Sheng for the idea. A > >> separate > >> >>> branch with it's own ARM CI seems a really good idea. > >> >>> By doing this we won't break any of the undergoing development in > >> trunk > >> >> and > >> >>> a CI can be a very good way to show what are the > >> >>> current problems and what have been fixed, it will also provide a > very > >> >> good > >> >>> view for contributors that are intrested to working on > >> >>> this. We can finally merge back the branch to trunk until the > >> community > >> >>> thinks it is good enough and stable enough. We can donate > >> >>> ARM machines to the existing CI system for the job. > >> >>> > >> >>> I wonder if this approch possible? > >> >>> > >> >>> BR, > >> >>> > >> >>>> On Thu, Aug 29, 2019 at 11:29 AM Sheng Liu <liusheng2...@gmail.com > > > >> >>> wrote: > >> >>> > >> >>>> Hi, > >> >>>> > >> >>>> Thanks Vinay for bring this up, I am a member of "Openlab" > community > >> >>>> mentioned by Vinay. I am working on building and > >> >>>> testing Hadoop components on aarch64 server these days, besides the > >> >>> missing > >> >>>> dependices of ARM platform issues #1 #2 #3 > >> >>>> mentioned by Vinay, other similar issue has also be found, such as > >> the > >> >>>> "PhantomJS" dependent package also missing for aarch64. > >> >>>> > >> >>>> To promote the ARM support for Hadoop, we have discussed and hoped > to > >> >> add > >> >>>> an ARM specific CI to Hadoop repo. we are not > >> >>>> sure about if there is any potential effect or confilict on the > trunk > >> >>>> branch, so maybe creating a ARM specific branch for doing these > stuff > >> >>>> is a better choice, what do you think? > >> >>>> > >> >>>> Hope to hear thoughts from you :) > >> >>>> > >> >>>> BR, > >> >>>> Liu sheng > >> >>>> > >> >>>> Vinayakumar B <vinayakum...@apache.org> 于2019年8月27日周二 上午5:34写道： > >> >>>> > >> >>>>> Hi Folks, > >> >>>>> > >> >>>>> ARM is becoming famous lately in its processing capability and has > >> >> got > >> >>>> the > >> >>>>> potential to run Bigdata workloads. > >> >>>>> Many users have been moving to ARM machines due to its low cost. > >> >>>>> > >> >>>>> In the past there were attempts to compile Hadoop on ARM (Rasberry > >> >> PI) > >> >>>> for > >> >>>>> experimental purposes. Today ARM architecture is taking some of > the > >> >>>>> serverside processing as well. So there will be/is a real need of > >> >>> Hadoop > >> >>>> to > >> >>>>> support ARM architecture as well. > >> >>>>> > >> >>>>> There are bunch of users who are trying out building Hadoop on > ARM, > >> >>>> trying > >> >>>>> to add ARM CI to hadoop and facing issues[1]. Also some > >> >>>>> > >> >>>>> As of today, Hadoop does not compile on ARM due to below issues, > >> >> found > >> >>>> from > >> >>>>> testing done in openlab in [2]. > >> >>>>> > >> >>>>> 1. Protobuf : > >> >>>>> ------------------- > >> >>>>> Hadoop project (also some downstream projects) stuck to > protobuf > >> >>>> 2.5.0 > >> >>>>> version, due to backward compatibility reasons. Protobuf-2.5.0 is > >> not > >> >>>> being > >> >>>>> maintained in the community. While protobuf 3.x is being actively > >> >>> adopted > >> >>>>> widely, still protobuf 3.x provides wire compatibility for proto2 > >> >>>> messages. > >> >>>>> Due to some compilation issues in the generated java code, which > can > >> >>>> induce > >> >>>>> problems in downstream. Due to this reason protobuf upgrade from > >> >> 2.5.0 > >> >>>> was > >> >>>>> not taken up. > >> >>>>> In 3.0.0 onwards, hadoop supports shading of libraries to avoid > >> >>> classpath > >> >>>>> problem in downstream projects. > >> >>>>> There are patches available to fix compilation in Hadoop. But > >> >> need > >> >>> to > >> >>>>> find a way to upgrade protobuf to latest version and still > maintain > >> >> the > >> >>>>> downstream's classpath using shading feature of Hadoop build. > >> >>>>> > >> >>>>> There is a Jira for protobuf upgrade[3] created even before > >> >> shade > >> >>>>> support was added to Hadoop. Now need to revisit the Jira and > >> >> continue > >> >>>>> explore possibilities. > >> >>>>> > >> >>>>> 2. leveldbjni: > >> >>>>> --------------- > >> >>>>> Current leveldbjni used in YARN doesnot support ARM > architecture, > >> >>>> need > >> >>>>> to check whether any of the future versions support ARM and can > >> >> hadoop > >> >>>>> upgrade to that version. > >> >>>>> > >> >>>>> > >> >>>>> 3. hadoop-yarn-csi's dependency 'protoc-gen-grpc-java:1.15.1' > >> >>>>> ------------------------- > >> >>>>> 'protoc-gen-grpc-java:1.15.1' does not provide ARM executable by > >> >>> default > >> >>>> in > >> >>>>> the maven repository. Workaround is to build it locally and keep > in > >> >>> local > >> >>>>> maven repository. > >> >>>>> Need to check whether any future versions of > 'protoc-gen-grpc-java' > >> >> is > >> >>>>> having ARM executable and whether hadoop-yarn-csi can upgrade it? > >> >>>>> > >> >>>>> > >> >>>>> Once the compilation issues are solved, then there might be many > >> >> native > >> >>>>> code related issues due to different architectures. > >> >>>>> So to explore everything, need to join hands together and proceed. > >> >>>>> > >> >>>>> > >> >>>>> Let us discuss and check, whether any body else out there who also > >> >> need > >> >>>> the > >> >>>>> support of Hadoop on ARM architectures and ready to lend their > hands > >> >>> and > >> >>>>> time in this work. > >> >>>>> > >> >>>>> > >> >>>>> [1] https://issues.apache.org/jira/browse/HADOOP-16358 > >> >>>>> [2] > >> >> > >> > https://issues.apache.org/jira/browse/HADOOP-16358?focusedCommentId=16904887&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16904887 > >> >>>>> [3] https://issues.apache.org/jira/browse/HADOOP-13363 > >> >>>>> > >> >>>>> -Vinay > >> >> > >> > > >
