[
https://issues.apache.org/jira/browse/HADOOP-13363?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16925234#comment-16925234
]
Vinayakumar B commented on HADOOP-13363:
----------------------------------------
Hi [~anu] and [~stack].
Apologies: Long reply ahead!!!
I looked at both HBase and Ozone's way of shading and relocating the generated
protobuf classes.
Technically, the latest PR does the samething. Features of both plugins (1.
downloading appropriate protoc exe, 2. replacing package with shaded prefix in
generated classes) is merged to hadoop-maven-plugin itself. regex to relocate
is hard coded only for protobuf, since the goal is specific to protoc.
Anyway I am open to replace the "hadoop-maven-plugin" with
"protoc-maven-plugin" and "replacer" plugins.
Problem is not in the Generated classes, but the dependency with shaded
protobuf classes.
In HBase and Ozone, shaded dependency is maintained in a separate repo.
[hbase-thirdparty|https://github.com/apache/hbase-thirdparty] for hbase and
[incubator-ratis-thirdparty|https://github.com/apache/incubator-ratis-thirdparty]
for ozone, its not maintained as a submodule.
Above compilation Problem is with only "mvn compile".
"mvn compile" will not download or refer to local repo for submodules from mvn
execution. It just looks for the compile classes.
But "mvn package" also will not refer to local repo for submodules, but it
builds and uses compiled jars for dependency. Here, shaded classes will be
included as well.
So current problem is with only "mvn compile" at root level.
To verify this, I have manually built *hadoop-shaded-thirdparty-dependency* and
deployed to central snapshot repo and excluded this module from normal build by
moving into separate profile in Latest commit.
This hack seems to have passed the compile phase in yetus test. But jenkins
itself timedout after 5hours, due to unit testing of all modules.
This is the latest build for PR with
[https://builds.apache.org/job/hadoop-multibranch/job/PR-1408/5/]
There are many javac warnings due to new protobuf-3.6.1 dependency due to
deprecated APIs usage.
To Reduce the jenkins time and to reduce huge changes together, following may
be the options.
# (Current way) Combine protobuf dependency upgrade + shading in a single PR
to reduce the impact for the other developments, but need to allow yetus to
take its own time for completion of tests.
#
** Upgrade protobuf dependency to 3.6.1 in hadoop-project/pom.xml (version
bump + compile fix) in a separate Jira
** Create separate module/repo to shade the dependencies. (com.google.protobuf)
** Relocate usage of protobuf module-by-module in module specific Jiras.
** Bring back protobuf dependency to 2.5.0 in hadoop-project/pom.xml for
downstreams.
# Do entire step #2 in a separate branch (just avoid impact on current
development) and merge back immediately to trunk once done.
Here I am fine with #2 or even #3.
Decision needs to be taken whether to keep
*hadoop-shaded-thirdparty-dependency* as a separate repo (like in hbase and
ozone/ratis) or separate sub-module within hadoop.
# Separate repo: comes with its own maintainance cost.
# Submodule-activated-on-profile, less maintainance as its within same repo,
but requires build steps changes. i.e. This module shoud be installed/packaged
first before otrhers. Also need to change yetus script.
Here I am leaning towards #2.
Any thoughts?
> Upgrade protobuf from 2.5.0 to something newer
> ----------------------------------------------
>
> Key: HADOOP-13363
> URL: https://issues.apache.org/jira/browse/HADOOP-13363
> Project: Hadoop Common
> Issue Type: Improvement
> Components: build
> Affects Versions: 3.0.0-alpha1, 3.0.0-alpha2
> Reporter: Allen Wittenauer
> Assignee: Vinayakumar B
> Priority: Major
> Labels: security
> Attachments: HADOOP-13363.001.patch, HADOOP-13363.002.patch,
> HADOOP-13363.003.patch, HADOOP-13363.004.patch, HADOOP-13363.005.patch
>
>
> Standard protobuf 2.5.0 does not work properly on many platforms. (See, for
> example, https://gist.github.com/BennettSmith/7111094 ). In order for us to
> avoid crazy work arounds in the build environment and the fact that 2.5.0 is
> starting to slowly disappear as a standard install-able package for even
> Linux/x86, we need to either upgrade or self bundle or something else.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
