[
https://issues.apache.org/jira/browse/MESOS-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13973157#comment-13973157
]
Benjamin Hindman commented on MESOS-1203:
-----------------------------------------
I took a step back this morning to try to understand the issues with protobuf
and Mesos and other projects and I'm not sure shading is going to solve any of
our problems.
The way protobuf is used in Mesos right now is to describe our _public_ types.
Furthermore, we include the _generated_ code for our protobufs in our JAR that
we expect people to depend on. So if we build against protobuf 2.4.1, we
include generated code for 2.4.1, likewise if we build against 2.5.0, we
include generated code for 2.5.0. IIUC there is no way we can _shade_ this
generated code as it is part of the public interface!
Interestingly enough, we user very few methods from protobuf in Java, we only
rely on 'toByteArray' and 'parseFrom', neither of which have changed (IIUC)
from protobuf 2.4.1 to protobuf 2.5.0. Moreover, these methods get called on
the protobufs from JNI code, at which point whichever protobuf library that is
loaded will get used. Said another way, the JNI code doesn't have a hard
dependency on either protobuf 2.4.1 or protobuf 2.5.0, as it's just dynamically
looking up and using the methods on what ever version of the protobuf that has
been loaded. Said yet another way, we "serialize" the protobufs between Java
and C++, so even if the C++ library was compiled against protobuf 2.4.1 the
Java code can be using protobuf 2.5.0 because we're just using the
'toByteArray' and 'parseFrom' methods to do the serialization/deserialization.
That is, we're already "shaded" through serialization.
So, why can't we simply ship two versions of the jar for each release? For
example, a mesos-0.16.0-protobuf-2.4.1.jar and a
mesos-0.16.0-protobuf.2.5.0.jar?
That's basically saying to a developer: if you're going to bring in protobuf
2.4.1 then depend on mesos-x.y.z-protobuf-2.4.1.jar and if your'e going to
bring in protobuf 2.5.0 then depend on mesos-x.y.z-protobuf-2.5.0.jar since we
expose compiled protos that were compiled with one of these libraries.
Another way to look at it would be to NOT include the org.apche.mesos.Protos.*
in the mesos-x.y.z.jar, but have a separate jar called
mesos-x.y.z-protos-2.4.1.jar which would only include the
org.apache.mesos.Protos.* that were built for 2.4.1. For simplicity I figured
there wasn't that much advantage in actually splitting this out though.
Unless I'm missing something here, this could be done TODAY for every release
without any issues. That is, even though mesos-0.16.0 shipped with protobuf
2.4.1, there should be no reason why we can't ship a
mesos-0.16.0-protobuf-2.5.0.jar since from our interface and JNI code there
shouldn't be any hard requirements on protobuf 2.5.0.
Am I missing something here? Feedback requested please.
> Shade protobuf dependency in Mesos Java library
> -----------------------------------------------
>
> Key: MESOS-1203
> URL: https://issues.apache.org/jira/browse/MESOS-1203
> Project: Mesos
> Issue Type: Improvement
> Components: build
> Reporter: Patrick Wendell
>
> Mesos's Java library uses the protobuf library which is also used by Hadoop.
> Unfortunately the protobuf library does not provide binary compatiblity
> between minor versions (for code compiled against 2.4.1 and 2.5.0 cannot run
> together in a single JVM classlaoder) .
> This makes use of Mesos via it's Java API, something that is required for
> Spark and I'm assuming other frameworks, fundamentally incompatible for
> certain Hadoop versions.
> Mesos could shade this jar using the maven shade plug-in. Take a look at the
> Parquet project for an example of shading:
> https://github.com/Parquet/parquet-format/blob/master/pom.xml#L140
> Without this fix Java users won't be able to use Mesos (< 0.17) with newer
> versions of Hadoop. Or Mesos 0.17+ with older versions of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
