[
https://issues.apache.org/jira/browse/MESOS-1203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13976063#comment-13976063
]
Benjamin Hindman commented on MESOS-1203:
-----------------------------------------
Thanks for the comments guys!
[~tknaup], what were the errors you were running into?
IIUC, the user experience is only better with shading _until_ you need to use
one of the generated protobuf objects that expects a class from the
com.google.protobuf package, because at that point you'll need update your code
to use org.apache.mesos.com.google.protobuf instead.
For example, once Spark wants to use Credential in order to do authentication
we'll need to go update the Spark code to use
org.apache.mesos.com.google.protobuf.ByteString instead of just
com.google.protobuf.ByteString. This is likely why it's been a better user
experience for you right now [~berngp].
Since most frameworks will likely want to do this, that means most frameworks
will be forced to use our shaded jar (forever). I suppose the upside is that
they'll only have to do this once? But if they actually do use protobuf in
their code they might find themselves needing to convert between
org.apache.mesos.com.google.protobuf and com.google.protobuf (if they
explicitly bring in protobuf themselves) or always use
org.apache.mesos.com.google.protobuf (which feels a bit dirty).
If this is less of a PITA then asking the user to determine which protobuf
version they want to use, then we should definitely go for it. Since Mesos
expects people to use things from com.google.protobuf my expectation was that
frameworks would explicitly call out the protobuf dependency themselves, and
thus they could just pick protobuf 2.4.1 or protobuf 2.5.0 and thus depend on
the appropriate Mesos jar.
It sounds like in Hadoop's case this _is_ a PITA because the protobuf
dependency is based on the Hadoop version, so it would require a bunch of build
time magic. Gross.
Okay, so assuming that we now know all the tradeoffs (and maybe you guys
already did and I'm just the slow one, I just wanted to make sure), it would be
great to hear everyone's vote for moving forward.
> Shade protobuf dependency in Mesos Java library
> -----------------------------------------------
>
> Key: MESOS-1203
> URL: https://issues.apache.org/jira/browse/MESOS-1203
> Project: Mesos
> Issue Type: Improvement
> Components: build
> Reporter: Patrick Wendell
>
> Mesos's Java library uses the protobuf library which is also used by Hadoop.
> Unfortunately the protobuf library does not provide binary compatiblity
> between minor versions (for code compiled against 2.4.1 and 2.5.0 cannot run
> together in a single JVM classlaoder) .
> This makes use of Mesos via it's Java API, something that is required for
> Spark and I'm assuming other frameworks, fundamentally incompatible for
> certain Hadoop versions.
> Mesos could shade this jar using the maven shade plug-in. Take a look at the
> Parquet project for an example of shading:
> https://github.com/Parquet/parquet-format/blob/master/pom.xml#L140
> Without this fix Java users won't be able to use Mesos (< 0.17) with newer
> versions of Hadoop. Or Mesos 0.17+ with older versions of Hadoop.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
