[
https://issues.apache.org/jira/browse/SPARK-22380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16223688#comment-16223688
]
Sean Owen commented on SPARK-22380:
-----------------------------------
The problem is that the tightly-coupled Hadoop dependencies depend on a 2.x
version, as far as I know. You'd have to research what's compatible with
Hadoop. You can always shade your dependencies.
> Upgrade protobuf-java (com.google.protobuf) version from 2.5.0 to 3.4.0
> -----------------------------------------------------------------------
>
> Key: SPARK-22380
> URL: https://issues.apache.org/jira/browse/SPARK-22380
> Project: Spark
> Issue Type: Dependency upgrade
> Components: Deploy
> Affects Versions: 1.6.1, 2.2.0
> Environment: Cloudera 5.13.x
> Spark 2.2.0.cloudera1-1.cdh5.12.0.p0.142354
> And anything beyond Spark 2.2.0
> Reporter: Maziyar PANAHI
> Priority: Blocker
>
> Hi,
> This upgrade is needed when we try to use CoreNLP 3.8 with Spark (1.6+ and
> 2.2+) due to incompatibilities in the protobuf version used by
> com.google.protobuf and the one is used in latest Stanford CoreNLP (3.8). The
> version of protobuf has been set to 2.5.0 in the global properties, and this
> is stated in the pom.xml file.
> The error that refers to this dependency:
> {code:java}
> java.lang.VerifyError: Bad type on operand stack
> Exception Details:
> Location:
>
> com/google/protobuf/GeneratedMessageV3$ExtendableMessage.getExtension(Lcom/google/protobuf/GeneratedMessage$GeneratedExtension;I)Ljava/lang/Object;
> @3: invokevirtual
> Reason:
> Type 'com/google/protobuf/GeneratedMessage$GeneratedExtension' (current
> frame, stack[1]) is not assignable to 'com/google/protobuf/ExtensionLite'
> Current Frame:
> bci: @3
> flags: { }
> locals: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> stack: { 'com/google/protobuf/GeneratedMessageV3$ExtendableMessage',
> 'com/google/protobuf/GeneratedMessage$GeneratedExtension', integer }
> Bytecode:
> 0x0000000: 2a2b 1cb6 0024 b0
> at edu.stanford.nlp.simple.Document.<init>(Document.java:433)
> at edu.stanford.nlp.simple.Sentence.<init>(Sentence.java:118)
> at edu.stanford.nlp.simple.Sentence.<init>(Sentence.java:126)
> ... 56 elided
> {code}
> Is it possible to upgrade this dependency to the latest (3.4) or any
> workaround besides manually removing protobuf-java-2.5.0.jar and adding
> protobuf-java-3.4.0.jar?
> You can follow the discussion of how this upgrade would fix the issue:
> https://github.com/stanfordnlp/CoreNLP/issues/556
> Many thanks,
> Maziyar
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
