[
https://issues.apache.org/jira/browse/FLINK-20350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Danny Cranmer updated FLINK-20350:
----------------------------------
Description:
*Problem*
Kinesis and GCP PubSub connector do not work together. The following error is
thrown.
{code}
java.lang.NoClassDefFoundError: Could not initialize class
io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder
at
org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:52)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:213)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:102)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
~[flink-core-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_252]
{code}
{code}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId>
<version>1.11.1</version>
</dependency>
{code}
*Cause*
This is caused by a Guava dependency conflict:
- Kinesis Consumer > {{18.0}}
- GCP PubSub > {{26.0-android}}
{{NettyChannelBuilder}} fails to initialise due to missing method in guava:
-
{{com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V}}
*Possible Fixes*
- Align Guava versions
- Shade Guava in either connector
was:
*Problem*
Kinesis and GCP PubSub connector do not work together. The following error is
thrown.
{code}
java.lang.NoClassDefFoundError: Could not initialize class
io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder
at
org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:52)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:213)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:102)
~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
~[flink-core-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
~[flink-runtime_2.11-1.11.1.jar:1.11.1]
at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_252]
{code}
{code}
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
<version>1.11.1</version>
</dependency>
<dependency>
<groupId>org.apache.flink</groupId>
<artifactId>flink-streaming-java_${scala.binary.version}</artifactId>
<version>1.11.1</version>
</dependency>
{code}
*Cause*
This is caused by a Guava dependency conflict:
- Kinesis Consumer > {{18.0}}
- GCP PubSub > {{26.0-android}}
{{NettyChannelBuilder}} fails to initialise due to missing method in guava:
-
{{com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V}}
*Possible Fixes*
- Align Guava versions
- Shade Guava in either connector
> [Kinesis][GCP PubSub] Incompatible Connectors due to Guava conflict
> -------------------------------------------------------------------
>
> Key: FLINK-20350
> URL: https://issues.apache.org/jira/browse/FLINK-20350
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Google Cloud PubSub, Connectors / Kinesis
> Affects Versions: 1.11.1, 1.11.2
> Reporter: Danny Cranmer
> Priority: Major
>
> *Problem*
> Kinesis and GCP PubSub connector do not work together. The following error is
> thrown.
> {code}
> java.lang.NoClassDefFoundError: Could not initialize class
> io.grpc.netty.shaded.io.grpc.netty.NettyChannelBuilder
> at
> org.apache.flink.streaming.connectors.gcp.pubsub.DefaultPubSubSubscriberFactory.getSubscriber(DefaultPubSubSubscriberFactory.java:52)
> ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.createAndSetPubSubSubscriber(PubSubSource.java:213)
> ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.connectors.gcp.pubsub.PubSubSource.open(PubSubSource.java:102)
> ~[flink-connector-gcp-pubsub_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.api.common.functions.util.FunctionUtils.openFunction(FunctionUtils.java:36)
> ~[flink-core-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.open(AbstractUdfStreamOperator.java:102)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.runtime.tasks.OperatorChain.initializeStateAndOpenOperators(OperatorChain.java:291)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$beforeInvoke$0(StreamTask.java:473)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$SynchronizedStreamTaskActionExecutor.runThrowing(StreamTaskActionExecutor.java:92)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.beforeInvoke(StreamTask.java:469)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:522)
> ~[flink-streaming-java_2.11-1.11.1.jar:1.11.1]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:721)
> ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:546)
> ~[flink-runtime_2.11-1.11.1.jar:1.11.1]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_252]
> {code}
> {code}
> <dependency>
> <groupId>org.apache.flink</groupId>
>
> <artifactId>flink-connector-gcp-pubsub_${scala.binary.version}</artifactId>
> <version>1.11.1</version>
> </dependency>
> <dependency>
> <groupId>org.apache.flink</groupId>
>
> <artifactId>flink-connector-kinesis_${scala.binary.version}</artifactId>
> <version>1.11.1</version>
> </dependency>
> {code}
> *Cause*
> This is caused by a Guava dependency conflict:
> - Kinesis Consumer > {{18.0}}
> - GCP PubSub > {{26.0-android}}
> {{NettyChannelBuilder}} fails to initialise due to missing method in guava:
> -
> {{com.google.common.base.Preconditions.checkArgument(ZLjava/lang/String;CLjava/lang/Object;)V}}
> *Possible Fixes*
> - Align Guava versions
> - Shade Guava in either connector
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
