Re: Kinesis Connector does not work
This is a general thing; see https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/connector/ The python documentation isn't particularly clear on how to use Java connectors. The easiest thing would be to use the "sql-*" connector jars I guess. On 08/11/2022 11:49, Matt Fysh wrote: Ok thanks, will give that a try. Is that something that should be added to the Kinesis connector docs page? There are existing instructions there for adding the flink-connector-kinesis jar as a dependency, but no instructions for adding commons-logging Or if this is something more general, it might be something to talk about in the Python section of the docs because most Python users are not going to understand the interplay between Java classes. On Tue, 8 Nov 2022 at 18:50, Chesnay Schepler wrote: Said dependency (on commons-logging) is not meant to be provided by the docker image, but bundled in your user-jar (along with the connector). On 08/11/2022 02:14, Matt Fysh wrote: > Hi, I'm following the kinesis connector instructions as documented > here: > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/ > > > I'm also running Flink in standalone session mode using docker compose > and the Python images, as described in the Flink docs (Deployment section) > > When I try to run a basic datastream.print() / env.execute() example > with a kinesis source, I get the following error. From my limited > understanding of Java, it seems the Kinesis connector is using a > shaded version of the AWS Java SDK, and that older version of the SDK > is trying to load a class that is no longer present in the 1.16.0 > Flink docker images. Is there a workaround for this? Thanks > > Caused by: java.lang.NoClassDefFoundError: > org/apache/commons/logging/LogFactory > at > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration.(ClientConfiguration.java:47) > at > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getDefaultConfig(ClientConfigurationFactory.java:46) > at > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36) > at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.createKinesisClient(KinesisProxy.java:268) > at > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.(KinesisProxy.java:152)
Re: Kinesis Connector does not work
Ok thanks, will give that a try. Is that something that should be added to the Kinesis connector docs page? There are existing instructions there for adding the flink-connector-kinesis jar as a dependency, but no instructions for adding commons-logging Or if this is something more general, it might be something to talk about in the Python section of the docs because most Python users are not going to understand the interplay between Java classes. On Tue, 8 Nov 2022 at 18:50, Chesnay Schepler wrote: > Said dependency (on commons-logging) is not meant to be provided by the > docker image, but bundled in your user-jar (along with the connector). > > On 08/11/2022 02:14, Matt Fysh wrote: > > Hi, I'm following the kinesis connector instructions as documented > > here: > > > https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/ > > > > > > I'm also running Flink in standalone session mode using docker compose > > and the Python images, as described in the Flink docs (Deployment > section) > > > > When I try to run a basic datastream.print() / env.execute() example > > with a kinesis source, I get the following error. From my limited > > understanding of Java, it seems the Kinesis connector is using a > > shaded version of the AWS Java SDK, and that older version of the SDK > > is trying to load a class that is no longer present in the 1.16.0 > > Flink docker images. Is there a workaround for this? Thanks > > > > Caused by: java.lang.NoClassDefFoundError: > > org/apache/commons/logging/LogFactory > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration.(ClientConfiguration.java:47) > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getDefaultConfig(ClientConfigurationFactory.java:46) > > at > > > org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36) > > at > > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.createKinesisClient(KinesisProxy.java:268) > > at > > > org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.(KinesisProxy.java:152) > > >
Re: Kinesis Connector does not work
Said dependency (on commons-logging) is not meant to be provided by the docker image, but bundled in your user-jar (along with the connector). On 08/11/2022 02:14, Matt Fysh wrote: Hi, I'm following the kinesis connector instructions as documented here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/ I'm also running Flink in standalone session mode using docker compose and the Python images, as described in the Flink docs (Deployment section) When I try to run a basic datastream.print() / env.execute() example with a kinesis source, I get the following error. From my limited understanding of Java, it seems the Kinesis connector is using a shaded version of the AWS Java SDK, and that older version of the SDK is trying to load a class that is no longer present in the 1.16.0 Flink docker images. Is there a workaround for this? Thanks Caused by: java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration.(ClientConfiguration.java:47) at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getDefaultConfig(ClientConfigurationFactory.java:46) at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36) at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.createKinesisClient(KinesisProxy.java:268) at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.(KinesisProxy.java:152)
Kinesis Connector does not work
Hi, I'm following the kinesis connector instructions as documented here: https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/connectors/datastream/kinesis/ I'm also running Flink in standalone session mode using docker compose and the Python images, as described in the Flink docs (Deployment section) When I try to run a basic datastream.print() / env.execute() example with a kinesis source, I get the following error. From my limited understanding of Java, it seems the Kinesis connector is using a shaded version of the AWS Java SDK, and that older version of the SDK is trying to load a class that is no longer present in the 1.16.0 Flink docker images. Is there a workaround for this? Thanks Caused by: java.lang.NoClassDefFoundError: org/apache/commons/logging/LogFactory at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfiguration.(ClientConfiguration.java:47) at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getDefaultConfig(ClientConfigurationFactory.java:46) at org.apache.flink.kinesis.shaded.com.amazonaws.ClientConfigurationFactory.getConfig(ClientConfigurationFactory.java:36) at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.createKinesisClient(KinesisProxy.java:268) at org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.(KinesisProxy.java:152)
