[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806797#comment-17806797
]
Danny Cranmer commented on FLINK-34076:
---
[~jiabao.sun] Ah, I missed that. Interesting.
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806783#comment-17806783
]
Jiabao Sun commented on FLINK-34076:
Hey [~dannycranmer], it's in dependencies.
!screenshot-4.png!
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png,
> screenshot-4.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806756#comment-17806756
]
Danny Cranmer commented on FLINK-34076:
---
> flink-connector-base has already been included as a parent dependency, and
> the submodules will also inherit this dependency.
This is in {{dependencyManagement}} so it will only take effect if there is a
{{dependency}} in the child pom.
It sounds like the problem/request is a convenience change for IntelliJ to add
a {{provided}} scoped dependency. [~khanhvu] can you confirm that adding the
{{provided}} dependency helps IntelliJ configure the project? If this is the
case, given there is no impact on the output jar, I +1 the change.
Alternatively we can promote {{flink-connector-base}} from
{{dependencyManagement}} to {{dependencies}} in the parent (as provided).
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806747#comment-17806747
]
Jiabao Sun commented on FLINK-34076:
flink-connector-base has already been included as a parent dependency, and the
submodules will also inherit this dependency.
[https://github.com/apache/flink-connector-aws/blob/38aafb44d3a8200e4ff41d87e0780338f40da258/pom.xml#L141-L146]
[https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kinesis/4.2.0-1.18/flink-connector-kinesis-4.2.0-1.18.pom]
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806736#comment-17806736
]
Khanh Vu commented on FLINK-34076:
--
The `flink-connector-base` is in `flink-connector-aws-base` dependency tree
([pom.xml|https://github.com/apache/flink-connector-aws/blob/v4.2/flink-connector-aws-base/pom.xml#L46]).
But because `flink-connector-aws-base` is
[shaded|https://github.com/apache/flink-connector-aws/blob/v4.2/flink-connector-aws/flink-connector-kinesis/pom.xml#L322]
in `flink-connector-kinesis` uber jar, the `flink-connector-base` is not
present in `flink-kinesis-connector`'s dependency tree anymore.
I think a simple fix here is to add `flink-connector-base` with `provided`
scope as direct dependency of `flink-connector-kinesis`.
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806727#comment-17806727
]
Khanh Vu commented on FLINK-34076:
--
> In fact, users do not need to explicitly declare the flink-connector-base
> dependency in the pom.xml file
Have you verified this? it does not work for me with 4.2.0-1.18 version.
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806725#comment-17806725
]
Khanh Vu commented on FLINK-34076:
--
hm, I dont see it's there:
https://github.com/apache/flink-connector-aws/blob/v4.2/flink-connector-aws/flink-connector-kinesis/pom.xml
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806723#comment-17806723
]
Jiabao Sun commented on FLINK-34076:
The purpose of "Stop bundling connector-base in externalized connectors" is to
prevent external connectors from having compile-time dependencies on specific
versions of flink-connector-base, but instead obtain it at runtime from
flink-dist to achieve better compatibility.
In fact, users do not need to explicitly declare the flink-connector-base
dependency in the pom.xml file. It is already included in the dependency tree
of the Kinesis connector and declared with a "provided" scope. Users only need
to add the provided dependency to the classpath to run the testing.
!screenshot-3.png!
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png, screenshot-3.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806720#comment-17806720
]
Khanh Vu commented on FLINK-34076:
--
I'm aware of the IntelliJ feature, it wont work with the fix I mentioned if I
dont enable it in IntelliJ.
I'm aware of the FLINK-30400 too, and I raised this one as we're creating the
issue for kinesis connector by the change. I was not aware when the decision
was made, but IMO it's the issue with the base, and we should not solve the
issue by transferring it to concrete connectors. For things which likely breaks
compatibility, it should be abstracted away from the base, here the coupling is
still there, and we cut the connection.
bq. The flink-connector-base is already included in flink-dist
You're not declaring `flink-dist` in your app's pom.xml, are you? it wont be
issue with prod environment, but it's an issue with local development. We have
a workaround, but it's a bad user experience to use the connector.
Again, it's programmatically incorrect as kinesis connector depends on
connector base to execute, but it's not in the dependency tree of kinesis
connector?! How am I (as a user) suppose to know I need to declare
flink-connector-base in my app pom.xml before I execute the app?
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806677#comment-17806677
]
Jiabao Sun commented on FLINK-34076:
Hi [~khanhvu],
The flink-connector-base is already included in flink-dist and we will not
package it in the externalized connectors[1].
The dependencies of "provided" will also be passed on. You just need to check
"add dependencies with 'provided' scope to classpath" in IDEA.
{code:xml}
1.17.0
4.2.0-1.17
2.12
org.apache.flink
flink-connector-kinesis
${flink.connector.kinesis.version}
org.apache.flink
flink-streaming-java
${flink.version}
provided
org.apache.flink
flink-table-api-java-bridge
${flink.version}
provided
org.apache.flink
flink-table-planner-loader
${flink.version}
provided
org.apache.flink
flink-table-runtime
${flink.version}
provided
{code}
!screenshot-2.png!
[1] https://issues.apache.org/jira/browse/FLINK-30400
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png, screenshot-2.png
>
>
> The following issue encounters with flink-kinesis-connector v4.2.0, Flink
> 1.17, it's working properly with kinesis connector v4.1.0 (I have not tested
> version pre v4.1.0).
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806669#comment-17806669
]
Khanh Vu commented on FLINK-34076:
--
Hi [~jiabao.sun],
Yes, if I have `flink-connector-base` in the dependency list, it will run
properly (it's the fix I mentioned), but if I leave the base out, it's failing.
Before the aforementioned commit, I just need to have `flink-connector-kinesis`
in the list (not along with the `flink-connector-base`).
It's actually not correct when the connector depends on `flink-connector-base`
to execute but does not have it as (transitive) dependency.
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png
>
>
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806605#comment-17806605
]
Jiabao Sun commented on FLINK-34076:
Hi [~khanhvu], do you add dependencies with "provided" scope to classpath?
I can run correctly locally.
{code:xml}
org.apache.flink
flink-connector-kinesis
${flink.connector.kinesis.version}
org.apache.flink
flink-connector-base
${flink.version}
provided
org.apache.flink
flink-streaming-java
${flink.version}
provided
org.apache.flink
flink-table-api-java-bridge
${flink.version}
provided
org.apache.flink
flink-table-planner-loader
${flink.version}
provided
org.apache.flink
flink-table-runtime
${flink.version}
provided
{code}
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png
>
>
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34076) flink-connector-base missing fails kinesis table sink to create
[
https://issues.apache.org/jira/browse/FLINK-34076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17806603#comment-17806603
]
Jiabao Sun commented on FLINK-34076:
Hi [~khanhvu], do you add dependencies with "provided" scope to classpath?
!screenshot-1.png!
> flink-connector-base missing fails kinesis table sink to create
> ---
>
> Key: FLINK-34076
> URL: https://issues.apache.org/jira/browse/FLINK-34076
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: aws-connector-4.2.0
>Reporter: Khanh Vu
>Priority: Major
> Attachments: screenshot-1.png
>
>
> The
> [commit|https://github.com/apache/flink-connector-aws/commit/01f112bd5a69f95cd5d2a4bc7e08d1ba9a81d56a]
> which stops bundling `flink-connector-base` with `flink-connector-kinesis`
> has caused kinesis sink failing to create when using Table API as required
> classes from `flink-connector-base` are not loaded in runtime.
> E.g. with following depenency only in pom.xml
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
> {code}
> and a minimal job definition:
> {code:java}
> public static void main(String[] args) throws Exception {
> // create data stream environment
> StreamExecutionEnvironment sEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> sEnv.setRuntimeMode(RuntimeExecutionMode.STREAMING);
> StreamTableEnvironment tEnv =
> StreamTableEnvironment.create(sEnv);
> Schema a = Schema.newBuilder().column("a",
> DataTypes.STRING()).build();
> TableDescriptor descriptor =
> TableDescriptor.forConnector("kinesis")
> .schema(a)
> .format("json")
> .build();
> tEnv.createTemporaryTable("sinkTable", descriptor);
> tEnv.executeSql("CREATE TABLE sinkTable " +
> descriptor.toString()).print();
> }
> {code}
> following exception will be thrown:
> {code:java}
> Caused by: java.lang.ClassNotFoundException:
> org.apache.flink.connector.base.table.AsyncDynamicTableSinkFactory
> at
> jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
> ~[?:?]
> at
> jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> ~[?:?]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:527) ~[?:?]
> ... 28 more
> {code}
> The fix is to explicitly specify `flink-connector-base` as dependency of the
> project:
> {code:java}
>
> org.apache.flink
> flink-connector-kinesis
> ${flink.connector.kinesis.version}
>
>
> org.apache.flink
> flink-connector-base
> ${flink.version}
> provided
>
> {code}
> In general, `flink-connector-base` should be pulled in by default when
> pulling in the kinesis connector, the current separation adds unnecessary
> hassle to use the connector.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
