[jira] [Commented] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756671#comment-17756671
]
Benchao Li commented on FLINK-32418:
[~m-kay] Sorry that I missed this one, I'll try to give it a review in the
following days, thanks for your patience.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Assignee: Michael Kreis
>Priority: Major
> Labels: pull-request-available
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17756660#comment-17756660
]
Michael Kreis commented on FLINK-32418:
---
[~libenchao] Could you may have a look at the pull-request?
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Assignee: Michael Kreis
>Priority: Major
> Labels: pull-request-available
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17739266#comment-17739266
]
Benchao Li commented on FLINK-32418:
Agree with [~yuxia] , connectors and formats should use the classloader from
the `DynamicTableFactory.Context`, instead of using thread local context
classloader.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Priority: Major
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17738524#comment-17738524
]
Michael Kreis commented on FLINK-32418:
---
For the deserialisation this works correctly because the correct classloader is
set in the context via the TemporaryClassLoaderContext. However this exception
is thrown in a validation phase where the classloader is not even available
from the context. I have added the [^full-stacktrace.log] to better see where
the exception is coming from.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Priority: Major
> Attachments: full-stacktrace.log
>
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32418) ClassNotFoundException when using flink-protobuf with sql-client
[
https://issues.apache.org/jira/browse/FLINK-32418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17736812#comment-17736812
]
luoyuxia commented on FLINK-32418:
--
I think `PbFormatUtils` should use passed userclassloader to load the class.
> ClassNotFoundException when using flink-protobuf with sql-client
>
>
> Key: FLINK-32418
> URL: https://issues.apache.org/jira/browse/FLINK-32418
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Client
>Affects Versions: 1.16.2
>Reporter: Michael Kreis
>Priority: Major
>
> When the protobuf format in the kafka connector is used via the sql-client it
> is not able to load the generated protobuf classes which are either passed
> via `-j /protobuf-classes.jar` or added in the script via ADD JAR
> '/protobuf-classes.jar'. The SHOW JARS command prints that the jar is loaded
> but when the protobuf classes are loaded a ClassNotFoundException occurs.
> executed command:
> {code:java}
> sql-client.sh -f protobuf-table.sql -j /protobuf-classes.jar
> {code}
> protobuf-table.sql
> {code:sql}
> ADD JAR '/opt/sql-client/lib/flink-sql-connector-kafka-1.16.2.jar';
> ADD JAR '/opt/sql-client/lib/flink-protobuf-1.16.2.jar';
> SHOW JARS;
> CREATE TABLE POSITIONS(id BIGINT) WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'topic' = 'protbuf-topic',
> 'properties.bootstrap.servers' = 'kafka:9092',
> 'properties.group.id' = 'flink-protobuf',
> 'properties.security.protocol' = 'SASL_PLAINTEXT',
> 'properties.sasl.mechanism' = 'SCRAM-SHA-512',
> 'properties.sasl.jaas.config' =
> 'org.apache.kafka.common.security.scram.ScramLoginModule required
> username="user" password="";',
> 'scan.startup.mode' = 'earliest-offset',
> 'protobuf.message-class-name' = 'com.example.protobuf.ProtoMessage',
> 'protobuf.ignore-parse-errors' = 'true'
> );
> SELECT * FROM POSITIONS;
> {code}
> exception in the log:
> {code:java}
> Caused by: java.lang.ClassNotFoundException: com.example.protobuf.ProtoMessage
> at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(Unknown
> Source)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(Unknown
> Source)
> at java.base/java.lang.ClassLoader.loadClass(Unknown Source)
> at java.base/java.lang.Class.forName0(Native Method)
> at java.base/java.lang.Class.forName(Unknown Source)
> at
> org.apache.flink.formats.protobuf.util.PbFormatUtils.getDescriptor(PbFormatUtils.java:89)
> ... 36 more
> {code}
> This also seems somehow related to FLINK-30318
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
