Hi Georg, I recently made a demo using flink sql client + schema registry + kafka as well to test my own schema registry catalog. To help us locate the root cause, I think you can add "SET 'sql-client.verbose' = 'true';" in your launched sql client to enable the sql client output more information or go to check the sql client logs under "/log" dir. "Caused by: java.lang.ClassNotFoundException: org.apache.avro.SchemaParseException" may not be the root cause as you have already tried to use -j option to specify flink-avro jar. For me, the root cause is that I miss the jar dependencies of jackson jars. I attach my command for starting sql client here and hope it can give you some hints:
sql-client.sh -j jackson-core-*.jar -j jackson-annotations-*.jar -j jackson-databind-*.jar -j ververica-connector-kafka-*.jar -j flink-avro-*-jar-with-dependencies.jar Note, you may need to substitute ververica-connector-kafka with open sourced flink-sql-connector-kafka jar. For your questions, to my best knowledge, '-j' and '-l' options are the only options for now. Maybe others in the community can provide more information. Best, Biao Geng Georg Heiler <[email protected]> 于2022年3月23日周三 23:59写道： > Hi, > > When trying to set up a demo for the kafka-sql-client reading an Avro > topic from Kafka I run into problems with regards to the additional > dependencies. > In the spark-shell there is a --packages option which automatically > resolves any additional required jars (transitively) using the provided > maven coordinates. So far, I could not find this function for flink. Am > I missing something? > > When now instead of trying to set this up manually I first get the > additional jars (for flink 1.14.1 scala 2.12) which are mentioned here: > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/kafka/ > and > > https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/connectors/table/formats/avro-confluent/ > > wget > https://repo1.maven.org/maven2/org/apache/flink/flink-connector-kafka_2.12/1.14.4/flink-connector-kafka_2.12-1.14.4.jar > -P lib/ > wget > https://repo1.maven.org/maven2/org/apache/kafka/kafka-clients/3.0.0/kafka-clients-3.0.0.jar > -P lib/ > wget > https://repo1.maven.org/maven2/org/apache/flink/flink-avro-confluent-registry/1.14.4/flink-avro-confluent-registry-1.14.4.jar > -P lib/ > wget > https://repo1.maven.org/maven2/org/apache/flink/flink-avro/1.14.4/flink-avro-1.14.4.jar > -P lib/ > > I still fail to get them loaded (even though they are put into the default > lib path. > When starting a local cluster: > > ./bin/start-cluster.sh local > > and the SQL client: > > ./bin/sql-client.sh > > Any option: > ./bin/sql-client.sh -j or ./bin/sql-client.sh -l (with the path to the lib > folder or the additional jars wich were added before) all fails with the > same reason: > > Caused by: java.lang.ClassNotFoundException: > org.apache.avro.SchemaParseException > > when trying to execute: > CREATE TABLE foo (foo string) WITH ( > 'connector' = 'kafka', > 'topic' = 'foo', > 'scan.startup.mode' = 'earliest-offset', > 'format' = 'avro-confluent', > 'avro-confluent.schema-registry.url' = 'http://localhost:8081/', > 'properties.group.id' = 'flink-test-001', > 'properties.bootstrap.servers' = 'localhost:9092' > ); > SELECT * FROM foo; > > (irrespective of any dummy data loaded) Though I have some dummy data in > the following structure available using the Kafka Connect dummy data > generator for the following Avro schema (and serialized using Avro into > Kafkas topic): > > { > "type": "record", > "name": "commercialrating", > "fields": [ > { > "name": "brand", > "type": { > "type": "string", > "arg.properties": { > "options": ["Acme", "Globex"] > } > } > }, > { > "name": "duration", > "type": { > "type": "int", > "arg.properties": { > "options": [30, 45, 60] > } > } > }, > { > "name": "rating", > "type": { > "type": "int", > "arg.properties": { > "range": { "min": 1, "max": 5 } > } > } > } > ] > } > > > *Questions:* > > *1) can I somehow specify maven coordinates directly? (for the naive > method of using the SQL client like in the spark-shell) to simplify the > setup of the required jars?* > > > *2) given the fact that I manually have downloaded the jars into the lib > folder of the flink installation - why are they not loaded by default? What > needs to change so the additional (required) jars for Avro + > confluent-schema-registry + Kafka are loaded by the flink SQL client?* > > Best, > Georg > > > >
