Re: Support avro-confluent format for FileSystem SQL connector

2022-10-06 Thread liuxiangcao
sorry I missed the references. [1] https://github.com/apache/flink/blob/fa161d3c5636370f5129320a9ca464e38f88fc6f/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/AvroFileFormatFactory.java [2] https://github.com/apache/flink/blob/fa161d3c5636370f5129320a9ca464e38f88fc6f/flink-f

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-06 Thread liuxiangcao
In the meantime, I looked into the code of avro-confluent format and compared it with avro format. The *avro* format implemented *BulkWriterFormatFactory & BulkReaderFormatFactory* (see [1]) for file system connector and also *DeserializationFormatFactory & SerializationFormatFactory* for other c

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-06 Thread liuxiangcao
Actually I got the previous option validation issue resolved after adding 'avro-confluent.url' = 'https://confluent-schema-registry-url', 'avro-confluent.subject' = '' Now I'm one step closer to verifying whether avro-confluent format works with filesystem connector or not. Unfortuantely,

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-06 Thread liuxiangcao
typo in my first sentence: I actually also noticed the reference of filesystem connector in the avro-confluent format doc which is confusing. On Thu, Oct 6, 2022 at 1:24 PM liuxiangcao wrote: > Hi Martin, > > Thank you for the reply. I actually also noticed the reference of > avro-confluent.subj

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-06 Thread liuxiangcao
Hi Martin, Thank you for the reply. I actually also noticed the reference of avro-confluent.subject in the avro-confluent format doc which is confusing. I tried using 'connector' = 'filesystem' and 'format' = 'avro-confluent' and it would fail with options validation when I tried to insert data

Re: Support avro-confluent format for FileSystem SQL connector

2022-10-04 Thread Martijn Visser
Hi, I'm wondering if the documentation is correct, because the documentation also mentions the option to use 'avro-confluent.subject' which refers to using it with filesystem [1]. Have you tried to use this already? Did you get an error message? Best regards, Martijn [1] https://nightlies.apac

Support avro-confluent format for FileSystem SQL connector

2022-10-04 Thread liuxiangcao
Hi Flink developer community, According to flink doc, avro-confluent([1]) is only supported for kafka sql connector and upsert kafka sql connector. I'm wondering if there is any reason this format is not supported for Filesystem sql connector ([2]) ? We are looking to use FileSystem sink to wri