Hi Hongjian Peng ~
For your question 1, it is not work as expected. If it is true, there is
definitely a bug. I would check and fix it later.
For your question 2, yes. This is an intent design. There is a routine in
the type inference: all the fields of a nullable struct type should also be
nullable.
Hongjian Peng <super...@163.com>于2020年11月25日 周三下午4:45写道：
> In Flink 1.10, we can pass this schema with 'format.avro-schema' property
> to SQL DDL, but in Flink 1.11, the Avro schema is always derived from
> the table schema.
> We have two questions about the Flink 1.11 Avro format:
>
> 1. Flink 1.11 maps nullable types to Avro union(something, null). How to
> map the nullable types to union(null, something)? In our schema definition,
> we follow the Avro recommended definition, list 'null' as the first type.
> With Flink default mapping, the AvroRowDataDeserializationSchema will get
> the wrong/error result when lists 'null' as the first type of union.
>
> 2. Now if we want to set the field in a Row to non-nullable, we have to
> set the Row to non-nullable firstly. Is it the expected behavior of
> the Avro format?
> For example,
> event ROW< createTimestamp BIGINT NOT NULL, sentTimestamp BIGINT NOT NULL,
> eventId VARCHAR> will be maped to Avro schema:
>
> {
> "type": "record",
> "name": "event",
> "fields": [
> {
> "name": "header",
> "type": {
> "type": "record",
> "name": "header",
> "fields": [
> {
> "name": "createTimestamp",
> "type": ["long",null]
> },
> {
> "name": "sentTimestamp",
> "type": ["long",null]
> },
> {
> "name": "eventId",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> }
> ]
> }
> }
> ]
> }
>
> We have to use "event ROW< createTimestamp BIGINT NOT NULL, sentTimestamp
> BIGINT NOT NULL, eventId VARCHAR> NOT NULL" to make the NOT NULL of
> fields: createTimestamp and sentTimestamp works.
>
>
>
>
>
> At 2020-11-25 16:12:58, "Hongjian Peng" <super...@163.com> wrote:
>
> Hi Flink Community,
>
> We are trying to upgrade our Flink SQL job from 1.10 to 1.11. We used
> Kafka source table, and the data is stored in Kafka in Avro format.
> Schema is like this:
>
> {
> "type": "record",
> "name": "event",
> "namespace": "busseniss.event",
> "fields": [
> {
> "name": "header",
> "type": {
> "type": "record",
> "name": "header",
> "fields": [
> {
> "name": "createTimestamp",
> "type": "long"
> },
> {
> "name": "sentTimestamp",
> "type": "long"
> },
> {
> "name": "eventId",
> "type": [
> "null",
> {
> "type": "string",
> "avro.java.string": "String"
> }
> ]
> }
> ]
> },
> "doc": "Rheos header "
> },
> {
> "name": "si",
> "type": [
> "null",
> "string"
> ]
> }
> ]
> }
>
>
>
>
>
>
> --
> Hongjian Peng
> Department of Computer Science and Engineering
> Shanghai Jiao Tong University
> Email: super...@163.com
>
>
