sarutak opened a new pull request, #7635: URL: https://github.com/apache/arrow-datafusion/pull/7635
## Which issue does this PR close? Closes #7634 ## Rationale for this change Avro defines three kinds of named types - `Record`, `Enum` and `Fixed`. https://avro.apache.org/docs/1.11.1/specification/#names But DataFusion currently supports only Record. So this PR aims to add support for the rest of the named types. ## What changes are included in this PR? It's not trivial that which Arrow type Avro's `Enum` or `Fixed` should be converted to. So I referred to how Apache Spark converts such types. Spark converts `Enum` type to String, and `Fixed` to Binary. Similarly, this PR introduces the conversion rule that converts `Enum` to `Utf8` and `Fixed` to `FixedSizeBinary`. For `Enum`, given we have the following schema and corresponding data (represented as JSON here for readability). ``` { "name": "record1", "namespace": "ns1", "type": "record", "fields": [ { "name": "f1", "type": { "name": "enum1", "type": "enum", "symbols": ["a", "b", "c", "d"], "default": "a" } }, { "name": "f2", "type": { "name": "enum2", "namespace": "ns2", "type": "enum", "symbols": ["e", "f", "g", "h"], "default": "e" } }, { "name": "f3", "type": { "name": "union1", "type": [ "null", { "name": "enum3", "type": "enum", "symbols": ["i", "j", "k"], "default": "i" } ] } } ] } ``` ``` {"f1":"a","f2":"g","f3":"j"} {"f1":"b","f2":"h","f3":"k"} {"f1":"c","f2":"e","f3":null} {"f1":"d","f2":"f","f3":"i"} ``` DataFusion converts like as follows. ``` CREATE EXTERNAL TABLE mytbl STORED AS AVRO LOCATION '/path/to/avro-file'; SELECT * FROM mytbl; +----+----+----+ | f1 | f2 | f3 | +----+----+----+ | a | g | j | | b | h | k | | c | e | | | d | f | i | +----+----+----+ ``` For `Fixed`, given we have the following schema and corresponding data. ``` { "name": "record1", "type": "record", "namespace": "ns1", "fields": [ { "name": "f1", "type": { "name": "fixed1", "type": "fixed", "size": 5 } }, { "name": "f2", "type": { "name": "fixed2", "namespace": "ns2", "type": "fixed", "size": 10 } }, { "name": "f3", "type": { "name": "union1", "type": [ "null", { "name": "fixed3", "type": "fixed", "size": 6 } ] } } ] } ``` ``` {"f1":[97,98,99,100,101],"f2":[102,103,104,105,106,107,108,109,110,111],"f3":[65,66,67,68,69,70]} {"f1":[49,50,51,52,53],"f2":[49,50,51,52,53,54,55,56,57,48],"f3":null} ``` DataFusion converts like as follows. ``` CREATE EXTERNAL TABLE mytbl STORED AS AVRO LOCATION '/path/to/avro-file'; SELECT * FROM mytbl; +------------+----------------------+--------------+ | f1 | f2 | f3 | +------------+----------------------+--------------+ | 6162636465 | 666768696a6b6c6d6e6f | 414243444546 | | 3132333435 | 31323334353637383930 | | +------------+----------------------+--------------+ ``` ## Are these changes tested? Added new tests. ## Are there any user-facing changes? No. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
