sarutak commented on issue #7828:
URL:
https://github.com/apache/arrow-datafusion/issues/7828#issuecomment-1784128469
@Asura7969
Sorry for the late confirmation.
I understand the problem you are talking about is DataFusion doesn't support
nullable top-level records.
@alamb
I'd like to discuss this problem.
Avro allows top-level records nullable.
So, the following records can be allowed in Avro.
```
{"x": "abc", "y": 100}
{"x": "def", "y": 200}
null
{"x": "ghi", "y": 300}
```
Notice that the third record is not `{"x": null, "y": null}` but the record
itself is `null`.
I think we would have the following options to treat the nullable top-level
records.
1. Allow nullable top-level records but skip null records
2. Disallow nullable top-level records. Nullable top-level records should be
converted to non-nullable beforehand.
3. Introduce a configuration option which controls whether nullable
top-level records is allowed or not.
BTW, Apache Spark has a similar feature that creates a table from an Avro
records but it doesn't currently support nullable top-level nullable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
