hanke580 opened a new issue, #26493:
URL: https://github.com/apache/datafusion/issues/26493
### Describe the bug
DataFusion's JSON sink builds arrow-json writers with the default
`explicit_nulls = false`, and `JsonOptions` has no
option to change it. For a struct this only drops a key whose field still
exists in the schema on read. For a
`MAP`, the entry *is* the data: arrow-json's `MapEncoder::encode`
(`arrow-json/src/writer/encoder.rs`, line 869 in 60.0.0 and 884 on main,
`if is_null && !self.explicit_nulls { continue; }`) skips every entry with a
NULL value. The key and the map's
cardinality are lost from the file and from every read of it. Parquet output
of the same table keeps the entries, and
DuckDB's JSON writer writes `{"color":"red","size":null}`.
The arrow-json writer with `with_explicit_nulls(true)` writes
`{"attrs":{"color":"red","size":null}}`, so the fix on
the DataFusion side could be to always enable it for maps, or to expose
`explicit_nulls` in `JsonOptions` (and maybe
default to `true` for map values, whose keys carry information).
### To Reproduce
```sql
CREATE TABLE src AS SELECT 1 AS id, MAP(['color', 'size', 'note'], ['red',
NULL, NULL]) AS attrs
UNION ALL SELECT 2, MAP(['flag'], [NULL]);
COPY src TO 'out.json' STORED AS JSON;
CREATE EXTERNAL TABLE back (id BIGINT, attrs MAP(VARCHAR, VARCHAR)) STORED
AS JSON LOCATION 'out.json';
SELECT id, attrs, cardinality(attrs) AS n FROM src ORDER BY id;
SELECT id, attrs, cardinality(attrs) AS n FROM back ORDER BY id;
```
```
source | 1 | {color: red, size: NULL, note: NULL} | 3
source | 2 | {flag: NULL} | 1
json back | 1 | {color: red} | 1 <-- size, note
gone
json back | 2 | {} | 0 <-- flag gone
```
`out.json`:
```
{"id":1,"attrs":{"color":"red"}}
{"id":2,"attrs":{}}
```
The same round trip through `STORED AS PARQUET` returns all entries.
### Expected behavior
`out.json` contains
`{"id":1,"attrs":{"color":"red","size":null,"note":null}}` and
`{"id":2,"attrs":{"flag":null}}`,
and the table read back equals the source.
### Additional context
- DataFusion main 290ad15 (2026-10-18), built from source; arrow-json 60.0.0.
- Plain arrow-rs shows the writer behaviour: a `MapArray` row `{color:
"red", size: null}` is written as
`{"attrs":{"color":"red"}}` by `LineDelimitedWriter`, and as
`{"attrs":{"color":"red","size":null}}` with
`WriterBuilder::new().with_explicit_nulls(true)`.
### Willingness to contribute
None
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
