Pastoray opened a new pull request, #27432: URL: https://github.com/apache/datafusion/pull/27432
> [!IMPORTANT] > > **Fork CI setup:** For pull requests from a fork to `main`, enable workflows in your fork's **Actions** tab before you push. > See the [fork CI setup guide](https://datafusion.apache.org/contributor-guide/#pull-requests-from-forks). ## Which issue does this PR close? - Closes #26493. ## Rationale for this change `COPY ... STORED AS JSON` and `INSERT INTO` a JSON table silently drop every `MAP` entry whose value is NULL. For example, the map `{ color: red, size: NULL }` is written as `{ "color": "red" }`, and `{ flag: NULL }` is written as `{}`. Unlike a struct field, which is filled back in from the schema on read, a map entry *is* the data. The key and the map's cardinality are lost from the file and from every read of it: `cardinality(attrs)` returns 1 instead of 2, and 0 instead of 1. Writing the same table to Parquet keeps all entries, so the two formats disagree. The cause is that DataFusion builds arrow-json writers with the default `explicit_nulls = false`, which makes arrow-json's `MapEncoder` skip NULL-valued entries. ## What changes are included in this PR? - Add `line_delimited_writer` in `datasource-json/src/file_format.rs`. It enables arrow-json's `explicit_nulls` when the schema contains a `MAP` at any nesting level (including inside lists, structs, unions, dictionaries and run-end encoded columns). Maps are now written as `{ "color": "red", "size": null }`. - Use it in both JSON write paths: `JsonSerializer` (used by `COPY` and `INSERT INTO`) and the sink in `source.rs`. - Schemas without a `MAP` are unaffected and keep the existing compact output. ## What is the testing strategy for this PR? - Five new unit tests in `datasource-json/src/file_format.rs`: map detection (including nesting), exact JSON output for the case in the issue, read-back preserving per-row entry counts, a map nested in a list, and unchanged output (NULLs omitted) for schemas without a map. - A new round-trip case in `sqllogictest/test_files/json.slt` taken from the issue's reproduction: `COPY` a table with NULL-valued maps to JSON, read it back through an external table, and check the values and `cardinality`. Without the fix it fails with `{ color: red } 1` / `{} 0`; with the fix it passes. ## Are there any user-facing changes? Yes, but only to the content of JSON written for schemas that contain a `MAP`. NULL-valued map entries are now written as `"key": null` instead of being omitted, so the data round-trips. - When a schema has a map, other NULL columns in the same file are also written as explicit `null` instead of being omitted. This is a limitation of arrow-json's single switch. Reading the files back gives identical data either way. - Output for schemas without a `MAP` is unchanged. - No public API or config changes, so no `api change` label is needed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
