cj-zhukov opened a new issue, #27515:
URL: https://github.com/apache/datafusion/issues/27515
### Is your feature request related to a problem or challenge?
There is no way to turn a struct, or a set of columns, into a JSON string
column without leaving the lazy plan: today this needs `collect()`, serializing
the batches with `arrow_json::ArrayWriter` and rebuilding the DataFrame, which
materializes the table and discards the original plan (see the helper in
#26185).
#26185 asked for this on the DataFrame API. It was closed when the
Spark-compatible functions moved to their own repository, and a
**Spark-flavored** `to_json` has since landed there
(apache/datafusion-spark#85, apache/datafusion-spark#86). That function follows
Spark's `JacksonGenerator` rules (Java `Double.toString` numbers, Base64
binary, millisecond timestamps in the session time zone, `"NaN"` as a string,
...), which is right for Spark users but not what a DataFusion user expects
from DataFusion's own JSON output.
DataFusion itself still has no `to_json`: `named_struct` / `struct` build
the value, `COPY ... TO 'file.json'` can write it, but there is no expression
that returns the JSON text as a column.
### Describe the solution you'd like
A `to_json(expression)` scalar function in `datafusion-functions` (`core`,
next to `named_struct`) whose output is exactly what DataFusion's JSON writer
produces for the value, i.e. arrow-json with its default options:
- a struct becomes an object keyed by its field names, a list an array, a
map an object, a scalar its JSON form; nested values are serialized recursively
- NULL struct fields and NULL map entries are omitted (as in `COPY TO
JSON`), NULL list elements are written as `null`, a NULL input returns NULL
- numbers as arrow-json writes them (`0.00001`, `1.0e10`, decimals with
their scale), non-finite floats as `null`, binary as hexadecimal, timestamps in
RFC 3339 form in the type's time zone
- types without a JSON form (unions, maps with non-string keys) are rejected
when the query is planned
```sql
SELECT id, to_json(named_struct('foo', foo, 'bar', bar)) AS json FROM t;
```
```rust
use datafusion::prelude::*;
let df = df.with_column("json", to_json(named_struct(vec![lit("foo"),
col("foo"), lit("bar"), col("bar")])))?;
// or an existing struct column
let df = df.with_column("json", to_json(col("event")))?;
```
The result stays a lazy projection: each batch is serialized with the
arrow-json row encoder, nothing is collected. The shape follows the merged
Spark implementation (apache/datafusion-spark#86), without the Spark-specific
encoders.
Not in scope for a first version: an options argument (explicit nulls,
date/time formats).
### Describe alternatives you've considered
- Keeping the `collect()`-based helper in user code: works, but executes the
plan and loses laziness, filter pushdown and the original lineage.
- `arrow_cast(struct_col, 'Utf8')`: produces Arrow's display format (`{a: 1,
b: x}`), not JSON.
- Using `datafusion-spark`'s `to_json`: gives Spark's text rules, which
differ from what DataFusion's own JSON writer produces, and pulls in the Spark
function set.
### Additional context
- Previous issue and PR: #26185, #26187 (closed when the Spark crate moved)
- Spark-flavored `to_json`, merged: apache/datafusion-spark#85,
apache/datafusion-spark#86
- arrow-json row encoder: `arrow::json::writer::make_encoder`
### Willingness to contribute
I plan to contribute to this feature independently
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]