cj-zhukov opened a new pull request, #86: URL: https://github.com/apache/datafusion-spark/pull/86
## Which issue does this PR close? Closes #https://github.com/apache/datafusion-spark/issues/85. Ported from apache/datafusion#26187, which was reviewed there (all review comments addressed, no further changes requested) but not merged before the crate moved to this repository. ## Rationale for this change There is no way to serialize a struct, or a set of columns, into a JSON string column without `collect()`-ing the DataFrame and rebuilding it. Spark has `to_json(struct)`. This PR adds a Spark-compatible `to_json` so the conversion stays a lazy projection and runs batch by batch. ```rust use datafusion_spark::expr_fn::to_json; df.with_column("json", to_json(&["foo", "bar"]))?; // {"foo":1,"bar":"bak"} df.with_column("json", to_json(col("event")))?; // existing struct column ``` ```sql SELECT to_json(named_struct('foo', foo, 'bar', bar)) AS json FROM t; ``` ## What changes are included in this PR? - `crates/spark/src/function/json/to_json.rs`: `SparkToJson` scalar UDF (`Struct -> Utf8`). Serializes each batch with the arrow-json row encoder (`encode_struct_rows`), with Spark-specific overrides supplied through an `EncoderFactory`: - NULL struct fields are omitted (`ignoreNullFields` default); NULL map entries are kept as `"key":null` - map keys are written as strings whatever their type (`{"1":"v"}`) - binary values are Base64 - non-finite floats are the strings `"NaN"`, `"Infinity"`, `"-Infinity"` - a NULL struct produces NULL; nested structs, lists and maps are handled recursively - non-struct input is rejected at plan time - `IntoJsonStruct` trait + `expr_fn::to_json(input)`: accepts either a struct `Expr` or a list of column names, which are packed with `named_struct` keyed by their unqualified names. The result is a plain `Expr`, so `DataFrame::with_column` needs no changes. - `crates/spark/src/function/json/mod.rs`: registration (`functions()`, `expr_fn`). The row-encoding loop is kept separate from the Spark options so a default (arrow-json flavored) `to_json` can share it later. Not in scope: the Spark `options` argument and map/array top-level inputs. `#[user_doc]` is not added yet since `datafusion-macros` is not a dependency of this crate; I will add it once #58 lands. ## What is the testing strategy for this PR? - Unit tests in `to_json.rs`: array/scalar input, null handling for structs and maps, nested struct/list/map, sliced input, Base64 binary, non-finite floats, map key stringification, unsupported key -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
