cj-zhukov opened a new pull request, #27518:
URL: https://github.com/apache/datafusion/pull/27518

   ## Which issue does this PR close?
   
   <!--
   We generally require a GitHub issue to be filed for all bug fixes and 
enhancements and this helps us generate change logs for our releases. You can 
link an issue to this PR using the GitHub syntax. For example `Closes #123` 
indicates that this PR will close issue #123.
   -->
   
   - Closes #https://github.com/apache/datafusion/issues/27515.
   
   ## Rationale for this change
   There is no way to serialize a struct, or a set of columns, into a JSON 
string column without `collect()`-ing the DataFrame and rebuilding it. 
DataFusion can build the value (`named_struct`, `struct`) and can write it 
(`COPY ... TO 'file.json'`), but has no expression that returns the JSON text 
as a column. This PR adds `to_json` so the conversion stays a lazy projection 
and runs batch by batch.
   
   ```rust
   use datafusion::prelude::*;
   
   df.with_column("json", to_json(named_struct(vec![lit("foo"), col("foo"), 
lit("bar"), col("bar")])))?; // {"foo":1,"bar":"bak"}
   df.with_column("json", to_json(col("event")))?;                              
                      // existing struct column
   ```
   
   ```sql
   SELECT to_json(named_struct('foo', foo, 'bar', bar)) AS json FROM t;
   ```
   <!--
    Why are you proposing this change? If this is already explained clearly in 
the issue then this section is not needed.
    Explaining clearly why changes are proposed helps reviewers understand your 
changes and offer better suggestions for fixes.
   
   Please explain the problem you are trying to solve in terms of the 
user-visible
   behavior, rather than the implementation.
   
   For example, "The code in `foo.rs` doesn't handle nulls" is a symptom of the
   implementation. "COUNT(DISTINCT) returns wrong results when the column 
contains
   nulls" is the user-visible problem.
   -->
   
   ## What changes are included in this PR?
   - `datafusion/functions/src/core/to_json.rs`: `ToJsonFunc` scalar UDF 
(`to_json(expression) -> Utf8`). Each batch is serialized with the arrow-json 
row encoder and default `EncoderOptions`, so the output is exactly what 
DataFusion's JSON writer (`COPY TO JSON`) produces for the value:
     - a struct becomes an object keyed by its field names, a list an array, a 
map an object, a scalar its JSON form; nested values are serialized recursively 
(dictionary-encoded values like their values)
     - NULL struct fields and NULL map entries are omitted, NULL list elements 
are written as `null`, a NULL input produces NULL
     - numbers as arrow-json writes them (`0.00001`, `1.0e10`, decimals with 
their scale), non-finite floats as `null`, binary as hexadecimal, dates as 
`yyyy-MM-dd`, timestamps in RFC 3339 form in the type's time zone, durations as 
ISO 8601
     - types without a JSON form (unions, maps with non-string keys, at any 
nesting level) are rejected at planning time; the check builds the encoder on 
an empty array of the argument type, so arrow-json's own rules decide
   - `datafusion/functions/src/core/mod.rs`: registration (`functions()`) and 
`expr_fn::to_json(Expr)` via `export_functions!`, which also exports it through 
`datafusion::prelude`.
   - `docs/source/user-guide/sql/scalar_functions.md`: regenerated from 
`#[user_doc]` (section "Struct Functions").
   <!--
   There is no need to duplicate the description in the issue here, but it is 
sometimes worth providing a summary of the individual changes in this PR.
   -->
   
   ## What is the testing strategy for this PR?
   - Unit tests in `to_json.rs`: struct/list/map/scalar input, array and scalar 
paths, NULL handling at every level, nested and sliced input, dictionaries, 
floats, decimals, large numbers, binary, dates, times, timestamps, durations 
and intervals, string escaping, and the planning-time rejections.
   - `datafusion/sqllogictest/test_files/to_json.slt`: the same from SQL (53 
queries).
   - `datafusion/core/tests/dataframe/mod.rs`: `with_column` tests on 
`test_table()`, including that the DataFrame and SQL paths produce identical 
results and that both reject unsupported types when the plan is built.
   <!--
   We typically require tests for all PRs in order to:
   1. Prevent the code from being accidentally broken by subsequent changes
   2. Serve as another way to document the expected behavior of the code
   
   Briefly describe how this PR is tested, and point to the specific tests you 
added. For example: 'This new feature is covered by the `sqllogictest` cases 
added in `foo.slt`'.
   
   If this PR does not add tests, explain why. For example, if the change is 
already covered by existing tests, please mention it.
   
   You should also check the `codecov` bot reply on this PR to confirm the 
changed code is exercised.
   -->
   
   ## Are there any user-facing changes?
   New scalar function `to_json`, available in SQL and as 
`datafusion::prelude::to_json` / `datafusion_functions::expr_fn::to_json`. No 
changes to existing APIs.
   
   In a session with the Spark function set enabled, `datafusion_spark`'s 
`to_json` keeps taking precedence by name, as other same-named functions 
already do.
   <!--
   If there are user-facing changes then we may require documentation to be 
updated before approving the PR.
   
   If there are any breaking changes to public APIs, please add the `api 
change` label.
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to