[ 
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yan Yan updated SPARK-61919:
----------------------------
    Affects Version/s: 3.1.1
             Priority: Minor  (was: Major)

> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
>                 Key: SPARK-61919
>                 URL: https://issues.apache.org/jira/browse/SPARK-61919
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.1.1, 5.0.0
>            Reporter: Yan Yan
>            Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while 
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}}, 
> and {{from_json}} initialize their writers or parsers lazily, after that 
> configuration scope has ended. Their option defaults can therefore come from 
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
> the view definition.
> This proposes definition-time binding for these defaults, consistent with 
> restoring them during analysis. The migration guide's existing capture 
> statement describes parsing and analysis; it does not promise to freeze every 
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to 
> the later writer/parser initialization. Preserve explicit function-option 
> precedence, case-insensitive option lookup, validation of malformed explicit 
> options, and CSV's null-as-unset handling. Preserve JSON fallback as 
> unset/true/false: when unset, parser initialization still derives the DATE 
> and TIMESTAMP behavior from the parser policy and their respective custom 
> formats.
> The patch includes an analysis-time rollback control, 
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which 
> follows the querying session for saved definitions. Existing analyzed 
> DataFrames and temporary views storing analyzed plans require reanalysis 
> after changing it. File-source default binding is preserved.
> Scope is the three functions/defaults above. Nested-value writers in 
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are 
> outside this change.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final 
> relevant local SBT validation passed 195 Catalyst tests and 407 SQL, cache, 
> view, and SQL-function tests (602 total), with clean Scala style checks. 
> Coverage includes explicit options, unset/true/false fallback, expression 
> copies and serialization, cache identity, and rollback behavior. The full 
> Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical 
> source inspection); the JSON fallback configuration was added in 3.4.0. 
> Native regression validation was performed against current master, 
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to