[
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-61919:
----------------------------
Affects Version/s: 3.1.1
Priority: Minor (was: Major)
> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
> Key: SPARK-61919
> URL: https://issues.apache.org/jira/browse/SPARK-61919
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1, 5.0.0
> Reporter: Yan Yan
> Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}},
> and {{from_json}} initialize their writers or parsers lazily, after that
> configuration scope has ended. Their option defaults can therefore come from
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
> the view definition.
> This proposes definition-time binding for these defaults, consistent with
> restoring them during analysis. The migration guide's existing capture
> statement describes parsing and analysis; it does not promise to freeze every
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to
> the later writer/parser initialization. Preserve explicit function-option
> precedence, case-insensitive option lookup, validation of malformed explicit
> options, and CSV's null-as-unset handling. Preserve JSON fallback as
> unset/true/false: when unset, parser initialization still derives the DATE
> and TIMESTAMP behavior from the parser policy and their respective custom
> formats.
> The patch includes an analysis-time rollback control,
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which
> follows the querying session for saved definitions. Existing analyzed
> DataFrames and temporary views storing analyzed plans require reanalysis
> after changing it. File-source default binding is preserved.
> Scope is the three functions/defaults above. Nested-value writers in
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
> outside this change.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final
> relevant local SBT validation passed 195 Catalyst tests and 407 SQL, cache,
> view, and SQL-function tests (602 total), with clean Scala style checks.
> Coverage includes explicit options, unset/true/false fallback, expression
> copies and serialization, cache identity, and rollback behavior. The full
> Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical
> source inspection); the JSON fallback configuration was added in 3.4.0.
> Native regression validation was performed against current master,
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]