[ 
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yan Yan updated SPARK-61919:
----------------------------
    Description: 
h3. Problem

Saved views and SQL functions restore captured SQL configurations while parsing 
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and 
{{from_json}} initialize their writers or parsers lazily, after that 
configuration scope has ended. Their option defaults can therefore come from 
the querying session instead of the restored definition settings.

The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}

h3. Reproduction

{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);

SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}

Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
the view definition.

This proposes definition-time binding for these defaults, consistent with 
restoring them during analysis. The migration guide's existing capture 
statement describes parsing and analysis; it does not promise to freeze every 
execution-time setting.

h3. Proposed change

Retain these defaults as expression state during analysis and pass them to the 
later writer/parser initialization. Preserve explicit function-option 
precedence, case-insensitive option lookup, validation of malformed explicit 
options, and CSV's null-as-unset handling. Preserve JSON fallback as 
unset/true/false: when unset, parser initialization still derives the DATE and 
TIMESTAMP behavior from the parser policy and their respective custom formats.

The patch includes an analysis-time rollback control, 
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows 
the querying session for saved definitions. Existing analyzed DataFrames and 
temporary views storing analyzed plans require reanalysis after changing it. 
File-source default binding is preserved.

Scope is the three functions/defaults above. Nested-value writers in 
{{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are 
outside this change.

h3. Validation

Behavioral failures were reproduced before the production fix. The final 
relevant local SBT validation passed 196 Catalyst tests and 408 SQL, cache, 
view, and SQL-function tests (604 total), with clean Scala style checks. 
Coverage includes explicit options, unset/true/false fallback, expression 
copies and serialization, cache identity, and rollback behavior. The full Spark 
suite was not run.

The {{to_json}} binding path is present in the 3.1.1 source (historical source 
inspection); the JSON fallback configuration was added in 3.4.0. Native 
regression validation was performed against current master, 5.0.0-SNAPSHOT.

PR: https://github.com/apache/spark/pull/60429


  was:
h3. Problem

Saved views and SQL functions restore captured SQL configurations while parsing 
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and 
{{from_json}} initialize their writers or parsers lazily, after that 
configuration scope has ended. Their option defaults can therefore come from 
the querying session instead of the restored definition settings.

The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}

h3. Reproduction

{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);

SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}

Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
the view definition.

This proposes definition-time binding for these defaults, consistent with 
restoring them during analysis. The migration guide's existing capture 
statement describes parsing and analysis; it does not promise to freeze every 
execution-time setting.

h3. Proposed change

Retain these defaults as expression state during analysis and pass them to the 
later writer/parser initialization. Preserve explicit function-option 
precedence, case-insensitive option lookup, validation of malformed explicit 
options, and CSV's null-as-unset handling. Preserve JSON fallback as 
unset/true/false: when unset, parser initialization still derives the DATE and 
TIMESTAMP behavior from the parser policy and their respective custom formats.

The patch includes an analysis-time rollback control, 
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows 
the querying session for saved definitions. Existing analyzed DataFrames and 
temporary views storing analyzed plans require reanalysis after changing it. 
File-source default binding is preserved.

Scope is the three functions/defaults above. Nested-value writers in 
{{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are 
outside this change.

h3. Validation

Behavioral failures were reproduced before the production fix. The final 
relevant local SBT validation passed 195 Catalyst tests and 408 SQL, cache, 
view, and SQL-function tests (603 total), with clean Scala style checks. 
Coverage includes explicit options, unset/true/false fallback, expression 
copies and serialization, cache identity, and rollback behavior. The full Spark 
suite was not run.

The {{to_json}} binding path is present in the 3.1.1 source (historical source 
inspection); the JSON fallback configuration was added in 3.4.0. Native 
regression validation was performed against current master, 5.0.0-SNAPSHOT.

PR: https://github.com/apache/spark/pull/60429



> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
>                 Key: SPARK-61919
>                 URL: https://issues.apache.org/jira/browse/SPARK-61919
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.1.1, 5.0.0
>            Reporter: Yan Yan
>            Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while 
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}}, 
> and {{from_json}} initialize their writers or parsers lazily, after that 
> configuration scope has ended. Their option defaults can therefore come from 
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
> the view definition.
> This proposes definition-time binding for these defaults, consistent with 
> restoring them during analysis. The migration guide's existing capture 
> statement describes parsing and analysis; it does not promise to freeze every 
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to 
> the later writer/parser initialization. Preserve explicit function-option 
> precedence, case-insensitive option lookup, validation of malformed explicit 
> options, and CSV's null-as-unset handling. Preserve JSON fallback as 
> unset/true/false: when unset, parser initialization still derives the DATE 
> and TIMESTAMP behavior from the parser policy and their respective custom 
> formats.
> The patch includes an analysis-time rollback control, 
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which 
> follows the querying session for saved definitions. Existing analyzed 
> DataFrames and temporary views storing analyzed plans require reanalysis 
> after changing it. File-source default binding is preserved.
> Scope is the three functions/defaults above. Nested-value writers in 
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are 
> outside this change.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final 
> relevant local SBT validation passed 196 Catalyst tests and 408 SQL, cache, 
> view, and SQL-function tests (604 total), with clean Scala style checks. 
> Coverage includes explicit options, unset/true/false fallback, expression 
> copies and serialization, cache identity, and rollback behavior. The full 
> Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical 
> source inspection); the JSON fallback configuration was added in 3.4.0. 
> Native regression validation was performed against current master, 
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to