[ 
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yan Yan updated SPARK-61919:
----------------------------
    Description: 
h3. Problem

Saved views and SQL functions restore captured SQL configurations while parsing 
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and 
{{from_json}} initialize their writers or parsers lazily, after that 
configuration scope has ended. Their option defaults can therefore come from 
the querying session instead of the restored definition settings.

The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}

h3. Reproduction

{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);

SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}

Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
the view definition.

This proposes definition-time binding for these defaults, consistent with 
restoring them during analysis. The migration guide's existing capture 
statement describes parsing and analysis; it does not promise to freeze every 
execution-time setting.

h3. Proposed change

Retain these defaults as expression state during analysis and pass them to the 
later writer/parser initialization. Preserve explicit function-option 
precedence, case-insensitive option lookup, validation of malformed explicit 
options, and CSV's null-as-unset handling. Preserve JSON fallback as 
unset/true/false: when unset, parser initialization still derives the DATE and 
TIMESTAMP behavior from the parser policy and their respective custom formats.

The patch includes an analysis-time rollback control, 
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows 
the querying session for saved definitions. Existing analyzed DataFrames and 
temporary views storing analyzed plans require reanalysis after changing it. 
File-source default binding is preserved.

Expression snapshots cover the three functions/defaults above; scalar argument 
handling also preserves the proper definition or compatibility context for 
omitted defaults. Nested-value writers in {{JSON_ARRAY}}/{{JSON_OBJECT}} and 
datetime fallback in {{from_csv}} are outside this change.

With capture enabled, omitted scalar SQL-function parameter defaults are parsed 
and resolved under the function's saved SQL configuration for both positional 
and named calls. Disabling capture restores the previous named-default caller 
context, including timestamp literals. Supplied arguments and parameter-name 
matching retain the caller's context. Named arguments are reordered and omitted 
defaults are resolved in the saved context before grouping compares calls, so 
an omitted default and the same explicitly supplied default match. Supplied 
expressions keep their caller bindings.

h3. Validation

Behavioral failures were reproduced before the production fix. The final 
relevant local SBT validation passed 203 Catalyst tests, 446 SQL, cache, view, 
and SQL-function tests, and 10 SQL golden tests (659 total), with clean Scala 
style checks. Coverage includes explicit options, unset/true/false fallback, 
expression copies and serialization, cache identity, rollback behavior, 
named/positional omitted defaults and caller parameter-name case-sensitivity, 
named aggregate/grouping behavior, nontrailing defaults and timestamp-literal 
rollback. Further regressions cover omitted versus explicit grouping defaults, 
multiple sparse named inputs, simultaneous saved/caller conversions, default 
dependencies and the original rollback evaluator. Targeted mutations confirmed 
that the mapping, context and evaluator-timing assertions fail on their 
corresponding regressions. Explicit file options skip unused fallback config 
reads; absent options select the fallback once, including captured unset JSON 
fallback. The full Spark suite was not run.

The {{to_json}} binding path is present in the 3.1.1 source (historical source 
inspection); the JSON fallback configuration was added in 3.4.0. Native 
regression validation was performed against current master, 5.0.0-SNAPSHOT.

PR: https://github.com/apache/spark/pull/60429


  was:
h3. Problem

Saved views and SQL functions restore captured SQL configurations while parsing 
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and 
{{from_json}} initialize their writers or parsers lazily, after that 
configuration scope has ended. Their option defaults can therefore come from 
the querying session instead of the restored definition settings.

The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}

h3. Reproduction

{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);

SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}

Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
the view definition.

This proposes definition-time binding for these defaults, consistent with 
restoring them during analysis. The migration guide's existing capture 
statement describes parsing and analysis; it does not promise to freeze every 
execution-time setting.

h3. Proposed change

Retain these defaults as expression state during analysis and pass them to the 
later writer/parser initialization. Preserve explicit function-option 
precedence, case-insensitive option lookup, validation of malformed explicit 
options, and CSV's null-as-unset handling. Preserve JSON fallback as 
unset/true/false: when unset, parser initialization still derives the DATE and 
TIMESTAMP behavior from the parser policy and their respective custom formats.

The patch includes an analysis-time rollback control, 
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows 
the querying session for saved definitions. Existing analyzed DataFrames and 
temporary views storing analyzed plans require reanalysis after changing it. 
File-source default binding is preserved.

Expression snapshots cover the three functions/defaults above; scalar argument 
handling also preserves the proper definition or compatibility context for 
omitted defaults. Nested-value writers in {{JSON_ARRAY}}/{{JSON_OBJECT}} and 
datetime fallback in {{from_csv}} are outside this change.

With capture enabled, omitted scalar SQL-function parameter defaults are parsed 
and resolved under the function's saved SQL configuration for both positional 
and named calls. Disabling capture restores the previous named-default caller 
context, including timestamp literals. Supplied arguments and parameter-name 
matching retain the caller's context. Supplied named arguments are normalized 
before aggregate and grouping analysis; only omitted defaults wait for the 
saved context.

h3. Validation

Behavioral failures were reproduced before the production fix. The final 
relevant local SBT validation passed 197 Catalyst tests, 433 SQL, cache, view, 
and SQL-function tests, and 10 SQL golden tests (640 total), with clean Scala 
style checks. Coverage includes explicit options, unset/true/false fallback, 
expression copies and serialization, cache identity, rollback behavior, 
named/positional omitted defaults and caller parameter-name case-sensitivity, 
named aggregate/grouping behavior, nontrailing defaults and timestamp-literal 
rollback. The full Spark suite was not run.

The {{to_json}} binding path is present in the 3.1.1 source (historical source 
inspection); the JSON fallback configuration was added in 3.4.0. Native 
regression validation was performed against current master, 5.0.0-SNAPSHOT.

PR: https://github.com/apache/spark/pull/60429



> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
>                 Key: SPARK-61919
>                 URL: https://issues.apache.org/jira/browse/SPARK-61919
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.1.1, 5.0.0
>            Reporter: Yan Yan
>            Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while 
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}}, 
> and {{from_json}} initialize their writers or parsers lazily, after that 
> configuration scope has ended. Their option defaults can therefore come from 
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from 
> the view definition.
> This proposes definition-time binding for these defaults, consistent with 
> restoring them during analysis. The migration guide's existing capture 
> statement describes parsing and analysis; it does not promise to freeze every 
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to 
> the later writer/parser initialization. Preserve explicit function-option 
> precedence, case-insensitive option lookup, validation of malformed explicit 
> options, and CSV's null-as-unset handling. Preserve JSON fallback as 
> unset/true/false: when unset, parser initialization still derives the DATE 
> and TIMESTAMP behavior from the parser policy and their respective custom 
> formats.
> The patch includes an analysis-time rollback control, 
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which 
> follows the querying session for saved definitions. Existing analyzed 
> DataFrames and temporary views storing analyzed plans require reanalysis 
> after changing it. File-source default binding is preserved.
> Expression snapshots cover the three functions/defaults above; scalar 
> argument handling also preserves the proper definition or compatibility 
> context for omitted defaults. Nested-value writers in 
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are 
> outside this change.
> With capture enabled, omitted scalar SQL-function parameter defaults are 
> parsed and resolved under the function's saved SQL configuration for both 
> positional and named calls. Disabling capture restores the previous 
> named-default caller context, including timestamp literals. Supplied 
> arguments and parameter-name matching retain the caller's context. Named 
> arguments are reordered and omitted defaults are resolved in the saved 
> context before grouping compares calls, so an omitted default and the same 
> explicitly supplied default match. Supplied expressions keep their caller 
> bindings.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final 
> relevant local SBT validation passed 203 Catalyst tests, 446 SQL, cache, 
> view, and SQL-function tests, and 10 SQL golden tests (659 total), with clean 
> Scala style checks. Coverage includes explicit options, unset/true/false 
> fallback, expression copies and serialization, cache identity, rollback 
> behavior, named/positional omitted defaults and caller parameter-name 
> case-sensitivity, named aggregate/grouping behavior, nontrailing defaults and 
> timestamp-literal rollback. Further regressions cover omitted versus explicit 
> grouping defaults, multiple sparse named inputs, simultaneous saved/caller 
> conversions, default dependencies and the original rollback evaluator. 
> Targeted mutations confirmed that the mapping, context and evaluator-timing 
> assertions fail on their corresponding regressions. Explicit file options 
> skip unused fallback config reads; absent options select the fallback once, 
> including captured unset JSON fallback. The full Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical 
> source inspection); the JSON fallback configuration was added in 3.4.0. 
> Native regression validation was performed against current master, 
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to