[
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-61919:
----------------------------
Description:
h3. Problem
Saved views and SQL functions restore captured SQL configurations while parsing
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and
{{from_json}} initialize their writers or parsers lazily, after that
configuration scope has ended. Their option defaults can therefore come from
the querying session instead of the restored definition settings.
The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
h3. Reproduction
{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);
SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}
Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
the view definition.
This proposes definition-time binding for these defaults, consistent with
restoring them during analysis. The migration guide's existing capture
statement describes parsing and analysis; it does not promise to freeze every
execution-time setting.
h3. Proposed change
Retain these defaults as expression state during analysis and pass them to the
later writer/parser initialization. Preserve explicit function-option
precedence, case-insensitive option lookup, validation of malformed explicit
options, and CSV's null-as-unset handling. Preserve JSON fallback as
unset/true/false: when unset, parser initialization still derives the DATE and
TIMESTAMP behavior from the parser policy and their respective custom formats.
The patch includes an analysis-time rollback control,
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows
the querying session for saved definitions. Existing analyzed DataFrames and
temporary views storing analyzed plans require reanalysis after changing it.
File-source default binding is preserved.
Expression snapshots cover the three functions/defaults above; scalar argument
handling also preserves the proper definition or compatibility context for
omitted defaults. Nested-value writers in {{JSON_ARRAY}}/{{JSON_OBJECT}} and
datetime fallback in {{from_csv}} are outside this change.
With capture enabled, omitted scalar SQL-function parameter defaults are parsed
and resolved under the function's saved SQL configuration for both positional
and named calls. Disabling capture restores the previous named-default caller
context, including timestamp literals. Supplied arguments and parameter-name
matching retain the caller's context. Supplied named arguments are reordered
before aggregate analysis. Named omissions are resolved once in the saved
context and reused for rendering and expansion. Grouping compares completed
argument lists only when different supplied lists need to match; positional
omissions keep their normal expansion timing. Existing generated column names
and the identity of calls with random defaults are preserved. Supplied
expressions keep their caller bindings.
h3. Validation
Behavioral failures were reproduced before the production fix. The final
relevant local SBT validation passed 203 Catalyst tests, 454 SQL, cache, view,
and SQL-function tests, and 10 SQL golden tests (667 total), with clean Scala
style checks. Coverage includes explicit options, unset/true/false fallback,
expression copies and serialization, cache identity, rollback behavior,
named/positional omitted defaults and caller parameter-name case-sensitivity,
named aggregate/grouping behavior, nontrailing defaults and timestamp-literal
rollback. Further regressions cover omitted versus explicit grouping defaults,
multiple sparse named inputs, simultaneous saved/caller conversions, default
dependencies and the original rollback evaluator. Targeted mutations confirmed
that the mapping, context and evaluator-timing assertions fail on their
corresponding regressions. Explicit file options skip unused fallback config
reads; absent options select the fallback once, including captured unset JSON
fallback. The full Spark suite was not run.
The {{to_json}} binding path is present in the 3.1.1 source (historical source
inspection); the JSON fallback configuration was added in 3.4.0. Native
regression validation was performed against current master, 5.0.0-SNAPSHOT.
PR: https://github.com/apache/spark/pull/60429
was:
h3. Problem
Saved views and SQL functions restore captured SQL configurations while parsing
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and
{{from_json}} initialize their writers or parsers lazily, after that
configuration scope has ended. Their option defaults can therefore come from
the querying session instead of the restored definition settings.
The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
h3. Reproduction
{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);
SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}
Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
the view definition.
This proposes definition-time binding for these defaults, consistent with
restoring them during analysis. The migration guide's existing capture
statement describes parsing and analysis; it does not promise to freeze every
execution-time setting.
h3. Proposed change
Retain these defaults as expression state during analysis and pass them to the
later writer/parser initialization. Preserve explicit function-option
precedence, case-insensitive option lookup, validation of malformed explicit
options, and CSV's null-as-unset handling. Preserve JSON fallback as
unset/true/false: when unset, parser initialization still derives the DATE and
TIMESTAMP behavior from the parser policy and their respective custom formats.
The patch includes an analysis-time rollback control,
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows
the querying session for saved definitions. Existing analyzed DataFrames and
temporary views storing analyzed plans require reanalysis after changing it.
File-source default binding is preserved.
Expression snapshots cover the three functions/defaults above; scalar argument
handling also preserves the proper definition or compatibility context for
omitted defaults. Nested-value writers in {{JSON_ARRAY}}/{{JSON_OBJECT}} and
datetime fallback in {{from_csv}} are outside this change.
With capture enabled, omitted scalar SQL-function parameter defaults are parsed
and resolved under the function's saved SQL configuration for both positional
and named calls. Disabling capture restores the previous named-default caller
context, including timestamp literals. Supplied arguments and parameter-name
matching retain the caller's context. Named arguments are reordered and omitted
defaults are resolved in the saved context before grouping compares calls, so
an omitted default and the same explicitly supplied default match. Supplied
expressions keep their caller bindings.
h3. Validation
Behavioral failures were reproduced before the production fix. The final
relevant local SBT validation passed 203 Catalyst tests, 446 SQL, cache, view,
and SQL-function tests, and 10 SQL golden tests (659 total), with clean Scala
style checks. Coverage includes explicit options, unset/true/false fallback,
expression copies and serialization, cache identity, rollback behavior,
named/positional omitted defaults and caller parameter-name case-sensitivity,
named aggregate/grouping behavior, nontrailing defaults and timestamp-literal
rollback. Further regressions cover omitted versus explicit grouping defaults,
multiple sparse named inputs, simultaneous saved/caller conversions, default
dependencies and the original rollback evaluator. Targeted mutations confirmed
that the mapping, context and evaluator-timing assertions fail on their
corresponding regressions. Explicit file options skip unused fallback config
reads; absent options select the fallback once, including captured unset JSON
fallback. The full Spark suite was not run.
The {{to_json}} binding path is present in the 3.1.1 source (historical source
inspection); the JSON fallback configuration was added in 3.4.0. Native
regression validation was performed against current master, 5.0.0-SNAPSHOT.
PR: https://github.com/apache/spark/pull/60429
> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
> Key: SPARK-61919
> URL: https://issues.apache.org/jira/browse/SPARK-61919
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1, 5.0.0
> Reporter: Yan Yan
> Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}},
> and {{from_json}} initialize their writers or parsers lazily, after that
> configuration scope has ended. Their option defaults can therefore come from
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
> the view definition.
> This proposes definition-time binding for these defaults, consistent with
> restoring them during analysis. The migration guide's existing capture
> statement describes parsing and analysis; it does not promise to freeze every
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to
> the later writer/parser initialization. Preserve explicit function-option
> precedence, case-insensitive option lookup, validation of malformed explicit
> options, and CSV's null-as-unset handling. Preserve JSON fallback as
> unset/true/false: when unset, parser initialization still derives the DATE
> and TIMESTAMP behavior from the parser policy and their respective custom
> formats.
> The patch includes an analysis-time rollback control,
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which
> follows the querying session for saved definitions. Existing analyzed
> DataFrames and temporary views storing analyzed plans require reanalysis
> after changing it. File-source default binding is preserved.
> Expression snapshots cover the three functions/defaults above; scalar
> argument handling also preserves the proper definition or compatibility
> context for omitted defaults. Nested-value writers in
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
> outside this change.
> With capture enabled, omitted scalar SQL-function parameter defaults are
> parsed and resolved under the function's saved SQL configuration for both
> positional and named calls. Disabling capture restores the previous
> named-default caller context, including timestamp literals. Supplied
> arguments and parameter-name matching retain the caller's context. Supplied
> named arguments are reordered before aggregate analysis. Named omissions are
> resolved once in the saved context and reused for rendering and expansion.
> Grouping compares completed argument lists only when different supplied lists
> need to match; positional omissions keep their normal expansion timing.
> Existing generated column names and the identity of calls with random
> defaults are preserved. Supplied expressions keep their caller bindings.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final
> relevant local SBT validation passed 203 Catalyst tests, 454 SQL, cache,
> view, and SQL-function tests, and 10 SQL golden tests (667 total), with clean
> Scala style checks. Coverage includes explicit options, unset/true/false
> fallback, expression copies and serialization, cache identity, rollback
> behavior, named/positional omitted defaults and caller parameter-name
> case-sensitivity, named aggregate/grouping behavior, nontrailing defaults and
> timestamp-literal rollback. Further regressions cover omitted versus explicit
> grouping defaults, multiple sparse named inputs, simultaneous saved/caller
> conversions, default dependencies and the original rollback evaluator.
> Targeted mutations confirmed that the mapping, context and evaluator-timing
> assertions fail on their corresponding regressions. Explicit file options
> skip unused fallback config reads; absent options select the fallback once,
> including captured unset JSON fallback. The full Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical
> source inspection); the JSON fallback configuration was added in 3.4.0.
> Native regression validation was performed against current master,
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]