[
https://issues.apache.org/jira/browse/SPARK-61919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yan Yan updated SPARK-61919:
----------------------------
Description:
h3. Problem
Saved views and SQL functions restore captured SQL configurations while parsing
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and
{{from_json}} initialize their writers or parsers lazily, after that
configuration scope has ended. Their option defaults can therefore come from
the querying session instead of the restored definition settings.
The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
h3. Reproduction
{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);
SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}
Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
the view definition.
This proposes definition-time binding for these defaults, consistent with
restoring them during analysis. The migration guide's existing capture
statement describes parsing and analysis; it does not promise to freeze every
execution-time setting.
h3. Proposed change
Retain these defaults as expression state during analysis and pass them to the
later writer/parser initialization. Preserve explicit function-option
precedence, case-insensitive option lookup, validation of malformed explicit
options, and CSV's null-as-unset handling. Preserve JSON fallback as
unset/true/false: when unset, parser initialization still derives the DATE and
TIMESTAMP behavior from the parser policy and their respective custom formats.
The patch includes an analysis-time rollback control,
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows
the querying session for saved definitions. Existing analyzed DataFrames and
temporary views storing analyzed plans require reanalysis after changing it.
File-source default binding is preserved.
Scope is the three functions/defaults above. Nested-value writers in
{{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
outside this change.
Omitted scalar SQL-function parameter defaults use the function's saved SQL
configuration for both positional and named calls. Supplied arguments and
parameter-name matching retain the caller's context.
h3. Validation
Behavioral failures were reproduced before the production fix. The final
relevant local SBT validation passed 197 Catalyst tests, 423 SQL, cache, view,
and SQL-function tests, and 10 SQL golden tests (630 total), with clean Scala
style checks. Coverage includes explicit options, unset/true/false fallback,
expression copies and serialization, cache identity, rollback behavior,
named/positional omitted defaults and caller parameter-name case-sensitivity.
The full Spark suite was not run.
The {{to_json}} binding path is present in the 3.1.1 source (historical source
inspection); the JSON fallback configuration was added in 3.4.0. Native
regression validation was performed against current master, 5.0.0-SNAPSHOT.
PR: https://github.com/apache/spark/pull/60429
was:
h3. Problem
Saved views and SQL functions restore captured SQL configurations while parsing
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and
{{from_json}} initialize their writers or parsers lazily, after that
configuration scope has ended. Their option defaults can therefore come from
the querying session instead of the restored definition settings.
The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
h3. Reproduction
{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);
SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}
Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
the view definition.
This proposes definition-time binding for these defaults, consistent with
restoring them during analysis. The migration guide's existing capture
statement describes parsing and analysis; it does not promise to freeze every
execution-time setting.
h3. Proposed change
Retain these defaults as expression state during analysis and pass them to the
later writer/parser initialization. Preserve explicit function-option
precedence, case-insensitive option lookup, validation of malformed explicit
options, and CSV's null-as-unset handling. Preserve JSON fallback as
unset/true/false: when unset, parser initialization still derives the DATE and
TIMESTAMP behavior from the parser policy and their respective custom formats.
The patch includes an analysis-time rollback control,
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows
the querying session for saved definitions. Existing analyzed DataFrames and
temporary views storing analyzed plans require reanalysis after changing it.
File-source default binding is preserved.
Scope is the three functions/defaults above. Nested-value writers in
{{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
outside this change.
h3. Validation
Behavioral failures were reproduced before the production fix. The final
relevant local SBT validation passed 197 Catalyst tests and 408 SQL, cache,
view, and SQL-function tests (605 total), with clean Scala style checks.
Coverage includes explicit options, unset/true/false fallback, expression
copies and serialization, cache identity, and rollback behavior. The full Spark
suite was not run.
The {{to_json}} binding path is present in the 3.1.1 source (historical source
inspection); the JSON fallback configuration was added in 3.4.0. Native
regression validation was performed against current master, 5.0.0-SNAPSHOT.
PR: https://github.com/apache/spark/pull/60429
> Preserve captured JSON and CSV option defaults
> ----------------------------------------------
>
> Key: SPARK-61919
> URL: https://issues.apache.org/jira/browse/SPARK-61919
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.1.1, 5.0.0
> Reporter: Yan Yan
> Priority: Minor
>
> h3. Problem
> Saved views and SQL functions restore captured SQL configurations while
> parsing and analyzing their definitions. However, {{to_json}}, {{to_csv}},
> and {{from_json}} initialize their writers or parsers lazily, after that
> configuration scope has ended. Their option defaults can therefore come from
> the querying session instead of the restored definition settings.
> The affected defaults are:
> * {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
> * {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
> * {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
> h3. Reproduction
> {code:sql}
> SET spark.sql.jsonGenerator.ignoreNullFields=false;
> CREATE VIEW captured_json AS
> SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
> FROM range(1, 2);
> SET spark.sql.jsonGenerator.ignoreNullFields=true;
> SELECT value FROM captured_json;
> {code}
> Actual result: {{ {"id":1} }}.
> Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
> the view definition.
> This proposes definition-time binding for these defaults, consistent with
> restoring them during analysis. The migration guide's existing capture
> statement describes parsing and analysis; it does not promise to freeze every
> execution-time setting.
> h3. Proposed change
> Retain these defaults as expression state during analysis and pass them to
> the later writer/parser initialization. Preserve explicit function-option
> precedence, case-insensitive option lookup, validation of malformed explicit
> options, and CSV's null-as-unset handling. Preserve JSON fallback as
> unset/true/false: when unset, parser initialization still derives the DATE
> and TIMESTAMP behavior from the parser policy and their respective custom
> formats.
> The patch includes an analysis-time rollback control,
> {{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which
> follows the querying session for saved definitions. Existing analyzed
> DataFrames and temporary views storing analyzed plans require reanalysis
> after changing it. File-source default binding is preserved.
> Scope is the three functions/defaults above. Nested-value writers in
> {{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
> outside this change.
> Omitted scalar SQL-function parameter defaults use the function's saved SQL
> configuration for both positional and named calls. Supplied arguments and
> parameter-name matching retain the caller's context.
> h3. Validation
> Behavioral failures were reproduced before the production fix. The final
> relevant local SBT validation passed 197 Catalyst tests, 423 SQL, cache,
> view, and SQL-function tests, and 10 SQL golden tests (630 total), with clean
> Scala style checks. Coverage includes explicit options, unset/true/false
> fallback, expression copies and serialization, cache identity, rollback
> behavior, named/positional omitted defaults and caller parameter-name
> case-sensitivity. The full Spark suite was not run.
> The {{to_json}} binding path is present in the 3.1.1 source (historical
> source inspection); the JSON fallback configuration was added in 3.4.0.
> Native regression validation was performed against current master,
> 5.0.0-SNAPSHOT.
> PR: https://github.com/apache/spark/pull/60429
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]