Yan Yan created SPARK-61919:
-------------------------------
Summary: Preserve captured JSON and CSV option defaults
Key: SPARK-61919
URL: https://issues.apache.org/jira/browse/SPARK-61919
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Yan Yan
h3. Problem
Saved views and SQL functions restore captured SQL configurations while parsing
and analyzing their definitions. However, {{to_json}}, {{to_csv}}, and
{{from_json}} initialize their writers or parsers lazily, after that
configuration scope has ended. Their option defaults can therefore come from
the querying session instead of the restored definition settings.
The affected defaults are:
* {{spark.sql.jsonGenerator.ignoreNullFields}} for {{to_json}}
* {{spark.sql.legacy.nullValueWrittenAsQuotedEmptyStringCsv}} for {{to_csv}}
* {{spark.sql.legacy.json.enableDateTimeParsingFallback}} for {{from_json}}
h3. Reproduction
{code:sql}
SET spark.sql.jsonGenerator.ignoreNullFields=false;
CREATE VIEW captured_json AS
SELECT to_json(named_struct('id', id, 'n', CAST(NULL AS INT))) AS value
FROM range(1, 2);
SET spark.sql.jsonGenerator.ignoreNullFields=true;
SELECT value FROM captured_json;
{code}
Actual result: {{ {"id":1} }}.
Proposed result: {{ {"id":1,"n":null} }}, honoring the default restored from
the view definition.
This proposes definition-time binding for these defaults, consistent with
restoring them during analysis. The migration guide's existing capture
statement describes parsing and analysis; it does not promise to freeze every
execution-time setting.
h3. Proposed change
Retain these defaults as expression state during analysis and pass them to the
later writer/parser initialization. Preserve explicit function-option
precedence, case-insensitive option lookup, validation of malformed explicit
options, and CSV's null-as-unset handling. Preserve JSON fallback as
unset/true/false: when unset, parser initialization still derives the DATE and
TIMESTAMP behavior from the parser policy and their respective custom formats.
The patch includes an analysis-time rollback control,
{{spark.sql.jsonCsvExpressions.useCapturedOptionDefaults=false}}, which follows
the querying session for saved definitions. Existing analyzed DataFrames and
temporary views storing analyzed plans require reanalysis after changing it.
File-source default binding is preserved.
Scope is the three functions/defaults above. Nested-value writers in
{{JSON_ARRAY}}/{{JSON_OBJECT}} and datetime fallback in {{from_csv}} are
outside this change.
h3. Validation
Behavioral failures were reproduced before the production fix. The final
relevant local SBT validation passed 195 Catalyst tests and 407 SQL, cache,
view, and SQL-function tests (602 total), with clean Scala style checks.
Coverage includes explicit options, unset/true/false fallback, expression
copies and serialization, cache identity, and rollback behavior. The full Spark
suite was not run.
The {{to_json}} binding path is present in the 3.1.1 source (historical source
inspection); the JSON fallback configuration was added in 3.4.0. Native
regression validation was performed against current master, 5.0.0-SNAPSHOT.
PR: https://github.com/apache/spark/pull/60429
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]