Dimitrije Zdravkovic created SPARK-61993:
--------------------------------------------
Summary: Resolve GAP FILL into a Generate with a GapFillGenerator
Key: SPARK-61993
URL: https://issues.apache.org/jira/browse/SPARK-61993
Project: Spark
Issue Type: Sub-task
Components: SQL
Affects Versions: 4.4.0
Reporter: Dimitrije Zdravkovic
Resolve the \{{GAP FILL}} relation operator in the analyzer, as the second step
of SPARK-61276.
* The analyzer rule \{{ResolveGapFill}} now turns \{{UnresolvedGapFill}} into a
\{{Generate}} with a new \{{GapFillGenerator}}, and a \{{Project}} above it
that keeps the input column order. Only the partition columns pass through the
\{{Generate}}. Every other column becomes a new nullable column that keeps the
name, type, metadata and table name of its input column.
* Add \{{GapFillResolution}}, which checks the clauses in a fixed order and
evaluates their constant values. The single-pass analyzer can use it too.
* Add the analysis errors: 15 new error classes, such as
\{{GAP_FILL_TIME_TYPE_MISMATCH}} and \{{GAP_FILL_INVALID_RANGE}}, and 4 new
subclasses of \{{UNSUPPORTED_FEATURE}}: \{{GAP_FILL_FILL_METHOD_TYPE}},
\{{GAP_FILL_PARTITION_KEY_TYPE}}, \{{GAP_FILL_TIMESTAMP_DAY_BIN_WIDTH}} and
\{{GAP_FILL_TIMESTAMP_NANOS}}.
* \{{CheckAnalysis}} rejects a correlated input, a streaming input, and a
metadata column read after \{{GAP FILL}}.
* \{{OptimizeOneRowRelationSubquery}} does not move a \{{GAP FILL}} onto the
rows of an outer query.
There is no physical execution yet. Until a later sub-task adds it, the planner
rejects a resolved \{{GAP FILL}} with \{{UNSUPPORTED_FEATURE.GAP_FILL}}. The
conf \{{spark.sql.gapFillRelationOperator.enabled}} stays internal and off by
default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]