Dimitrije Zdravkovic created SPARK-61993:
--------------------------------------------

             Summary: Resolve GAP FILL into a Generate with a GapFillGenerator
                 Key: SPARK-61993
                 URL: https://issues.apache.org/jira/browse/SPARK-61993
             Project: Spark
          Issue Type: Sub-task
          Components: SQL
    Affects Versions: 4.4.0
            Reporter: Dimitrije Zdravkovic


Resolve the \{{GAP FILL}} relation operator in the analyzer, as the second step 
of SPARK-61276.

* The analyzer rule \{{ResolveGapFill}} now turns \{{UnresolvedGapFill}} into a 
\{{Generate}} with a new \{{GapFillGenerator}}, and a \{{Project}} above it 
that keeps the input column order. Only the partition columns pass through the 
\{{Generate}}. Every other column becomes a new nullable column that keeps the 
name, type, metadata and table name of its input column.
* Add \{{GapFillResolution}}, which checks the clauses in a fixed order and 
evaluates their constant values. The single-pass analyzer can use it too.
* Add the analysis errors: 15 new error classes, such as 
\{{GAP_FILL_TIME_TYPE_MISMATCH}} and \{{GAP_FILL_INVALID_RANGE}}, and 4 new 
subclasses of \{{UNSUPPORTED_FEATURE}}: \{{GAP_FILL_FILL_METHOD_TYPE}}, 
\{{GAP_FILL_PARTITION_KEY_TYPE}}, \{{GAP_FILL_TIMESTAMP_DAY_BIN_WIDTH}} and 
\{{GAP_FILL_TIMESTAMP_NANOS}}.
* \{{CheckAnalysis}} rejects a correlated input, a streaming input, and a 
metadata column read after \{{GAP FILL}}.
* \{{OptimizeOneRowRelationSubquery}} does not move a \{{GAP FILL}} onto the 
rows of an outer query.

There is no physical execution yet. Until a later sub-task adds it, the planner 
rejects a resolved \{{GAP FILL}} with \{{UNSUPPORTED_FEATURE.GAP_FILL}}. The 
conf \{{spark.sql.gapFillRelationOperator.enabled}} stays internal and off by 
default.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to