[
https://issues.apache.org/jira/browse/SPARK-61993?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-61993:
-----------------------------------
Labels: pull-request-available (was: )
> Resolve GAP FILL into a Generate with a GapFillGenerator
> --------------------------------------------------------
>
> Key: SPARK-61993
> URL: https://issues.apache.org/jira/browse/SPARK-61993
> Project: Spark
> Issue Type: Sub-task
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Dimitrije Zdravkovic
> Priority: Major
> Labels: pull-request-available
>
> Resolve the \{{GAP FILL}} relation operator in the analyzer, as the second
> step of SPARK-61276.
> * The analyzer rule \{{ResolveGapFill}} now turns \{{UnresolvedGapFill}} into
> a \{{Generate}} with a new \{{GapFillGenerator}}, and a \{{Project}} above it
> that keeps the input column order. Only the partition columns pass through
> the \{{Generate}}. Every other column becomes a new nullable column that
> keeps the name, type, metadata and table name of its input column.
> * Add \{{GapFillResolution}}, which checks the clauses in a fixed order and
> evaluates their constant values. The single-pass analyzer can use it too.
> * Add the analysis errors: 15 new error classes, such as
> \{{GAP_FILL_TIME_TYPE_MISMATCH}} and \{{GAP_FILL_INVALID_RANGE}}, and 4 new
> subclasses of \{{UNSUPPORTED_FEATURE}}: \{{GAP_FILL_FILL_METHOD_TYPE}},
> \{{GAP_FILL_PARTITION_KEY_TYPE}}, \{{GAP_FILL_TIMESTAMP_DAY_BIN_WIDTH}} and
> \{{GAP_FILL_TIMESTAMP_NANOS}}.
> * \{{CheckAnalysis}} rejects a correlated input, a streaming input, and a
> metadata column read after \{{GAP FILL}}.
> * \{{OptimizeOneRowRelationSubquery}} does not move a \{{GAP FILL}} onto the
> rows of an outer query.
> There is no physical execution yet. Until a later sub-task adds it, the
> planner rejects a resolved \{{GAP FILL}} with
> \{{UNSUPPORTED_FEATURE.GAP_FILL}}. The conf
> \{{spark.sql.gapFillRelationOperator.enabled}} stays internal and off by
> default.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]