Noémi Pap-Takács created IMPALA-15072:
-----------------------------------------
Summary: Exclude OPTIMIZE TABLE maintenance operation from lineage
graph
Key: IMPALA-15072
URL: https://issues.apache.org/jira/browse/IMPALA-15072
Project: IMPALA
Issue Type: Bug
Reporter: Noémi Pap-Takács
Assignee: Noémi Pap-Takács
Running OPTIMIZE TABLE on Iceberg V3 tables migrated from V2 throws an
IllegalStateException in ColumnLineageGraph:
{noformat}
java.lang.RuntimeException: java.lang.IllegalStateException at
org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2473)
at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2140)
at
org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:180)
Caused by: java.lang.IllegalStateException at
com.google.common.base.Preconditions.checkState(Preconditions.java:497) at
org.apache.impala.analysis.ColumnLineageGraph.computeProjectionDependencies(ColumnLineageGraph.java:637)
at
org.apache.impala.analysis.ColumnLineageGraph.computeLineageGraph(ColumnLineageGraph.java:607)
at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:281)
at org.apache.impala.planner.Planner.createPlans(Planner.java:314) at
org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2078){noformat}
The Precondition check
{code:java}
Preconditions.checkState(resultExprs.size() ==
targetColumnLabels_.size());{code}
fails because V3 tables have 2 hidden columns: {{_file_row_id}} and
{{_file_last_updated_sequence_number.}}
'targetColumnLabels_' contains them but 'resultExprs' filters out hidden
columns.
Proposed solution:
We could exclude OPTIMIZE TABLE from the lineage graph, since it is a
maintenance operation that unnecessarily pollutes the graph.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
