[
https://issues.apache.org/jira/browse/IMPALA-15072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18086992#comment-18086992
]
ASF subversion and git services commented on IMPALA-15072:
----------------------------------------------------------
Commit 2be15d0d9f1d0be64428f3b4f4dd1569dd37782c in impala's branch
refs/heads/master from Noemi Pap-Takacs
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=2be15d0d9 ]
IMPALA-15072: Exclude OPTIMIZE TABLE maintenance operation from lineage graph
OPTIMIZE TABLE is a maintenance operation on Iceberg tables that
rewrites the table to a compact form preserving all its content
unchanged.
Maintenance operations can unnecessarily pollute the lineage graph
without adding value, therefore this patch excludes OPTIMIZE TABLE
from the column lineage graph.
Testing:
- added custom cluster test
Change-Id: I7f50e9e719d290440e2386c15f8f6517bf2d09ae
Reviewed-on: http://gerrit.cloudera.org:8080/24398
Reviewed-by: Impala Public Jenkins <[email protected]>
Tested-by: Impala Public Jenkins <[email protected]>
> Exclude OPTIMIZE TABLE maintenance operation from lineage graph
> ---------------------------------------------------------------
>
> Key: IMPALA-15072
> URL: https://issues.apache.org/jira/browse/IMPALA-15072
> Project: IMPALA
> Issue Type: Bug
> Reporter: Noémi Pap-Takács
> Assignee: Noémi Pap-Takács
> Priority: Major
>
> Running OPTIMIZE TABLE on Iceberg V3 tables migrated from V2 throws an
> IllegalStateException in ColumnLineageGraph:
> {noformat}
> java.lang.RuntimeException: java.lang.IllegalStateException at
> org.apache.impala.service.Frontend.getTExecRequestWithFallback(Frontend.java:2473)
> at org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2140)
> at
> org.apache.impala.service.JniFrontend.createExecRequest(JniFrontend.java:180)
> Caused by: java.lang.IllegalStateException at
> com.google.common.base.Preconditions.checkState(Preconditions.java:497) at
> org.apache.impala.analysis.ColumnLineageGraph.computeProjectionDependencies(ColumnLineageGraph.java:637)
> at
> org.apache.impala.analysis.ColumnLineageGraph.computeLineageGraph(ColumnLineageGraph.java:607)
> at org.apache.impala.planner.Planner.createPlanFragments(Planner.java:281)
> at org.apache.impala.planner.Planner.createPlans(Planner.java:314) at
> org.apache.impala.service.Frontend.createExecRequest(Frontend.java:2078){noformat}
> The Precondition check
> {code:java}
> Preconditions.checkState(resultExprs.size() ==
> targetColumnLabels_.size());{code}
> fails because V3 tables have 2 hidden columns: {{_file_row_id}} and
> {{_file_last_updated_sequence_number.}}
> 'targetColumnLabels_' contains them but 'resultExprs' filters out hidden
> columns.
>
> Proposed solution:
> We could exclude OPTIMIZE TABLE from the lineage graph, since it is a
> maintenance operation that unnecessarily pollutes the graph.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
