[
https://issues.apache.org/jira/browse/HIVE-27101?focusedWorklogId=850126&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-850126
]
ASF GitHub Bot logged work on HIVE-27101:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 09/Mar/23 17:20
Start Date: 09/Mar/23 17:20
Worklog Time Spent: 10m
Work Description: amansinha100 commented on code in PR #4079:
URL: https://github.com/apache/hive/pull/4079#discussion_r1131321197
##########
ql/src/java/org/apache/hadoop/hive/ql/metadata/Hive.java:
##########
@@ -2101,6 +2102,55 @@ public boolean
validateMaterializedViewsFromRegistry(List<Table> cachedMateriali
}
}
+ private Materialization
getMaterializationInvalidationInfo(MaterializedViewMetadata metadata)
+ throws TException, HiveException {
+ Optional<SourceTable> first =
metadata.getSourceTables().stream().findFirst();
+ if (!first.isPresent()) {
+ // This is unexpected: all MV must have at least one source
+ Materialization materialization = new Materialization();
+ materialization.setSourceTablesCompacted(true);
+ materialization.setSourceTablesUpdateDeleteModified(true);
+ return new Materialization();
+ } else {
+ Table table = getTable(first.get().getTable().getDbName(),
first.get().getTable().getTableName());
+ if (!(table.isNonNative() &&
table.getStorageHandler().areSnapshotsSupported())) {
+ // Mixing native and non-native acid source tables are not supported.
If the first source is native acid
+ // the rest is expected to be native acid
+ return getMSC().getMaterializationInvalidationInfo(
+ metadata.creationMetadata,
conf.get(ValidTxnList.VALID_TXNS_KEY));
+ }
+ }
+
+ MaterializationSnapshot mvSnapshot =
MaterializationSnapshot.fromJson(metadata.creationMetadata.getValidTxnList());
+
+ boolean hasDelete = false;
+ for (SourceTable sourceTable : metadata.getSourceTables()) {
+ Table table = getTable(sourceTable.getTable().getDbName(),
sourceTable.getTable().getTableName());
+ HiveStorageHandler storageHandler = table.getStorageHandler();
+ if (storageHandler == null) {
+ Materialization materialization = new Materialization();
+ materialization.setSourceTablesCompacted(true);
+ return materialization;
+ }
+ Boolean b = storageHandler.hasDeletes(
Review Comment:
Does hasDeletes() api cover all types of CRUD operations that are not insert
operations ? e.g updates, truncate table, drop partition etc.
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/views/HivePushdownSnapshotFilterRule.java:
##########
@@ -0,0 +1,156 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to you under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.hadoop.hive.ql.optimizer.calcite.rules.views;
+
+import org.apache.calcite.plan.RelOptRule;
+import org.apache.calcite.plan.RelOptRuleCall;
+import org.apache.calcite.plan.RelOptTable;
+import org.apache.calcite.plan.RelRule;
+import org.apache.calcite.rel.RelNode;
+import org.apache.calcite.rel.metadata.RelMetadataQuery;
+import org.apache.calcite.rel.type.RelDataTypeField;
+import org.apache.calcite.rex.RexBuilder;
+import org.apache.calcite.rex.RexCall;
+import org.apache.calcite.rex.RexInputRef;
+import org.apache.calcite.rex.RexLiteral;
+import org.apache.calcite.rex.RexNode;
+import org.apache.calcite.rex.RexShuttle;
+import org.apache.calcite.rex.RexTableInputRef;
+import org.apache.calcite.sql.SqlKind;
+import org.apache.calcite.sql.type.SqlTypeFamily;
+import org.apache.hadoop.hive.ql.metadata.VirtualColumn;
+import org.apache.hadoop.hive.ql.optimizer.calcite.HiveRelFactories;
+import org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable;
+import org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveFilter;
+
+import java.util.Set;
+
+/**
+ * Calcite rule to push down predicates contains {@link
VirtualColumn#SNAPSHOT_ID} reference to TableScan.
+ * <p>
+ * This rule traverse the logical expression in {@link HiveFilter} operators
and search for
+ * predicates like
+ * <p>
+ * <code>
+ * snapshotId <= 12345677899
+ * </code>
+ * <p>
+ * The literal is set in the {@link RelOptHiveTable#getHiveTableMD()} object
wrapped by
+ * {@link
org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan}
+ * and the original predicate in the {@link HiveFilter} is replaced with
literal true.
Review Comment:
nit: At another place I saw a mention that the HiveFilter would be dropped.
If that is done by a different rule, you may want to mention that here for
clarification.
##########
iceberg/iceberg-handler/src/test/queries/positive/mv_iceberg_orc4.q:
##########
@@ -0,0 +1,42 @@
+
Issue Time Tracking
-------------------
Worklog Id: (was: 850126)
Time Spent: 7h 20m (was: 7h 10m)
> Support incremental materialized view rebuild when Iceberg source tables have
> insert operation only.
> ----------------------------------------------------------------------------------------------------
>
> Key: HIVE-27101
> URL: https://issues.apache.org/jira/browse/HIVE-27101
> Project: Hive
> Issue Type: Improvement
> Components: Iceberg integration, Materialized views
> Reporter: Krisztian Kasa
> Assignee: Krisztian Kasa
> Priority: Major
> Labels: pull-request-available
> Time Spent: 7h 20m
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
