cshuo commented on code in PR #20672:
URL: https://github.com/apache/hudi/pull/20672#discussion_r4613438986


##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/BloomFiltersIndexSupport.scala:
##########
@@ -23,16 +23,32 @@ import org.apache.hudi.HoodieConversionUtils.toScalaOption
 import org.apache.hudi.common.config.HoodieMetadataConfig
 import org.apache.hudi.common.model.FileSlice
 import org.apache.hudi.common.table.HoodieTableMetaClient
+import org.apache.hudi.config.HoodieIndexConfig
 import org.apache.hudi.core.read.BaseHoodieTableFileIndex
+import org.apache.hudi.index.HoodieIndex.IndexType
 import org.apache.hudi.metadata.HoodieTableMetadataUtil
 
 import org.apache.spark.sql.SparkSession
 import org.apache.spark.sql.catalyst.expressions.Expression
 
+import scala.collection.JavaConverters._
+
 class BloomFiltersIndexSupport(spark: SparkSession,
                                metadataConfig: HoodieMetadataConfig,
                                metaClient: HoodieTableMetaClient) extends 
SparkBaseIndexSupport(spark, metadataConfig, metaClient) {
 
+  // Like bucket pruning, trust the declared reader index type to match the 
writer. These Spark
+  // indexes write new keys to base files. A custom class overrides the type 
and has no such guarantee.
+  // Do not infer the writer's default SIMPLE index when the reader has not 
declared a type.
+  private val canPruneLogSlices = {
+    val indexType = 
metadataConfig.getStringOrDefault(HoodieIndexConfig.INDEX_TYPE, "")
+    metadataConfig.getStringOrDefault(HoodieIndexConfig.INDEX_CLASS_NAME, 
"").isEmpty &&
+      
BloomFiltersIndexSupport.BASE_BLOOM_COVERED_INDEX_TYPES.exists(_.name().equalsIgnoreCase(indexType))
+  }
+
+  private def mustRetainSlice(fileSlice: FileSlice): Boolean =
+    fileSlice.hasLogFiles && (fileSlice.getBaseFile.isEmpty || 
!canPruneLogSlices)

Review Comment:
   The allowlist assumes that the reader’s hoodie.index.type matches the index 
used to write the file slice. The reader has no reliable way to determine that 
from the slice itself. Spark’s simple bucket pruning makes a similar assumption 
when it trusts the reader’s bucket configuration to match the on-disk layout.
   
   As you pointed out, a reader configured with SIMPLE can still encounter logs 
written earlier with BUCKET. If we want snapshot queries to be correct 
regardless of that history, the conservative approach is to retain every file 
slice that contains logs and use base blooms to prune base-only slices. The 
trade-off is less effective pruning when the logs only update keys already 
present in the base.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to