codope commented on code in PR #11162:
URL: https://github.com/apache/hudi/pull/11162#discussion_r1628207181
##########
hudi-common/src/main/java/org/apache/hudi/metadata/HoodieMetadataLogRecordReader.java:
##########
@@ -253,7 +253,7 @@ public HoodieMetadataLogRecordReader build() {
}
private boolean shouldUseMetadataMergedLogRecordScanner() {
- return PARTITION_NAME_SECONDARY_INDEX.equals(partitionName);
+ return partitionName.startsWith(PARTITION_NAME_SECONDARY_INDEX_PREFIX);
Review Comment:
note to reviewer: this is amin fix in the latest commit. The issue was not
caught in the testing previously because I was only asserting count of records.
I have improved the test. Now, we assert the secondary index records as well as
check that file pruning happens. Please see `TestSecondaryIndexPruning`
##########
hudi-spark-datasource/hudi-spark/src/test/scala/org/apache/hudi/functional/SecondaryIndexTestBase.scala:
##########
@@ -62,4 +69,54 @@ class SecondaryIndexTestBase extends
HoodieSparkClientTestBase {
cleanupResources()
}
+ def verifyQueryPredicate(hudiOpts: Map[String, String], columnName: String):
Unit = {
+ mergedDfList =
spark.read.format("hudi").options(hudiOpts).load(basePath).repartition(1).cache()
:: mergedDfList
+ val secondaryKey = mergedDfList.last.limit(1).collect().map(row =>
row.getAs(columnName).toString)
+ val dataFilter = EqualTo(attribute(columnName), Literal(secondaryKey(0)))
+ verifyFilePruning(hudiOpts, dataFilter)
+ }
+
+ private def attribute(partition: String): AttributeReference = {
+ AttributeReference(partition, StringType, nullable = true)()
+ }
+
+
+ private def verifyFilePruning(opts: Map[String, String], dataFilter:
Expression): Unit = {
Review Comment:
note to reviewer: the method here and below just help with verifying file
pruning happens correctly i.e. with data skipping enabled, filtered file count
< data files in latest snapshot, and with data skipping disabled they should be
equal.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
