rahil-c commented on code in PR #13650:
URL: https://github.com/apache/hudi/pull/13650#discussion_r2357273362
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/SparkBaseIndexSupport.scala:
##########
@@ -174,13 +173,14 @@ abstract class SparkBaseIndexSupport(spark: SparkSession,
var recordKeyQueries: List[Expression] = List.empty
var compositeRecordKeys: List[String] = List.empty
val recordKeyOpt = getRecordKeyConfig
- val isComplexRecordKey = {
- val fieldCount = recordKeyOpt.map(recordKeys =>
recordKeys.length).getOrElse(0)
- // Consider as complex if:
- // 1. Multiple fields (> 1), OR
- // 2. Using complex key generator with single record key field
- fieldCount > 1 ||
isComplexKeyGeneratorWithSingleRecordKeyField(metaClient.getTableConfig)
+
+ if
(KeyGenUtils.mayUseNewEncodingForComplexKeyGen(metaClient.getTableConfig)) {
+ // Avoid using the index for this case
+ return (List.empty, List.empty)
}
+ // for other cases though we should be able to handle complex key
+ // a complex key can be a single field or multiple fields based on hudi
docs, regardless of encoding
+ val isComplexRecordKey = recordKeyOpt.map(recordKeys =>
recordKeys.length).getOrElse(0) >= 1
Review Comment:
Discussed with @yihua, for now the condition should be > 1, or user is using
complexKey with single record field. His recommendation is to use the helper
function he introduced in this pr.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
