nsivabalan commented on code in PR #18205:
URL: https://github.com/apache/hudi/pull/18205#discussion_r2829817790
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -1076,7 +1076,18 @@ object DataSourceOptionsHelper {
def parametersWithReadDefaults(parameters: Map[String, String]): Map[String,
String] = {
// First check if the ConfigUtils.IS_QUERY_AS_RO_TABLE has set by
HiveSyncTool,
// or else use query type from QUERY_TYPE.
- val paramsWithGlobalProps =
DFSPropertiesConfiguration.getGlobalProps.asScala.toMap ++ parameters
+ // Config precedence (low -> high):
+ // 1) global DFS props
+ // 2) spark.hoodie.* (normalized to hoodie.*)
+ // 3) hoodie.* / explicit data source options
+ // NOTE: If both spark.hoodie.X and hoodie.X are set, hoodie.X wins.
+ val normalizedSparkHoodieConfigs = parameters.collect {
+ case (key, value) if key.startsWith("spark.hoodie.") =>
(key.stripPrefix("spark."), value)
Review Comment:
can we declare static variables for "spark.hoodie." and "spark." and use it
everywhere rather than hard coding it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
