nsivabalan commented on code in PR #7951:
URL: https://github.com/apache/hudi/pull/7951#discussion_r1114942532
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DataSourceOptions.scala:
##########
@@ -295,30 +295,35 @@ object DataSourceWriteOptions {
def translateSqlOptions(optParams: Map[String, String]): Map[String, String]
= {
var translatedOptParams = optParams
// translate the api partitionBy of spark DataFrameWriter to
PARTITIONPATH_FIELD
- if (optParams.contains(SparkDataSourceUtils.PARTITIONING_COLUMNS_KEY)) {
+ // we should set hoodie's partition path only if its not set by the user.
+ if (optParams.contains(SparkDataSourceUtils.PARTITIONING_COLUMNS_KEY)
Review Comment:
this might be backwards incompatible change. but not sure if previous
behavior was supported by mistake.
for eg, if some sets hoodie partiiton path field to col1, but incoming df
had col2, prior to this patch, col2 will be considered as partitioning col for
hudi. but after this patch, it will be col1.
only if user did not explicitly set hoodie partition path config, we will
use col2.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
