yihua commented on code in PR #8082:
URL: https://github.com/apache/hudi/pull/8082#discussion_r1187725527
##########
hudi-client/hudi-spark-client/src/main/scala/org/apache/hudi/HoodieSparkUtils.scala:
##########
@@ -58,6 +58,7 @@ private[hudi] trait SparkVersionsSupport {
def gteqSpark3_2_1: Boolean = getSparkVersion >= "3.2.1"
def gteqSpark3_2_2: Boolean = getSparkVersion >= "3.2.2"
def gteqSpark3_3: Boolean = getSparkVersion >= "3.3"
+ def gteqSpark3_3_2: Boolean = getSparkVersion >= "3.3.2"
Review Comment:
Is this based on compile-time Spark version or runtime Spark version?
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/io/storage/HoodieSparkFileReaderFactory.java:
##########
@@ -33,6 +33,8 @@ protected HoodieFileReader newParquetFileReader(Configuration
conf, Path path) {
conf.setIfUnset(SQLConf.PARQUET_BINARY_AS_STRING().key(),
SQLConf.PARQUET_BINARY_AS_STRING().defaultValueString());
conf.setIfUnset(SQLConf.PARQUET_INT96_AS_TIMESTAMP().key(),
SQLConf.PARQUET_INT96_AS_TIMESTAMP().defaultValueString());
conf.setIfUnset(SQLConf.CASE_SENSITIVE().key(),
SQLConf.CASE_SENSITIVE().defaultValueString());
+ // Using string value of this conf to preserve compatibility across spark
versions.
+ conf.setIfUnset("spark.sql.legacy.parquet.nanosAsLong", "false");
Review Comment:
Could you use a variable to define the default value `false`?
##########
hudi-spark-datasource/hudi-spark3.2plus-common/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/Spark32PlusHoodieParquetFileFormat.scala:
##########
@@ -95,7 +95,11 @@ class Spark32PlusHoodieParquetFileFormat(private val
shouldAppendPartitionValues
hadoopConf.setBoolean(
SQLConf.PARQUET_INT96_AS_TIMESTAMP.key,
sparkSession.sessionState.conf.isParquetINT96AsTimestamp)
-
+ // Using string value of this conf to preserve compatibility across spark
versions.
+ hadoopConf.setBoolean(
+ "spark.sql.legacy.parquet.nanosAsLong",
+
sparkSession.sessionState.conf.getConfString("spark.sql.legacy.parquet.nanosAsLong",
"false").toBoolean
Review Comment:
Similar here for the default value `false`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
