umehrot2 commented on code in PR #6154:
URL: https://github.com/apache/hudi/pull/6154#discussion_r927913811
##########
hudi-spark-datasource/hudi-spark-common/src/main/scala/org/apache/hudi/DefaultSource.scala:
##########
@@ -56,6 +56,9 @@ class DefaultSource extends RelationProvider
// Enable "passPartitionByAsOptions" to support "write.partitionBy(...)"
spark.conf.set("spark.sql.legacy.sources.write.passPartitionByAsOptions",
"true")
}
+ // Revisit EMR Spark and EMRFS incompatibilities, for now disable
+ spark.conf.set("spark.sql.dataPrefetch.enabled", "false")
+
spark.sparkContext.hadoopConfiguration.set("fs.s3.metadata.cache.expiration.seconds",
"0")
Review Comment:
@codope Yes the property is specific to `EmrFS` and its not present in open
source implementations. This is done as a temporary fix to make sure Hudi 0.12
can work out of the box on EMR environment. This property should act as No Op
for other environments.
We are also working internally to have a fix for this on EmrFS side instead,
and should get rid of this in next release. It was tested using EMR's internal
integration tests. With the current hudi master it was breaking several of our
tests, and with this fix we get past them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
