zhedoubushishi commented on a change in pull request #2208:
URL: https://github.com/apache/hudi/pull/2208#discussion_r533108221
##########
File path:
hudi-spark/src/main/scala/org/apache/hudi/MergeOnReadSnapshotRelation.scala
##########
@@ -113,9 +113,6 @@ class MergeOnReadSnapshotRelation(val sqlContext:
SQLContext,
hadoopConf = sqlContext.sparkSession.sessionState.newHadoopConf()
)
- // Follow the implementation of Spark internal HadoopRDD to handle the
broadcast configuration.
Review comment:
```SparkHadoopUtil``` becomes ```private``` in spark 3, so I need to see
if these lines are necessary.
The reason this is needed in internal implementation is to guard against
case where user passes a custom Configuration which doesn't contain credentials
to access secure HDFS (https://github.com/apache/spark/pull/2676). Since the
Configuration being used here was created as part of spark context the
credentials should already be loaded. So we can remove it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
