gatorsmile commented on a change in pull request #23328:
[SPARK-26327][SQL][FOLLOW-UP] Refactor the code and restore the metrics name
URL: https://github.com/apache/spark/pull/23328#discussion_r242005335
##########
File path:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
##########
@@ -298,12 +308,14 @@ case class FileSourceScanExec(
options = relation.options,
hadoopConf =
relation.sparkSession.sessionState.newHadoopConfWithOptions(relation.options))
- relation.bucketSpec match {
+ val readRDD = relation.bucketSpec match {
case Some(bucketing) if
relation.sparkSession.sessionState.conf.bucketingEnabled =>
createBucketedReadRDD(bucketing, readFile, selectedPartitions,
relation)
case _ =>
createNonBucketedReadRDD(readFile, selectedPartitions, relation)
}
+ sendDriverMetrics()
Review comment:
Line 313 and line 315, both are calling `selectedPartitions`. Thus, it is
safer to say `selectedPartitions` is initialized before we send the driver-side
metrics
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
