cloud-fan commented on code in PR #48172:
URL: https://github.com/apache/spark/pull/48172#discussion_r1768265554
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -609,6 +611,16 @@ case class FileSourceScanExec(
override val disableBucketedScan: Boolean = false)
extends FileSourceScanLike {
+ /** SQL metrics generated for JSON scans involving variants. */
+ protected lazy val variantBuilderMetrics: Map[String, SQLMetric] =
+ VariantConstructionMetrics.createSQLMetrics(sparkContext)
+
+ /** Only report variant metrics if the data source file format is JSON */
+ override lazy val metrics: Map[String, SQLMetric] = super.metrics ++ {
+ if (relation.fileFormat.isInstanceOf[JsonFileFormat]) variantBuilderMetrics
Review Comment:
does json scan always produce variants? If not we should only display these
metrics when variant will be produced by the scan.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
