Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/22905#discussion_r230616072
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala
---
@@ -306,7 +306,15 @@ case class FileSourceScanExec(
withOptPartitionCount
}
- withSelectedBucketsCount
+ val withOptColumnCount = relation.fileFormat match {
+ case columnar: ColumnarFileFormat =>
+ val sqlConf = relation.sparkSession.sessionState.conf
+ val columnCount = columnar.columnCountForSchema(sqlConf,
requiredSchema)
+ withSelectedBucketsCount + ("ColumnCount" -> columnCount.toString)
--- End diff --
I was wondering how important to know if the columns are pruned or not. In
that way, other logs should be put in metadata. For instance, we're not even
showing the actual filters (not cayalyst but I mean the actual pushed filters
that are going to apply at each source implementation level such as filters
from `ParquetFilters.createFilter`) in Spark side.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
