Mister-Meeseeks commented on a change in pull request #23830: Decrease
processing overhead on DataFrameReader CSV calls with specif…
URL: https://github.com/apache/spark/pull/23830#discussion_r258121214
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/DataFrameReader.scala
##########
@@ -508,7 +508,9 @@ class DataFrameReader private[sql](sparkSession:
SparkSession) extends Logging {
sparkSession.sessionState.conf.sessionLocalTimeZone)
val filteredLines: Dataset[String] =
CSVUtils.filterCommentAndEmpty(csvDataset, parsedOptions)
- val maybeFirstLine: Option[String] = filteredLines.take(1).headOption
+ val maybeFirstLine: Option[String] =
+ if (userSpecifiedSchema.isEmpty)
+ filteredLines.take(1).headOption else None
Review comment:
Thanks guys for pointing that out. I pushed a new commit to the PR to
address that.
To go into more detail. It seems that the first line scan accomplishes three
goals:
- Schema inference
- Schema enforcement when schema is specified by the user.
- Scanning the header string, so we know which lines to drop.
If the user specifies a schema and header is false, then none of the above
implies. The `linesWithoutHeader` is just a computationally wasteful noop. When
header is false, `CSVHeaderChecker` takes no action. And `CSVUtils` just passes
through the unfiltered iterator.
Therefore as long as schema is set, and header is false, then
short-circuiting `maybeFirstLine` does not change any downstream behavior.
And for that that, we get major efficiency gains. The entire executor (or
entire cluster if dynamic allocation isn't enabled) is sitting idle, waiting
for 1 core to run an entire job just to get 1 line.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
