Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21004#discussion_r181269206
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/DataSource.scala
---
@@ -95,6 +95,14 @@ case class DataSource(
lazy val sourceInfo: SourceInfo = sourceSchema()
private val caseInsensitiveOptions = CaseInsensitiveMap(options)
private val equality = sparkSession.sessionState.conf.resolver
+ // The operations below are expensive therefore try not to do them if we
don't need to, e.g.,
+ // in streaming mode, we have already inferred and registered partition
columns, we will
+ // never have to materialize the lazy val below
+ private lazy val tempFileIndex = {
--- End diff --
it's only used once, no need to be a lazy val, we can just inline it.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
