Github user gatorsmile commented on a diff in the pull request:
https://github.com/apache/spark/pull/21415#discussion_r190701039
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/csv/CSVOptions.scala
---
@@ -29,17 +29,20 @@ import org.apache.spark.sql.catalyst.util._
class CSVOptions(
@transient val parameters: CaseInsensitiveMap[String],
defaultTimeZoneId: String,
- defaultColumnNameOfCorruptRecord: String)
+ defaultColumnNameOfCorruptRecord: String,
+ val columnPruning: Boolean)
extends Logging with Serializable {
def this(
parameters: Map[String, String],
defaultTimeZoneId: String,
- defaultColumnNameOfCorruptRecord: String = "") = {
+ defaultColumnNameOfCorruptRecord: String = "",
+ columnPruning: Boolean = false) = {
--- End diff --
always enabling it is also not right. Can we remove the default?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
