Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/17436#discussion_r151473518
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
---
@@ -364,8 +372,10 @@ class ParquetFileFormat
if (pushed.isDefined) {
ParquetInputFormat.setFilterPredicate(hadoopAttemptContext.getConfiguration,
pushed.get)
}
+ val taskContext = Option(TaskContext.get())
val parquetReader = if (enableVectorizedReader) {
- val vectorizedReader = new VectorizedParquetRecordReader()
+ val vectorizedReader =
+ new VectorizedParquetRecordReader(enableOffHeapColumnVector)
--- End diff --
only enable it when taskContext exist?
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
