Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/19907#discussion_r155313387
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/orc/OrcFileFormat.scala
---
@@ -167,8 +169,10 @@ class OrcFileFormat
val iter = new RecordReaderIterator[OrcStruct](orcRecordReader)
Option(TaskContext.get()).foreach(_.addTaskCompletionListener(_ =>
iter.close()))
- val unsafeProjection = UnsafeProjection.create(requiredSchema)
- val deserializer = new OrcDeserializer(dataSchema, requiredSchema,
requestedColIds)
+ val colIds = requestedColIds ++
List.fill(partitionSchema.length)(-1).toArray[Int]
+ val unsafeProjection = UnsafeProjection.create(resultSchema)
--- End diff --
Oh, I see. you meant non-vectorized path. Sorry, I was confused since I
focused too much on vectorized path. I'll do.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
