Ryan Blue created SPARK-28878:
---------------------------------
Summary: DataSourceV2 should not insert extra projection for
columnar batches
Key: SPARK-28878
URL: https://issues.apache.org/jira/browse/SPARK-28878
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 3.0.0
Reporter: Ryan Blue
SPARK-23325 added an extra physical projection when reading from a DSv2 source
because some Spark operators assume that InternalRow instances are actually
UnsafeRow. The projection ensures that InternalRow is converted to UnsafeRow.
This isn't needed for the columnar batch read path because this is already done
when converting from columnar operators to row-based operators in
InputRDDCodegen.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
