Github user marmbrus commented on a diff in the pull request:
https://github.com/apache/spark/pull/11509#discussion_r55257029
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/ExistingRDD.scala ---
@@ -226,16 +226,17 @@ private[sql] object PhysicalRDD {
rdd: RDD[InternalRow],
relation: BaseRelation,
metadata: Map[String, String] = Map.empty): PhysicalRDD = {
- val outputUnsafeRows = if (relation.isInstanceOf[ParquetRelation]) {
- // The vectorized parquet reader does not produce unsafe rows.
-
!SQLContext.getActive().get.conf.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED)
- } else {
- // All HadoopFsRelations output UnsafeRows
- relation.isInstanceOf[HadoopFsRelation]
+
+ val outputUnsafeRows = relation match {
+ case r: HadoopFsRelation if r.fileFormat.isInstanceOf[ParquetSource]
=>
+
!SQLContext.getActive().get.conf.getConf(SQLConf.PARQUET_VECTORIZED_READER_ENABLED)
+ case _: HadoopFsRelation => true
+ case _ => false
--- End diff --
Yes, there is not interface to do that today since the only version returns
an RDD[InternalRow], but
[SPARK-13682](https://issues.apache.org/jira/browse/SPARK-13682) tracks fixing
that.
When we do this we should avoid hacks that rely on erasure in the bytecode.
Instead just have an internal function that returns `InternalRow` by
converting the result of calling the public version, which returns `Row`.
Internal implementations like parquet can circumvent this extra step and just
override the conversion function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
