Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/21118#discussion_r204030846
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/DataSourceV2Strategy.scala
---
@@ -125,16 +125,13 @@ object DataSourceV2Strategy extends Strategy {
val filterCondition = postScanFilters.reduceLeftOption(And)
val withFilter = filterCondition.map(FilterExec(_,
scan)).getOrElse(scan)
- val withProjection = if (withFilter.output != project) {
- ProjectExec(project, withFilter)
- } else {
- withFilter
- }
-
- withProjection :: Nil
+ // always add the projection, which will produce unsafe rows
required by some operators
+ ProjectExec(project, withFilter) :: Nil
case r: StreamingDataSourceV2Relation =>
- DataSourceV2ScanExec(r.output, r.source, r.options, r.pushedFilters,
r.reader) :: Nil
+ // ensure there is a projection, which will produce unsafe rows
required by some operators
+ ProjectExec(r.output,
--- End diff --
For now I think it's safer to still require `DataSourceV2ScanExec` to
return unsafe rows and move the unsafe conversion to here in a followup PR.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
