Github user cloud-fan commented on a diff in the pull request:
    --- Diff: 
    @@ -99,15 +100,22 @@ object PushDownOperatorsToDataSource extends 
Rule[LogicalPlan] with PredicateHel
           case relation: DataSourceV2Relation => relation.reader match {
             case reader: SupportsPushDownRequiredColumns =>
    +          // TODO: Enable the below assert after we make 
`DataSourceV2Relation` immutable. Fow now
    +          // it's possible that the mutable reader being updated by 
someone else, and we need to
    +          // always call `reader.pruneColumns` here to correct it.
    +          // assert(relation.output.toStructType == reader.readSchema(),
    +          //  "Schema of data source reader does not match the relation 
               val requiredColumns = 
    +          relation.copy(output = requiredColumns)
    --- End diff --
    @rdblue This is the bug I mentioned before. Finally I figured out a way to 
fix it surgically: always run column pruning even no column needs to be pruned. 
This helps us correct the required schema of the reader, if it's updated by 
someone else.


To unsubscribe, e-mail:
For additional commands, e-mail:

Reply via email to