alamb commented on a change in pull request #619:
URL: https://github.com/apache/arrow-datafusion/pull/619#discussion_r658863004
##########
File path: datafusion/src/optimizer/projection_push_down.rs
##########
@@ -75,9 +78,12 @@ fn get_projected_schema(
//
// we discard non-existing columns because some column names are not part
of the schema,
// e.g. when the column derives from an aggregation
- let mut projection: Vec<usize> = required_columns
+ //
+ // Use BTreeSet to remove potential duplicates (e.g. union) as
+ // well as to sort the projection to ensure deterministic behavior
+ let mut projection: BTreeSet<usize> = required_columns
.iter()
- .filter(|c| c.relation.as_ref() == table_name)
+ .filter(|c| c.relation.is_none() || c.relation.as_ref() == table_name)
Review comment:
This is the core change -- don't compare the relation qualifier if there
is none -- otherwise if `c = Column { relation: None, name: "a"}` and the table
name is `Some("foo")` the column will be filtered, even if `foo` has a column
named `a`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
