james-willis commented on code in PR #27143:
URL: https://github.com/apache/datafusion/pull/27143#discussion_r4605033618
##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -685,11 +688,35 @@ impl ExprSchemable for Expr {
))),
}
}
+ Expr::Case(case) => {
+ let fields = case
+ .when_then_expr
+ .iter()
+ .map(|(_, then)| then)
+ .chain(case.else_expr.iter())
+ .map(|expr| expr.to_field(schema).map(|(_, field)| field))
+ .collect::<Result<Vec<_>>>()?;
+ let metadata =
+ coerced_fields_metadata(fields.iter().map(|f|
f.as_ref()))?;
+ // Same as `get_type`: the first non-NULL branch type. Reading
it from
+ // `fields` avoids a second walk of the branches, which
multiplies with
+ // the depth of nested CASE and function calls.
+ let data_type = fields
+ .iter()
+ .map(|f| f.data_type())
+ .find(|t| !t.is_null())
+ .cloned()
+ .unwrap_or(DataType::Null);
+ Ok(Arc::new(metadata.add_to_field(Field::new(
+ &schema_name,
+ data_type,
+ self.nullable(schema)?,
Review Comment:
Current main is exponential on this shape. Bisected our base and main and
found #27139, merged Nov 6. It changed CASE get_type to read every branch, and
main’s to_field calls get_type and then nullable(), so the ELSE is read twice
per level.
Coincidentally, once I was looking at 27139, I noticed its change also
conflicts with this PR. Merged as-is, our to_field would undo it: DESCRIBE
SELECT CASE WHEN false THEN 1 ELSE 2.5 END reports Int64 instead of Float64. A
test already on main catches this, but this PR’s CI skips the full test jobs,
so it would first have failed in the merge queue.
So I merged main and put 27139’s type rule now lives in one helper that
get_type and our to_field both call so each branch is still read only once.
I also timed another 17 nesting shapes, inspired by this one you found. 16
were fine but one had a similar regression which I've handled here as well as
added a test.
Will post the updated commit + description update here in a second.
I didn't appreciate how complex this part of the code base was when I
started working on this 😅
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]