james-willis commented on code in PR #27143:
URL: https://github.com/apache/datafusion/pull/27143#discussion_r4605033618


##########
datafusion/expr/src/expr_schema.rs:
##########
@@ -685,11 +688,35 @@ impl ExprSchemable for Expr {
                     ))),
                 }
             }
+            Expr::Case(case) => {
+                let fields = case
+                    .when_then_expr
+                    .iter()
+                    .map(|(_, then)| then)
+                    .chain(case.else_expr.iter())
+                    .map(|expr| expr.to_field(schema).map(|(_, field)| field))
+                    .collect::<Result<Vec<_>>>()?;
+                let metadata =
+                    coerced_fields_metadata(fields.iter().map(|f| 
f.as_ref()))?;
+                // Same as `get_type`: the first non-NULL branch type. Reading 
it from
+                // `fields` avoids a second walk of the branches, which 
multiplies with
+                // the depth of nested CASE and function calls.
+                let data_type = fields
+                    .iter()
+                    .map(|f| f.data_type())
+                    .find(|t| !t.is_null())
+                    .cloned()
+                    .unwrap_or(DataType::Null);
+                Ok(Arc::new(metadata.add_to_field(Field::new(
+                    &schema_name,
+                    data_type,
+                    self.nullable(schema)?,

Review Comment:
   Current main is exponential on this shape too. I bisected between this PR’s 
merge base and main and landed on 
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139), 
merged Nov 6. It changed CASE `get_type` to read every branch, and main’s 
`to_field` calls `get_type` and then `nullable()`, so the ELSE is read twice 
per level. At 18 levels, the commit before 
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139) 
plans your query in 0.06 s and 
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139) 
takes 22 s.
   
   While looking at 
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139), I 
noticed it also conflicts with this PR. Merged as-is, our `to_field` would undo 
it: `DESCRIBE SELECT CASE WHEN false THEN 1 ELSE 2.5 END` reports `Int64` 
instead of `Float64`. A test already on main catches this 
(`union_infers_types_before_analysis`), but this PR’s CI skips the full test 
jobs, so it would first have failed in the merge queue.
   
   So I merged main and moved 
[apache/datafusion#27139](https://github.com/apache/datafusion/pull/27139)’s 
type rule into one helper that `get_type` and our `to_field` both call, so each 
branch is still read only once. Your `case_nullable` change is in too.
   
   I also timed 17 nesting shapes, including this one. Against current main, 10 
take the same time, and 6 that are exponential on main are linear here. The 
last one, COALESCE nested in its first argument, was about 1.4x slower than 
main because the constant-WHEN simplification read the branches several times. 
It now reads each once and is about 1.06x main. That shape is still exponential 
on both, because the simplifier rewrites COALESCE to a CASE that repeats the 
argument.
   
   Pushed in afd47cb39, and the description’s performance section has the full 
numbers.
   
   I didn’t appreciate how complex this part of the code base was when I 
started working on this 😅
   



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to