Jefffrey commented on code in PR #5612:
URL: https://github.com/apache/arrow-datafusion/pull/5612#discussion_r1137606275
##########
datafusion/core/src/dataframe.rs:
##########
@@ -627,22 +627,12 @@ impl DataFrame {
/// # }
/// ```
pub async fn count(self) -> Result<usize> {
- let rows = self
- .aggregate(
- vec![],
- vec![datafusion_expr::count(Expr::Literal(ScalarValue::Null))],
- )?
- .collect()
- .await?;
- let len = *rows
- .first()
- .and_then(|r| r.columns().first())
- .and_then(|c| c.as_any().downcast_ref::<Int64Array>())
- .and_then(|a| a.values().first())
- .ok_or(DataFusionError::Internal(
- "Unexpected output when collecting for count()".to_string(),
- ))? as usize;
- Ok(len)
+ let mut stream = self.execute_stream().await?;
Review Comment:
You could keep the old behaviour and use
`datafusion_exr::utils::COUNT_STAR_EXPANSION` instead of the
`ScalarValue::Null`, since `count(*)` is expanded to that in SQL planner
(related bug https://github.com/apache/arrow-datafusion/pull/5518 prevents
directly using `Count(Expr:Wildcard)` at the moment)
The reason I had it that way was to let the count optimize out any column
projections to only the minimal needed, which I think theoretically is faster
than just executing the plan as is and counting the rows
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
