alamb commented on code in PR #17837:
URL: https://github.com/apache/datafusion/pull/17837#discussion_r2395104936
##########
datafusion/functions-aggregate/src/string_agg.rs:
##########
@@ -138,7 +145,21 @@ impl AggregateUDFImpl for StringAgg {
}
fn state_fields(&self, args: StateFieldsArgs) -> Result<Vec<FieldRef>> {
- self.array_agg.state_fields(args)
+ // See comments in `impl AggregateUDFImpl ...` for more detail
+ let no_order_no_distinct =
+ (args.ordering_fields.is_empty()) && (!args.is_distinct);
+ if no_order_no_distinct {
+ // Case `SimpleStringAggAccumulator`
+ Ok(vec![Field::new(
+ format_state_name(args.name, "string_agg"),
+ DataType::LargeUtf8,
+ true,
+ )
+ .into()])
Review Comment:
It would be nice to put this as part of SimpleStringAggAccumulator,
something like
```rust
SimpleStringAggAccumulator::state_fields(args)
```
##########
datafusion/functions-aggregate/src/string_agg.rs:
##########
@@ -161,21 +182,31 @@ impl AggregateUDFImpl for StringAgg {
);
};
- let array_agg_acc = self.array_agg.accumulator(AccumulatorArgs {
- return_field: Field::new(
- "f",
- DataType::new_list(acc_args.return_field.data_type().clone(),
true),
- true,
- )
- .into(),
- exprs: &filter_index(acc_args.exprs, 1),
- ..acc_args
- })?;
+ // See comments in `impl AggregateUDFImpl ...` for more detail
+ let no_order_no_distinct =
+ acc_args.order_bys.is_empty() && (!acc_args.is_distinct);
- Ok(Box::new(StringAggAccumulator::new(
- array_agg_acc,
- delimiter,
- )))
+ if no_order_no_distinct {
+ // simple case (more efficient)
+ Ok(Box::new(SimpleStringAggAccumulator::new(delimiter)))
+ } else {
+ // general case
+ let array_agg_acc = self.array_agg.accumulator(AccumulatorArgs {
Review Comment:
ditto here for encapsulating this
##########
datafusion/functions-aggregate/src/string_agg.rs:
##########
@@ -269,6 +301,105 @@ fn filter_index<T: Clone>(values: &[T], index: usize) ->
Vec<T> {
.collect::<Vec<_>>()
}
+/// StringAgg accumulator for the simple case (no order or distinct specified)
+/// This accumulator is more efficient than `StringAggAccumulator`
+/// because it accumulates the string directly,
+/// whereas `StringAggAccumulator` uses `ArrayAggAccumulator`.
+#[derive(Debug)]
+pub(crate) struct SimpleStringAggAccumulator {
+ delimiter: String,
+ /// Updated during `update_batch()`. e.g. "foo,bar"
+ accumulated_string: String,
Review Comment:
Rater than `has_value` perhaps using an option would be better / more rust
idomatic and harder to misuse
```rust
accumulated_string: Option<String>,
```
##########
datafusion/functions-aggregate/src/string_agg.rs:
##########
@@ -269,6 +301,105 @@ fn filter_index<T: Clone>(values: &[T], index: usize) ->
Vec<T> {
.collect::<Vec<_>>()
}
+/// StringAgg accumulator for the simple case (no order or distinct specified)
+/// This accumulator is more efficient than `StringAggAccumulator`
+/// because it accumulates the string directly,
+/// whereas `StringAggAccumulator` uses `ArrayAggAccumulator`.
+#[derive(Debug)]
+pub(crate) struct SimpleStringAggAccumulator {
+ delimiter: String,
+ /// Updated during `update_batch()`. e.g. "foo,bar"
+ accumulated_string: String,
+ has_value: bool,
+}
+
+impl SimpleStringAggAccumulator {
+ pub fn new(delimiter: &str) -> Self {
+ Self {
+ delimiter: delimiter.to_string(),
+ accumulated_string: "".to_string(),
+ has_value: false,
+ }
+ }
+
+ #[inline]
+ fn append_strings<'a, I>(&mut self, iter: I)
+ where
+ I: Iterator<Item = Option<&'a str>>,
+ {
+ for value in iter.flatten() {
+ if self.has_value {
+ self.accumulated_string.push_str(&self.delimiter);
+ }
+
+ self.accumulated_string.push_str(value);
+ self.has_value = true;
Review Comment:
If you used an option, this could be like
```suggestion
if let Some(accumulated_value) = self.accumulated_value.as_mut()
{
accumulated_string.push_str(&self.delimiter);
} else {
self.accumulated_valie = Some(String::from(&value))
}
```
##########
datafusion/functions-aggregate/src/string_agg.rs:
##########
@@ -269,6 +301,105 @@ fn filter_index<T: Clone>(values: &[T], index: usize) ->
Vec<T> {
.collect::<Vec<_>>()
}
+/// StringAgg accumulator for the simple case (no order or distinct specified)
+/// This accumulator is more efficient than `StringAggAccumulator`
+/// because it accumulates the string directly,
+/// whereas `StringAggAccumulator` uses `ArrayAggAccumulator`.
+#[derive(Debug)]
+pub(crate) struct SimpleStringAggAccumulator {
Review Comment:
Yes, this is likely much better than what we have. We can probably do better
still with a GroupsAccumulator as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
