jayzhan211 commented on code in PR #7461:
URL: https://github.com/apache/arrow-datafusion/pull/7461#discussion_r1322972238
##########
datafusion/expr/src/expr.rs:
##########
@@ -1030,6 +1031,47 @@ impl Expr {
pub fn contains_outer(&self) -> bool {
!find_out_reference_exprs(self).is_empty()
}
+
+ /// Flatten the nested array expressions until the base array is reached.
+ /// For example:
+ /// [[1, 2, 3], [4, 5, 6]] => [1, 2, 3, 4, 5, 6]
+ /// [[[1, 2], [3, 4]], [[5, 6], [7, 8]]] => [1, 2, 3, 4, 5, 6, 7, 8]
+ /// Panics if the expression cannot be flattened.
+ pub fn flatten(&self) -> Self {
+ self.try_flatten().unwrap_or_else(|_| self.clone())
+ }
+
+ /// Flatten the nested array expressions until the base array is reached.
+ /// For example:
+ /// [[1, 2, 3], [4, 5, 6]] => [1, 2, 3, 4, 5, 6]
+ /// [[[1, 2], [3, 4]], [[5, 6], [7, 8]]] => [1, 2, 3, 4, 5, 6, 7, 8]
+ /// Returns an error if the expression cannot be flattened.
+ pub fn try_flatten(&self) -> Result<Self> {
Review Comment:
This function is mainly used in the unnest function currently, and I don't
think it is a helpful expression optimization, it is too specialized.
Not only that, in my current design for unnest, I will process them before
the optimization phase, specifically in the planning phase. If I move this
function into the optimization phase, I still need to process this function
prior, and since only the unnest function needs this now, we do nothing in the
optimization phase.
Only If this optimization is generally used, I think moving it to the
optimization phase makes sense.
You can look to the PR for unnest
https://github.com/apache/arrow-datafusion/pull/6796 too, it is ready for view
actually. I planned to reopen it after the flatten function merged, but since
this function is tightly bind to unnest function, maybe review them together is
better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
