Light-City opened a new issue, #37765:
URL: https://github.com/apache/arrow/issues/37765
### Describe the bug, including details regarding any error messages,
version, and platform.
when we execute casewhen expression, such as:
CASE WHEN i > 0 THEN j / i ELSE j END
If column i has values( 0 ), we will get division overflow problem, j / 0 ->
overflow。
Why does this problem occur?
I read the source code and found that the masks for these conditions are
calculated separately, and the previous mask is not used to participate in
subsequent operations. This will cause the subsequent division to overflow.
Currently, we have two solutions.
1. Regardless of performance
Using the previous mask to participate in subsequent operations will cause
performance degradation in all cases.
```cpp
for (size_t i = 0; i < arguments.size(); ++i) {
// get SelectorVector from arguments before computed;
std::shared_ptr<ArrayData> filtered_mask =
GetExpressionMask(call->function_name, arguments, i);
if (filtered_mask != nullptr) {
auto values = input.values;
// construct filtered batch
for (auto& value : values) {
if (value.is_array()) {
std::vector<Datum> invert_args;
invert_args.push_back(filtered_mask);
ARROW_ASSIGN_OR_RAISE(auto null_mask,
CallFunction("invert", invert_args,
exec_context));
ARROW_ASSIGN_OR_RAISE(
value, ReplaceWithMask(value, null_mask,
MakeNullScalar(value.type())));
}
}
ARROW_ASSIGN_OR_RAISE(auto new_batch,
ExecBatch::Make(std::move(values)));
// execute and pass child expr
ARROW_ASSIGN_OR_RAISE(
arguments[i],
ExecuteScalarExpression(call->arguments[i], new_batch,
exec_context));
} else {
ARROW_ASSIGN_OR_RAISE(
arguments[i], ExecuteScalarExpression(call->arguments[i], input,
exec_context));
}
ARROW_ASSIGN_OR_RAISE(
arguments[i], ExecuteScalarExpression(call->arguments[i], input,
exec_context));
}
```
2. Considering performance, only division expressions are treated specially.
We currently apply this method to the application layer, which constructs a
complex expression to avoid division overflow problems.
case when j > 0 then i / j else i end
->
case when j > 0 then i / ((case when j > 0 then j else null end)) else i end
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
