eugenegujing opened a new issue, #7531:
URL: https://github.com/apache/texera/issues/7531
### What happened?
The `max` aggregation computes its running maximum starting from the type's
minimum value, and at the end it should check "is the result still the starting
value?" to detect a group that had no real data. But the check compares against
the type's **maximum** value instead — the line was copied from `min` (where
that comparison is correct, because `min` starts from the maximum).
Two results come out silently wrong, with the workflow completing normally
and no error shown:
- **A group with only null values returns the starting value instead of
`null`**: `-2147483648` for INTEGER, `-Infinity` for DOUBLE, `1970-01-01
00:00:00` for TIMESTAMP. `min` on the same data correctly returns `null`.
- **A real maximum equal to the type's maximum value is discarded**: `max`
over `{1, 5, 2147483647}` computes `2147483647` correctly, then the final check
mistakes it for "no data seen" and drops it, so a wrong smaller number is
reported instead.
**Expected:** `max` returns `null` for a group with no non-null values (like
`min` does), and returns `2147483647` when that is the largest value present.
The bug is in
`common/workflow-operator/src/main/scala/org/apache/texera/amber/operator/aggregate/AggregationOperation.scala:240`
— `maxAgg` initialises with `AttributeTypeUtils.minValue(attributeType)` (line
231) but its finaliser tests `partial ==
AttributeTypeUtils.maxValue(attributeType)`, which should be `minValue`.
Introduced by #1840, which generalised the hard-coded `Double` sentinels to
per-type helpers and updated `maxAgg`'s initialiser but not its finaliser.
### How to reproduce?
1. Add a **1-out Python UDF** source with output columns `k` (string) and
`v` (integer):
```python
from pytexera import *
class GenerateOperator(UDFSourceOperator):
@overrides
def produce(self) -> Iterator[Union[TupleLike, TableLike, None]]:
yield {"k": "g1", "v": None}
yield {"k": "g1", "v": None}
yield {"k": "g2", "v": 1}
yield {"k": "g2", "v": 5}
yield {"k": "g2", "v": 2147483647}
```
2. Connect an **Aggregate** operator: Aggregate Func = `max`, Attribute =
`v`, Result attribute = `max_v`, Group By Keys = `k`.
3. Run and view the result.
Observed: `g1` shows `max_v = -2147483648` (expected `null`), and `g2` shows
a wrong smaller value (expected `2147483647`).
Replacing `max` with `min` in the same workflow gives the correct behaviour:
`null` for `g1`, `1` for `g2`.
### Version/Branch
1.3.0-incubating-SNAPSHOT (main)
### Commit Hash (Optional)
_No response_
### What browsers are you seeing the problem on?
_No response_
### Relevant log output
```shell
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
