yashmayya opened a new pull request, #14089: URL: https://github.com/apache/pinot/pull/14089
- Implement polymorphic binary arithmetic scalar functions (+, -, *, /) based on the framework added in https://github.com/apache/pinot/pull/13573/ (similar to https://github.com/apache/pinot/pull/13711). - Before this, all such functions relied on double based calculations regardless of the input operand types. After this, functions involving only integer / long values will use a long based calculation. - This solves a whole class of bugs along with bringing standard SQL-like semantics. - The added test shows a simple case that would've failed without these changes. A query like: `SELECT * FROM mytable WHERE ts BETWEEN now() - 1000 AND now() ` fails on the v1 single-stage query engine without these changes. The reason is that during query compilation, the [CompileTimeFunctionsInvoker](https://github.com/apache/pinot/blob/852e4f3fdbbb97e27c2b819c8e73fe005ae1ead9/pinot-common/src/main/java/org/apache/pinot/sql/parsers/rewriter/CompileTimeFunctionsInvoker.java#L62) query rewriter will replace `now() - 1000` with the equivalent double value since that's the only available implementation of the minus function - https://github.com/apache/pinot/blob/852e4f3fdbbb97e27c2b819c8e73fe005ae1ead9/pinot-common/src/main/java/org/apache/pinot/common/function/scalar/ArithmeticFunctions.java#L38-L41 - This fails later on either in the [MergeRangeFilterOptimizer](https://github.com/apache/pinot/blob/852e4f3fdbbb97e27c2b819c8e73fe005ae1ead9/pinot-core/src/main/java/org/apache/pinot/core/query/optimizer/filter/MergeRangeFilterOptimizer.java#L149) in the broker or in the [ColumnValueSegmentPruner](https://github.com/apache/pinot/blob/852e4f3fdbbb97e27c2b819c8e73fe005ae1ead9/pinot-core/src/main/java/org/apache/pinot/core/query/pruner/ColumnValueSegmentPruner.java#L176) in the server. The root cause of both failures is that the literal representing the range bound gets converted to a string (see [here](https://github.com/apache/pinot/blob/852e4f3fdbbb97e27c2b819c8e73fe005ae1ead9/pinot-common/src/main/java/org/apache/pinot/common/request/context/RequestContextUtils.java#L225-L227) for instance) and later when converting back to a `LONG`, we run into an issue since the string could be the scientific representation of the double value: ``` Exception: java.lang.IllegalArgumentException: Cannot convert value: '1.726807808874E12' to type: LONG at org.apache.pinot.spi.data.FieldSpec$DataType.convertInternal(FieldSpec.java:724) at org.apache.pinot.core.query.optimizer.filter.MergeRangeFilterOptimizer.getComparable(MergeRangeFilterOptimizer.java:149) at org.apache.pinot.core.query.optimizer.filter.MergeRangeFilterOptimizer.getRange(MergeRangeFilterOptimizer.java:135) at org.apache.pinot.core.query.optimizer.filter.MergeRangeFilterOptimizer.optimize(MergeRangeFilterOptimizer.java:85) ``` ``` org.apache.pinot.spi.exception.BadQueryRequestException: java.lang.IllegalArgumentException: Cannot convert value: '1.727344079699E12' to type: LONG at org.apache.pinot.core.query.pruner.ValueBasedSegmentPruner.convertValue(ValueBasedSegmentPruner.java:157) at org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneRangePredicate(ColumnValueSegmentPruner.java:176) at org.apache.pinot.core.query.pruner.ColumnValueSegmentPruner.pruneSegmentWithPredicate(ColumnValueSegmentPruner.java:88) at org.apache.pinot.core.query.pruner.ValueBasedSegmentPruner.pruneSegment(ValueBasedSegmentPruner.java:144) ``` - We should also look into fixing the above root cause by using a type specific bound in the range predicate but that likely has a lot more touch points in sensitive code. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
