timgrein opened a new pull request, #3855:
URL: https://github.com/apache/calcite/pull/3855
The javadocs of `RelDataTypeSystem.deriveDecimalDivideType` state the
following algorithm for deriving the decimal type of a decimal division:
```
d = p1 - s1 + s2
s < max(6, s1 + p2 + 1)
p = d + s
```
The inequality here doesn't make 100% sense to me as the result of an
inequality is not a number, but a boolean. Should it be the following?
```
d = p1 - s1 + s2
s = max(6, s1 + p2 + 1)
p = d + s
```
I don't have access to the original SQL standard from 2003, but this would
align at least with how MS SQL Server defines type derivation for decimal
division: [Precision, Scale and Length (Transact
SQL)](https://learn.microsoft.com/en-us/sql/t-sql/data-types/precision-scale-and-length-transact-sql?view=sql-server-ver16).
If so I propose to fix and follow the docs and adjust the previous
implementation from:
```
final int maxNumericPrecision = getMaxNumericPrecision();
int dout =
Math.min(
p1 - s1 + s2,
maxNumericPrecision);
int scale = Math.max(6, s1 + p2 + 1);
scale =
Math.min(
scale,
maxNumericPrecision - dout);
scale = Math.min(scale, getMaxNumericScale());
int precision = dout + scale;
```
to:
```
int scale = Math.max(6, s1 + p2 + 1);
int precision = p1 - s1 + s2 + scale;
precision = Math.min(precision, getMaxNumericPrecision());
scale = Math.min(scale, getMaxNumericScale());
```
This also fixes the issue described in [[CALCITE-6464] Type inference for
DECIMAL division seems
incorrect](https://issues.apache.org/jira/browse/CALCITE-6464). I added an
explicit regression test to check this.
I've also adjusted one test case name to reflect that it tests subtraction
and not addition and added an explicit test case for addition. I can also
remove the test changes from this PR and open a separate one, if needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
