hcrosse opened a new issue, #1512:
URL: https://github.com/apache/arrow-go/issues/1512
### Describe the bug, including details regarding any error messages,
version, and platform.
`decimal128.FromString` and `decimal256.FromString` panic on infinities and
exponents that overflow `big.Float`, such as `1e1000000000`. A finite value can
also panic if multiplying it by the scale's power of ten produces infinity.
These functions also allocate hundreds of megabytes for short strings before
returning an error or zero. On macOS arm64, I measured about 480 MB peak RSS
for `1e600000000` at scale 2 in both widths, and about 240 MB at scale −2 in
all four widths. Parsing `1e-600000000` at scale zero used about 480 MB in all
four widths, only to return zero.
The panic reaches the decimal builders' `AppendValueFromString`,
`array.FromJSON`, `scalar.ParseScalar` and the CSV reader.
I reproduced this on main at `eb1781e7`, on macOS arm64 with Go 1.26.10.
**Reproduction**
Save this as `main.go` in a new subdirectory of an arrow-go checkout at
`eb1781e7faa2b1f6f7499ded6fce7b86f804f303`, then run `go run . Inf` and `go run
. 1e1000000000` from that directory:
```go
package main
import (
"fmt"
"os"
"github.com/apache/arrow-go/v18/arrow/decimal128"
)
func main() {
_, err := decimal128.FromString(os.Args[1], 38, 2)
fmt.Println(err)
}
```
Actual output starts with this for both inputs:
```text
panic: runtime error: invalid memory address or nil pointer dereference
```
Expected output for `Inf`:
```text
cannot convert Inf to decimal128
```
Expected output for `1e1000000000`:
```text
cannot convert 1e1000000000 to decimal128
```
The same program with `1e600000000` or `1e-600000000` demonstrates the
memory growth. I expect an error or zero without allocating memory proportional
to the exponent's magnitude.
**Cause**
`big.ParseFloat` accepts infinities, but `big.Float.Int` returns `nil` for
them. Large finite values are converted to `big.Int` before their size is
checked. Rounding tiny values adds or subtracts one half, which allocates in
proportion to the exponent gap.
https://github.com/apache/arrow-go/blob/eb1781e7faa2b1f6f7499ded6fce7b86f804f303/arrow/decimal128/decimal128.go#L273-L311
https://github.com/apache/arrow-go/blob/eb1781e7faa2b1f6f7499ded6fce7b86f804f303/arrow/decimal256/decimal256.go#L153-L190
I have a patch with tests and can open a PR.
### Component(s)
Other
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
