zeroshade commented on code in PR #1553:
URL: https://github.com/apache/arrow-go/pull/1553#discussion_r4617832276
##########
parquet/internal/encoding/alp_scaling.go:
##########
@@ -242,3 +233,130 @@ func (s *alpScaling[T]) tryEncode(v T) (encoded int64, ok
bool) {
}
return encoded, s.decode(encoded) == v
}
+
+// alpScale64 scales every value to the integer alpScaling.encode gives it and
+// checks, as tryEncode does, that the integer decodes back to the value. It
+// writes all the integers to encoded and the positions of the values that
+// fail, in order, to excPos. Both must hold len(values). It returns how many
+// failed.
+//
+// This is the loop every value of a column goes through, so it is written for
+// one type and avoids what alpScaling cannot. Nothing in it branches on the
+// data: an exception costs what any other value costs, where a branch is
+// mispredicted wherever the two mix. And it leaves the range of the integers
+// to alpIntRange, because tracking a minimum and a maximum here would make
+// every value wait for the one before.
+func alpScale64(values []float64, encExp, encFac, decFac, decExp float64,
encoded []int64, excPos []uint16) (numExc int) {
+ encoded = encoded[:len(values)]
+ excPos = excPos[:len(values)]
+ // The same values as integers, so that comparing bits and taking the
sign
+ // do not have to move each one out of a floating-point register.
+ raw :=
arrow.Uint64Traits.CastFromBytes(arrow.Float64Traits.CastToBytes(values))[:len(values)]
+ magicBits := math.Float64bits(alpMagicFloat64)
+ for i, v := range values {
+ bits := raw[i]
+ // The conversion keeps the product from fusing with the
addition
+ // below, so every platform rounds the same product.
+ scaled := float64(v * encExp * encFac)
+ // The magic constant takes the value's sign, which is the
scaled
+ // value's sign too: adding and subtracting it then rounds as
+ // alpScaling.round does.
+ magic := math.Float64frombits(magicBits | bits&(1<<63))
+ e := int64((scaled + magic) - magic)
+ // A NaN fails the comparison too.
+ if !(math.Abs(scaled) <= alpEncodingUpperFloat64) {
+ e = int64(alpEncodingUpperFloat64)
+ }
+ encoded[i] = e
+ excPos[numExc] = uint16(i)
+ // Comparing bits rather than values also rejects -0.0, which
scales
+ // to 0 and decodes to 0.0.
+ if math.Float64bits(float64(e)*decFac*decExp) != bits {
+ numExc++
+ }
+ }
+ return numExc
+}
+
+// alpScale32 is the FLOAT form of alpScale64.
+func alpScale32(values []float32, encExp, encFac, decFac, decExp float32,
encoded []int64, excPos []uint16) (numExc int) {
+ encoded = encoded[:len(values)]
+ excPos = excPos[:len(values)]
+ raw :=
arrow.Uint32Traits.CastFromBytes(arrow.Float32Traits.CastToBytes(values))[:len(values)]
+ magicBits := math.Float32bits(alpMagicFloat32)
+ for i, v := range values {
+ bits := raw[i]
+ scaled := float32(v * encExp * encFac)
+ magic := math.Float32frombits(magicBits | bits&(1<<31))
+ e := int64((scaled + magic) - magic)
Review Comment:
Yes, good catch. I merged main and `alpScale64`/`alpScale32` now round with
`math.RoundToEven`, the same as `alpScaling.round` since #1555. With that
rounding an integer can no longer pass the int32 limit, so the separate check
for one in `alpScale32` is gone too, as on main.
`TestALPScaleMatchesTryEncode` holds both loops to `tryEncode` either side
of the magnitudes that fix was about (2^22, 2^24, 2^51, 2^53), and I re-checked
that the pages are still byte for byte what main writes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]