This is an automated email from the ASF dual-hosted git repository.
uros-b pushed a commit to branch branch-4.x
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.x by this push:
new 8a68e94c90af [SPARK-60421][SQL] Fix overflow and non-finite handling
in vector_avg for float vectors
8a68e94c90af is described below
commit 8a68e94c90af5975e1549e773d2477a680988584
Author: Uros Stankovic <[email protected]>
AuthorDate: Thu Nov 5 13:07:45 2026 +0000
[SPARK-60421][SQL] Fix overflow and non-finite handling in vector_avg for
float vectors
### What changes were proposed in this pull request?
Fix overflow and non-finite handling when updating `vector_avg` for
`ARRAY<FLOAT>` inputs.
Keep the existing running-average arithmetic for finite deltas. When
subtracting two finite operands overflows, scale them before subtraction. When
either operand is non-finite, use IEEE addition to preserve infinities and
propagate NaN.
Check four deltas together before writing a block. An exceptional block
uses the scalar helper with the values already loaded, then processing
continues with the next block. Trailing elements use the same helper. The merge
implementation is unchanged.
Add regression coverage for overflow, cumulative weights, input order,
exact finite rounding, both array representations, and block boundaries.
Document the behavior change in the SQL migration guide.
This change covers float vectors. The `ARRAY<DOUBLE>` work in #59590 needs
equivalent handling separately.
### Why are the changes needed?
The existing update formula can overflow its intermediate subtraction or
turn an infinite running average into NaN after a finite input. This makes
updates disagree with merging singleton buffers and makes the result depend on
input order.
The following cases show successive inputs for one vector element:
| Input sequence | Before | After |
|---|---|---|
| `Float.MaxValue`, then `-Float.MaxValue` | `-Infinity` | `0.0f` |
| `-Float.MaxValue`, then `Float.MaxValue` | `Infinity` | `0.0f` |
| `Infinity`, then `Infinity` | `NaN` | `Infinity` |
| `-Infinity`, then `-Infinity` | `NaN` | `-Infinity` |
| `Infinity`, then a finite value | `NaN` | `Infinity` |
| `-Infinity`, then a finite value | `NaN` | `-Infinity` |
| Opposite infinities, in either order | `NaN` | `NaN` |
| A finite value, then an infinity | That infinity | That infinity |
| `NaN` and any value, in either order | `NaN` | `NaN` |
### Does this PR introduce _any_ user-facing change?
Yes. Updating running averages with extreme finite inputs avoids
intermediate subtraction overflow. An infinite running average absorbs finite
values and same-sign infinities. Opposite infinities and NaN inputs continue to
produce NaN.
Finite updates whose subtraction does not overflow retain the original
arithmetic and rounding. The result remains `ARRAY<FLOAT>`; scalar `avg(FLOAT)`
uses double accumulation and can have different finite-input rounding.
### How was this patch tested?
All **65 tests** in `VectorAggSuite` and both Catalyst Scala style checks
passed on this revision:
```bash
build/sbt 'catalyst/testOnly *VectorAggSuite' \
'catalyst/Compile/scalastyle' 'catalyst/Test/scalastyle'
```
Regression tests cover finite subtraction overflow with a one-third weight,
both signs of infinity, opposite infinities, NaN operand combinations, and
exact finite rounding. The table-driven cases exercise both input orders,
updates and singleton-buffer merges, both array representations, and
scalar/block boundaries. The weighted-overflow test covers dimensions 2, 4, 5,
and 8 with both array representations.
**Performance**
Fresh local measurements compare the original `VectorAvg` at PR merge-base
`9b17867be1f3` with PR head `e7eeef693981`. The merge method source is
identical
in both revisions.
Times are ns/element; lower is better. Positive percentage changes mean
slower.
Each value is the median of five fresh-JVM medians, with three steady timed
windows
per JVM. Brackets show the minimum and maximum of the five JVM medians.
`unsafe` and `generic` mean `UnsafeArrayData` and `GenericArrayData`;
merges use
binary aggregate buffers.
**Finite updates**
| Operation / input | Array | Dim | Before | After | Change |
|---|---|---:|---:|---:|---:|
| update: All finite | unsafe | 128 | 0.851 [0.812-0.852] | 0.898
[0.896-0.899] | +5.5% |
| update: All finite | unsafe | 1536 | 0.775 [0.775-0.776] | 0.798
[0.798-0.798] | +3.0% |
| update: All finite | generic | 128 | 0.965 [0.941-0.979] | 0.978
[0.978-0.984] | +1.4% |
| update: All finite | generic | 1536 | 0.862 [0.857-0.864] | 0.956
[0.953-0.957] | +10.8% |
**Updates with special values (dimension 1536)**
| Operation / input | Array | Dim | Before | After | Change |
|---|---|---:|---:|---:|---:|
| update: First row: NaN at index 0 | unsafe | 1536 | 0.775 [0.774-0.775] |
0.841 [0.841-0.841] | +8.5% |
| update: First row: NaN at index 0 | generic | 1536 | 0.862 [0.858-0.863]
| 1.139 [0.947-1.140] | +32.1% |
| update: First row: +Inf at index 0 | unsafe | 1536 | 0.774 [0.774-0.776]
| 0.842 [0.841-0.843] | +8.7% |
| update: First row: +Inf at index 0 | generic | 1536 | 0.860 [0.859-0.864]
| 0.950 [0.948-1.141] | +10.4% |
| update: First row: all NaN | unsafe | 1536 | 0.775 [0.774-0.778] | 1.184
[1.184-1.185] | +52.7% |
| update: First row: all NaN | generic | 1536 | 0.861 [0.858-0.863] | 1.407
[1.404-1.412] | +63.5% |
| update: 75% non-finite later inputs | unsafe | 1536 | 0.775 [0.774-0.775]
| 1.178 [1.167-1.233] | +52.1% |
| update: 75% non-finite later inputs | generic | 1536 | 0.861
[0.855-0.863] | 1.333 [1.327-1.334] | +54.9% |
**Finite merges**
| Operation / input | Array | Dim | Before | After | Change |
|---|---|---:|---:|---:|---:|
| merge: All finite | binary | 128 | 0.371 [0.367-0.373] | 0.370
[0.367-0.395] | -0.3% |
| merge: All finite | binary | 1536 | 0.344 [0.343-0.344] | 0.344
[0.343-0.345] | -0.1% |
Method: Intel Xeon Platinum 8375C, one pinned CPU; sequential JVM jobs with
randomized workload order and randomized before/after order within each
pair.
Both revisions use Temurin OpenJDK 21.0.13+9, Scala 2.13.18, Java 21 target,
identical dependencies/compiler/JVM flags, and a fixed 1 GiB heap. Each JVM
first
executes repeated 0.5s warmup windows for at least 8s, then measures three
windows
of at least 0.6s. This warms the enclosing caller's loop exits as well as
the kernel.
All 140 timed kernels, aggregate APIs, and enclosing timed callers reached
C2
before the timed windows, with no relevant C2 compilation or invalidation
during
measurement. No prior benchmark timings are reused.
Update measures real `VectorAvg.update`: 100,000 calls per group, 64 seeded
vectors,
finite values in [-100,100), and a non-nullable array type. Construction,
first row,
buffer reset, and result consumption are outside timing. Early NaN/Inf and
all-NaN-first
cases receive finite later rows; mostly-nonfinite input has 75% NaN/+Inf
elements
in every later row. Null-element scanning is excluded.
Merge measures real `VectorAvg.merge` over 512 independent pairs per batch,
with
counts 1,200 and 9,800 in both orders. Every target byte array and count is
restored
before timing; source buffers remain unchanged. Reset, result consumption,
and count
checks are outside timing. The left/right/pristine vector bytes occupy
approximately
0.75 MiB at dim128 and 9 MiB at dim1536, plus row objects; resetting warms
the input data.
Finite update results match over 8,000 groups; finite and mixed Inf/NaN
merge results
match over 10,000 groups each. The existing finite merge overflow is
deliberately
unchanged. Early +Inf updates have intentionally different results: the
original
returns NaN after later finite rows; this PR preserves +Inf.
These are warmed single-thread aggregate microbenchmarks; they do not
measure SQL
execution, shuffle, or end-to-end queries. Other JVMs, cache footprints,
null scans,
and data distributions can change the result. Overlapping fork ranges
should be
treated as variation rather than a reliable improvement or regression.
In particular, the 128-dimensional generic finite-update ranges overlap;
its 1.4%
median change is within variation. Generic early-NaN/Inf cases have visibly
different
timing modes across fresh JVMs, so their medians should be read alongside
the ranges.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex (GPT-6)
Closes #59615 from urosstan-db/vector-avg-float-correctness.
Authored-by: Uros Stankovic <[email protected]>
Signed-off-by: Uros Bojanic <[email protected]>
(cherry picked from commit f3199630c25a32b5510f415922182d2251beb224)
Signed-off-by: Uros Bojanic <[email protected]>
---
docs/sql-migration-guide.md | 1 +
.../catalyst/expressions/vectorExpressions.scala | 77 +++++++++-
.../expressions/aggregate/VectorAggSuite.scala | 158 ++++++++++++++++++++-
3 files changed, 228 insertions(+), 8 deletions(-)
diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index dc7469c8d052..3fe7636b75c0 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -36,6 +36,7 @@ license: |
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a column referenced in a lambda function, such as `i` in `transform(a, x
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value
stored in that column. Previously, it read the value of the query before the
store assignment cast, so it could silently store a value that does not match
the stored column, and a later write of that row failed with
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817). [...]
- Since Spark 4.4, an auto-filled generated column of a Data Source V2 table
reads a field of another column, such as `s.f`, from the value stored in that
column. Previously, it read the value of the query before the store assignment
cast, so it could silently store a value that does not match the stored column,
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION`
(SPARK-60803). Rows that are already stored keep their values, and data derived
from them may need to be re [...]
- Since Spark 4.4, `spark.addArtifact` and `spark.addArtifacts` take a file
whose name ends with `.sparkpkg` as a [native data source
package](sql-native-data-sources.html), also when it is added with `file=True`,
`pyfile=True` or `archive=True` in Python without Spark Connect, or with
`file=True` from a Spark Connect client, and fail with
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` for such a file that is not a valid
package, or whose version Spark does not support, or that has no library for
[...]
+- Since Spark 4.4, `vector_avg` avoids intermediate subtraction overflow when
updating running averages for `ARRAY<FLOAT>` inputs. Extreme finite inputs that
previously produced infinity or `NaN` during updates may now produce a finite
average. An infinite running average combined with a finite value or an
infinity of the same sign now produces infinity with that sign instead of
`NaN`. Opposite infinities and `NaN` inputs continue to produce `NaN`.
- Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled`
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before
Spark 4.2. With columnar input, they kept the columns that they pass through
until the results of their batch came back from Python, and failed or returned
wrong results when the operator reused or released the vectors of the batch in
the meantime. To enab [...]
- Since Spark 4.4, the conversion of columnar batches to rows releases each
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when
it is not whole-stage code-generated, e.g. when the query has more than
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A
`ColumnVector` that a Data Source V2 source reuses across batches, such as an
`ArrowColumnVector` over a `VectorSchemaRoot [...]
- Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default
`false`) is a preview feature that may change in future releases. When true, it
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family,
explicit casts can return constrained types, least-common-type resolution for
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string
expressions that transform thei [...]
diff --git
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
index ef2cb94c932b..a56acc4b6c22 100644
---
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
+++
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
@@ -17,6 +17,8 @@
package org.apache.spark.sql.catalyst.expressions
+import java.lang.{Float => JFloat}
+
import org.apache.spark.sql.catalyst.InternalRow
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
@@ -672,17 +674,88 @@ object VectorAvg {
inputArray: ArrayData,
dim: Int,
newCount: Long): Unit = {
- // Update running average: new_avg = old_avg + (new_value - old_avg) /
new_count
+ // Finite deltas use: new_avg = old_avg + (new_value - old_avg) *
invCount.
+ // updateElement handles subtraction overflow and non-finite operands.
val invCount = 1.0f / newCount
+ updateElementsInBlocks(accBytes, inputArray, dim, invCount)
+ }
+
+ private def updateElementsInBlocks(
+ accBytes: Array[Byte],
+ inputArray: ArrayData,
+ dim: Int,
+ invCount: Float): Unit = {
var i = 0
+ while (i <= dim - 4) {
+ val off = Platform.BYTE_ARRAY_OFFSET + i.toLong * 4
+ val oldAvg0 = Platform.getFloat(accBytes, off)
+ val value0 = inputArray.getFloat(i)
+ val delta0 = value0 - oldAvg0
+ val oldAvg1 = Platform.getFloat(accBytes, off + 4)
+ val value1 = inputArray.getFloat(i + 1)
+ val delta1 = value1 - oldAvg1
+ val oldAvg2 = Platform.getFloat(accBytes, off + 8)
+ val value2 = inputArray.getFloat(i + 2)
+ val delta2 = value2 - oldAvg2
+ val oldAvg3 = Platform.getFloat(accBytes, off + 12)
+ val value3 = inputArray.getFloat(i + 3)
+ val delta3 = value3 - oldAvg3
+ // A finite sum guarantees finite deltas; sum overflow only triggers a
scalar fallback.
+ val deltaSum = (delta0 + delta1) + (delta2 + delta3)
+ if (JFloat.isFinite(deltaSum)) {
+ Platform.putFloat(accBytes, off, oldAvg0 + delta0 * invCount)
+ Platform.putFloat(accBytes, off + 4, oldAvg1 + delta1 * invCount)
+ Platform.putFloat(accBytes, off + 8, oldAvg2 + delta2 * invCount)
+ Platform.putFloat(accBytes, off + 12, oldAvg3 + delta3 * invCount)
+ } else {
+ Platform.putFloat(accBytes, off, updateElement(oldAvg0, value0,
invCount))
+ Platform.putFloat(accBytes, off + 4, updateElement(oldAvg1, value1,
invCount))
+ Platform.putFloat(accBytes, off + 8, updateElement(oldAvg2, value2,
invCount))
+ Platform.putFloat(accBytes, off + 12, updateElement(oldAvg3, value3,
invCount))
+ }
+ i += 4
+ }
+ if (i < dim) {
+ updateElementsScalar(accBytes, inputArray, i, dim, invCount)
+ }
+ }
+
+ private def updateElementsScalar(
+ accBytes: Array[Byte],
+ inputArray: ArrayData,
+ start: Int,
+ dim: Int,
+ invCount: Float): Unit = {
+ var i = start
while (i < dim) {
val off = Platform.BYTE_ARRAY_OFFSET + i.toLong * 4
val oldAvg = Platform.getFloat(accBytes, off)
- Platform.putFloat(accBytes, off, oldAvg + (inputArray.getFloat(i) -
oldAvg) * invCount)
+ val value = inputArray.getFloat(i)
+ Platform.putFloat(accBytes, off, updateElement(oldAvg, value,
invCount))
i += 1
}
}
+ private def updateElement(oldAvg: Float, value: Float, invCount: Float):
Float = {
+ val delta = value - oldAvg
+ if (JFloat.isFinite(delta)) {
+ oldAvg + delta * invCount
+ } else if (JFloat.isFinite(oldAvg) && JFloat.isFinite(value)) {
+ // Scale before subtracting when opposite finite values overflow the
delta.
+ // For oldAvg = Float.MaxValue, value = -Float.MaxValue and invCount =
0.5f,
+ // the delta is -Infinity, but scaling first gives the correct average
of 0.0f.
+ oldAvg + (value * invCount - oldAvg * invCount)
+ } else {
+ // At least one operand is non-finite. Addition gives the IEEE result
+ // regardless of which operand is the running average:
+ // Inf + Inf -> Inf; -Inf + (-Inf) -> -Inf.
+ // Inf + (-Inf) -> NaN.
+ // NaN + any value -> NaN.
+ // Inf + finite -> Inf; -Inf + finite -> -Inf.
+ oldAvg + value
+ }
+ }
+
override def mergeElements(
accBytes: Array[Byte],
inputBytes: Array[Byte],
diff --git
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
index d509bfd56a85..624518a9dce8 100644
---
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
+++
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
@@ -17,7 +17,7 @@
package org.apache.spark.sql.catalyst.expressions.aggregate
-import java.lang.{Double => JDouble}
+import java.lang.{Double => JDouble, Float => JFloat}
import org.apache.spark.{SparkFunSuite, SparkRuntimeException}
import org.apache.spark.sql.catalyst.InternalRow
@@ -48,8 +48,15 @@ class VectorAggSuite extends SparkFunSuite {
}
// Helper to create input row with float array
- def createInputRow(values: Array[Float]): InternalRow = {
- InternalRow(ArrayData.toArrayData(values))
+ def createInputRow(
+ values: Array[Float],
+ useGenericArray: Boolean = false): InternalRow = {
+ val array = if (useGenericArray) {
+ new GenericArrayData(values)
+ } else {
+ ArrayData.toArrayData(values)
+ }
+ InternalRow(array)
}
// Helper to create input row with null
@@ -60,7 +67,7 @@ class VectorAggSuite extends SparkFunSuite {
}
// Helper to create input row with array containing null elements
- def createInputRowWithNullElement(values: Array[java.lang.Float]):
InternalRow = {
+ def createInputRowWithNullElement(values: Array[JFloat]): InternalRow = {
val arrayData = new GenericArrayData(values.map {
case null => null
case v => v.floatValue().asInstanceOf[AnyRef]
@@ -86,7 +93,7 @@ class VectorAggSuite extends SparkFunSuite {
// Asserts that two floats are approximately equal within a tolerance.
def assertFloatEquals(actual: Float, expected: Float, tolerance: Float =
1e-5f): Unit = {
assert(
- java.lang.Float.isNaN(expected) && java.lang.Float.isNaN(actual) ||
+ actual == expected || JFloat.isNaN(expected) && JFloat.isNaN(actual) ||
math.abs(actual - expected) <= tolerance,
s"Expected $expected but got $actual (tolerance: $tolerance)"
)
@@ -405,6 +412,145 @@ class VectorAggSuite extends SparkFunSuite {
assert(result(1) === 3.0f)
}
+ test("VectorAvg - opposite extreme values do not overflow") {
+ val (agg, buffer, _) = createVectorAvg()
+ val expected = Array(0.0f, 0.0f).map(JFloat.floatToIntBits)
+ agg.update(buffer, createInputRow(Array(Float.MaxValue, -Float.MaxValue)))
+ agg.update(buffer, createInputRow(Array(-Float.MaxValue, Float.MaxValue)))
+ assertResult(expected)(evalAsFloatArray(agg,
buffer).map(JFloat.floatToIntBits))
+
+ val (_, buffer1, _) = createVectorAvg()
+ val (_, buffer2, _) = createVectorAvg()
+ agg.update(buffer1, createInputRow(Array(Float.MaxValue, -Float.MaxValue)))
+ agg.update(buffer2, createInputRow(Array(-Float.MaxValue, Float.MaxValue)))
+ agg.merge(buffer1, buffer2)
+ assertResult(expected)(evalAsFloatArray(agg,
buffer1).map(JFloat.floatToIntBits))
+ }
+
+ test("VectorAvg - finite overflow preserves cumulative weights") {
+ for {
+ dim <- Seq(2, 4, 5, 8)
+ useGenericArray <- Seq(false, true)
+ } {
+ val (agg, buffer, _) = createVectorAvg()
+ val first = Array.tabulate(dim)(i => if (i % 2 == 0) Float.MaxValue else
-Float.MaxValue)
+ agg.update(buffer, createInputRow(first, useGenericArray))
+ agg.update(buffer, createInputRow(first, useGenericArray))
+ // The third update overflows the delta. An unweighted midpoint would
incorrectly return zero.
+ agg.update(buffer, createInputRow(first.map(-_), useGenericArray))
+ val expected = (Float.MaxValue.toDouble / 3).toFloat
+ assertFloatArrayEquals(
+ evalAsFloatArray(agg, buffer),
+ Array.tabulate(dim)(i => if (i % 2 == 0) expected else -expected),
+ tolerance = math.ulp(expected) * 2)
+ }
+ }
+
+ test("VectorAvg - non-finite results do not depend on input order") {
+ val first =
+ Array(Float.PositiveInfinity, 1.0f, Float.NegativeInfinity,
Float.PositiveInfinity)
+ val second = Array(1.0f, Float.PositiveInfinity, 1.0f,
Float.NegativeInfinity)
+ def checkResult(result: Array[Float]): Unit = {
+ assertFloatArrayEquals(result,
+ Array(Float.PositiveInfinity, Float.PositiveInfinity,
Float.NegativeInfinity, Float.NaN))
+ }
+
+ Seq(Seq(first, second), Seq(second, first)).foreach { vectors =>
+ val (agg, buffer, _) = createVectorAvg()
+ vectors.foreach(v => agg.update(buffer, createInputRow(v)))
+ checkResult(evalAsFloatArray(agg, buffer))
+ agg.update(buffer, createInputRow(Array(2.0f, 2.0f, 2.0f, 2.0f)))
+ checkResult(evalAsFloatArray(agg, buffer))
+
+ val (_, buffer1, _) = createVectorAvg()
+ val (_, buffer2, _) = createVectorAvg()
+ agg.update(buffer1, createInputRow(vectors.head))
+ agg.update(buffer2, createInputRow(vectors(1)))
+ agg.merge(buffer1, buffer2)
+ checkResult(evalAsFloatArray(agg, buffer1))
+ }
+ }
+
+ test("VectorAvg - finite elements around non-finite deltas") {
+ val first = Array(1.0f, Float.MaxValue, 3.0f, Float.PositiveInfinity, 5.0f,
+ Float.NegativeInfinity, 7.0f, Float.PositiveInfinity, 9.0f, Float.NaN,
11.0f)
+ val second = Array(3.0f, -Float.MaxValue, 5.0f, 1.0f, 7.0f, 1.0f, 9.0f,
+ Float.NegativeInfinity, 11.0f, 2.0f, 13.0f)
+ val expected = Array(2.0f, 0.0f, 4.0f, Float.PositiveInfinity, 6.0f,
+ Float.NegativeInfinity, 8.0f, Float.NaN, 10.0f, Float.NaN, 12.0f)
+
+ Seq(false, true).foreach { useGenericArray =>
+ val (agg, buffer, _) = createVectorAvg()
+ Seq(first, second).foreach { values =>
+ agg.update(buffer, createInputRow(values, useGenericArray))
+ }
+ assertResult(expected.map(JFloat.floatToIntBits))(
+ evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+ }
+ }
+
+ test("VectorAvg - running-average rounding with and without infinity") {
+ Seq(false, true).foreach { includeInfinity =>
+ val (agg, buffer, _) = createVectorAvg()
+ val first = Array.tabulate(17)(i => if (i % 2 == 0) 1.0f else -1.0f)
+ val last = first.map(_ * 6)
+ val expected = Array.tabulate(17)(i => if (i % 2 == 0) 0x402aaaab else
0xc02aaaab)
+ if (includeInfinity) {
+ first(8) = Float.PositiveInfinity
+ last(8) = 1.0f
+ expected(8) = 0x7f800000
+ }
+ agg.update(buffer, createInputRow(first))
+ agg.update(buffer, createInputRow(first))
+ agg.update(buffer, createInputRow(last))
+ assertResult(expected)(
+ evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+ }
+ }
+
+ test("VectorAvg - non-finite deltas at different positions") {
+ val cases = Seq(
+ (Float.MaxValue, -Float.MaxValue, 0.0f),
+ (Float.PositiveInfinity, 1.0f, Float.PositiveInfinity),
+ (Float.NegativeInfinity, 1.0f, Float.NegativeInfinity),
+ (Float.PositiveInfinity, Float.PositiveInfinity, Float.PositiveInfinity),
+ (Float.NegativeInfinity, Float.NegativeInfinity, Float.NegativeInfinity),
+ (Float.PositiveInfinity, Float.NegativeInfinity, Float.NaN),
+ (Float.NaN, Float.NaN, Float.NaN),
+ (Float.NaN, Float.PositiveInfinity, Float.NaN),
+ (Float.NaN, Float.NegativeInfinity, Float.NaN),
+ (Float.NaN, 1.0f, Float.NaN))
+ for {
+ dim <- Seq(1, 3, 4, 5, 7, 8, 9, 17)
+ index <- 0 until dim
+ (left, right, expectedValue) <- cases
+ (oldAvg, value) <- Seq((left, right), (right, left))
+ useGenericArray <- Seq(false, true)
+ } {
+ val (agg, buffer, _) = createVectorAvg()
+ val first = Array.fill(dim)(1.0f)
+ val second = Array.fill(dim)(3.0f)
+ val expected = Array.fill(dim)(2.0f)
+ first(index) = oldAvg
+ second(index) = value
+ expected(index) = expectedValue
+ val rows = Seq(first, second).map { values =>
+ createInputRow(values, useGenericArray)
+ }
+ rows.foreach(row => agg.update(buffer, row))
+ assertResult(expected.map(JFloat.floatToIntBits))(
+ evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+
+ val (_, buffer1, _) = createVectorAvg()
+ val (_, buffer2, _) = createVectorAvg()
+ agg.update(buffer1, rows.head)
+ agg.update(buffer2, rows(1))
+ agg.merge(buffer1, buffer2)
+ assertResult(expected.map(JFloat.floatToIntBits))(
+ evalAsFloatArray(agg, buffer1).map(JFloat.floatToIntBits))
+ }
+ }
+
test("VectorAvg - numerical stability with running average") {
val (agg, buffer, _) = createVectorAvg()
// Add many vectors to test numerical stability of running average
@@ -516,7 +662,7 @@ class VectorAggSuite extends SparkFunSuite {
test("VectorSum - large vector with null element is skipped") {
val (agg, buffer, _) = createVectorSum()
- val vec1 = Array[java.lang.Float](1.0f, 2.0f, 3.0f, 4.0f, 5.0f, null,
7.0f, 8.0f,
+ val vec1 = Array[JFloat](1.0f, 2.0f, 3.0f, 4.0f, 5.0f, null, 7.0f, 8.0f,
9.0f, 10.0f, 11.0f, 12.0f, 13.0f, 14.0f, 15.0f, 16.0f)
val vec2 = Array(16.0f, 15.0f, 14.0f, 13.0f, 12.0f, 11.0f, 10.0f, 9.0f,
8.0f, 7.0f, 6.0f, 5.0f, 4.0f, 3.0f, 2.0f, 1.0f)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]