This is an automated email from the ASF dual-hosted git repository.

uros-b pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new f3199630c25a [SPARK-60421][SQL] Fix overflow and non-finite handling 
in vector_avg for float vectors
f3199630c25a is described below

commit f3199630c25a32b5510f415922182d2251beb224
Author: Uros Stankovic <[email protected]>
AuthorDate: Thu Nov 5 13:07:45 2026 +0000

    [SPARK-60421][SQL] Fix overflow and non-finite handling in vector_avg for 
float vectors
    
    ### What changes were proposed in this pull request?
    
    Fix overflow and non-finite handling when updating `vector_avg` for 
`ARRAY<FLOAT>` inputs.
    
    Keep the existing running-average arithmetic for finite deltas. When 
subtracting two finite operands overflows, scale them before subtraction. When 
either operand is non-finite, use IEEE addition to preserve infinities and 
propagate NaN.
    
    Check four deltas together before writing a block. An exceptional block 
uses the scalar helper with the values already loaded, then processing 
continues with the next block. Trailing elements use the same helper. The merge 
implementation is unchanged.
    
    Add regression coverage for overflow, cumulative weights, input order, 
exact finite rounding, both array representations, and block boundaries. 
Document the behavior change in the SQL migration guide.
    
    This change covers float vectors. The `ARRAY<DOUBLE>` work in #59590 needs 
equivalent handling separately.
    
    ### Why are the changes needed?
    
    The existing update formula can overflow its intermediate subtraction or 
turn an infinite running average into NaN after a finite input. This makes 
updates disagree with merging singleton buffers and makes the result depend on 
input order.
    
    The following cases show successive inputs for one vector element:
    
    | Input sequence | Before | After |
    |---|---|---|
    | `Float.MaxValue`, then `-Float.MaxValue` | `-Infinity` | `0.0f` |
    | `-Float.MaxValue`, then `Float.MaxValue` | `Infinity` | `0.0f` |
    | `Infinity`, then `Infinity` | `NaN` | `Infinity` |
    | `-Infinity`, then `-Infinity` | `NaN` | `-Infinity` |
    | `Infinity`, then a finite value | `NaN` | `Infinity` |
    | `-Infinity`, then a finite value | `NaN` | `-Infinity` |
    | Opposite infinities, in either order | `NaN` | `NaN` |
    | A finite value, then an infinity | That infinity | That infinity |
    | `NaN` and any value, in either order | `NaN` | `NaN` |
    
    ### Does this PR introduce _any_ user-facing change?
    
    Yes. Updating running averages with extreme finite inputs avoids 
intermediate subtraction overflow. An infinite running average absorbs finite 
values and same-sign infinities. Opposite infinities and NaN inputs continue to 
produce NaN.
    
    Finite updates whose subtraction does not overflow retain the original 
arithmetic and rounding. The result remains `ARRAY<FLOAT>`; scalar `avg(FLOAT)` 
uses double accumulation and can have different finite-input rounding.
    
    ### How was this patch tested?
    
    All **65 tests** in `VectorAggSuite` and both Catalyst Scala style checks 
passed on this revision:
    
    ```bash
    build/sbt 'catalyst/testOnly *VectorAggSuite' \
      'catalyst/Compile/scalastyle' 'catalyst/Test/scalastyle'
    ```
    
    Regression tests cover finite subtraction overflow with a one-third weight, 
both signs of infinity, opposite infinities, NaN operand combinations, and 
exact finite rounding. The table-driven cases exercise both input orders, 
updates and singleton-buffer merges, both array representations, and 
scalar/block boundaries. The weighted-overflow test covers dimensions 2, 4, 5, 
and 8 with both array representations.
    
    **Performance**
    
    Fresh local measurements compare the original `VectorAvg` at PR merge-base
    `9b17867be1f3` with PR head `e7eeef693981`. The merge method source is 
identical
    in both revisions.
    
    Times are ns/element; lower is better. Positive percentage changes mean 
slower.
    Each value is the median of five fresh-JVM medians, with three steady timed 
windows
    per JVM. Brackets show the minimum and maximum of the five JVM medians.
    `unsafe` and `generic` mean `UnsafeArrayData` and `GenericArrayData`; 
merges use
    binary aggregate buffers.
    
    **Finite updates**
    
    | Operation / input | Array | Dim | Before | After | Change |
    |---|---|---:|---:|---:|---:|
    | update: All finite | unsafe | 128 | 0.851 [0.812-0.852] | 0.898 
[0.896-0.899] | +5.5% |
    | update: All finite | unsafe | 1536 | 0.775 [0.775-0.776] | 0.798 
[0.798-0.798] | +3.0% |
    | update: All finite | generic | 128 | 0.965 [0.941-0.979] | 0.978 
[0.978-0.984] | +1.4% |
    | update: All finite | generic | 1536 | 0.862 [0.857-0.864] | 0.956 
[0.953-0.957] | +10.8% |
    
    **Updates with special values (dimension 1536)**
    
    | Operation / input | Array | Dim | Before | After | Change |
    |---|---|---:|---:|---:|---:|
    | update: First row: NaN at index 0 | unsafe | 1536 | 0.775 [0.774-0.775] | 
0.841 [0.841-0.841] | +8.5% |
    | update: First row: NaN at index 0 | generic | 1536 | 0.862 [0.858-0.863] 
| 1.139 [0.947-1.140] | +32.1% |
    | update: First row: +Inf at index 0 | unsafe | 1536 | 0.774 [0.774-0.776] 
| 0.842 [0.841-0.843] | +8.7% |
    | update: First row: +Inf at index 0 | generic | 1536 | 0.860 [0.859-0.864] 
| 0.950 [0.948-1.141] | +10.4% |
    | update: First row: all NaN | unsafe | 1536 | 0.775 [0.774-0.778] | 1.184 
[1.184-1.185] | +52.7% |
    | update: First row: all NaN | generic | 1536 | 0.861 [0.858-0.863] | 1.407 
[1.404-1.412] | +63.5% |
    | update: 75% non-finite later inputs | unsafe | 1536 | 0.775 [0.774-0.775] 
| 1.178 [1.167-1.233] | +52.1% |
    | update: 75% non-finite later inputs | generic | 1536 | 0.861 
[0.855-0.863] | 1.333 [1.327-1.334] | +54.9% |
    
    **Finite merges**
    
    | Operation / input | Array | Dim | Before | After | Change |
    |---|---|---:|---:|---:|---:|
    | merge: All finite | binary | 128 | 0.371 [0.367-0.373] | 0.370 
[0.367-0.395] | -0.3% |
    | merge: All finite | binary | 1536 | 0.344 [0.343-0.344] | 0.344 
[0.343-0.345] | -0.1% |
    
    Method: Intel Xeon Platinum 8375C, one pinned CPU; sequential JVM jobs with
    randomized workload order and randomized before/after order within each 
pair.
    Both revisions use Temurin OpenJDK 21.0.13+9, Scala 2.13.18, Java 21 target,
    identical dependencies/compiler/JVM flags, and a fixed 1 GiB heap. Each JVM 
first
    executes repeated 0.5s warmup windows for at least 8s, then measures three 
windows
    of at least 0.6s. This warms the enclosing caller's loop exits as well as 
the kernel.
    All 140 timed kernels, aggregate APIs, and enclosing timed callers reached 
C2
    before the timed windows, with no relevant C2 compilation or invalidation 
during
    measurement. No prior benchmark timings are reused.
    
    Update measures real `VectorAvg.update`: 100,000 calls per group, 64 seeded 
vectors,
    finite values in [-100,100), and a non-nullable array type. Construction, 
first row,
    buffer reset, and result consumption are outside timing. Early NaN/Inf and 
all-NaN-first
    cases receive finite later rows; mostly-nonfinite input has 75% NaN/+Inf 
elements
    in every later row. Null-element scanning is excluded.
    
    Merge measures real `VectorAvg.merge` over 512 independent pairs per batch, 
with
    counts 1,200 and 9,800 in both orders. Every target byte array and count is 
restored
    before timing; source buffers remain unchanged. Reset, result consumption, 
and count
    checks are outside timing. The left/right/pristine vector bytes occupy 
approximately
    0.75 MiB at dim128 and 9 MiB at dim1536, plus row objects; resetting warms 
the input data.
    
    Finite update results match over 8,000 groups; finite and mixed Inf/NaN 
merge results
    match over 10,000 groups each. The existing finite merge overflow is 
deliberately
    unchanged. Early +Inf updates have intentionally different results: the 
original
    returns NaN after later finite rows; this PR preserves +Inf.
    
    These are warmed single-thread aggregate microbenchmarks; they do not 
measure SQL
    execution, shuffle, or end-to-end queries. Other JVMs, cache footprints, 
null scans,
    and data distributions can change the result. Overlapping fork ranges 
should be
    treated as variation rather than a reliable improvement or regression.
    In particular, the 128-dimensional generic finite-update ranges overlap; 
its 1.4%
    median change is within variation. Generic early-NaN/Inf cases have visibly 
different
    timing modes across fresh JVMs, so their medians should be read alongside 
the ranges.
    
    ### Was this patch authored or co-authored using generative AI tooling?
    
    Generated-by: OpenAI Codex (GPT-6)
    
    Closes #59615 from urosstan-db/vector-avg-float-correctness.
    
    Authored-by: Uros Stankovic <[email protected]>
    Signed-off-by: Uros Bojanic <[email protected]>
---
 docs/sql-migration-guide.md                        |   1 +
 .../catalyst/expressions/vectorExpressions.scala   |  77 +++++++++-
 .../expressions/aggregate/VectorAggSuite.scala     | 158 ++++++++++++++++++++-
 3 files changed, 228 insertions(+), 8 deletions(-)

diff --git a/docs/sql-migration-guide.md b/docs/sql-migration-guide.md
index 1766bb3d0f5d..e18d02cdf6f8 100644
--- a/docs/sql-migration-guide.md
+++ b/docs/sql-migration-guide.md
@@ -41,6 +41,7 @@ license: |
 - Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a column referenced in a lambda function, such as `i` in `transform(a, x 
-> x + i)`, or with `IDENTIFIER`, such as `IDENTIFIER('i')`, from the value 
stored in that column. Previously, it read the value of the query before the 
store assignment cast, so it could silently store a value that does not match 
the stored column, and a later write of that row failed with 
`CHECK_CONSTRAINT_VIOLATION` (SPARK-60817).  [...]
 - Since Spark 4.4, an auto-filled generated column of a Data Source V2 table 
reads a field of another column, such as `s.f`, from the value stored in that 
column. Previously, it read the value of the query before the store assignment 
cast, so it could silently store a value that does not match the stored column, 
and a later write of that row failed with `CHECK_CONSTRAINT_VIOLATION` 
(SPARK-60803). Rows that are already stored keep their values, and data derived 
from them may need to be re [...]
 - Since Spark 4.4, `spark.addArtifact` and `spark.addArtifacts` take a file 
whose name ends with `.sparkpkg` as a [native data source 
package](sql-native-data-sources.html), also when it is added with `file=True`, 
`pyfile=True` or `archive=True` in Python without Spark Connect, or with 
`file=True` from a Spark Connect client, and fail with 
`INVALID_NATIVE_DATA_SOURCE_PACKAGE` for such a file that is not a valid 
package, or whose version Spark does not support, or that has no library for  
[...]
+- Since Spark 4.4, `vector_avg` avoids intermediate subtraction overflow when 
updating running averages for `ARRAY<FLOAT>` inputs. Extreme finite inputs that 
previously produced infinity or `NaN` during updates may now produce a finite 
average. An infinite running average combined with a finite value or an 
infinity of the same sign now produces infinity with that sign instead of 
`NaN`. Opposite infinities and `NaN` inputs continue to produce `NaN`.
 - Since Spark 4.4, `spark.sql.execution.arrow.pythonUDF.columnarInput.enabled` 
is `false` by default: Arrow-based Python UDFs read rows from an Arrow-backed 
upstream operator, such as a Data Source V2 scan or the Arrow cache, as before 
Spark 4.2. With columnar input, they kept the columns that they pass through 
until the results of their batch came back from Python, and failed or returned 
wrong results when the operator reused or released the vectors of the batch in 
the meantime. To enab [...]
 - Since Spark 4.4, the conversion of columnar batches to rows releases each 
batch with `ColumnVector.closeIfFreeable` once its rows are consumed also when 
it is not whole-stage code-generated, e.g. when the query has more than 
`spark.sql.codegen.maxFields` fields or `spark.sql.codegen.wholeStage` is 
`false`, as the generated code has done since Spark 3.5.4 and 4.0. A 
`ColumnVector` that a Data Source V2 source reuses across batches, such as an 
`ArrowColumnVector` over a `VectorSchemaRoot [...]
 - Since Spark 4.4, `spark.sql.charVarchar.standardSemantics.enabled` (default 
`false`) is a preview feature that may change in future releases. When true, it 
enables first-class `CHAR(n)` and `VARCHAR(n)` semantics throughout schemas and 
query plans: `CHAR` -> `VARCHAR` -> `STRING` form a type-precedence family, 
explicit casts can return constrained types, least-common-type resolution for 
`COALESCE`/`CASE`/`UNION` can return `CHAR` or `VARCHAR`, and string 
expressions that transform thei [...]
diff --git 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
index ef2cb94c932b..a56acc4b6c22 100644
--- 
a/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
+++ 
b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/vectorExpressions.scala
@@ -17,6 +17,8 @@
 
 package org.apache.spark.sql.catalyst.expressions
 
+import java.lang.{Float => JFloat}
+
 import org.apache.spark.sql.catalyst.InternalRow
 import org.apache.spark.sql.catalyst.analysis.TypeCheckResult
 import org.apache.spark.sql.catalyst.analysis.TypeCheckResult.DataTypeMismatch
@@ -672,17 +674,88 @@ object VectorAvg {
         inputArray: ArrayData,
         dim: Int,
         newCount: Long): Unit = {
-      // Update running average: new_avg = old_avg + (new_value - old_avg) / 
new_count
+      // Finite deltas use: new_avg = old_avg + (new_value - old_avg) * 
invCount.
+      // updateElement handles subtraction overflow and non-finite operands.
       val invCount = 1.0f / newCount
+      updateElementsInBlocks(accBytes, inputArray, dim, invCount)
+    }
+
+    private def updateElementsInBlocks(
+        accBytes: Array[Byte],
+        inputArray: ArrayData,
+        dim: Int,
+        invCount: Float): Unit = {
       var i = 0
+      while (i <= dim - 4) {
+        val off = Platform.BYTE_ARRAY_OFFSET + i.toLong * 4
+        val oldAvg0 = Platform.getFloat(accBytes, off)
+        val value0 = inputArray.getFloat(i)
+        val delta0 = value0 - oldAvg0
+        val oldAvg1 = Platform.getFloat(accBytes, off + 4)
+        val value1 = inputArray.getFloat(i + 1)
+        val delta1 = value1 - oldAvg1
+        val oldAvg2 = Platform.getFloat(accBytes, off + 8)
+        val value2 = inputArray.getFloat(i + 2)
+        val delta2 = value2 - oldAvg2
+        val oldAvg3 = Platform.getFloat(accBytes, off + 12)
+        val value3 = inputArray.getFloat(i + 3)
+        val delta3 = value3 - oldAvg3
+        // A finite sum guarantees finite deltas; sum overflow only triggers a 
scalar fallback.
+        val deltaSum = (delta0 + delta1) + (delta2 + delta3)
+        if (JFloat.isFinite(deltaSum)) {
+          Platform.putFloat(accBytes, off, oldAvg0 + delta0 * invCount)
+          Platform.putFloat(accBytes, off + 4, oldAvg1 + delta1 * invCount)
+          Platform.putFloat(accBytes, off + 8, oldAvg2 + delta2 * invCount)
+          Platform.putFloat(accBytes, off + 12, oldAvg3 + delta3 * invCount)
+        } else {
+          Platform.putFloat(accBytes, off, updateElement(oldAvg0, value0, 
invCount))
+          Platform.putFloat(accBytes, off + 4, updateElement(oldAvg1, value1, 
invCount))
+          Platform.putFloat(accBytes, off + 8, updateElement(oldAvg2, value2, 
invCount))
+          Platform.putFloat(accBytes, off + 12, updateElement(oldAvg3, value3, 
invCount))
+        }
+        i += 4
+      }
+      if (i < dim) {
+        updateElementsScalar(accBytes, inputArray, i, dim, invCount)
+      }
+    }
+
+    private def updateElementsScalar(
+        accBytes: Array[Byte],
+        inputArray: ArrayData,
+        start: Int,
+        dim: Int,
+        invCount: Float): Unit = {
+      var i = start
       while (i < dim) {
         val off = Platform.BYTE_ARRAY_OFFSET + i.toLong * 4
         val oldAvg = Platform.getFloat(accBytes, off)
-        Platform.putFloat(accBytes, off, oldAvg + (inputArray.getFloat(i) - 
oldAvg) * invCount)
+        val value = inputArray.getFloat(i)
+        Platform.putFloat(accBytes, off, updateElement(oldAvg, value, 
invCount))
         i += 1
       }
     }
 
+    private def updateElement(oldAvg: Float, value: Float, invCount: Float): 
Float = {
+      val delta = value - oldAvg
+      if (JFloat.isFinite(delta)) {
+        oldAvg + delta * invCount
+      } else if (JFloat.isFinite(oldAvg) && JFloat.isFinite(value)) {
+        // Scale before subtracting when opposite finite values overflow the 
delta.
+        // For oldAvg = Float.MaxValue, value = -Float.MaxValue and invCount = 
0.5f,
+        // the delta is -Infinity, but scaling first gives the correct average 
of 0.0f.
+        oldAvg + (value * invCount - oldAvg * invCount)
+      } else {
+        // At least one operand is non-finite. Addition gives the IEEE result
+        // regardless of which operand is the running average:
+        // Inf + Inf -> Inf; -Inf + (-Inf) -> -Inf.
+        // Inf + (-Inf) -> NaN.
+        // NaN + any value -> NaN.
+        // Inf + finite -> Inf; -Inf + finite -> -Inf.
+        oldAvg + value
+      }
+    }
+
     override def mergeElements(
         accBytes: Array[Byte],
         inputBytes: Array[Byte],
diff --git 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
index d509bfd56a85..624518a9dce8 100644
--- 
a/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
+++ 
b/sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/expressions/aggregate/VectorAggSuite.scala
@@ -17,7 +17,7 @@
 
 package org.apache.spark.sql.catalyst.expressions.aggregate
 
-import java.lang.{Double => JDouble}
+import java.lang.{Double => JDouble, Float => JFloat}
 
 import org.apache.spark.{SparkFunSuite, SparkRuntimeException}
 import org.apache.spark.sql.catalyst.InternalRow
@@ -48,8 +48,15 @@ class VectorAggSuite extends SparkFunSuite {
   }
 
   // Helper to create input row with float array
-  def createInputRow(values: Array[Float]): InternalRow = {
-    InternalRow(ArrayData.toArrayData(values))
+  def createInputRow(
+      values: Array[Float],
+      useGenericArray: Boolean = false): InternalRow = {
+    val array = if (useGenericArray) {
+      new GenericArrayData(values)
+    } else {
+      ArrayData.toArrayData(values)
+    }
+    InternalRow(array)
   }
 
   // Helper to create input row with null
@@ -60,7 +67,7 @@ class VectorAggSuite extends SparkFunSuite {
   }
 
   // Helper to create input row with array containing null elements
-  def createInputRowWithNullElement(values: Array[java.lang.Float]): 
InternalRow = {
+  def createInputRowWithNullElement(values: Array[JFloat]): InternalRow = {
     val arrayData = new GenericArrayData(values.map {
       case null => null
       case v => v.floatValue().asInstanceOf[AnyRef]
@@ -86,7 +93,7 @@ class VectorAggSuite extends SparkFunSuite {
   // Asserts that two floats are approximately equal within a tolerance.
   def assertFloatEquals(actual: Float, expected: Float, tolerance: Float = 
1e-5f): Unit = {
     assert(
-      java.lang.Float.isNaN(expected) && java.lang.Float.isNaN(actual) ||
+      actual == expected || JFloat.isNaN(expected) && JFloat.isNaN(actual) ||
         math.abs(actual - expected) <= tolerance,
       s"Expected $expected but got $actual (tolerance: $tolerance)"
     )
@@ -405,6 +412,145 @@ class VectorAggSuite extends SparkFunSuite {
     assert(result(1) === 3.0f)
   }
 
+  test("VectorAvg - opposite extreme values do not overflow") {
+    val (agg, buffer, _) = createVectorAvg()
+    val expected = Array(0.0f, 0.0f).map(JFloat.floatToIntBits)
+    agg.update(buffer, createInputRow(Array(Float.MaxValue, -Float.MaxValue)))
+    agg.update(buffer, createInputRow(Array(-Float.MaxValue, Float.MaxValue)))
+    assertResult(expected)(evalAsFloatArray(agg, 
buffer).map(JFloat.floatToIntBits))
+
+    val (_, buffer1, _) = createVectorAvg()
+    val (_, buffer2, _) = createVectorAvg()
+    agg.update(buffer1, createInputRow(Array(Float.MaxValue, -Float.MaxValue)))
+    agg.update(buffer2, createInputRow(Array(-Float.MaxValue, Float.MaxValue)))
+    agg.merge(buffer1, buffer2)
+    assertResult(expected)(evalAsFloatArray(agg, 
buffer1).map(JFloat.floatToIntBits))
+  }
+
+  test("VectorAvg - finite overflow preserves cumulative weights") {
+    for {
+      dim <- Seq(2, 4, 5, 8)
+      useGenericArray <- Seq(false, true)
+    } {
+      val (agg, buffer, _) = createVectorAvg()
+      val first = Array.tabulate(dim)(i => if (i % 2 == 0) Float.MaxValue else 
-Float.MaxValue)
+      agg.update(buffer, createInputRow(first, useGenericArray))
+      agg.update(buffer, createInputRow(first, useGenericArray))
+      // The third update overflows the delta. An unweighted midpoint would 
incorrectly return zero.
+      agg.update(buffer, createInputRow(first.map(-_), useGenericArray))
+      val expected = (Float.MaxValue.toDouble / 3).toFloat
+      assertFloatArrayEquals(
+        evalAsFloatArray(agg, buffer),
+        Array.tabulate(dim)(i => if (i % 2 == 0) expected else -expected),
+        tolerance = math.ulp(expected) * 2)
+    }
+  }
+
+  test("VectorAvg - non-finite results do not depend on input order") {
+    val first =
+      Array(Float.PositiveInfinity, 1.0f, Float.NegativeInfinity, 
Float.PositiveInfinity)
+    val second = Array(1.0f, Float.PositiveInfinity, 1.0f, 
Float.NegativeInfinity)
+    def checkResult(result: Array[Float]): Unit = {
+      assertFloatArrayEquals(result,
+        Array(Float.PositiveInfinity, Float.PositiveInfinity, 
Float.NegativeInfinity, Float.NaN))
+    }
+
+    Seq(Seq(first, second), Seq(second, first)).foreach { vectors =>
+      val (agg, buffer, _) = createVectorAvg()
+      vectors.foreach(v => agg.update(buffer, createInputRow(v)))
+      checkResult(evalAsFloatArray(agg, buffer))
+      agg.update(buffer, createInputRow(Array(2.0f, 2.0f, 2.0f, 2.0f)))
+      checkResult(evalAsFloatArray(agg, buffer))
+
+      val (_, buffer1, _) = createVectorAvg()
+      val (_, buffer2, _) = createVectorAvg()
+      agg.update(buffer1, createInputRow(vectors.head))
+      agg.update(buffer2, createInputRow(vectors(1)))
+      agg.merge(buffer1, buffer2)
+      checkResult(evalAsFloatArray(agg, buffer1))
+    }
+  }
+
+  test("VectorAvg - finite elements around non-finite deltas") {
+    val first = Array(1.0f, Float.MaxValue, 3.0f, Float.PositiveInfinity, 5.0f,
+      Float.NegativeInfinity, 7.0f, Float.PositiveInfinity, 9.0f, Float.NaN, 
11.0f)
+    val second = Array(3.0f, -Float.MaxValue, 5.0f, 1.0f, 7.0f, 1.0f, 9.0f,
+      Float.NegativeInfinity, 11.0f, 2.0f, 13.0f)
+    val expected = Array(2.0f, 0.0f, 4.0f, Float.PositiveInfinity, 6.0f,
+      Float.NegativeInfinity, 8.0f, Float.NaN, 10.0f, Float.NaN, 12.0f)
+
+    Seq(false, true).foreach { useGenericArray =>
+      val (agg, buffer, _) = createVectorAvg()
+      Seq(first, second).foreach { values =>
+        agg.update(buffer, createInputRow(values, useGenericArray))
+      }
+      assertResult(expected.map(JFloat.floatToIntBits))(
+        evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+    }
+  }
+
+  test("VectorAvg - running-average rounding with and without infinity") {
+    Seq(false, true).foreach { includeInfinity =>
+      val (agg, buffer, _) = createVectorAvg()
+      val first = Array.tabulate(17)(i => if (i % 2 == 0) 1.0f else -1.0f)
+      val last = first.map(_ * 6)
+      val expected = Array.tabulate(17)(i => if (i % 2 == 0) 0x402aaaab else 
0xc02aaaab)
+      if (includeInfinity) {
+        first(8) = Float.PositiveInfinity
+        last(8) = 1.0f
+        expected(8) = 0x7f800000
+      }
+      agg.update(buffer, createInputRow(first))
+      agg.update(buffer, createInputRow(first))
+      agg.update(buffer, createInputRow(last))
+      assertResult(expected)(
+        evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+    }
+  }
+
+  test("VectorAvg - non-finite deltas at different positions") {
+    val cases = Seq(
+      (Float.MaxValue, -Float.MaxValue, 0.0f),
+      (Float.PositiveInfinity, 1.0f, Float.PositiveInfinity),
+      (Float.NegativeInfinity, 1.0f, Float.NegativeInfinity),
+      (Float.PositiveInfinity, Float.PositiveInfinity, Float.PositiveInfinity),
+      (Float.NegativeInfinity, Float.NegativeInfinity, Float.NegativeInfinity),
+      (Float.PositiveInfinity, Float.NegativeInfinity, Float.NaN),
+      (Float.NaN, Float.NaN, Float.NaN),
+      (Float.NaN, Float.PositiveInfinity, Float.NaN),
+      (Float.NaN, Float.NegativeInfinity, Float.NaN),
+      (Float.NaN, 1.0f, Float.NaN))
+    for {
+      dim <- Seq(1, 3, 4, 5, 7, 8, 9, 17)
+      index <- 0 until dim
+      (left, right, expectedValue) <- cases
+      (oldAvg, value) <- Seq((left, right), (right, left))
+      useGenericArray <- Seq(false, true)
+    } {
+      val (agg, buffer, _) = createVectorAvg()
+      val first = Array.fill(dim)(1.0f)
+      val second = Array.fill(dim)(3.0f)
+      val expected = Array.fill(dim)(2.0f)
+      first(index) = oldAvg
+      second(index) = value
+      expected(index) = expectedValue
+      val rows = Seq(first, second).map { values =>
+        createInputRow(values, useGenericArray)
+      }
+      rows.foreach(row => agg.update(buffer, row))
+      assertResult(expected.map(JFloat.floatToIntBits))(
+        evalAsFloatArray(agg, buffer).map(JFloat.floatToIntBits))
+
+      val (_, buffer1, _) = createVectorAvg()
+      val (_, buffer2, _) = createVectorAvg()
+      agg.update(buffer1, rows.head)
+      agg.update(buffer2, rows(1))
+      agg.merge(buffer1, buffer2)
+      assertResult(expected.map(JFloat.floatToIntBits))(
+        evalAsFloatArray(agg, buffer1).map(JFloat.floatToIntBits))
+    }
+  }
+
   test("VectorAvg - numerical stability with running average") {
     val (agg, buffer, _) = createVectorAvg()
     // Add many vectors to test numerical stability of running average
@@ -516,7 +662,7 @@ class VectorAggSuite extends SparkFunSuite {
 
   test("VectorSum - large vector with null element is skipped") {
     val (agg, buffer, _) = createVectorSum()
-    val vec1 = Array[java.lang.Float](1.0f, 2.0f, 3.0f, 4.0f, 5.0f, null, 
7.0f, 8.0f,
+    val vec1 = Array[JFloat](1.0f, 2.0f, 3.0f, 4.0f, 5.0f, null, 7.0f, 8.0f,
                      9.0f, 10.0f, 11.0f, 12.0f, 13.0f, 14.0f, 15.0f, 16.0f)
     val vec2 = Array(16.0f, 15.0f, 14.0f, 13.0f, 12.0f, 11.0f, 10.0f, 9.0f,
                      8.0f, 7.0f, 6.0f, 5.0f, 4.0f, 3.0f, 2.0f, 1.0f)


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to