Ruifeng Zheng created SPARK-61364:
-------------------------------------

             Summary: Check vector schema nullability in VectorDotProduct
                 Key: SPARK-61364
                 URL: https://issues.apache.org/jira/browse/SPARK-61364
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Ruifeng Zheng


VectorDotProduct currently uses StructType directly with ExpectsInputTypes, 
which accepts any struct without validating the vector schema or its nested 
nullability. In particular, nullable indices or values elements can reach code 
that assumes primitive array elements are non-null.

Validate both operands against the vector SQL schema with compatible 
nullability: require the expected field names and types and non-null 
indices/values elements, while allowing stricter non-nullable struct fields. 
Preserve the vector_dot_product(column, Vector) overload by constructing its 
primitive arrays with typed literals.

Add Catalyst coverage for both operands, malformed schemas, nullable array 
elements, and compatible stricter fields. Run the existing ML 
vector_dot_product test for dense and sparse constant vectors.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to