pavanreddy02 created SPARK-61866:
------------------------------------
Summary: Vector validation in checkNonNanVectors is slow on JVMs
that do not inline virtual calls
Key: SPARK-61866
URL: https://issues.apache.org/jira/browse/SPARK-61866
Project: Spark
Issue Type: Improvement
Components: ML, MLlib
Affects Versions: 4.0.5
Reporter: pavanreddy02
SPARK-45547 replaced the compiled validateVector UDF in
DatasetUtils.checkNonNanVectors with the Catalyst higher-order function
exists(). ArrayExists extends CodegenFallback, so its lambda is interpreted per
element, costing several virtual calls per element of every vector. HotSpot
inlines these calls so no regression was visible there, but on JVMs that do not
(e.g. OpenJ9 on s390x) the cost is proportional to the vector size on every
row, for every algorithm that validates its features. Proposal: restore the
compiled UDF in checkNonNanVectors only. The VectorSizeHint and NaiveBayes
changes from SPARK-45547 are untouched. Benchmark details will be added as
comments.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]