ngtanvan opened a new pull request, #58417:
URL: https://github.com/apache/spark/pull/58417
### What changes were proposed in this pull request?
Fix `MinHashLSHModel.hashFunction` to return `Array.empty[Vector]` when
encountering empty sparse vectors (0 non-zero entries) instead of throwing an
unhandled `IllegalArgumentException`.
### Why are the changes needed?
When pipelines combining `CountVectorizer` and `MinHashLSH` process empty
sparse vectors (e.g. text containing terms outside vocabulary), `MinHashLSH`
threw `java.lang.IllegalArgumentException: requirement failed: Must have at
least 1 non zero entry.`, crashing the entire Spark job.
Returning an empty vector array allows `posexplode` in
`approxSimilarityJoin` to cleanly skip empty vectors without crashing.
### Does this PR introduce _any_ user-facing change?
No breaking change. It fixes an unexpected crash when transforming or
joining empty vectors with `MinHashLSH`.
### How was this patch tested?
Updated unit test `MinHashLSHSuite` ("hashFunction: empty vector") to verify
returning empty vector array, and verified all 15 tests in `MinHashLSHSuite`
pass cleanly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
