HyukjinKwon commented on PR #58799:
URL: https://github.com/apache/spark/pull/58799#issuecomment-5672778408
Reviewed as part of a batch pass (read-only). LGTM.
The fix makes `PythonUserDefinedType` identity depend on `sqlType` in
addition to
`pyUDT`, keeping `equals`/`hashCode` consistent (`Objects.hash(pyUDT,
sqlType)` matches
the widened `equals`). This is what the
`KeyedPartitioning.groupedKeyRowOrdering` cache
needs: two Python UDTs sharing a `pyUDT` but differing in `sqlType` (e.g.
`StringType`
vs `StringType("UTF8_LCASE")`) previously collided in the cache and reused
an ordering
built for the wrong collation. The `InternalRowComparableWrapperSuite` cases
exercise
exactly that — and using a UUID for `pyUDT` to dodge JVM-lifetime cache
pollution is a
nice touch — while the `DataTypeSuite` case pins the equals/hashCode
contract.
One optional, non-blocking note: `acceptsType` still compares only `pyUDT`.
That reads
as intentional (type-acceptance is a different question from identity), but
a one-line
comment there would make the asymmetry explicit for future readers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
