zahed1994 commented on PR #59095:
URL: https://github.com/apache/spark/pull/59095#issuecomment-5888717288
Thanks for the thorough review and suggestions, @uros-b!
I have addressed all the feedback:
1. **PR Description**:
- Fixed the parent JIRA reference to **SPARK-59459** (PR #58758).
- Updated the `ClassTag` resolution details to reflect that only array
keys are specialized, while non-array keys and all values retain `fakeClassTag`.
- Scoped the user-facing section specifically to `parallelizePairs` with
array keys and `fromJavaRDD(rdd, keyClass)`.
- Removed the reference to the 3-arg overload.
2. **`collectFirst` in `parallelizePairs`**:
- Switched from `find` to `seq.iterator.collectFirst` so array keys
anywhere in the collection are detected, safely handling null tuples, null
keys, and non-array keys.
3. **Public API Surface**:
- Dropped the 3-arg `fromJavaRDD(rdd, keyClass, valueClass)` overload to
keep the public API surface minimal and focused strictly on the key-only guard.
4. **Test Coverage (`JavaAPISuite`)**:
- **Homogeneous `byte[]`**: Verified that `keys().collect()` and
`values().collect()` succeed and preserve array contents via
`assertArrayEquals`.
- **Array key not first**: Verified that an array key preceded by a
non-array key or a `null` key still triggers
`UNSUPPORTED_ARRAY_KEY.HASH_PARTITIONER`.
- **False-positive guard check**: Verified that `partitionBy(new
HashPartitioner(2))` on legal heterogeneous `Number` keys succeeds and collects
cleanly.
Regarding `mapToPair`, `flatMapToPair`, and untyped `fromJavaRDD`: because
lazy transformations have their types erased without driver-side collections to
inspect at construction time, handling them will require a separate design
(such as explicit type manifest overloads). I agree that this should be tracked
in a separate follow-up JIRA/PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
