HyukjinKwon commented on code in PR #59095:
URL: https://github.com/apache/spark/pull/59095#discussion_r4172950580
##########
core/src/main/scala/org/apache/spark/api/java/JavaSparkContext.scala:
##########
@@ -152,9 +152,14 @@ class JavaSparkContext(val sc: SparkContext) extends
Closeable {
/** Distribute a local Scala collection to form an RDD. */
def parallelizePairs[K, V](list: java.util.List[Tuple2[K, V]], numSlices:
Int)
: JavaPairRDD[K, V] = {
- implicit val ctagK: ClassTag[K] = fakeClassTag
+ val seq = list.asScala.toSeq
+ implicit val ctagK: ClassTag[K] =
+ seq.iterator.collectFirst {
Review Comment:
**Non-blocking (P2):** Following up on the leaf-ClassTag point: reverting
`ctagV` to `fakeClassTag` and narrowing `ctagK` to array keys only resolves the
`[Integer, Double]` value case, but the key side still regresses for a mixed
array/non-array key RDD. For `[("nonArrayKey", 1), (new byte[]{2}, 2)]`,
`collectFirst` picks `ClassTag(byte[])`, so `keys().collect()` allocates
`Array[byte[]]` and throws `ArrayStoreException` on the String key -- where the
old `fakeClassTag` (Object) produced `Array[AnyRef]` and succeeded. The added
test builds exactly this mixed RDD but only calls `partitionBy` (which throws
at the array-key guard first), so the `collect()` regression is uncovered.
Consider keeping an `AnyRef`/`Object` key ClassTag for the RDD and detecting
array keys for the guard from the actual keys (or specializing only when every
key shares the array class), and add a `keys().collect()` assertion over a
mixed-key RDD.
**Verification:**
- **Regression:** A test asserting keys().collect() on a mixed-key RDD
succeeds (or behaves as intended) fails before the fix and passes after, while
the array-key guard still fires.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
