Github user tdas commented on the pull request:
https://github.com/apache/spark/pull/3798#issuecomment-72756440
I spent some time talking to Patrick offline about this. If we expose the
KafkaRDD as is (while keeping its constructor private), then the simplified
createRDD would be
```
KafkaUtils.createRDD[K, V, KD, VD](....): KafkaRDD[K, V, KD, VD, (K, V)]
```
Imagine how one would use it in Java.
```
KafkaRDD<String, String, StringDecoder, StringDecode, Product2<String,
String>> rdd = KafkaUtils.createRDD(....)
```
That's not very Java friendly if you ask a Java developer. And we a huge
fraction of the community as Java developers. Furthemore, we want to add Python
API as well, and that also requires the interfaces to be Java-friendly. Here is
the alternative (I think) with what I proposed.
```
KafkaRDD<String, String> rdd = KafkaUtils.createRDD(....)
```
Much simpler.
Regarding casting, there are two cases,
1. casting RDD generated from createRDD - If we take off `HasOffsetRanges`
(`KafkaUtils.createRDD` returns only RDD), then user have to cast. But if we
return abstract class KafkaRDD, then no casting necessary.
2. casting RDD in `DStream.foreachRDD` - Casting is necessary either ways.
But isnt it more intuitive to write `rdd.asInstanceOf[KafkaRDD]` than
`rdd.asInstanceOf[HasOffsetRanges]`?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
