Github user koeninger commented on the pull request:
https://github.com/apache/spark/pull/7185#issuecomment-118188461
I don't have a problem with the static method, especially if it prevents
yet more wrapper classes.
My concern was more about the difference in semantics, not a static vs
member method.
On Jul 2, 2015 3:15 PM, "Tathagata Das" <[email protected]> wrote:
> I agree with most of @koeninger <https://github.com/koeninger> points. I
> think the issue is that there is that the python KafkaDirectStream
> generates python RDDs which is effectively [python map on the JavaPairRDD
> wrapping the KafkaRDD].
>
> How about this.
>
> -
>
> Create a KafkaRDD class in python which is a subclass of python's RDD
> class. It adds the method getOffsetRanges(), which will find the offset
> ranges by traversing through the from the Scala RDD inside associated
> JavaPairRDD.
> -
>
> Create a KafkaDirectDStream in python which is a subclass of python's
> DStream, and override transform and foreachRDD method to apply the user
> specified function on the python's KafkaRDD objects instead of RDD
objects.
> This will ensure that the user will be able to call
> rdd.getOffsetRanges() inside the transform and foreach functions.
>
> With this I think it is doable to maintain consistency with Scala API,
> that is, rdd.getOffsetRanges() in python gives the offset ranges.
>
> â
> Reply to this email directly or view it on GitHub
> <https://github.com/apache/spark/pull/7185#issuecomment-118153576>.
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
