[
https://issues.apache.org/jira/browse/SPARK-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14600010#comment-14600010
]
Juan RodrĂguez Hortalá commented on SPARK-8337:
-----------------------------------------------
Hi,
As I said above, I don't know much about the internals of pyspark, and
currently the original RDD from Scala is wrapped by several wrappers for the
communication with python, and so the RDD implementing HasOffsetRanges is
hidden by those layers. However, after its merge with SPARK-8389, it looks like
this issue has got the attention of several Spark committers, and I'm sure they
will be able to come up with a solution that makes OffsetRanges accessible from
pyspark.
Greetings,
Juan
> KafkaUtils.createDirectStream for python is lacking API/feature parity with
> the Scala/Java version
> --------------------------------------------------------------------------------------------------
>
> Key: SPARK-8337
> URL: https://issues.apache.org/jira/browse/SPARK-8337
> Project: Spark
> Issue Type: Bug
> Components: PySpark, Streaming
> Affects Versions: 1.4.0
> Reporter: Amit Ramesh
> Priority: Critical
>
> See the following thread for context.
> http://apache-spark-developers-list.1001551.n3.nabble.com/Re-Spark-1-4-Python-API-for-getting-Kafka-offsets-in-direct-mode-tt12714.html
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
