[
https://issues.apache.org/jira/browse/SPARK-15406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15486131#comment-15486131
]
Cody Koeninger commented on SPARK-15406:
----------------------------------------
I've got a minimal working Source and SourceProvider, at least for topics that
are String key and value only, at
https://github.com/apache/spark/compare/master...koeninger:SPARK-15406
If you haven't already attempted an implementation, I'd suggest at least
looking at that before writing up a design doc that may or may not address some
of the pragmatic issues.
The big thing I'm running into, and maybe I'm just not understanding the
intention behind the SourceProvider interface, is that putting all
configuration through a Map[String, String] makes it super awkward to configure
types, or classes, or collections of offsets, or... anything really.
Another significant issue is that I have no idea how rate limiting is supposed
to work.
> Structured streaming support for consuming from Kafka
> -----------------------------------------------------
>
> Key: SPARK-15406
> URL: https://issues.apache.org/jira/browse/SPARK-15406
> Project: Spark
> Issue Type: New Feature
> Reporter: Cody Koeninger
>
> Structured streaming doesn't have support for kafka yet. I personally feel
> like time based indexing would make for a much better interface, but it's
> been pushed back to kafka 0.10.1
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-33+-+Add+a+time+based+log+index
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
