[
https://issues.apache.org/jira/browse/SPARK-1981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14053037#comment-14053037
]
Chris Fregly commented on SPARK-1981:
-------------------------------------
[~matei] [~pwendell] i'm in the process of making the Kinesis Streaming
component an optional module similar to ganglia per
https://issues.apache.org/jira/browse/LEGAL-198
unlike the ganglia component, however, this component has tests and examples
similar to the other streaming implementations such as Kafka and Flume. these
other implementations have their tests and examples in external/ and examples/,
respectively.
if i understand correctly, i need to put the kinesis streaming code, test, and
examples all within extras/, correct?
this will cause a bit of confusion for people searching the examples/ source,
but - unless i'm missing something - this is the best we can do given the
current build scripts and directory structure.
is this the correct approach?
the other option is to stick with the base AWS Java SDK which is under Apache
2.0 license (https://github.com/aws/aws-sdk-java/blob/master/LICENSE.txt)
we'd lose some of the convenience goodies that the Kinesis Client Library gives
us like worker load balancing, shard autoscaling, checkpointing, etc but would
simplify the build.
definitely not optimal, but throwing it out as an option.
thoughts?
> Add AWS Kinesis streaming support
> ---------------------------------
>
> Key: SPARK-1981
> URL: https://issues.apache.org/jira/browse/SPARK-1981
> Project: Spark
> Issue Type: New Feature
> Components: Streaming
> Reporter: Chris Fregly
> Assignee: Chris Fregly
>
> Add AWS Kinesis support to Spark Streaming.
> Initial discussion occured here: https://github.com/apache/spark/pull/223
> I discussed this with Parviz from AWS recently and we agreed that I would
> take this over.
> Look for a new PR that takes into account all the feedback from the earlier
> PR including spark-1.0-compliant implementation, AWS-license-aware build
> support, tests, comments, and style guide compliance.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
