Takeshi Yamamuro created SPARK-18942:
----------------------------------------
Summary: Support output operations for kinesis
Key: SPARK-18942
URL: https://issues.apache.org/jira/browse/SPARK-18942
Project: Spark
Issue Type: New Feature
Components: DStreams
Affects Versions: 2.0.2
Reporter: Takeshi Yamamuro
Priority: Trivial
Spark does not support output operations (e.g. DStream#saveAsTextFile) for
Kinesis. So, officially supporting this is useful for some AWS users, I think.
An usage of the output operations is assumed as follows;
{code}
// Import a class that includes an output function
scala> import org.apache.spark.streaming.kinesis.KinesisDStreamFunctions._
// Create a DStream
scala> val stream: DStream[String] = ...
// Define a handler to convert the DStream type for output
scala> val msgHandler = (s: String) => s.getBytes("UTF-8")
// Define the output operation
scala> kinesisStream.count().saveAsKinesisStream(streamName, endpointUrl,
msgHandler)
{code}
A prototype I made is here:
https://github.com/apache/spark/compare/master...maropu:OutputOpForKinesis
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
