Github user cfregly commented on a diff in the pull request:
https://github.com/apache/spark/pull/6147#discussion_r30775310
--- Diff:
extras/kinesis-asl/src/main/scala/org/apache/spark/streaming/kinesis/KinesisUtils.scala
---
@@ -79,19 +181,116 @@ object KinesisUtils {
* per Kinesis' limit of 24 hours
*
(InitialPositionInStream.TRIM_HORIZON) or
* the tip of the stream
(InitialPositionInStream.LATEST).
- * @param storageLevel Storage level to use for storing the received
objects
+ * @param storageLevel Storage level to use for storing the received
objects.
+ * StorageLevel.MEMORY_AND_DISK_2 is recommended.
+ */
+ def createStream(
+ jssc: JavaStreamingContext,
+ kinesisAppName: String,
+ streamName: String,
+ endpointUrl: String,
+ regionName: String,
+ initialPositionInStream: InitialPositionInStream,
+ checkpointInterval: Duration,
+ storageLevel: StorageLevel
+ ): JavaReceiverInputDStream[Array[Byte]] = {
+ createStream(jssc.ssc, kinesisAppName, streamName, endpointUrl,
regionName,
+ initialPositionInStream, checkpointInterval, storageLevel)
+ }
+
+ /**
+ * Create an input stream that pulls messages from a Kinesis stream.
+ * This uses the Kinesis Client Library (KCL) to pull messages from
Kinesis.
*
- * @return JavaReceiverInputDStream[Array[Byte]]
+ * Note:
+ * The given AWS credentials will get saved in DStream checkpoints if
checkpointing
+ * is enabled. Make sure that your checkpoint directory is secure.
+ *
+ * @param jssc Java StreamingContext object
+ * @param kinesisAppName Kinesis application name used by the Kinesis
Client Library
+ * (KCL) to update DynamoDB
+ * @param streamName Kinesis stream name
+ * @param endpointUrl Url of Kinesis service (e.g.,
https://kinesis.us-east-1.amazonaws.com)
+ * @param regionName Name of region used by the Kinesis Client Library
(KCL) to update
+ * DynamoDB (lease coordination and checkpointing)
and CloudWatch (metrics)
+ * @param awsAccessKeyId AWS AccessKeyId (if null, will use
DefaultAWSCredentialsProviderChain)
--- End diff --
move these to the end
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
