varun senthilnathan created SPARK-31413:
-------------------------------------------
Summary: Accessing the sequence number and partition id for
records in Kinesis adapter
Key: SPARK-31413
URL: https://issues.apache.org/jira/browse/SPARK-31413
Project: Spark
Issue Type: Question
Components: DStreams
Affects Versions: 2.4.3
Environment: We are using spark 2.4.3 in java. We would like to log
the partition key and the sequence number of every event. The overloaded create
stream function of the kinesis utils always throws a compilation error.
{{Function<Record,Record> printSeq = s -> s;}}
{{KinesisUtils.createStream(}}jssc, appName, streamName, endPointUrl,
regionName, InitialPositionInStream.TRIM_HORIZON, kinesisCheckpointInterval,
StorageLevel.MEMORY_AND_DISK_SER(), printSeq, Record.class);
The exception is as follows:
{quote}no suitable method found for
createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
{quote}
JAVA DOCS:
[https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]
Is there a way out?
Reporter: varun senthilnathan
We
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
