[
https://issues.apache.org/jira/browse/SPARK-31413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
varun senthilnathan updated SPARK-31413:
----------------------------------------
Description:
We are using spark 2.4.3 in java. We would like to log the partition key and
the sequence number of every event. The overloaded create stream function of
the kinesis utils always throws a compilation error.
Function<Record,Record> printSeq = s -> s;
KinesisUtils.createStream(
jssc,
appName,
streamName,
endPointUrl,
regionName,
InitialPositionInStream.TRIM_HORIZON,
kinesisCheckpointInterval,
StorageLevel.MEMORY_AND_DISK_SER(),
printSeq,
Record.class);
Record is of type com.amazonaws.services.kinesis.model.Record
The exception is as follows:
{quote}no suitable method found for
createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
{quote}
JAVA DOCS :
[https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]
Is there a way out?
was:
We are using spark 2.4.3 in java. We would like to log the partition key and
the sequence number of every event. The overloaded create stream function of
the kinesis utils always throws a compilation error.
Function<Record,Record> printSeq = s -> s;
KinesisUtils.createStream(
jssc,
appName,
streamName,
endPointUrl,
regionName,
InitialPositionInStream.TRIM_HORIZON,
kinesisCheckpointInterval,
StorageLevel.MEMORY_AND_DISK_SER(),
printSeq,
Record.class);
The exception is as follows:
{quote}no suitable method found for
createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
{quote}
JAVA DOCS :
[https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]
Is there a way out?
> Accessing the sequence number and partition id for records in Kinesis adapter
> -----------------------------------------------------------------------------
>
> Key: SPARK-31413
> URL: https://issues.apache.org/jira/browse/SPARK-31413
> Project: Spark
> Issue Type: Question
> Components: DStreams
> Affects Versions: 2.4.3
> Reporter: varun senthilnathan
> Priority: Critical
>
> We are using spark 2.4.3 in java. We would like to log the partition key and
> the sequence number of every event. The overloaded create stream function of
> the kinesis utils always throws a compilation error.
>
> Function<Record,Record> printSeq = s -> s;
> KinesisUtils.createStream(
> jssc,
> appName,
> streamName,
> endPointUrl,
> regionName,
> InitialPositionInStream.TRIM_HORIZON,
> kinesisCheckpointInterval,
> StorageLevel.MEMORY_AND_DISK_SER(),
> printSeq,
> Record.class);
> Record is of type com.amazonaws.services.kinesis.model.Record
>
> The exception is as follows:
> {quote}no suitable method found for
> createStream(org.apache.spark.streaming.api.java.JavaStreamingContext,java.lang.String,java.lang.String,java.lang.String,java.lang.String,com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream,org.apache.spark.streaming.Duration,org.apache.spark.storage.StorageLevel,java.util.function.Function,java.lang.Class)
> {quote}
> JAVA DOCS :
> [https://spark.apache.org/docs/latest/api/java/org/apache/spark/streaming/kinesis/KinesisUtils.html#createStream-org.apache.spark.streaming.api.java.JavaStreamingContext-java.lang.String-java.lang.String-java.lang.String-java.lang.String-com.amazonaws.services.kinesis.clientlibrary.lib.worker.InitialPositionInStream-org.apache.spark.streaming.Duration-org.apache.spark.storage.StorageLevel-org.apache.spark.api.java.function.Function-java.lang.Class-]
> Is there a way out?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
