Taukir created SPARK-19976:
------------------------------
Summary: DirectStream API throws OffsetOutOfRange Exception
Key: SPARK-19976
URL: https://issues.apache.org/jira/browse/SPARK-19976
Project: Spark
Issue Type: Bug
Components: DStreams
Affects Versions: 1.6.2
Reporter: Taukir
I am using following code
def functionToCreateContext(sc:SparkContext):StreamingContext = {
val kafkaParams = new mutable.HashMap[String, Object]()
kafkaParams.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, kafkaBrokers)
kafkaParams.put(ConsumerConfig.GROUP_ID_CONFIG, KafkaConsumerGrp)
kafkaParams.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
classOf[StringDeserializer])
kafkaParams.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
classOf[StringDeserializer])
kafkaParams.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG,"true")
kafkaParams.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest")
val consumerStrategy = ConsumerStrategies.Subscribe[String,
String](threeparEventLogTopic.split(",").map(_.trim).filter(!_.isEmpty).toSet,
kafkaParams)
val kafkaStream =
KafkaUtils.createDirectStream(ssc,LocationStrategies.PreferConsistent,consumerStrategy)
}
spark throws error and crash once OffsetOutOf RangeException
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
