Hi I am processing kafka messages using spark streaming 1.3.
I am using mapPartitions function to process kafka message. How can I access offset no of individual message getting being processed. JavaPairInputDStream<byte[], byte[]> directKafkaStream =KafkaUtils.createDirectStream(..); directKafkaStream.mapPartitions(new FlatMapFunction<Iterator<Tuple2<byte[],byte[]>>, String>() { public Iterable<String> call(Iterator<Tuple2<byte[], byte[]>> t) throws Exception { while(t.hasNext()){ Tuple2<byte[], byte[]> tuple = t.next(); byte[] key = tuple._1(); byte[] msg = tuple._2(); ///how to get kafka partition no and offset of this message } } });