Re: Kafka 0.8 + Spark 2.0 Partition Issue

[Cody Koeninger]

Kafka is designed to only allow reads from leaders.  You need to fix
this at the kafka level not the spark level.

On Fri, Jan 6, 2017 at 7:33 AM, Raghu Vadapalli  wrote:
>
> My spark 2.0 +  kafka 0.8 streaming job fails with error partition leaderset
> exception. When I check the kafka topic the partition, it is indeed in error
> with Leader = -1 and empty ISR.  I did lot of google and all of them point
> to either restarting or deleting the topic.  To do any of those two in
> production system while other topics are in heavy use is next to impossible.
> Now my question, is there way to force spark to read from leaderless
> partition accepting some dataloss or inconsistency ? Or force the immediate
> sync followed by election ( a kafka users group question but I am pushing my
> luck:) here )
>
> Topic: vzecsapplog Partition: 8 Leader: -1 Replicas: 5,4 Isr:
>
> --
> Regards,
> Raghu
>

-
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Kafka 0.8 + Spark 2.0 Partition Issue

[Raghu Vadapalli]

My spark 2.0 +  kafka 0.8 streaming job fails with error partition leaderset 
exception. When I check the kafka topic the partition, it is indeed in error 
with Leader = -1 and empty ISR.  I did lot of google and all of them point to 
either restarting or deleting the topic.  To do any of those two in production 
system while other topics are in heavy use is next to impossible.  Now my 
question, is there way to force spark to read from leaderless partition 
accepting some dataloss or inconsistency ? Or force the immediate sync followed 
by election ( a kafka users group question but I am pushing my luck:) here ) 

Topic: vzecsapplog  Partition: 8Leader: -1  Replicas: 5,4   Isr:



--
Regards,
Raghu