>From the logs you've attached, my guess is it's most likely due to KAFKA-1382.
Thanks, Neha On Wed, Sep 24, 2014 at 10:48 AM, Seshadri, Balaji <balaji.sesha...@dish.com > wrote: > Hi, > > > > We got the below error in our logs and our consumers stopped consuming any > data ?.It worked only after restart. > > > > We would like to confirm that it’s because we are running with 0.8-beta > version and not 0.8 release version to convince “THE MGMT” guys. > > > > Please let me know if it’s this KAFKA-1382 causing the issue. > > > > Thanks, > > > > Balaji > > > > *From:* Gulia, Vikram > *Sent:* Wednesday, September 24, 2014 8:43 AM > *To:* Sharma, Navdeep; #IT-MAD DES; #IT-MAA > *Cc:* Alam, Mohammad Shah > *Subject:* RE: 9/23 prod issue - offline kafka partitions. > > > > Adding full MAA distro. > > > > DES Offshore looked in to the logs on kafka servers and seems like the > issue we encountered yesterday may be described in these threads, please > have a look - > > > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 > > > > https://issues.apache.org/jira/browse/KAFKA-1382 (it describes the > fix/patch which is available in 0.8.1.2/0.8.2) > > > > Thank You, > > Vikram Gulia > > > > *From:* Sharma, Navdeep > *Sent:* Wednesday, September 24, 2014 6:53 AM > *To:* Gulia, Vikram; #IT-MAD DES > *Cc:* #IT-MAA Offshore; Alam, Mohammad Shah > *Subject:* RE: 9/23 prod issue - offline kafka partitions. > > > > Hi Vikram, > > > > We analyzed below mentioned issue with MAA-Offshore (Abhishek) and found > that the error occurred only on 23 Sept. This is not historical as we > checked last 4 days logs. > > > > It looks like that consumer got stopped on September 22 2014 for Linux > patching activty.MAA started consumer September 23 2014 at 1:00 AM. > > > > *Issue *in server log *"* *BadVersion for > /brokers/topics/rain-burn-in/partitions/121/state"* but it is not > present in previous 4 days logs. > > More detail of this error can be found at- > > http://permalink.gmane.org/gmane.comp.apache.kafka.user/1904 > > > > We are not sure about data loss in this scenario and working on this. > > > > > > > > Let us know if any concerns. > > > > [image: cid:image001.gif@01CF7B0A.03F21580] > > Navdeep Sharma > Developer - offshore, Middleware Applications & Development > o India: 0120-4532000 - 2234 > c: +91-9911698102 > > > > > > > > > > > > > > > > *From:* Gulia, Vikram > *Sent:* Tuesday, September 23, 2014 6:17 PM > *To:* #IT-MAD DES > *Subject:* FW: 9/23 prod issue - offline kafka partitions. > > > > DES Offshore dev, > > > > Please work with MAA offshore to monitor the kafka broker as we had this > incident where lot of partitions went offline around 1.45 PM MST and MAA > has to restart the kafka servers. We may have lost messages and we need to > see if there is a way to figure out what was the impact. > > > > Also, check the logs for kafka servers and see if we can figure out why > did partitions go offline or are un-available? Let us know if you find > anything relevant. > > > > Thank You, > > Vikram Gulia > > > > *From:* Nielsen, Andy > *Sent:* Tuesday, September 23, 2014 5:04 PM > *To:* #IT-MAD DES; Gulia, Vikram > *Cc:* #IT-MAA > *Subject:* 9/23 prod issue - offline kafka partitions. > > > > desadmin@pc1mwdpl01 ~/bin $ ./kafka.sh topic --unavailable-partitions > > topic: account-access partition: 21 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: account-access partition: 51 leader: -1 replicas: 4,6,1 > isr: > > topic: account-access partition: 81 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: account-access partition: 111 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: account-activated partition: 13 leader: -1 replicas: > 4,6,1 isr: > > topic: account-activated partition: 43 leader: -1 replicas: > 4,6,1 isr: > > topic: account-activated partition: 73 leader: -1 replicas: > 4,6,1 isr: > > topic: account-activated partition: 103 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: account-adjustment-issued partition: 27 leader: -1 > replicas: 4,6,1 isr: > > topic: account-adjustment-issued partition: 57 leader: -1 > replicas: 4,6,1 isr: > > topic: account-adjustment-issued partition: 87 leader: -1 > replicas: 4,6,1 isr: 1 > > topic: account-adjustment-issued partition: 117 leader: -1 > replicas: 4,6,1 isr: > > topic: account-created partition: 11 leader: -1 replicas: 4,6,1 > isr: > > topic: account-created partition: 41 leader: -1 replicas: 4,6,1 > isr: > > topic: account-created partition: 71 leader: -1 replicas: 4,6,1 > isr: > > topic: account-created partition: 101 leader: -1 replicas: 4,6,1 > isr: > > topic: account-info-updated partition: 7 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: account-info-updated partition: 37 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: account-info-updated partition: 67 leader: -1 replicas: > 4,6,1 isr: > > topic: account-info-updated partition: 97 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: account-info-updated partition: 127 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: application-access partition: 21 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: application-access partition: 51 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: application-access partition: 81 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: application-access partition: 111 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: bill-generated partition: 3 leader: -1 replicas: 4,6,1 > isr: > > topic: bill-generated partition: 33 leader: -1 replicas: 4,6,1 > isr: > > topic: bill-generated partition: 63 leader: -1 replicas: 4,6,1 > isr: > > topic: bill-generated partition: 93 leader: -1 replicas: 4,6,1 > isr: > > topic: bill-generated partition: 123 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: collected-event partition: 29 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: collected-event partition: 59 leader: -1 replicas: 4,6,1 > isr: > > topic: collected-event partition: 89 leader: -1 replicas: 4,6,1 > isr: > > topic: collected-event partition: 119 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: customer-cues partition: 27 leader: -1 replicas: 4,6,1 > isr: > > topic: customer-cues partition: 57 leader: -1 replicas: 4,6,1 > isr: > > topic: customer-cues partition: 87 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: customer-cues partition: 117 leader: -1 replicas: 4,6,1 > isr: > > topic: dish-promo-application-access partition: 23 leader: -1 > replicas: 4,6,1 isr: > > topic: dish-promo-application-access partition: 53 leader: -1 > replicas: 4,6,1 isr: > > topic: dish-promo-application-access partition: 83 leader: -1 > replicas: 4,6,1 isr: > > topic: dish-promo-application-access partition: 113 leader: -1 > replicas: 4,6,1 isr: > > topic: event-response partition: 2 leader: -1 replicas: 4,6,1 > isr: > > topic: event-response partition: 32 leader: -1 replicas: 4,6,1 > isr: > > topic: event-response partition: 62 leader: -1 replicas: 4,6,1 > isr: > > topic: event-response partition: 92 leader: -1 replicas: 4,6,1 > isr: > > topic: event-response partition: 122 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: leads-service partition: 24 leader: -1 replicas: 4,6,1 > isr: > > topic: leads-service partition: 54 leader: -1 replicas: 4,6,1 > isr: > > topic: leads-service partition: 84 leader: -1 replicas: 4,6,1 > isr: > > topic: leads-service partition: 114 leader: -1 replicas: 4,6,1 > isr: > > topic: logprod_v3 partition: 3 leader: -1 replicas: 4,6,1 > isr: > > topic: logprod_v3 partition: 33 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: logprod_v3 partition: 63 leader: -1 replicas: 4,6,1 > isr: > > topic: logprod_v3 partition: 93 leader: -1 replicas: 4,6,1 > isr: > > topic: logprod_v3 partition: 123 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: online-account-registration-attempted partition: 21 leader: > -1 replicas: 4,6,1 isr: > > topic: online-account-registration-attempted partition: 51 leader: > -1 replicas: 4,6,1 isr: 1 > > topic: online-account-registration-attempted partition: 81 leader: > -1 replicas: 4,6,1 isr: > > topic: online-account-registration-attempted partition: 111 leader: > -1 replicas: 4,6,1 isr: > > topic: order-cancelled partition: 29 leader: -1 replicas: 4,6,1 > isr: > > topic: order-cancelled partition: 59 leader: -1 replicas: 4,6,1 > isr: > > topic: order-cancelled partition: 89 leader: -1 replicas: 4,6,1 > isr: > > topic: order-cancelled partition: 119 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-completed partition: 24 leader: -1 replicas: 4,6,1 > isr: > > topic: order-completed partition: 54 leader: -1 replicas: 4,6,1 > isr: > > topic: order-completed partition: 84 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-completed partition: 114 leader: -1 replicas: 4,6,1 > isr: > > topic: order-created partition: 25 leader: -1 replicas: 4,6,1 > isr: > > topic: order-created partition: 55 leader: -1 replicas: 4,6,1 > isr: > > topic: order-created partition: 85 leader: -1 replicas: 4,6,1 > isr: > > topic: order-created partition: 115 leader: -1 replicas: 4,6,1 > isr: > > topic: order-modified partition: 8 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-modified partition: 38 leader: -1 replicas: 4,6,1 > isr: > > topic: order-modified partition: 68 leader: -1 replicas: 4,6,1 > isr: > > topic: order-modified partition: 98 leader: -1 replicas: 4,6,1 > isr: > > topic: order-modified partition: 128 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-request partition: 24 leader: -1 replicas: 4,6,1 > isr: > > topic: order-request partition: 54 leader: -1 replicas: 4,6,1 > isr: > > topic: order-request partition: 84 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-request partition: 114 leader: -1 replicas: 4,6,1 > isr: > > topic: order-response partition: 27 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: order-response partition: 57 leader: -1 replicas: 4,6,1 > isr: > > topic: order-response partition: 87 leader: -1 replicas: 4,6,1 > isr: > > topic: order-response partition: 117 leader: -1 replicas: 4,6,1 > isr: > > topic: outbound-call-attempted partition: 13 leader: -1 replicas: > 4,6,1 isr: > > topic: outbound-call-attempted partition: 43 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: outbound-call-attempted partition: 73 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: outbound-call-attempted partition: 103 leader: -1 replicas: > 4,6,1 isr: > > topic: outbound-communications partition: 4 leader: -1 replicas: > 4,6,1 isr: > > topic: outbound-communications partition: 34 leader: -1 replicas: > 4,6,1 isr: > > topic: outbound-communications partition: 64 leader: -1 replicas: > 4,6,1 isr: > > topic: outbound-communications partition: 94 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: outbound-communications partition: 124 leader: -1 replicas: > 4,6,1 isr: 1 > > topic: postal-mail-undeliverable partition: 15 leader: -1 > replicas: 4,6,1 isr: 1 > > topic: postal-mail-undeliverable partition: 45 leader: -1 > replicas: 4,6,1 isr: > > topic: postal-mail-undeliverable partition: 75 leader: -1 > replicas: 4,6,1 isr: > > topic: postal-mail-undeliverable partition: 105 leader: -1 > replicas: 4,6,1 isr: > > topic: rain-burn-in partition: 4 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-burn-in partition: 34 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-burn-in partition: 64 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-burn-in partition: 94 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-burn-in partition: 124 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-enhanced partition: 26 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-enhanced partition: 56 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-enhanced partition: 86 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-enhanced partition: 116 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-listener partition: 23 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-listener partition: 53 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-listener partition: 83 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-listener partition: 113 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-load-test partition: 8 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-load-test partition: 38 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-load-test partition: 68 leader: -1 replicas: 4,6,1 > isr: > > topic: rain-load-test partition: 98 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: rain-load-test partition: 128 leader: -1 replicas: 4,6,1 > isr: > > topic: submit-agreement partition: 2 leader: -1 replicas: 4,6,1 > isr: > > topic: submit-agreement partition: 32 leader: -1 replicas: 4,6,1 > isr: 1 > > topic: submit-agreement partition: 62 leader: -1 replicas: 4,6,1 > isr: > > topic: submit-agreement partition: 92 leader: -1 replicas: 4,6,1 > isr: > > topic: submit-agreement partition: 122 leader: -1 replicas: 4,6,1 > isr: > > topic: threshold-exceeded partition: 14 leader: -1 replicas: > 4,6,1 isr: > > topic: threshold-exceeded partition: 44 leader: -1 replicas: > 4,6,1 isr: > > topic: threshold-exceeded partition: 74 leader: -1 replicas: > 4,6,1 isr: > > topic: threshold-exceeded partition: 104 leader: -1 replicas: > 4,6,1 isr: 1 > > > > *Andy Nielsen* > > *Middleware Application Admin* > > *303-723-2347 <303-723-2347>* > > *cell:720-971-2856 <720-971-2856>* > > >
