Hi Harsha, The Log which I am processing is containing ~220 Million lines which are transformed as events to Kafka, which indeed is getting processed by Storm.
The numbers while running Storm topology which I could see says, Id Executors Tasks Emitted Transferred Complete latency (ms) Acked Failed Error Host Error Port Last error KafkaSpout <http://localhost:8080/component.html?id=KafkaSpout&topology_id=KafkaConsumerTopologyy-19-1423568605> 882179187402179187403.4942179377800 <http://:8000/log?file=worker-.log> Which indeed is little less(~10 Million) than the total number of events, hence I expect the Emitted Counts to be around ~220 Million which is currently getting around ~217 Million. Could you brief me through the nature of processing or why is the number less than the expected count! Thanks! On Sun, Feb 15, 2015 at 9:53 PM, Harsha <[email protected]> wrote: > Vineet, > How are you looking at number of events in kafka. Did you checked > storm worker logs for any errors and what you mean by "the acknowledgement > of 190 million events in storm" are you looking at number of acked messages? > -Harsha > > > On Sun, Feb 15, 2015, at 04:40 AM, Vineet Mishra wrote: > > Hi All, > > I am having a Kafka Storm Topology which is ingesting events published to > Kafka and processing on top of that data. > > Although apart from some latency I found that everything was going good. > But recently I came across a issue which I couldn't get any solution yet. > > I publishing some events from Logstash to Kafka and which is being > Subscribed by Storm Topology for further processing, I could see that the > source record count and the events processed by Storm is varying by a > reasonable number. So I have around 200 Million events to be processed out > of which 10 Million Events are getting lost as I could see the > acknowledgement of 190 Million events in the Storm. > > Stuck at this issue, looking for expert advise. > > Thanks! > > >
