Hi Stephane, seeing this only now, so it might be a little late. Have you resolved it?
If not: How many Kafka nodes are you using? I had a similar issue using only one broker, while the default config expects more brokers. The way you describe this issue it might be that your configuration expects 3 or more nodes (resulting in offset issues). If this matches your problem could you try setting "offsets.topic.replication.factor" to your actual number of brokers? (if < 3). Are your Kafka brokers running without errors? (before and after changing this config) Best, Stefan On Tue, 9 Apr 2019, 13:44 Simon Elliston Ball, <[email protected]> wrote: > One thing worth noting is that group.id is essentially a client > identifier, so if you specify one that matches another consumer (such as > Metron topologies) then they will interfere, and you are likely to balance > across your console and the actual Metron processes, so generally when > watching a Kafka topic for debugging you should let Kafka choose a random > group.id. You should not have to specify partition either if you want the > console to show all messages. > > Simon > > On 9 Apr 2019, at 11:17, <[email protected]> < > [email protected]> wrote: > > Hello, > > > > I haven’t sorted out yet this issue, but I think I’ve narrowed it. > Actually, after many tests with Kafka console-consumer and basic Python > scripts, I realize that I can only consume messages when I specify the > partition number and not the group.id. This is of course not what storm > tries to do, it tries do dynamically fetch the right partition and commit > the offset. > > > > My Kafka cluster is a fresh one installed with Hortonworks data platform > with 3 brokers. I can’t find any kind of option around this behavior. > Moreover, we regularly use Kafka for some other purpose with our docker > images and have never faced issues like this… > > > > Any idea? > > > > Thanks, > > > > Stéphane > > > > > > *From:* DAVY Stephane OBS/CSO > *Sent:* Monday, April 08, 2019 17:56 > *To:* [email protected] > *Subject:* RE: Metron concept > > > > Well, I realize that the console-consumer works with the—zookeeper option, > which is the “old consumer”, while it doesn’t work when I specify > –bootstrap-server, which is the “new consumer” way. So, it looks like a > Kafka issue… > > > > > > *From:* DAVY Stephane OBS/CSO > *Sent:* Monday, April 08, 2019 16:45 > *To:* '[email protected]' > *Subject:* RE: Metron concept > > > > Hello Simon, > > > > I send just one line at a time, and the line has been validated in the > Metron UI. I see no message in the topology logs. I switched to DEBUG mode, > and I can see the following sequence again and again: > > > > 2019-04-08 16:35:50.463 o.a.k.c.c.i.AbstractCoordinator > Thread-14-kafkaSpout-executor[4 4] [DEBUG] Sending coordinator request for > group forti1_parser to broker r-petya:6667 (id: 1011 rack: /default-rack) > > 2019-04-08 16:35:50.463 o.a.k.c.c.i.AbstractCoordinator > Thread-14-kafkaSpout-executor[4 4] [DEBUG] Received group coordinator > response ClientResponse(receivedTimeMs=1554734150463, disconnected=false, > request=ClientRequest(expectResponse=true, > callback=org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler@35437dce, > request=RequestSend(header={api_key=10,api_version=0,correlation_id=61518,client_id=consumer-1}, > body={group_id=forti1_parser}), createdTimeMs=1554734150463, > sendTimeMs=1554734150463), > responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}}) > > 2019-04-08 16:35:50.562 o.a.k.c.NetworkClient > Thread-14-kafkaSpout-executor[4 4] [DEBUG] Sending metadata request > {topics=[forti1]} to node 1011 > > 2019-04-08 16:35:50.562 o.a.k.c.Metadata Thread-14-kafkaSpout-executor[4 > 4] [DEBUG] Updated cluster metadata version 30761 to Cluster(nodes = > [r-petya:6667 (id: 1011 rack: /default-rack), r-jigsaw:6667 (id: 1012 rack: > /default-rack), r-wannacry.rd.francetelecom.fr:6667 (id: 1010 rack: > /default-rack)], partitions = [Partition(topic = forti1, partition = 0, > leader = 1012, replicas = [1012,], isr = [1012,]]) > > > > > > Is it normal to have > “responseBody={error_code=15,coordinator={node_id=-1,host=,port=-1}})” in > the response? > > > > Thanks, > > > > Stéphane > > > > > > *From:* Simon Elliston Ball [mailto:[email protected] > <[email protected]>] > *Sent:* Monday, April 08, 2019 16:29 > *To:* [email protected] > *Subject:* Re: Metron concept > > > > Are you seeing events on the enrichments topic, and if so, are they > getting to indexing? Any messages in the storm logs for these topologies? > > > > Are you also certain the parser is correct, and there are no invalid or > error messages being sent to the error index? > > > > Simon > > > > On Mon, 8 Apr 2019 at 15:26, <[email protected]> wrote: > > Hello Nick, > > > > Thanks for your answer. I went through this post and see that all my > events should go in Elastic, which is what I want, but which it isn’t what > I get L > > > > I have setup the following basic setup: > > - New telemetry with grok parser (validated in UI with sample) > and a kafka topic => the topic didn’t exist before, and it is created > automatically as I can see with the kafka-topics.sh CLI utility > > - A simple Nifi flow to push data in this topic => I can see > some data in the topic with the kafka-console-consumer.sh CLI utility. > > > > But I have the feeling that my topology never consume Kafka messages. The > Storm UI shows “0” figure nearly everywhere in my topology, and the Elastic > index is not created (_cat/indices). I see also nothing in the “indexing” > Kafka topic. > > > > But I see no error message, I don’t really know how to go on… > > > > Does anybody have a suggestion for me? I guess I’m not the first one with > kind of issue but I cannot find any case close to mine. > > > > > > > > *From:* Nick Allen [mailto:[email protected]] > *Sent:* Monday, April 08, 2019 15:17 > *To:* [email protected] > *Subject:* Re: Metron concept > > > > All events are indexed by default. > > > > See if this guide helps you any. > https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source > > > > On Mon, Apr 8, 2019 at 2:49 AM <[email protected]> wrote: > > Hello all, > > > > There is one my point that isn’t clear for me. When sending data into > Metron, are all the events all indexed sent to Elastic and / or HDFS, or > only the events that trigger a triage rule? > > > > For now I’m trying to send some FW logs in Metron, I feed a Kafka topic > with Nifi, I can see that the topic has data thanks to Kafka CLI, but > nothing more happens after I’ve configured a new source from UI management… > > > > Stéphane > > _________________________________________________________________________________________________________________________ > > > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > > they should not be distributed, used or copied without authorisation. > > If you have received this email in error, please notify the sender and delete > this message and its attachments. > > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > > Thank you. > > _________________________________________________________________________________________________________________________ > > > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > > they should not be distributed, used or copied without authorisation. > > If you have received this email in error, please notify the sender and delete > this message and its attachments. > > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > > Thank you. > > -- > > -- > > simon elliston ball > > @sireb > > _________________________________________________________________________________________________________________________ > > Ce message et ses pieces jointes peuvent contenir des informations > confidentielles ou privilegiees et ne doivent donc > pas etre diffuses, exploites ou copies sans autorisation. Si vous avez recu > ce message par erreur, veuillez le signaler > a l'expediteur et le detruire ainsi que les pieces jointes. Les messages > electroniques etant susceptibles d'alteration, > Orange decline toute responsabilite si ce message a ete altere, deforme ou > falsifie. Merci. > > This message and its attachments may contain confidential or privileged > information that may be protected by law; > they should not be distributed, used or copied without authorisation. > If you have received this email in error, please notify the sender and delete > this message and its attachments. > As emails may be altered, Orange is not liable for messages that have been > modified, changed or falsified. > Thank you. > >
