Can you check /kafaka-logs on your VM box (It should have a folder named your topic). Can you check if it is there ?
On Mon, Jan 15, 2018 at 3:49 PM, Gaurav Bapat <[email protected]> wrote: > I am not getting data into my Kafka topic > > I have used i5 4 Core Processor with 16 GB RAM and I have allocated 12 GB > RAM to my vagrant VM. > > I dont understand how to configure Kafka broker because it is giving me > failed while waiting for acks to Kafka > > > > On 15 January 2018 at 16:10, Farrukh Naveed Anjum <[email protected] > > wrote: > >> Can you tell me is your KAFKA Topic getting data ? What are you machine >> specifications ? >> >> >> On Mon, Jan 15, 2018 at 2:56 PM, Gaurav Bapat <[email protected]> >> wrote: >> >>> Thanks Farrukh, >>> >>> I am not getting data in my kafka topic even after creating one, the >>> issue seems to be with broker config, how to configure Kafka and Zookeeper >>> port? >>> >>> On 15 January 2018 at 13:23, Farrukh Naveed Anjum < >>> [email protected]> wrote: >>> >>>> Hi, >>>> >>>> I had similar issue it turned out to be the issue in STROM >>>> >>>> No worker is assigned to togolgoy all you need is to add additional >>>> port in >>>> >>>> Ambari -> Storm -> Configs -> supervisor.slot.ports by assigning an >>>> additional port to the list >>>> >>>> >>>> https://community.hortonworks.com/questions/32499/no-workers >>>> -in-storm-for-squid-topology.html >>>> >>>> >>>> I had similar issue and finally got it fixed >>>> >>>> On Mon, Jan 15, 2018 at 8:45 AM, Gaurav Bapat <[email protected]> >>>> wrote: >>>> >>>>> Storm UI >>>>> >>>>> On 15 January 2018 at 08:59, Gaurav Bapat <[email protected]> >>>>> wrote: >>>>> >>>>>> Hey Jon, >>>>>> >>>>>> I have Storm UI and the logs are coming from firewalls, servers, etc >>>>>> from other machines(HP ArcSight Logger). >>>>>> >>>>>> I have attached the NiFi screenshots, my logs are coming but there is >>>>>> some error with Kafka and I am having issues with configuring Kafka >>>>>> broker >>>>>> >>>>>> >>>>>> >>>>>> On 12 January 2018 at 18:14, [email protected] <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> In Ambari under storm you can find the UI under quick links at the >>>>>>> top. That said, the issue seems to be upstream of Metron, in NiFi. >>>>>>> That >>>>>>> is something I can't help with as much, but if you can share the >>>>>>> listensyslog processor config that would be a start. Also, share the >>>>>>> config of the thing that is sending syslog as well (are these local >>>>>>> syslog, >>>>>>> is that machine aggregating syslog from other machines, etc.). Thanks, >>>>>>> >>>>>>> Jon >>>>>>> >>>>>>> On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> I have created a Kafka topic "cef" but my Listen Syslogs is not >>>>>>>> getting logs in the processor. >>>>>>>> >>>>>>>> Also I checked using tcpdump -i and it is getting logs in my >>>>>>>> machine but ListenSyslogs is not getting the logs >>>>>>>> >>>>>>>> On 12 January 2018 at 11:13, Gaurav Bapat <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> [root@metron incubator-metron]# ./metron-deployment/scripts/pl >>>>>>>>> atform-info.sh >>>>>>>>> Metron 0.4.3 >>>>>>>>> -- >>>>>>>>> * master >>>>>>>>> -- >>>>>>>>> commit c559ed7e1838ec71344eae3d9e37771db2641635 >>>>>>>>> Author: cstella <[email protected]> >>>>>>>>> Date: Tue Jan 9 15:28:47 2018 -0500 >>>>>>>>> >>>>>>>>> METRON-1379: Add an OBJECT_GET stellar function closes >>>>>>>>> apache/incubator-metron#880 >>>>>>>>> -- >>>>>>>>> metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- >>>>>>>>> 1 file changed, 1 insertion(+), 1 deletion(-) >>>>>>>>> -- >>>>>>>>> ansible 2.0.0.2 >>>>>>>>> config file = >>>>>>>>> configured module search path = Default w/o overrides >>>>>>>>> -- >>>>>>>>> Vagrant 1.9.6 >>>>>>>>> -- >>>>>>>>> Python 2.7.5 >>>>>>>>> -- >>>>>>>>> Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; >>>>>>>>> 2015-11-10T22:11:47+05:30) >>>>>>>>> Maven home: /opt/maven/current >>>>>>>>> Java version: 1.8.0_151, vendor: Oracle Corporation >>>>>>>>> Java home: /opt/jdk1.8.0_151/jre >>>>>>>>> Default locale: en_US, platform encoding: UTF-8 >>>>>>>>> OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: >>>>>>>>> "amd64", family: "unix" >>>>>>>>> -- >>>>>>>>> Docker version 1.12.6, build ec8512b/1.12.6 >>>>>>>>> -- >>>>>>>>> node >>>>>>>>> v8.9.3 >>>>>>>>> -- >>>>>>>>> npm >>>>>>>>> 5.5.1 >>>>>>>>> -- >>>>>>>>> g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) >>>>>>>>> Copyright (C) 2015 Free Software Foundation, Inc. >>>>>>>>> This is free software; see the source for copying conditions. >>>>>>>>> There is NO >>>>>>>>> warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR >>>>>>>>> PURPOSE. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Compiler is C++11 compliant >>>>>>>>> -- >>>>>>>>> Linux metron.com 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 >>>>>>>>> 01:06:37 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux >>>>>>>>> -- >>>>>>>>> Total System Memory = 15773.3 MB >>>>>>>>> Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz >>>>>>>>> Processor Speed: 3320.875 MHz >>>>>>>>> Processor Speed: 3307.191 MHz >>>>>>>>> Processor Speed: 3376.699 MHz >>>>>>>>> Processor Speed: 3338.917 MHz >>>>>>>>> Total Physical Processors: 4 >>>>>>>>> Total cores: 16 >>>>>>>>> Disk information: >>>>>>>>> /dev/mapper/centos-root 200G 22G 179G 11% / >>>>>>>>> /dev/sda1 2.0G 224M 1.8G 11% /boot >>>>>>>>> /dev/sda2 1022M 12K 1022M 1% /boot/efi >>>>>>>>> /dev/mapper/centos-home 247G 10G 237G 5% /home >>>>>>>>> This CPU appears to support virtualization >>>>>>>>> >>>>>>>>> On 12 January 2018 at 09:25, Gaurav Bapat <[email protected]> >>>>>>>>> wrote: >>>>>>>>> >>>>>>>>>> Hey Jon, >>>>>>>>>> >>>>>>>>>> Appreciate your timely reply. >>>>>>>>>> >>>>>>>>>> I gone through your answer but still I can't figure out how do I >>>>>>>>>> do parsing/indexing in Storm UI as I cant find any option for the >>>>>>>>>> same. >>>>>>>>>> >>>>>>>>>> Is there any other UI to do parsing/indexing? >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On 11 January 2018 at 21:22, [email protected] <[email protected]> >>>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>>> So, you created a new cef topic, and set up the appropriate >>>>>>>>>>> parser config for it (if not, this >>>>>>>>>>> <https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source> >>>>>>>>>>> may be helpful)? If so: >>>>>>>>>>> >>>>>>>>>>> Here are some basic troubleshooting steps: >>>>>>>>>>> 1. Validate that the logs are getting onto the kafka topic that >>>>>>>>>>> you are sending to. If they aren't there, the problem is upstream >>>>>>>>>>> from >>>>>>>>>>> Metron. >>>>>>>>>>> 2. If they are getting onto the kafka topic they are being >>>>>>>>>>> directly sent to, check the indexing kafka topic for an enriched >>>>>>>>>>> version of >>>>>>>>>>> those same logs. >>>>>>>>>>> 3. Do a binary search of the various components involved with >>>>>>>>>>> ingest. >>>>>>>>>>> a. If the logs are *not* on the indexing kafka topic, check >>>>>>>>>>> the enrichments topic for those logs. >>>>>>>>>>> b. If the logs are *not* on the enrichments topic, check >>>>>>>>>>> the parser storm topology. >>>>>>>>>>> c. If the logs are on the enrichments topic, but *not* >>>>>>>>>>> indexing, check the enrichments storm topology. >>>>>>>>>>> d. If the logs are on the indexing but *not* Kibana, check >>>>>>>>>>> the indexing storm topic. >>>>>>>>>>> e. If the logs are in on the indexing topic and indexing >>>>>>>>>>> storm topic is in good shape, check elasticsearch directly. >>>>>>>>>>> 4. You should have identified where the issue is at this >>>>>>>>>>> point. Report back here with what you observed, any relevant error >>>>>>>>>>> messages, etc. >>>>>>>>>>> >>>>>>>>>>> Side note: We should document a decision tree for >>>>>>>>>>> troubleshooting data ingest. It is fairly straightforward and >>>>>>>>>>> makes me >>>>>>>>>>> wonder if we already have this somewhere and I'm not aware of it? >>>>>>>>>>> It would >>>>>>>>>>> also be a good place to put pointers to some common errors. >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>> >>>>>>>>>>> On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat < >>>>>>>>>>> [email protected]> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hello everyone, I have deployed Metron on a single node machine >>>>>>>>>>>> and I would like to know how do I get Syslogs from NiFi into Kibana >>>>>>>>>>>> dashboard? >>>>>>>>>>>> >>>>>>>>>>>> I have created a Kafka topic by the name "cef" and I can see >>>>>>>>>>>> that the topic exists in >>>>>>>>>>>> Metron Configuration but I am unable to connect it with Kibana >>>>>>>>>>>> >>>>>>>>>>>> Need Help!! >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> >>>>>>>>>>> Jon >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> -- >>>>>>> >>>>>>> Jon >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>>> >>>> -- >>>> With Regards >>>> Farrukh Naveed Anjum >>>> >>> >>> >> >> >> -- >> With Regards >> Farrukh Naveed Anjum >> > > -- With Regards Farrukh Naveed Anjum
