Hi Gaurav, If you click on the red squares in the upper right corners of your processors, what error messages do you see?
On 2018-01-14 19:29, Gaurav Bapat wrote: > Hey Jon, > > I have Storm UI and the logs are coming from firewalls, servers, etc from > other machines(HP ArcSight Logger). > > I have attached the NiFi screenshots, my logs are coming but there is some > error with Kafka and I am having issues with configuring Kafka broker > > On 12 January 2018 at 18:14, [email protected] <[email protected]> wrote: > > In Ambari under storm you can find the UI under quick links at the top. That > said, the issue seems to be upstream of Metron, in NiFi. That is something I > can't help with as much, but if you can share the listensyslog processor > config that would be a start. Also, share the config of the thing that is > sending syslog as well (are these local syslog, is that machine aggregating > syslog from other machines, etc.). Thanks, > > Jon > > On Fri, Jan 12, 2018, 01:00 Gaurav Bapat <[email protected]> wrote: > > I have created a Kafka topic "cef" but my Listen Syslogs is not getting logs > in the processor. > > Also I checked using tcpdump -i and it is getting logs in my machine but > ListenSyslogs is not getting the logs > > On 12 January 2018 at 11:13, Gaurav Bapat <[email protected]> wrote: > > [root@metron incubator-metron]# ./metron-deployment/scripts/platform-info.sh > Metron 0.4.3 > -- > * master > -- > commit c559ed7e1838ec71344eae3d9e37771db2641635 > Author: cstella <[email protected]> > Date: Tue Jan 9 15:28:47 2018 -0500 > > METRON-1379: Add an OBJECT_GET stellar function closes > apache/incubator-metron#880 > -- > metron-deployment/vagrant/full-dev-platform/Vagrantfile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > -- > ansible 2.0.0.2 > config file = > configured module search path = Default w/o overrides > -- > Vagrant 1.9.6 > -- > Python 2.7.5 > -- > Apache Maven 3.3.9 (bb52d8502b132ec0a5a3f4c09453c07478323dc5; > 2015-11-10T22:11:47+05:30) > Maven home: /opt/maven/current > Java version: 1.8.0_151, vendor: Oracle Corporation > Java home: /opt/jdk1.8.0_151/jre > Default locale: en_US, platform encoding: UTF-8 > OS name: "linux", version: "3.10.0-693.11.6.el7.x86_64", arch: "amd64", > family: "unix" > -- > Docker version 1.12.6, build ec8512b/1.12.6 > -- > node > v8.9.3 > -- > npm > 5.5.1 > -- > g++ (GCC) 4.8.5 20150623 (Red Hat 4.8.5-16) > Copyright (C) 2015 Free Software Foundation, Inc. > This is free software; see the source for copying conditions. There is NO > warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. > > -- > Compiler is C++11 compliant > -- > Linux metron.com [1] 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC > 2018 x86_64 x86_64 x86_64 GNU/Linux > -- > Total System Memory = 15773.3 MB > Processor Model: Intel(R) Core(TM) i5-3450 CPU @ 3.10GHz > Processor Speed: 3320.875 MHz > Processor Speed: 3307.191 MHz > Processor Speed: 3376.699 MHz > Processor Speed: 3338.917 MHz > Total Physical Processors: 4 > Total cores: 16 > Disk information: > /dev/mapper/centos-root 200G 22G 179G 11% / > /dev/sda1 2.0G 224M 1.8G 11% /boot > /dev/sda2 1022M 12K 1022M 1% /boot/efi > /dev/mapper/centos-home 247G 10G 237G 5% /home > This CPU appears to support virtualization > > On 12 January 2018 at 09:25, Gaurav Bapat <[email protected]> wrote: > > Hey Jon, > > Appreciate your timely reply. > > I gone through your answer but still I can't figure out how do I do > parsing/indexing in Storm UI as I cant find any option for the same. > > Is there any other UI to do parsing/indexing? > > On 11 January 2018 at 21:22, [email protected] <[email protected]> wrote: > > So, you created a new cef topic, and set up the appropriate parser config for > it (if not, this [2] may be helpful)? If so: > Here are some basic troubleshooting steps: > 1. Validate that the logs are getting onto the kafka topic that you are > sending to. If they aren't there, the problem is upstream from Metron. > 2. If they are getting onto the kafka topic they are being directly sent to, > check the indexing kafka topic for an enriched version of those same logs. > 3. Do a binary search of the various components involved with ingest. > a. If the logs are NOT on the indexing kafka topic, check the enrichments > topic for those logs. > b. If the logs are NOT on the enrichments topic, check the parser storm > topology. > c. If the logs are on the enrichments topic, but NOT indexing, check the > enrichments storm topology. > d. If the logs are on the indexing but NOT Kibana, check the indexing storm > topic. > e. If the logs are in on the indexing topic and indexing storm topic is in > good shape, check elasticsearch directly. > 4. You should have identified where the issue is at this point. Report back > here with what you observed, any relevant error messages, etc. > > Side note: We should document a decision tree for troubleshooting data > ingest. It is fairly straightforward and makes me wonder if we already have > this somewhere and I'm not aware of it? It would also be a good place to put > pointers to some common errors. > > Jon > > On Thu, Jan 11, 2018 at 1:44 AM Gaurav Bapat <[email protected]> wrote: > > Hello everyone, I have deployed Metron on a single node machine and I would > like to know how do I get Syslogs from NiFi into Kibana dashboard? > > I have created a Kafka topic by the name "cef" and I can see that the topic > exists in Metron Configuration but I am unable to connect it with Kibana > > Need Help!! > > -- > > Jon -- Jon Links: ------ [1] http://metron.com [2] https://cwiki.apache.org/confluence/display/METRON/Adding+a+New+Telemetry+Data+Source
