We now have 2 topologies for indexing - random access and batch. Double check that both are currently running - our full dev environment is pretty full with resources currently. random_access_indexing batch_indexing
random_access_indexing is responsible for getting data into Elasticsearch. You can also check ES has indexes by going into Ambari -> Elasticsearch -> Quick Links -> Elasticsearch Indexes. You should see something like the following: health status index uuid pri rep docs.count docs.deleted store.size pri.store.size yellow open .kibana qbpdYf_RTMa_Rd2dB9q7oA 1 1 44 0 120kb 120kb yellow open bro_index_2018.02.06.22 -FiQxEGEQtSec0sC4oGAFA 5 1 7990 0 12.8mb 12.8mb yellow open bro_index_2018.02.06.23 AS4DHjrBQNyFrzDOxpGFeQ 5 1 8100 0 12.7mb 12.7mb yellow open snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg 5 1 7530 0 11.9mb 11.9mb yellow open bro_index_2018.02.06.18 U1RTmFnpTCCDAicwWxc7Mg 5 1 4640 0 8mb 8mb ... On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma <rksu...@gmail.com> wrote: > Thanks Ryan...I see some data fro Snort & Bro sensors. Another problem > which I have is that there is no information from Kibana dashboard.Do I > need to deploy some additional component to populate kibana ? > > Regards > RK Sharma > > On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merriman <merrim...@gmail.com> wrote: > >> I think you need to go one level deeper, those are directories. Here is >> what I see in my dev environment: >> >> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed >> Found 2 items >> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >> /apps/metron/indexing/indexed/bro >> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >> /apps/metron/indexing/indexed/snort >> >> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed/bro >> Found 1 items >> -rw-r--r-- 1 storm hadoop 12842043 2018-02-07 01:20 >> /apps/metron/indexing/indexed/bro/enrichment-hdfsIndexingBol >> t-3-0-1517966421778.json >> >> On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma <rksu...@gmail.com> wrote: >> >>> Hi, >>> I have deployed Full Development VM on Virtual Box and all >>> services including metron, kafka, storm etc. are started. However, when I >>> check if there is some data written into HDFS ( >>> /apps/metron/indexing/indexed/yaf|bro|snort ) for any data sources, I >>> don't see any data. Hereby below is output. >>> >>> >>> [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/ >>> Found 3 items >>> drwxrwxr-x - storm hadoop 0 2018-02-06 13:03 >>> /apps/metron/indexing/in >>> dexed/bro >>> drwxrwxr-x - storm hadoop 0 2018-01-31 13:35 >>> /apps/metron/indexing/in >>> dexed/error >>> drwxrwxr-x - storm hadoop 0 2018-02-07 04:53 >>> /apps/metron/indexing/in >>> dexed/snort >>> >>> On other-hand, I can see sensors (Snort & Bro) started on >>> http://node1:4200 and is having some throughput, although very low. >>> Hereby below is sensor status. >>> >>> >>> GrokWebSphere Stopped - - >>> jsonMap JSONMap Stopped - - >>> squid Grok Stopped - - >>> snort Snort Running 3.862s 1.89kb/s >>> asa Asa Stopped - - >>> bro Bro Running 4.25s 1.94kb/s >>> yaf Grok Running 0s 0kb/s >>> Can anybody guide me what should I check to ensure sensors produce data >>> and HDFS should be populated with this data ? >>> >>> Thanks & Regards >>> RK Sharma >>> >>> >> >