Related: The documentation <https://github.com/apache/metron/tree/master/metron-platform/metron-indexing#indexing-architecture> should probably be updated to account for METRON-1302.
On Fri, Feb 9, 2018 at 8:18 AM R K Sharma <rksu...@gmail.com> wrote: > Thanks Michael... I can see similar information for my setup as well. What > I could notice is that MySQL service is not running on my machine Which I > think could be an issue for kibana. What's your view ? > > Regards > RK Sharma > > On Thu, Feb 8, 2018 at 9:11 PM, Michael Miklavcic < > michael.miklav...@gmail.com> wrote: > >> We now have 2 topologies for indexing - random access and batch. Double >> check that both are currently running - our full dev environment is pretty >> full with resources currently. >> random_access_indexing >> batch_indexing >> >> random_access_indexing is responsible for getting data into >> Elasticsearch. You can also check ES has indexes by going into Ambari -> >> Elasticsearch -> Quick Links -> Elasticsearch Indexes. You should see >> something like the following: >> >> health status index uuid pri rep >> docs.count docs.deleted store.size pri.store.size >> yellow open .kibana qbpdYf_RTMa_Rd2dB9q7oA 1 1 >> 44 0 120kb 120kb >> yellow open bro_index_2018.02.06.22 -FiQxEGEQtSec0sC4oGAFA 5 1 >> 7990 0 12.8mb 12.8mb >> yellow open bro_index_2018.02.06.23 AS4DHjrBQNyFrzDOxpGFeQ 5 1 >> 8100 0 12.7mb 12.7mb >> yellow open snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg 5 1 >> 7530 0 11.9mb 11.9mb >> yellow open bro_index_2018.02.06.18 U1RTmFnpTCCDAicwWxc7Mg 5 1 >> 4640 0 8mb 8mb >> >> ... >> >> >> >> >> On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma <rksu...@gmail.com> wrote: >> >>> Thanks Ryan...I see some data fro Snort & Bro sensors. Another problem >>> which I have is that there is no information from Kibana dashboard.Do I >>> need to deploy some additional component to populate kibana ? >>> >>> Regards >>> RK Sharma >>> >>> On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merriman <merrim...@gmail.com> >>> wrote: >>> >>>> I think you need to go one level deeper, those are directories. Here >>>> is what I see in my dev environment: >>>> >>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed >>>> Found 2 items >>>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/bro >>>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/snort >>>> >>>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed/bro >>>> Found 1 items >>>> -rw-r--r-- 1 storm hadoop 12842043 2018-02-07 01:20 >>>> /apps/metron/indexing/indexed/bro/enrichment-hdfsIndexingBolt-3-0-1517966421778.json >>>> >>>> On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma <rksu...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> I have deployed Full Development VM on Virtual Box and all >>>>> services including metron, kafka, storm etc. are started. However, when I >>>>> check if there is some data written into HDFS ( >>>>> /apps/metron/indexing/indexed/yaf|bro|snort ) for any data sources, I >>>>> don't see any data. Hereby below is output. >>>>> >>>>> >>>>> [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/ >>>>> Found 3 items >>>>> drwxrwxr-x - storm hadoop 0 2018-02-06 13:03 >>>>> /apps/metron/indexing/in >>>>> dexed/bro >>>>> drwxrwxr-x - storm hadoop 0 2018-01-31 13:35 >>>>> /apps/metron/indexing/in >>>>> >>>>> dexed/error >>>>> drwxrwxr-x - storm hadoop 0 2018-02-07 04:53 >>>>> /apps/metron/indexing/in >>>>> >>>>> dexed/snort >>>>> >>>>> On other-hand, I can see sensors (Snort & Bro) started on >>>>> http://node1:4200 and is having some throughput, although very low. >>>>> Hereby below is sensor status. >>>>> >>>>> >>>>> GrokWebSphere Stopped - - >>>>> jsonMap JSONMap Stopped - - >>>>> squid Grok Stopped - - >>>>> snort Snort Running 3.862s 1.89kb/s >>>>> asa Asa Stopped - - >>>>> bro Bro Running 4.25s 1.94kb/s >>>>> yaf Grok Running 0s 0kb/s >>>>> Can anybody guide me what should I check to ensure sensors produce >>>>> data and HDFS should be populated with this data ? >>>>> >>>>> Thanks & Regards >>>>> RK Sharma >>>>> >>>>> >>>> >>> >> > -- Jon
