We now have 2 topologies for indexing - random access and batch. Double
check that both are currently running - our full dev environment is pretty
full with resources currently.
random_access_indexing
batch_indexing
random_access_indexing is responsible for getting data into Elasticsearch.
You can also check ES has indexes by going into Ambari -> Elasticsearch ->
Quick Links -> Elasticsearch Indexes. You should see something like the
following:
health status index uuid pri rep
docs.count docs.deleted store.size pri.store.size
yellow open .kibana qbpdYf_RTMa_Rd2dB9q7oA 1 1
44 0 120kb 120kb
yellow open bro_index_2018.02.06.22 -FiQxEGEQtSec0sC4oGAFA 5 1
7990 0 12.8mb 12.8mb
yellow open bro_index_2018.02.06.23 AS4DHjrBQNyFrzDOxpGFeQ 5 1
8100 0 12.7mb 12.7mb
yellow open snort_index_2018.02.06.20 Sxg-JGI3SAeXdg-V11BNkg 5 1
7530 0 11.9mb 11.9mb
yellow open bro_index_2018.02.06.18 U1RTmFnpTCCDAicwWxc7Mg 5 1
4640 0 8mb 8mb
...
On Thu, Feb 8, 2018 at 3:19 AM, R K Sharma <[email protected]> wrote:
> Thanks Ryan...I see some data fro Snort & Bro sensors. Another problem
> which I have is that there is no information from Kibana dashboard.Do I
> need to deploy some additional component to populate kibana ?
>
> Regards
> RK Sharma
>
> On Wed, Feb 7, 2018 at 3:38 PM, Ryan Merriman <[email protected]> wrote:
>
>> I think you need to go one level deeper, those are directories. Here is
>> what I see in my dev environment:
>>
>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed
>> Found 2 items
>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20
>> /apps/metron/indexing/indexed/bro
>> drwxrwxr-x - storm hadoop 0 2018-02-07 01:20
>> /apps/metron/indexing/indexed/snort
>>
>> [root@node1 ~]# hdfs dfs -ls /apps/metron/indexing/indexed/bro
>> Found 1 items
>> -rw-r--r-- 1 storm hadoop 12842043 2018-02-07 01:20
>> /apps/metron/indexing/indexed/bro/enrichment-hdfsIndexingBol
>> t-3-0-1517966421778.json
>>
>> On Wed, Feb 7, 2018 at 3:58 AM, R K Sharma <[email protected]> wrote:
>>
>>> Hi,
>>> I have deployed Full Development VM on Virtual Box and all
>>> services including metron, kafka, storm etc. are started. However, when I
>>> check if there is some data written into HDFS (
>>> /apps/metron/indexing/indexed/yaf|bro|snort ) for any data sources, I
>>> don't see any data. Hereby below is output.
>>>
>>>
>>> [vagrant@node1 bin]$ hdfs dfs -ls /apps/metron/indexing/indexed/
>>> Found 3 items
>>> drwxrwxr-x - storm hadoop 0 2018-02-06 13:03
>>> /apps/metron/indexing/in
>>> dexed/bro
>>> drwxrwxr-x - storm hadoop 0 2018-01-31 13:35
>>> /apps/metron/indexing/in
>>> dexed/error
>>> drwxrwxr-x - storm hadoop 0 2018-02-07 04:53
>>> /apps/metron/indexing/in
>>> dexed/snort
>>>
>>> On other-hand, I can see sensors (Snort & Bro) started on
>>> http://node1:4200 and is having some throughput, although very low.
>>> Hereby below is sensor status.
>>>
>>>
>>> GrokWebSphere Stopped - -
>>> jsonMap JSONMap Stopped - -
>>> squid Grok Stopped - -
>>> snort Snort Running 3.862s 1.89kb/s
>>> asa Asa Stopped - -
>>> bro Bro Running 4.25s 1.94kb/s
>>> yaf Grok Running 0s 0kb/s
>>> Can anybody guide me what should I check to ensure sensors produce data
>>> and HDFS should be populated with this data ?
>>>
>>> Thanks & Regards
>>> RK Sharma
>>>
>>>
>>
>
