I see the below error in the logs. The url that is tried
http://<StormInstance>:8983/solr/demo_unix_shard1_replica1<http://%3cStormInstance%3e:8983/solr/demo_unix_shard1_replica1>
seems doesn’t work. When I tried from browser
http://<StormInstance>:8983/solr/#/demo_unix_shard1_replica1<http://%3cStormInstance%3e:8983/solr/#/demo_unix_shard1_replica1>
worked.
I see the collections list it tries to get from solr is fine. Issue seems to be
the url tried to access the shard should be changed.
***************Logs***********
18/08/23 09:55:17 ERROR impl.CloudSolrClient: Request to collection
demo_asa,demo_windowsnxlog,demo_bluecoat,demo_unix,demo_sysmon,bro,yaf,demo_msexchange,demo_wgdhcp,demo_wgtraffic,snort,metaalert
failed due to (503)
org.apache.solr.client.solrj.impl.HttpSolrClient$RemoteSolrException: Error
from server at http://<StormInstance>:8983/solr/demo_unix_shard1_replica1: no
servers hosting shard: snort_shard1, retry? 0
18/08/23 09:55:17 ERROR dao.SolrSearchDao: No live SolrServers available to
handle this request:[http://<StormInstance>:8983/solr/demo_unix_shard1_replica1]
org.apache.solr.client.solrj.SolrServerException: No live SolrServers available
to handle this
request:[http://<StormInstance>:8983/solr/demo_unix_shard1_replica1]
at
org.apache.solr.client.solrj.impl.LBHttpSolrClient.request(LBHttpSolrClient.java:426)
*****************************
From: Ryan Merriman [mailto:[email protected]]
Sent: Thursday, August 23, 2018 7:12 PM
To: [email protected]
Subject: Re: Unable to see alerts in metron alert UI with solr
You can pass credentials in curl with the -u flag: curl -u user:password -X
GET --header 'Accept: application/json' 'http://{rest
host}:8082/api/v1/sensor/indexing/config/list/indices/solr'. I prefer to use
Swagger.
We need to dig into the 500 response. An error message should be included in
the response or you can try running it separately in Swagger. I would also
check the log at /var/log/metron/metron-rest.log.
On Thu, Aug 23, 2018 at 6:35 AM Anil Donthireddy
<[email protected]<mailto:[email protected]>> wrote:
@Ryan: I tried the api call from swagger UI. I can see the list of indices is
being returned fine as present in Solr.
From: Anil Donthireddy
[mailto:[email protected]<mailto:[email protected]>]
Sent: Thursday, August 23, 2018 5:03 PM
To: [email protected]<mailto:[email protected]>
Subject: RE: Unable to see alerts in metron alert UI with solr
Hi,
Thanks a lot for the inputs.
@Stefan: We don’t have ranger setup for access permissions.
@Ryan: Below are the issues I observed from your suggestion
For the search request in network tab, I see the response is
“500 Internal Server Error”
In the request payload I see all the params are emtpy as below.
{indices: [], facetFields: [], query: "*",
from: 0, size: 25}
When I try to run the curl command, I am getting unauthorized error as
below. How can I pass credentials for authentication along with the request or
how can I authenticate via curl command and store the jessionid as cookie and
pass through your curl command?
{"timestamp":"2018-08-23
11:13:22","status":401,"error":"Unauthorized","message":"Unauthorized","path":"/api/v1/sensor/indexing/config/list/indices/solr"}
Thanks,
Anil.
From: Ryan Merriman [mailto:[email protected]]
Sent: Thursday, August 23, 2018 3:41 AM
To: [email protected]<mailto:[email protected]>
Subject: Re: Unable to see alerts in metron alert UI with solr
The first thing I would do is figure out which indices are included in the
Alerts UI. If you open the network tab in your browser there should be a
"search" request sent to REST. There is an "indices" property that will have a
list of indices include in the search. If it's empty ("[]") then this call
should tell you what the default indices are:
curl -X GET --header 'Accept: application/json' 'http://{rest
host}:8082/api/v1/sensor/indexing/config/list/indices/solr'
Verify your collection is in the list and we can go from there.
On Wed, Aug 22, 2018 at 4:47 PM Stefan Kupstaitis-Dunkler
<[email protected]<mailto:[email protected]>> wrote:
Hey Anil,
do you use any kind of policies to prevent access by the metron user? Do you
use Ranger to manage access policies?
The user running the Metron REST service (normally "metron") needs to have
access to all collections.
Check the (Ranger) audit log to see if access to the collections is
denied/allowed.
If you use Ranger + Solr with Metron you might run into a current Solr plugin
bug. I've run into that a few weeks ago and provided a workaround here:
https://community.hortonworks.com/articles/203326/searching-in-multiple-collections-in-one-query-wit.html<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fcommunity.hortonworks.com%2farticles%2f203326%2fsearching-in-multiple-collections-in-one-query-wit.html&c=E,1,9dD6y1u3ennoQvuZCZ45l9VNmxiTrI6YNYlhGiC0_1XXX0pMj7qHTj6yBi6drFMrosDMlFfiJthaQqZTUjr6MVXrJwuRPuh_eNO6-s9QM6lnTnhpPaw,&typo=1>
Best,
Stefan
On Wed, Aug 22, 2018, 18:40 Anil Donthireddy
<[email protected]<mailto:[email protected]>> wrote:
Yes, necessary two fields are set properly as below
"source.type.field" : "source.type",
"threat.triage.score.field" : "threat.triage.score"
From: Anand Subramanian
[mailto:[email protected]<mailto:[email protected]>]
Sent: Wednesday, August 22, 2018 10:07 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Unable to see alerts in metron alert UI with solr
Hey Anil,
You might also want to check if the “source.type.field” is set to “source.type”
(and NOT source:type) in global config.
See:
https://github.com/apache/metron/blob/master/metron-interface/metron-alerts/README.md#global-configuration-properties
HTH,
Anand
From: Anil Donthireddy
<[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Wednesday, August 22, 2018 at 10:04 PM
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: RE: Unable to see alerts in metron alert UI with solr
Hi Stefan,
I can see the events being written to solr collection. There are no issues upto
storing data to Solr. There are events which have is_alert=true. But in the
metron alerts UI I dint see any events.
Thanks,
Anil.
From: Stefan Kupstaitis-Dunkler [mailto:[email protected]]
Sent: Wednesday, August 22, 2018 9:52 PM
To: [email protected]<mailto:[email protected]>
Subject: Re: Unable to see alerts in metron alert UI with solr
Hi Anil,
The alerts UI just queries your Solr collections and displays them
Things you could do:
• You could check the collection the event is supposed to be directly
• If it’s not there you could check the Solr “error” collection.
• Use the Kafka console consumer and
o check the parser topic (you define the name in the management ui) if events
are being written to it
o check the “enrichments” topic if events are being written to it
o check the “indexing” topic if events are being written to it.
Hope that helps as a start.
Best,
Stefan
From: Anil Donthireddy
<[email protected]<mailto:[email protected]>>
Reply-To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Date: Wednesday, 22. August 2018 at 18:07
To: "[email protected]<mailto:[email protected]>"
<[email protected]<mailto:[email protected]>>
Subject: Unable to see alerts in metron alert UI with solr
Hi,
I have created a collection in solr for my source as documented in metron-slor
readme in git to enable solr and writing data to solr. I see the data is being
loaded to Solr for my new source and is_alert is set to tru. But still I am not
seeing any data in metron alerts UI.
I checked the metron alerts UI logs, but could not find any error logs. I am
blocked to proceed further to display alerts in the metron alerts ui. May I get
any suggestions to debug the issue.
Thanks,
Anil.
--
Stefan Kupstaitis-Dunkler
https://datahovel.com/<https://linkprotect.cudasvc.com/url?a=https%3a%2f%2fdatahovel.com%2f&c=E,1,rgB6RnyVWsWRLyz4Pfv-ZJ0kCNc2qsDK9f0qPgKhSLPGI8cUtFb3JzjKtCJ0C5P5533BhkBS_k9oAum3K-N7e8fYNQHryxJrRnCjR5uHcw,,&typo=1>
https://www.meetup.com/Hadoop-User-Group-Vienna/
https://twitter.com/StefanDunkler
