Thank you Joshua Garnett. I've switched from syslog to localhost to reading the log file directly.
Few questions: - Is there are way to filter with "Alert Level X or above"? (This is more generic Kibana question) - Which field did you use for the Bettermap Panel? I've added the panel with geoip.lattitude however the panel fails to load without any errors - Is there a reason why you choose to remove fields? for me syslog_timestamp is much cleaner than @timestamp Cheers On Thursday, 20 March 2014 17:02:52 UTC, vic hargrave wrote: > > Since writing my blog on using Elasticseach for OSSEC log management, I've > upgraded to Elasticsearch 1.0.1 which does not seem to be able get logs > data from Logstash 1.3.2 or 1.3.3. The solution is to use > "elasticsearch_http" in the "output" section of the logstash configuration > file. When you do that all is well. > > For more information on better log ingestion rates, check out Brad > Lhotsky's article - > http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html. > > > On Thu, Mar 20, 2014 at 3:43 AM, Chris H <[email protected]<javascript:> > > wrote: > >> Thanks, I'll have a look. For me the default template created each field >> as a multi-field, with the regular, analysed field and an additional "raw" >> un-analysed field. I'm extracting quite a lot of fields from the different >> log types, which is something I was doing in Splunk before trying >> elasticsearch. >> >> "Alert_Level" : { >> "type" : "multi_field", >> "fields" : { >> "Alert_Level" : { >> "type" : "string", >> "omit_norms" : true >> }, >> "raw" : { >> >> "type" : "string", >> "index" : "not_analyzed", >> "omit_norms" : true, >> "index_options" : "docs", >> "include_in_all" : false, >> "ignore_above" : 256 >> } >> } >> }, >> >> I created a new default template in elasticsearch: >> >> curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d '{ >> "template": "logstash-*", >> "settings": { >> "index.store.compress.stored": true >> }, >> "mappings": { >> "_default_": { >> "_source": { "compress": "true" }, >> "_all" : { >> "enabled" : false >> } >> } >> } >> }' >> >> This has applied, but the compression doesn't seem to do much. I'm at >> the point where I might only be able to store a limited amount of data in >> elasticsearch :( >> >> Chris >> >> >> >> On Wednesday, March 19, 2014 7:37:41 PM UTC, Joshua Garnett wrote: >> >>> Chris, >>> >>> Yeah digging into the templates was another big win for me. For >>> instance, if you try to do a topN query on signature with the default >>> template, you end up with words like the and and as your top hits. Setting >>> signature to not_analyzed ensures the field isn't tokenized. Below is my >>> template. >>> >>> --Josh >>> >>> Logstash settings: >>> >>> output { >>> elasticsearch { >>> host => "10.0.0.1" >>> cluster => "mycluster" >>> index => "logstash-ossec-%{+YYYY.MM.dd}" >>> index_type => "ossec" >>> template_name => "template-ossec" >>> template => "/etc/logstash/elasticsearch_template.json" >>> template_overwrite => true >>> } >>> } >>> >>> elasticsearch_template.json >>> >>> { >>> "template":"logstash-ossec-*", >>> "settings":{ >>> "index.analysis.analyzer.default.stopwords":"_none_", >>> "index.refresh_interval":"5s", >>> "index.analysis.analyzer.default.type":"standard" >>> }, >>> "mappings":{ >>> "ossec":{ >>> "properties":{ >>> "@fields.hostname":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "@fields.product":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "@message":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "@timestamp":{ >>> "type":"date" >>> }, >>> "@version":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "acct":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "ossec_group":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "ossec_server":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "raw_message":{ >>> "type":"string", >>> "index":"analyzed" >>> }, >>> "reporting_ip":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "reporting_source":{ >>> "type":"string", >>> "index":"analyzed" >>> }, >>> "rule_number":{ >>> "type":"integer" >>> }, >>> "severity":{ >>> "type":"integer" >>> }, >>> "signature":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "src_ip":{ >>> "type":"string", >>> "index":"not_analyzed" >>> }, >>> "geoip":{ >>> "type" : "object", >>> "dynamic": true, >>> "path": "full", >>> "properties" : { >>> "location" : { "type" : "geo_point" } >>> } >>> } >>> }, >>> "_all":{ >>> "enabled":true >>> } >>> } >>> } >>> } >>> >>> >>> On Wed, Mar 19, 2014 at 10:54 AM, Chris H <[email protected]> wrote: >>> >>>> Hi, Joshua. >>>> >>>> I'm using a very similar technique. Are you applying a mapping >>>> template, or using the default? I'm using the default automatic >>>> templates, >>>> because frankly I don't fully understand templates. What this means >>>> though >>>> is that my daily indexes are larger than the uncompressed alerts.log, >>>> between 2-4GB per day, and I'm rapidly running out of disk space. I >>>> gather >>>> than this can be optimised by enabling compression and excluding the >>>> _source and _all fields through the mapping template, but I'm not sure >>>> exactly how this works. Just wondered if you've come across the same >>>> problem. >>>> >>>> Thanks. >>>> >>>> >>>> On Saturday, March 8, 2014 10:02:35 PM UTC, Joshua Garnett wrote: >>>>> >>>>> All, >>>>> >>>>> I'll probably write a blog post on this, but I wanted to share some >>>>> work I've done today. http://vichargrave.com/ossec- >>>>> log-management-with-elasticsearch/ shows how to use OSSEC's syslog >>>>> output to route messages to Elasticsearch. The problem with this method >>>>> is >>>>> it uses UDP. Even when sending packets to a local process UDP by >>>>> definition is unreliable. Garbage collections and other system events >>>>> can >>>>> cause packets to be lost. I've found it tends to cap out at around 1,500 >>>>> messages per minute. >>>>> >>>>> To address this issue I've put together a logstash config that will >>>>> read the alerts from /var/ossec/logs/alerts/alerts.log. On top of >>>>> solving the reliability issue, it also fixes issues with multi-lines >>>>> being >>>>> lost, and adds geoip lookups for the src_ip. I tested it against >>>>> approximately 1GB of alerts (3M events). >>>>> >>>>> input { >>>>> file { >>>>> type => "ossec" >>>>> path => "/var/ossec/logs/alerts/alerts.log" >>>>> sincedb_path => "/opt/logstash/" >>>>> codec => multiline { >>>>> pattern => "^\*\*" >>>>> negate => true >>>>> what => "previous" >>>>> } >>>>> } >>>>> } >>>>> >>>>> filter { >>>>> if [type] == "ossec" { >>>>> # Parse the header of the alert >>>>> grok { >>>>> # Matches 2014 Mar 08 00:57:49 (some.server.com) >>>>> 10.1.2.3->ossec >>>>> # (?m) fixes issues with multi-lines see >>>>> https://logstash.jira.com/browse/LOGSTASH-509 >>>>> match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{ >>>>> SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group}\n%{YEAR} >>>>> %{SYSLOGTIMESTAMP:syslog_timestamp} \(%{DATA:reporting_host}\) >>>>> %{IP:reporting_ip}\-\>%{DATA:reporting_source}\nRule: >>>>> %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> >>>>> '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] >>>>> >>>>> # Matches 2014 Mar 08 00:00:00 ossec-server01->/var/log/auth. >>>>> log >>>>> match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{ >>>>> SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group}\n%{YEAR} >>>>> %{SYSLOGTIMESTAMP:syslog_timestamp} >>>>> %{DATA:reporting_host}\-\>%{DATA:reporting_source}\nRule: >>>>> %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> >>>>> '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] >>>>> } >>>>> >>>>> # Attempt to parse additional data from the alert >>>>> grok { >>>>> match => ["remaining_message", "(?m)(Src IP: >>>>> %{IP:src_ip}%{SPACE})?(Src Port: %{NONNEGINT:src_port}%{SPACE})?(Dst >>>>> IP: %{IP:dst_ip}%{SPACE})?(Dst Port: >>>>> %{NONNEGINT:dst_port}%{SPACE})?(User: >>>>> %{USER:acct}%{SPACE})?%{GREEDYDATA:real_message}"] >>>>> } >>>>> >>>>> geoip { >>>>> source => "src_ip" >>>>> } >>>>> >>>>> mutate { >>>>> convert => [ "severity", "integer"] >>>>> replace => [ "@message", "%{real_message}" ] >>>>> replace => [ "@fields.hostname", "%{reporting_host}"] >>>>> add_field => [ "@fields.product", "ossec"] >>>>> add_field => [ "raw_message", "%{message}"] >>>>> add_field => [ "ossec_server", "%{host}"] >>>>> remove_field => [ "type", "syslog_program", "syslog_timestamp", >>>>> "reporting_host", "message", "timestamp_seconds", "real_message", >>>>> "remaining_message", "path", "host", "tags"] >>>>> } >>>>> } >>>>> } >>>>> >>>>> output { >>>>> elasticsearch { >>>>> host => "10.0.0.1" >>>>> cluster => "mycluster" >>>>> } >>>>> } >>>>> >>>>> Here are a few examples of the output this generates. >>>>> >>>>> { >>>>> "@timestamp":"2014-03-08T20:34:08.847Z", >>>>> "@version":"1", >>>>> "ossec_group":"syslog,sshd,invalid_login,authentication_failed,", >>>>> "reporting_ip":"10.1.2.3", >>>>> "reporting_source":"/var/log/auth.log", >>>>> "rule_number":"5710", >>>>> "severity":5, >>>>> "signature":"Attempt to login using a non-existent user", >>>>> "src_ip":"112.65.211.164", >>>>> "geoip":{ >>>>> "ip":"112.65.211.164", >>>>> "country_code2":"CN", >>>>> "country_code3":"CHN", >>>>> "country_name":"China", >>>>> "continent_code":"AS", >>>>> "region_name":"23", >>>>> "city_name":"Shanghai", >>>>> "latitude":31.045600000000007, >>>>> "longitude":121.3997, >>>>> "timezone":"Asia/Shanghai", >>>>> "real_region_name":"Shanghai", >>>>> "location":[ >>>>> 121.3997, >>>>> 31.045600000000007 >>>>> ] >>>>> }, >>>>> "@message":"Mar 8 01:00:59 someserver sshd[22874]: Invalid user >>>>> oracle from 112.65.211.164\n", >>>>> "@fields.hostname":"someserver.somedomain.com", >>>>> "@fields.product":"ossec", >>>>> "raw_message":"** Alert 1394240459.2305861: - >>>>> syslog,sshd,invalid_login,authentication_failed,\n2014 Mar 08 >>>>> 01:00:59 (someserver.somedomain.com) 10.1.2.3->/var/log/auth.log\nRule: >>>>> 5710 (level 5) -> 'Attempt to login using a non-existent user'\nSrc IP: >>>>> 112.65.211.164\nMar 8 01:00:59 someserver sshd[22874]: Invalid user >>>>> oracle >>>>> from 112.65.211.164\n", >>>>> "ossec_server":"ossec-server.somedomain.com" >>>>> } >>>>> >>>>> and >>>>> >>>>> { >>>>> "@timestamp":"2014-03-08T21:15:23.278Z", >>>>> "@version":"1", >>>>> "ossec_group":"syslog,sudo", >>>>> "reporting_source":"/var/log/auth.log", >>>>> "rule_number":"5402", >>>>> "severity":3, >>>>> "signature":"Successful sudo to ROOT executed", >>>>> "acct":"nagios", >>>>> "@message":"Mar 8 00:00:03 ossec-server sudo: nagios : >>>>> TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/lib/some/command", >>>>> "@fields.hostname":"ossec-server", >>>>> "@fields.product":"ossec", >>>>> "raw_message":"** Alert 1394236804.1451: - syslog,sudo\n2014 Mar 08 >>>>> 00:00:04 ossec-server->/var/log/auth.log\nRule: 5402 (level 3) -> >>>>> 'Successful sudo to ROOT executed'\nUser: nagios\nMar 8 00:00:03 >>>>> ossec-server sudo: nagios : TTY=unknown ; PWD=/ ; USER=root ; >>>>> COMMAND=/usr/lib/some/command", >>>>> "ossec_server":"ossec-server.somedomain.com" >>>>> } >>>>> >>>>> If you combine the above with a custom Elasticsearch template, you can >>>>> put together some really nice Kibana dashboards. >>>>> >>>>> >>>>> --Josh >>>>> >>>>> >>>>> -- >>>> >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "ossec-list" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> >> --- >> You received this message because you are subscribed to the Google Groups >> "ossec-list" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
