Since writing my blog on using Elasticseach for OSSEC log management, I've upgraded to Elasticsearch 1.0.1 which does not seem to be able get logs data from Logstash 1.3.2 or 1.3.3. The solution is to use "elasticsearch_http" in the "output" section of the logstash configuration file. When you do that all is well.
For more information on better log ingestion rates, check out Brad Lhotsky's article - http://edgeofsanity.net/article/2012/12/26/elasticsearch-for-logging.html. On Thu, Mar 20, 2014 at 3:43 AM, Chris H <[email protected]> wrote: > Thanks, I'll have a look. For me the default template created each field > as a multi-field, with the regular, analysed field and an additional "raw" > un-analysed field. I'm extracting quite a lot of fields from the different > log types, which is something I was doing in Splunk before trying > elasticsearch. > > "Alert_Level" : { > "type" : "multi_field", > "fields" : { > "Alert_Level" : { > "type" : "string", > "omit_norms" : true > }, > "raw" : { > > "type" : "string", > "index" : "not_analyzed", > "omit_norms" : true, > "index_options" : "docs", > "include_in_all" : false, > "ignore_above" : 256 > } > } > }, > > I created a new default template in elasticsearch: > > curl -XPUT 'http://localhost:9200/_template/template_logstash/' -d '{ > "template": "logstash-*", > "settings": { > "index.store.compress.stored": true > }, > "mappings": { > "_default_": { > "_source": { "compress": "true" }, > "_all" : { > "enabled" : false > } > } > } > }' > > This has applied, but the compression doesn't seem to do much. I'm at the > point where I might only be able to store a limited amount of data in > elasticsearch :( > > Chris > > > > On Wednesday, March 19, 2014 7:37:41 PM UTC, Joshua Garnett wrote: > >> Chris, >> >> Yeah digging into the templates was another big win for me. For >> instance, if you try to do a topN query on signature with the default >> template, you end up with words like the and and as your top hits. Setting >> signature to not_analyzed ensures the field isn't tokenized. Below is my >> template. >> >> --Josh >> >> Logstash settings: >> >> output { >> elasticsearch { >> host => "10.0.0.1" >> cluster => "mycluster" >> index => "logstash-ossec-%{+YYYY.MM.dd}" >> index_type => "ossec" >> template_name => "template-ossec" >> template => "/etc/logstash/elasticsearch_template.json" >> template_overwrite => true >> } >> } >> >> elasticsearch_template.json >> >> { >> "template":"logstash-ossec-*", >> "settings":{ >> "index.analysis.analyzer.default.stopwords":"_none_", >> "index.refresh_interval":"5s", >> "index.analysis.analyzer.default.type":"standard" >> }, >> "mappings":{ >> "ossec":{ >> "properties":{ >> "@fields.hostname":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "@fields.product":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "@message":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "@timestamp":{ >> "type":"date" >> }, >> "@version":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "acct":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "ossec_group":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "ossec_server":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "raw_message":{ >> "type":"string", >> "index":"analyzed" >> }, >> "reporting_ip":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "reporting_source":{ >> "type":"string", >> "index":"analyzed" >> }, >> "rule_number":{ >> "type":"integer" >> }, >> "severity":{ >> "type":"integer" >> }, >> "signature":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "src_ip":{ >> "type":"string", >> "index":"not_analyzed" >> }, >> "geoip":{ >> "type" : "object", >> "dynamic": true, >> "path": "full", >> "properties" : { >> "location" : { "type" : "geo_point" } >> } >> } >> }, >> "_all":{ >> "enabled":true >> } >> } >> } >> } >> >> >> On Wed, Mar 19, 2014 at 10:54 AM, Chris H <[email protected]> wrote: >> >>> Hi, Joshua. >>> >>> I'm using a very similar technique. Are you applying a mapping >>> template, or using the default? I'm using the default automatic templates, >>> because frankly I don't fully understand templates. What this means though >>> is that my daily indexes are larger than the uncompressed alerts.log, >>> between 2-4GB per day, and I'm rapidly running out of disk space. I gather >>> than this can be optimised by enabling compression and excluding the >>> _source and _all fields through the mapping template, but I'm not sure >>> exactly how this works. Just wondered if you've come across the same >>> problem. >>> >>> Thanks. >>> >>> >>> On Saturday, March 8, 2014 10:02:35 PM UTC, Joshua Garnett wrote: >>>> >>>> All, >>>> >>>> I'll probably write a blog post on this, but I wanted to share some >>>> work I've done today. http://vichargrave.com/ossec- >>>> log-management-with-elasticsearch/ shows how to use OSSEC's syslog >>>> output to route messages to Elasticsearch. The problem with this method is >>>> it uses UDP. Even when sending packets to a local process UDP by >>>> definition is unreliable. Garbage collections and other system events can >>>> cause packets to be lost. I've found it tends to cap out at around 1,500 >>>> messages per minute. >>>> >>>> To address this issue I've put together a logstash config that will >>>> read the alerts from /var/ossec/logs/alerts/alerts.log. On top of >>>> solving the reliability issue, it also fixes issues with multi-lines being >>>> lost, and adds geoip lookups for the src_ip. I tested it against >>>> approximately 1GB of alerts (3M events). >>>> >>>> input { >>>> file { >>>> type => "ossec" >>>> path => "/var/ossec/logs/alerts/alerts.log" >>>> sincedb_path => "/opt/logstash/" >>>> codec => multiline { >>>> pattern => "^\*\*" >>>> negate => true >>>> what => "previous" >>>> } >>>> } >>>> } >>>> >>>> filter { >>>> if [type] == "ossec" { >>>> # Parse the header of the alert >>>> grok { >>>> # Matches 2014 Mar 08 00:57:49 (some.server.com) 10.1.2.3->ossec >>>> # (?m) fixes issues with multi-lines see >>>> https://logstash.jira.com/browse/LOGSTASH-509 >>>> match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{ >>>> SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group}\n%{YEAR} >>>> %{SYSLOGTIMESTAMP:syslog_timestamp} \(%{DATA:reporting_host}\) >>>> %{IP:reporting_ip}\-\>%{DATA:reporting_source}\nRule: >>>> %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> >>>> '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] >>>> >>>> # Matches 2014 Mar 08 00:00:00 ossec-server01->/var/log/auth.log >>>> match => ["message", "(?m)\*\* Alert %{DATA:timestamp_seconds}:%{ >>>> SPACE}%{WORD}?%{SPACE}\- %{DATA:ossec_group}\n%{YEAR} >>>> %{SYSLOGTIMESTAMP:syslog_timestamp} >>>> %{DATA:reporting_host}\-\>%{DATA:reporting_source}\nRule: >>>> %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> >>>> '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] >>>> } >>>> >>>> # Attempt to parse additional data from the alert >>>> grok { >>>> match => ["remaining_message", "(?m)(Src IP: >>>> %{IP:src_ip}%{SPACE})?(Src Port: %{NONNEGINT:src_port}%{SPACE})?(Dst >>>> IP: %{IP:dst_ip}%{SPACE})?(Dst Port: %{NONNEGINT:dst_port}%{SPACE})?(User: >>>> %{USER:acct}%{SPACE})?%{GREEDYDATA:real_message}"] >>>> } >>>> >>>> geoip { >>>> source => "src_ip" >>>> } >>>> >>>> mutate { >>>> convert => [ "severity", "integer"] >>>> replace => [ "@message", "%{real_message}" ] >>>> replace => [ "@fields.hostname", "%{reporting_host}"] >>>> add_field => [ "@fields.product", "ossec"] >>>> add_field => [ "raw_message", "%{message}"] >>>> add_field => [ "ossec_server", "%{host}"] >>>> remove_field => [ "type", "syslog_program", "syslog_timestamp", >>>> "reporting_host", "message", "timestamp_seconds", "real_message", >>>> "remaining_message", "path", "host", "tags"] >>>> } >>>> } >>>> } >>>> >>>> output { >>>> elasticsearch { >>>> host => "10.0.0.1" >>>> cluster => "mycluster" >>>> } >>>> } >>>> >>>> Here are a few examples of the output this generates. >>>> >>>> { >>>> "@timestamp":"2014-03-08T20:34:08.847Z", >>>> "@version":"1", >>>> "ossec_group":"syslog,sshd,invalid_login,authentication_failed,", >>>> "reporting_ip":"10.1.2.3", >>>> "reporting_source":"/var/log/auth.log", >>>> "rule_number":"5710", >>>> "severity":5, >>>> "signature":"Attempt to login using a non-existent user", >>>> "src_ip":"112.65.211.164", >>>> "geoip":{ >>>> "ip":"112.65.211.164", >>>> "country_code2":"CN", >>>> "country_code3":"CHN", >>>> "country_name":"China", >>>> "continent_code":"AS", >>>> "region_name":"23", >>>> "city_name":"Shanghai", >>>> "latitude":31.045600000000007, >>>> "longitude":121.3997, >>>> "timezone":"Asia/Shanghai", >>>> "real_region_name":"Shanghai", >>>> "location":[ >>>> 121.3997, >>>> 31.045600000000007 >>>> ] >>>> }, >>>> "@message":"Mar 8 01:00:59 someserver sshd[22874]: Invalid user >>>> oracle from 112.65.211.164\n", >>>> "@fields.hostname":"someserver.somedomain.com", >>>> "@fields.product":"ossec", >>>> "raw_message":"** Alert 1394240459.2305861: - >>>> syslog,sshd,invalid_login,authentication_failed,\n2014 Mar 08 01:00:59 >>>> (someserver.somedomain.com) 10.1.2.3->/var/log/auth.log\nRule: 5710 >>>> (level 5) -> 'Attempt to login using a non-existent user'\nSrc IP: >>>> 112.65.211.164\nMar 8 01:00:59 someserver sshd[22874]: Invalid user oracle >>>> from 112.65.211.164\n", >>>> "ossec_server":"ossec-server.somedomain.com" >>>> } >>>> >>>> and >>>> >>>> { >>>> "@timestamp":"2014-03-08T21:15:23.278Z", >>>> "@version":"1", >>>> "ossec_group":"syslog,sudo", >>>> "reporting_source":"/var/log/auth.log", >>>> "rule_number":"5402", >>>> "severity":3, >>>> "signature":"Successful sudo to ROOT executed", >>>> "acct":"nagios", >>>> "@message":"Mar 8 00:00:03 ossec-server sudo: nagios : >>>> TTY=unknown ; PWD=/ ; USER=root ; COMMAND=/usr/lib/some/command", >>>> "@fields.hostname":"ossec-server", >>>> "@fields.product":"ossec", >>>> "raw_message":"** Alert 1394236804.1451: - syslog,sudo\n2014 Mar 08 >>>> 00:00:04 ossec-server->/var/log/auth.log\nRule: 5402 (level 3) -> >>>> 'Successful sudo to ROOT executed'\nUser: nagios\nMar 8 00:00:03 >>>> ossec-server sudo: nagios : TTY=unknown ; PWD=/ ; USER=root ; >>>> COMMAND=/usr/lib/some/command", >>>> "ossec_server":"ossec-server.somedomain.com" >>>> } >>>> >>>> If you combine the above with a custom Elasticsearch template, you can >>>> put together some really nice Kibana dashboards. >>>> >>>> >>>> --Josh >>>> >>>> >>>> -- >>> >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "ossec-list" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- > > --- > You received this message because you are subscribed to the Google Groups > "ossec-list" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
