Hi Joshua, Thank you for your post. I'am also concern with Ossec and Logstash for a business. I used your configuration in my test environment with somes changes in the name of the cluster and ip adress. This my test environment: I have two virtual host in the same network and the ping together, one let's say A with logstash and the other let's say B with Elasticsearch. In host A, I have copied somes ossec log and aggregate them in an unique file in other to get what you have with "alert.log". The other parameter in the configuration file remain the same with what you mentionned. But this is the issue I have: Iogstash doesn't create any index in Elasticsearch cluster and I don't know why. Have you met this issue? However, Elasticsearch instance détectes logstash instance. And when I configure in logstash's file config, to send output in stdout, I get something, the result you have. So, why logstash can't send result of the parsing Elasticsearch. I use logstash-1.4.0 and Elasticsearch-1.3.0 Thank you for any help! Le samedi 8 mars 2014 23:02:35 UTC+1, Joshua Garnett a écrit : > > All, > > I'll probably write a blog post on this, but I wanted to share some work > I've done today. > http://vichargrave.com/ossec-log-management-with-elasticsearch/ shows how > to use OSSEC's syslog output to route messages to Elasticsearch. The > problem with this method is it uses UDP. Even when sending packets to a > local process UDP by definition is unreliable. Garbage collections and > other system events can cause packets to be lost. I've found it tends to > cap out at around 1,500 messages per minute. > > To address this issue I've put together a logstash config that will read > the alerts from /var/ossec/logs/alerts/alerts.log. On top of solving the > reliability issue, it also fixes issues with multi-lines being lost, and > adds geoip lookups for the src_ip. I tested it against approximately 1GB > of alerts (3M events). > > input { > file { > type => "ossec" > path => "/var/ossec/logs/alerts/alerts.log" > sincedb_path => "/opt/logstash/" > codec => multiline { > pattern => "^\*\*" > negate => true > what => "previous" > } > } > } > > filter { > if [type] == "ossec" { > # Parse the header of the alert > grok { > # Matches 2014 Mar 08 00:57:49 (some.server.com) 10.1.2.3->ossec > # (?m) fixes issues with multi-lines see > https://logstash.jira.com/browse/LOGSTASH-509 > match => ["message", "(?m)\*\* Alert > %{DATA:timestamp_seconds}:%{SPACE}%{WORD}?%{SPACE}\- > %{DATA:ossec_group}\n%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp} > \(%{DATA:reporting_host}\) > %{IP:reporting_ip}\-\>%{DATA:reporting_source}\nRule: > %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> > '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] > > # Matches 2014 Mar 08 00:00:00 ossec-server01->/var/log/auth.log > match => ["message", "(?m)\*\* Alert > %{DATA:timestamp_seconds}:%{SPACE}%{WORD}?%{SPACE}\- > %{DATA:ossec_group}\n%{YEAR} %{SYSLOGTIMESTAMP:syslog_timestamp} > %{DATA:reporting_host}\-\>%{DATA:reporting_source}\nRule: > %{NONNEGINT:rule_number} \(level %{NONNEGINT:severity}\) \-\> > '%{DATA:signature}'\n%{GREEDYDATA:remaining_message}"] > } > > # Attempt to parse additional data from the alert > grok { > match => ["remaining_message", "(?m)(Src IP: > %{IP:src_ip}%{SPACE})?(Src Port: %{NONNEGINT:src_port}%{SPACE})?(Dst IP: > %{IP:dst_ip}%{SPACE})?(Dst Port: %{NONNEGINT:dst_port}%{SPACE})?(User: > %{USER:acct}%{SPACE})?%{GREEDYDATA:real_message}"] > } > > geoip { > source => "src_ip" > } > > mutate { > convert => [ "severity", "integer"] > replace => [ "@message", "%{real_message}" ] > replace => [ "@fields.hostname", "%{reporting_host}"] > add_field => [ "@fields.product", "ossec"] > add_field => [ "raw_message", "%{message}"] > add_field => [ "ossec_server", "%{host}"] > remove_field => [ "type", "syslog_program", "syslog_timestamp", > "reporting_host", "message", "timestamp_seconds", "real_message", > "remaining_message", "path", "host", "tags"] > } > } > } > > output { > elasticsearch { > host => "10.0.0.1" > cluster => "mycluster" > } > } > > Here are a few examples of the output this generates. > > { > "@timestamp":"2014-03-08T20:34:08.847Z", > "@version":"1", > "ossec_group":"syslog,sshd,invalid_login,authentication_failed,", > "reporting_ip":"10.1.2.3", > "reporting_source":"/var/log/auth.log", > "rule_number":"5710", > "severity":5, > "signature":"Attempt to login using a non-existent user", > "src_ip":"112.65.211.164", > "geoip":{ > "ip":"112.65.211.164", > "country_code2":"CN", > "country_code3":"CHN", > "country_name":"China", > "continent_code":"AS", > "region_name":"23", > "city_name":"Shanghai", > "latitude":31.045600000000007, > "longitude":121.3997, > "timezone":"Asia/Shanghai", > "real_region_name":"Shanghai", > "location":[ > 121.3997, > 31.045600000000007 > ] > }, > "@message":"Mar 8 01:00:59 someserver sshd[22874]: Invalid user oracle > from 112.65.211.164\n", > "@fields.hostname":"someserver.somedomain.com", > "@fields.product":"ossec", > "raw_message":"** Alert 1394240459.2305861: - > syslog,sshd,invalid_login,authentication_failed,\n2014 Mar 08 01:00:59 ( > someserver.somedomain.com) 10.1.2.3->/var/log/auth.log\nRule: 5710 (level > 5) -> 'Attempt to login using a non-existent user'\nSrc IP: > 112.65.211.164\nMar 8 01:00:59 someserver sshd[22874]: Invalid user oracle > from 112.65.211.164\n", > "ossec_server":"ossec-server.somedomain.com" > } > > and > > { > "@timestamp":"2014-03-08T21:15:23.278Z", > "@version":"1", > "ossec_group":"syslog,sudo", > "reporting_source":"/var/log/auth.log", > "rule_number":"5402", > "severity":3, > "signature":"Successful sudo to ROOT executed", > "acct":"nagios", > "@message":"Mar 8 00:00:03 ossec-server sudo: nagios : TTY=unknown ; > PWD=/ ; USER=root ; COMMAND=/usr/lib/some/command", > "@fields.hostname":"ossec-server", > "@fields.product":"ossec", > "raw_message":"** Alert 1394236804.1451: - syslog,sudo\n2014 Mar 08 > 00:00:04 ossec-server->/var/log/auth.log\nRule: 5402 (level 3) -> > 'Successful sudo to ROOT executed'\nUser: nagios\nMar 8 00:00:03 > ossec-server sudo: nagios : TTY=unknown ; PWD=/ ; USER=root ; > COMMAND=/usr/lib/some/command", > "ossec_server":"ossec-server.somedomain.com" > } > > If you combine the above with a custom Elasticsearch template, you can put > together some really nice Kibana dashboards. > > > --Josh > > >
-- --- You received this message because you are subscribed to the Google Groups "ossec-list" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
