Hi Jochen, I initially tried the Raw/Plaintext Kafka input but although the Graylog input showed the messages coming in nothing was going into elasticsearch and the logs were empty?
Thanks Daniel On Tuesday, 27 October 2015 10:15:41 UTC, Jochen Schalanda wrote: > > Hi Daniel, > > the message format cries for either a Regular Expression or a Grok > Extractor. Given that the message already contains a description of the > fields in the "#Fields: " line, it should be relatively straight forward to > come up with a matching Grok pattern. > > In general, I'd recommend using a Raw/Plaintext Kafka input for this kind > of message. > > > Cheers, > Jochen > > On Tuesday, 27 October 2015 10:50:17 UTC+1, Daniel Niasoff wrote: >> >> Hi. >> >> I just installed Graylog 1.2.2. >> >> I have a HTTP proxy that sends logs via Kafka and the only input that >> seemed to work is the Kafka syslog input (although it's not a valid syslog >> message). >> >> This is a sample of the input >> >> full_message >> Toggle dropdown >> #Software: SGOS 6.6.2.3 #Version: 1.0 #Start-Date: 2015-10-27 09:25:42 >> #Date: 2015-10-26 15:00:55 #Fields: date time time-taken c-ip cs-username >> cs-auth-group s-supplier-name s-supplier-ip s-supplier-country >> s-supplier-failures x-exception-id sc-filter-result cs-categories >> cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme >> cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension >> cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id >> x-bluecoat-application-name x-bluecoat-application-operation cs-threat-risk >> #Remark: 1006318175 "192.168.40.20 - Blue Coat SG-VA Series" >> "192.168.40.20" "main" 2015-10-27 09:25:38 16 82.69.3.231 dniasoff >> CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL >> >> graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" >> http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 >> >> 200 TCP_NC_MISS GET application/json;%20charset=utf-8 http >> graylog1.redactus.co.uk 9000 /a/system/notifications - - "Mozilla/5.0 >> (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) >> Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 203 717 - "none" "none" >> none 2015-10-27 09:25:38 976 82.69.3.231 dniasoff >> CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL >> >> graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" >> http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 >> >> 200 TCP_NC_MISS POST application/javascript;%20charset=UTF-8 http >> graylog1.redactus.co.uk 9000 /a/metrics/346/2knjgujd/xhr >> ?t=1445937937553 - "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 >> (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 1388 >> 723 - "none" "none" none >> >> How can I parse it so that the each line is a accessible individually and >> the fields are all broken up and searchable against? >> >> Any pointers will be really useful. >> >> Thanks >> >> Daniel >> >> >> -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/04e69f44-f3af-46ed-8b3b-05c97aca6938%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
