Hi Daniel, the message format cries for either a Regular Expression or a Grok Extractor. Given that the message already contains a description of the fields in the "#Fields: " line, it should be relatively straight forward to come up with a matching Grok pattern.
In general, I'd recommend using a Raw/Plaintext Kafka input for this kind of message. Cheers, Jochen On Tuesday, 27 October 2015 10:50:17 UTC+1, Daniel Niasoff wrote: > > Hi. > > I just installed Graylog 1.2.2. > > I have a HTTP proxy that sends logs via Kafka and the only input that > seemed to work is the Kafka syslog input (although it's not a valid syslog > message). > > This is a sample of the input > > full_message > Toggle dropdown > #Software: SGOS 6.6.2.3 #Version: 1.0 #Start-Date: 2015-10-27 09:25:42 > #Date: 2015-10-26 15:00:55 #Fields: date time time-taken c-ip cs-username > cs-auth-group s-supplier-name s-supplier-ip s-supplier-country > s-supplier-failures x-exception-id sc-filter-result cs-categories > cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme > cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension > cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id > x-bluecoat-application-name x-bluecoat-application-operation cs-threat-risk > #Remark: 1006318175 "192.168.40.20 - Blue Coat SG-VA Series" > "192.168.40.20" "main" 2015-10-27 09:25:38 16 82.69.3.231 dniasoff > CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL > > graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" > http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 > > 200 TCP_NC_MISS GET application/json;%20charset=utf-8 http > graylog1.redactus.co.uk 9000 /a/system/notifications - - "Mozilla/5.0 > (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) > Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 203 717 - "none" "none" > none 2015-10-27 09:25:38 976 82.69.3.231 dniasoff > CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL > > graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" > http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 > > 200 TCP_NC_MISS POST application/javascript;%20charset=UTF-8 http > graylog1.redactus.co.uk 9000 /a/metrics/346/2knjgujd/xhr ?t=1445937937553 > - "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like > Gecko) Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 1388 723 - "none" > "none" none > > How can I parse it so that the each line is a accessible individually and > the fields are all broken up and searchable against? > > Any pointers will be really useful. > > Thanks > > Daniel > > > -- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/ea4896eb-a3ab-490e-9d40-98dc6c93a454%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
