Ignore that, just tried again and it works :) On Tuesday, 27 October 2015 10:30:15 UTC, Daniel Niasoff wrote: > > Hi Jochen, > > I initially tried the Raw/Plaintext Kafka input but although the Graylog > input showed the messages coming in nothing was going into elasticsearch > and the logs were empty? > > Thanks > > Daniel > > On Tuesday, 27 October 2015 10:15:41 UTC, Jochen Schalanda wrote: >> >> Hi Daniel, >> >> the message format cries for either a Regular Expression or a Grok >> Extractor. Given that the message already contains a description of the >> fields in the "#Fields: " line, it should be relatively straight forward to >> come up with a matching Grok pattern. >> >> In general, I'd recommend using a Raw/Plaintext Kafka input for this kind >> of message. >> >> >> Cheers, >> Jochen >> >> On Tuesday, 27 October 2015 10:50:17 UTC+1, Daniel Niasoff wrote: >>> >>> Hi. >>> >>> I just installed Graylog 1.2.2. >>> >>> I have a HTTP proxy that sends logs via Kafka and the only input that >>> seemed to work is the Kafka syslog input (although it's not a valid syslog >>> message). >>> >>> This is a sample of the input >>> >>> full_message >>> Toggle dropdown >>> #Software: SGOS 6.6.2.3 #Version: 1.0 #Start-Date: 2015-10-27 09:25:42 >>> #Date: 2015-10-26 15:00:55 #Fields: date time time-taken c-ip cs-username >>> cs-auth-group s-supplier-name s-supplier-ip s-supplier-country >>> s-supplier-failures x-exception-id sc-filter-result cs-categories >>> cs(Referer) sc-status s-action cs-method rs(Content-Type) cs-uri-scheme >>> cs-host cs-uri-port cs-uri-path cs-uri-query cs-uri-extension >>> cs(User-Agent) s-ip sc-bytes cs-bytes x-virus-id >>> x-bluecoat-application-name x-bluecoat-application-operation cs-threat-risk >>> #Remark: 1006318175 "192.168.40.20 - Blue Coat SG-VA Series" >>> "192.168.40.20" "main" 2015-10-27 09:25:38 16 82.69.3.231 dniasoff >>> CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL >>> >>> graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" >>> http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 >>> >>> 200 TCP_NC_MISS GET application/json;%20charset=utf-8 http >>> graylog1.redactus.co.uk 9000 /a/system/notifications - - "Mozilla/5.0 >>> (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) >>> Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 203 717 - "none" "none" >>> none 2015-10-27 09:25:38 976 82.69.3.231 dniasoff >>> CN=BC_Allow_Newsgroups_Forums,OU=Allow,OU=BlueCoat,OU=Categories,OU=Groups,DC=SafeSurf,DC=LOCAL >>> >>> graylog1.redactus.co.uk 149.202.163.197 None - - PROXIED "none" >>> http://graylog1.redactus.co.uk:9000/search?q=gl2_source_input%3A562ef6fde4b093bfac41a012&relative=28800 >>> >>> 200 TCP_NC_MISS POST application/javascript;%20charset=UTF-8 http >>> graylog1.redactus.co.uk 9000 /a/metrics/346/2knjgujd/xhr >>> ?t=1445937937553 - "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 >>> (KHTML, like Gecko) Chrome/46.0.2490.80 Safari/537.36" 192.168.40.20 1388 >>> 723 - "none" "none" none >>> >>> How can I parse it so that the each line is a accessible individually >>> and the fields are all broken up and searchable against? >>> >>> Any pointers will be really useful. >>> >>> Thanks >>> >>> Daniel >>> >>> >>>
-- You received this message because you are subscribed to the Google Groups "Graylog Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/graylog2/5aeb8cbe-e284-474a-9861-df18005f91e9%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
