Hi Masoom, I don't get why Logstash was missing your data. The only idea I have is that Logstash couldn't process it fast enough, and because it has a tiny queue, the fallback is rsyslog's queue. And if rsyslog queue fills, the fallback is... I don't know. How are you generating logs? If you send a burst to rsyslog via UDP or the local /dev/log, messages are lost if the queue is full. Or maybe rsyslog wasn't fast enough in processing the burst in the first place (and only a chunk of logs got transmitted to Logstash).
On Tue, Apr 15, 2014 at 1:59 PM, masoom alam <[email protected]> wrote: > Hi Every one, > > We have sent log traffic to Rsyslog on TCP port 514. > > It is configured to forward log traffic to Logtash at TCP port 520. However > it sent a burst of logs to logtash which misses a lot of data. The burst of > logs is mentioned below. However if we send log traffic at UDP port 514, > every thing is ok. Its a dummy traffic generated via LOIC. > > Please guide. > > > <13>Apr 15 23:46:33 p = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reason: User Requested Dec 5 01:00:22 1.1.1.10 > %ASA-4-113019: Group = 2.3.4.5, Username = 2.3.4.5, IP = 2.3.4.5, Session > disconnected. Session Type: IPSecLAN2LAN, Duration: 2h:59m:59s, Bytes xmt: > 19992, Bytes rcv: 0, Reaso > > > > On Tue, Apr 15, 2014 at 1:20 AM, David Lang <[email protected]> wrote: > > > On Mon, 14 Apr 2014, DuyLong Le wrote: > > > > Hi, > >> > >> Hi Radu, > >>> > >>> I think using JASON at the logtash is not a solution for me, as we > cannot > >>> use Ruby code as filter in the JASON. We have to place a lot of ruby > code > >>> inside the Ruby filters. I just want to quote an answer -see below, > >>> which I > >>> got from logtash ML here which describe what we want to achieve with > Ruby > >>> filters in Logtash (from a guy from Microsoft) -- in order to balance > the > >>> performance we have Rsyslog with Logtash: > >>> > >>> 1. Yes, you can imbed a ruby filter > >>> > >>> 2. Actually, logstash has a small buffer in the pipeline.rb but > >>> does > >>> not persist to disk. Some of us have drastically increased this buffer > to > >>> great benefit when processing large volumes of logs or dealing with > >>> bursts > >>> and output limitations (like rabbitmq). > >>> > >>> a. W.r.t. performance, it really depends on what your filters do. > >>> We > >>> have around 1000 lines of filter logic, but it has a lot of if/elseif > >>> ordering so that it escapes as soon as possible. We also do large > >>> dictionary lookups (40k entries) and are able to get around 250 msg/sec > >>> per > >>> CPU core with this "big" config. We scale out indexers as needed. > >>> > >>> > >> You can use also the "translate" plugin in order to simplify the code. > >> > >> Now how can we do the same in Rsyslog? > >>> > >>> > >> We use Redis and RabbitMQ for this work. With a custom script, we can > >> extend messages easily. Unfortunately Rsyslog has not RabbitMQ module, > we > >> use Logstash as intermediary. > >> > > > > there is an omrabbitmq module in the source. I don't see a doc page for > it > > though. > > > > David Lang > > > > > > -- > >> Dle > >> > >> Rainer: Performance is not an issue for Rsyslog but including support > for > >>> languages like Ruby might not be possible for Rsyslog. > >>> > >>> > >>> Thanks > >>> > >>> > >>> > >>> > >>> On Fri, Apr 11, 2014 at 11:28 AM, Radu Gheorghe > >>> <[email protected]>wrote: > >>> > >>> Hi, > >>>> > >>>> Rainer - here's where Grok seems to live: > >>>> https://github.com/jordansissel/grok > >>>> > >>>> It is indeed a library, and there are also other implementations (I > >>>> know of > >>>> one in Ruby and one in Java). > >>>> > >>>> Masoom - I think Richard and David already answered your questions > about > >>>> what you can use in rsyslog. If that's not enough, please give some > >>>> examples of what kind of filtering you need to do. > >>>> > >>>> Best regards, > >>>> Radu > >>>> > >>>> > >>>> On Fri, Apr 11, 2014 at 8:03 AM, Rainer Gerhards > >>>> <[email protected]>wrote: > >>>> > >>>> I have yet to look into it, but maybe someone knows if grok is a > >>>>> stand-alone component. If so, we could probably very easily make it > >>>>> available in rsyslog via the new external message modification plugin > >>>>> capability. > >>>>> > >>>>> Anyone in the know (else I'll try to find out)? > >>>>> > >>>>> Rainer > >>>>> > >>>>> > >>>>> On Fri, Apr 11, 2014 at 5:51 AM, David Lang <[email protected]> wrote: > >>>>> > >>>>> On Fri, 11 Apr 2014, masoom alam wrote: > >>>>>> > >>>>>> Very detailed answer. Thanks!!! > >>>>>> > >>>>>>> > >>>>>>> Since it is related with both Rsyslog and Logtash, thats why I am > >>>>>>> > >>>>>> asking > >>>> > >>>>> here. After your kind guidance, Its now clear that we should use > JASON > >>>>>>> template in Rsyslog, and then use JASON in logtash. I did not find > >>>>>>> any > >>>>>>> significant difference at the logtash end regarding Grok and JASON, > >>>>>>> > >>>>>> except > >>>>> > >>>>>> the word Jason in the filter instead of Grok, am I right? -- I mean > as > >>>>>>> > >>>>>> for > >>>>> > >>>>>> as the syntax is concerned. For the execution it will definitely > have > >>>>>>> performance gains, as you suggested. > >>>>>>> > >>>>>>> Another thing which I think I did not explain well in my email is > >>>>>>> that > >>>>>>> > >>>>>> we > >>>>> > >>>>>> are thinking to place some regex at the Rsyslog end too. Suppose we > >>>>>>> > >>>>>> have > >>>> > >>>>> > >>>>>>> 200 filters defined in Logtash, so will happen that when a log > entry > >>>>>>>> > >>>>>>> will > >>>>> > >>>>>> > >>>>>>>> arrive at the Logtash, it will have to match it against all the > 200 > >>>>>>> filters > >>>>>>> -- worst case, and/or some thing matches earlier and we compose the > >>>>>>> configuration file of Logtash in a way that it escapes. Any ideas > how > >>>>>>> > >>>>>> to > >>>> > >>>>> optimize the log deep/fancy parsing at this end? > >>>>>>> > >>>>>>> > >>>>>> I don't know about the logstash side, but I suspect that you are > >>>>>> > >>>>> correct. > >>>> > >>>>> On the rsyslog side, the equivalent would be mmlognorm, and with it > the > >>>>>> number of rules doesn't matter because they get compiled into a > parse > >>>>>> > >>>>> tree, > >>>>> > >>>>>> you go through the log message once. > >>>>>> > >>>>>> David Lang > >>>>>> > >>>>>> > >>>>>> Once Again thanks Radu. You are very helpful. > >>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Fri, Apr 11, 2014 at 12:36 AM, Radu Gheorghe > >>>>>>> <[email protected]>wrote: > >>>>>>> > >>>>>>> I've never actually tried this, but I think the best way for > >>>>>>> > >>>>>> performance > >>>>> > >>>>>> is > >>>>>>>> to send over TCP, but make the template a JSON with everything > >>>>>>>> > >>>>>>> rsyslog > >>>> > >>>>> can > >>>>>>>> parse (by default, stuff like severity, date, etc). On the > Logstash > >>>>>>>> > >>>>>>> side, > >>>>> > >>>>>> you'll use the JSON that should parse much faster than grok can > parse > >>>>>>>> syslog. After that, you'd set the rest of the Logstash filters you > >>>>>>>> > >>>>>>> want > >>>> > >>>>> to > >>>>>>>> use for fancy processing. > >>>>>>>> > >>>>>>>> Also, sending over TCP allows you to use rsyslog for buffering, > and > >>>>>>>> > >>>>>>> if > >>>> > >>>>> you're using in-memory queues (or disk-assisted, assuming those > >>>>>>>> > >>>>>>> rarely > >>>> > >>>>> spill out to disk), this means you'll avoid the I/O penalty of > >>>>>>>> > >>>>>>> writing > >>>> > >>>>> to > >>>>> > >>>>>> disks and having Logstash poll from disk periodically. > >>>>>>>> > >>>>>>>> If you need help with any of those, please write here (or on the > >>>>>>>> > >>>>>>> Logstash > >>>>> > >>>>>> ML for the Logstash part, people are really helpful there). > >>>>>>>> > >>>>>>>> Best regards, > >>>>>>>> Radu > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Apr 10, 2014 at 6:13 PM, masoom alam < > [email protected] > >>>>>>>> > > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>> Is it necessary to fill the templates inside rsyslog so that > >>>>>>>> rsyslog > >>>>>>>> > >>>>>>>>> > >>>>>>>>> should > >>>>>>>> > >>>>>>>> write each log source to a separate file for logtash - will be > easy > >>>>>>>>> > >>>>>>>> for > >>>>> > >>>>>> > >>>>>>>>> it > >>>>>>>> > >>>>>>>> for parsing? - also due to the reason logrtash has to catch > >>>>>>>>> > >>>>>>>> rsyslog? . > >>>> > >>>>> > >>>>>>>>> What > >>>>>>>> > >>>>>>>> is the alternative if we are doing extensive parsing in logtash? > - > >>>>>>>>> simply > >>>>>>>>> directing log on to a port and ask logtash to pick it up - match > it > >>>>>>>>> > >>>>>>>>> against > >>>>>>>> > >>>>>>>> 200 plugins? > >>>>>>>>> > >>>>>>>>> from phone thus brief. > >>>>>>>>> On Apr 10, 2014 5:06 PM, "Radu Gheorghe" < > >>>>>>>>> > >>>>>>>> [email protected]> > >>>> > >>>>> wrote: > >>>>>>>>> > >>>>>>>>> Here's an article that explains how to configure squeeze > >>>>>>>>> > >>>>>>>> performance > >>>> > >>>>> > >>>>>>>>>> from a > >>>>>>>>> > >>>>>>>>> rsyslog>ES>Kibana setup, and the numbers I got (20-30K EPS on my > >>>>>>>>>> > >>>>>>>>>> good-old > >>>>>>>>> > >>>>>>>> > >>>>>>>> laptop): > http://www.rsyslog.com/performance-tuning-elasticsearch/ > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> You also have links there about other articles in this are (that > >>>>>>>>>> > >>>>>>>>> also > >>>> > >>>>> > >>>>>>>>>> have > >>>>>>>>> > >>>>>>>>> config snippets and explanations). > >>>>>>>>>> > >>>>>>>>>> On Tue, Apr 8, 2014 at 11:34 PM, Josh Bitto < > >>>>>>>>>> > >>>>>>>>> [email protected]> > >>>> > >>>>> wrote: > >>>>>>>>>> > >>>>>>>>>> If I'm reading this right your saying that you did > >>>>>>>>>> > >>>>>>>>>>> Rsyslog->Elasticsearch->gui? > >>>>>>>>>>> > >>>>>>>>>>> I've tried installing the rpm on centos and it installs but > >>>>>>>>>>> > >>>>>>>>>>> apparently > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> it > >>>>>>>>> > >>>>>>>>> doesn't come with a config file and so the daemon starts it > errors > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> out > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> in > >>>>>>>>> > >>>>>>>>> the logs and just shuts down after that. > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> -----Original Message----- > >>>>>>>>>>> From: [email protected] [mailto: > >>>>>>>>>>> [email protected]] On Behalf Of Rick Brown > >>>>>>>>>>> Sent: Tuesday, April 08, 2014 11:31 AM > >>>>>>>>>>> To: rsyslog-users > >>>>>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ logstash-elasticsearch-kibana > >>>>>>>>>>> > >>>>>>>>>>> server > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>>> Today I've setup my central rsyslog server to replay the logs > via > >>>>>>>>>>> omudpspoof to a logstash server -> ES. It's already indexing > >>>>>>>>>>> > >>>>>>>>>> about > >>>> > >>>>> > >>>>>>>>>>> twice > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> as much as just rsyslog -> ES was using the recipe in the first > >>>>>>>>>> > >>>>>>>>> link > >>>> > >>>>> > >>>>>>>>>>> below, > >>>>>>>>>> > >>>>>>>>>> and I haven't even begun to dig into the scads of plugins > >>>>>>>>>>> > >>>>>>>>>> available > >>>> > >>>>> > >>>>>>>>>>> for > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> logstash. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> http://blog.sematext.com/2013/07/01/recipe-rsyslog- > >>>>>>>>> > >>>>>>>> elasticsearch-kibana/isagoodplace to start, although you can > >>>>>>>> replace > >>>>>>>> the omelasticsearch OM > >>>>>>>> > >>>>>>>> with omudpspoof if you want to do logstash. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> http://cookbook.logstash.net/recipes/rsyslog-agent/ is a good > >>>>>>>>>>> > >>>>>>>>>> place > >>>> > >>>>> > >>>>>>>>>>> to > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> start with rsyslog -> logstash, although I did UDP instead of > TCP, > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> and > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> used > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> the elasticsearch output module instead of stdout, which is > >>>>>>>>>>> > >>>>>>>>>>> documented > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> here: http://cookbook.logstash.net/recipes/central-syslog/ > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> Good luck to you! Those three links is basically all I needed, > >>>>>>>>>>> > >>>>>>>>>> and > >>>> > >>>>> > >>>>>>>>>>> should > >>>>>>>>>> > >>>>>>>>>> set you down the right path, regardless of how your path > differs > >>>>>>>>>>> > >>>>>>>>>> from > >>>>> > >>>>>> > >>>>>>>>>>> mine > >>>>>>>>>> > >>>>>>>>>> ;) > >>>>>>>>>>> > >>>>>>>>>>> ----- Original Message ----- > >>>>>>>>>>> > >>>>>>>>>>> From: "Orangepeel Beef" <[email protected]> > >>>>>>>>>>>> To: "rsyslog-users" <[email protected]> > >>>>>>>>>>>> Sent: Tuesday, April 8, 2014 2:17:42 PM > >>>>>>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ > logstash-elasticsearch-kibana > >>>>>>>>>>>> > >>>>>>>>>>>> server > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>>> it works, but I find it overly complex for my environment. > read: > >>>>>>>>>>>> > >>>>>>>>>>>> I > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>> don't need it ;) On Apr 8, 2014 11:13 AM, "Josh Bitto" > >>>>>>>>> > >>>>>>>>>> <[email protected]> wrote: > >>>>>>>>>>>> > >>>>>>>>>>>> I have read about Redis as being the "broker" thoughts? > >>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> > >>>>>>>>>>>>> -----Original Message----- > >>>>>>>>>>>>> From: [email protected] [mailto: > >>>>>>>>>>>>> [email protected]] On Behalf Of Orangepeel > >>>>>>>>>>>>> Beef > >>>>>>>>>>>>> Sent: Tuesday, April 08, 2014 11:11 AM > >>>>>>>>>>>>> To: rsyslog-users > >>>>>>>>>>>>> Subject: Re: [rsyslog] Rsyslog w/ > logstash-elasticsearch-kibana > >>>>>>>>>>>>> server > >>>>>>>>>>>>> > >>>>>>>>>>>>> I use rsyslog to pipe into sec, and then use logstash file > >>>>>>>>>>>>> input > >>>>>>>>>>>>> > >>>>>>>>>>>>> to > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> index. > >>>>>>>>> > >>>>>>>>>> could be done without SEC as well. I don't like delivering > >>>>>>>>>>>>> > >>>>>>>>>>>>> syslog > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> right into logstash. > >>>>>>>>> > >>>>>>>>>> On Apr 8, 2014 11:09 AM, "Sphonic" <[email protected]> > >>>>>>>>>>>>> > >>>>>>>>>>>>> wrote: > >>>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>> > >>>>>>>>> I use rsyslog to send all items to logstash which has a syslog > >>>>>>>>>>>>> > >>>>>>>>>>>>>> listener enabled. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> Sent from my iPhone > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> On 8 Apr 2014, at 18:05, Josh Bitto < > [email protected] > >>>>>>>>>>>>>> > > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> wrote: > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> Hello Everyone, > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> I'm wanting to setup a syslog server that combines the > three > >>>>>>>>>>>>>>> programs > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> listed above with rsyslog. Has anyone had any success > using > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> this? > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> I'm > >>>>>>>>> > >>>>>>>>>> running on a CentOS 6.5 and finding adequate instructions on > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> how > >>>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>> to not only setup all three PLUS rsyslog has been somewhat of a > >>>>>>>>> > >>>>>>>>>> challenge. > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> This issue that I run into is on how to get > >>>>>>>>>>>>>>> logstash/elasticsearch and > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> kibana to talk with rsyslog. Halp meh! Please! > >>>>>>>>>>>>>> > >>>>>>>>>>>>>> > >>>>>>>>>>>>>>> > >>>>>>>>>>>>>>> rsyslog mailing list > >>>>>>>>>>> > >>>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/ > >>>>>>>>>>>> rgerhardsNOTE > >>>>>>>>>>>> WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>>>>>> > >>>>>>>>>>> myriad > >>>> > >>>>> > >>>>>>>>>>>> of > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>> sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>>>>> you > >>>>>>>>>> > >>>>>>>>>>> DON'T LIKE THAT. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> -- > >>>>>>>>>>> Rick Brown > >>>>>>>>>>> Office of Information Technology > >>>>>>>>>>> Georgia Institute of Technology > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> rsyslog mailing list > >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>>> What's up with rsyslog? Follow > https://twitter.com/rgerhardsNOTE > >>>>>>>>>>> > >>>>>>>>>>> WELL: > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> sites > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> LIKE > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> THAT. > >>>>>>>>>> > >>>>>>>>>>> _______________________________________________ > >>>>>>>>>>> rsyslog mailing list > >>>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED > by a > >>>>>>>>>>> > >>>>>>>>>>> myriad > >>>>>>>>>> > >>>>>>>>> > >>>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> you > >>>>>>>>>> > >>>>>>>>> > >>>>>>>> DON'T LIKE THAT. > >>>>>>>>> > >>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> Performance Monitoring * Log Analytics * Search Analytics > >>>>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> rsyslog mailing list > >>>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by > a > >>>>>>>>>> > >>>>>>>>>> myriad > >>>>>>>>> > >>>>>>>> > >>>>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST > if > >>>>>>>>> > >>>>>>>> you > >>>> > >>>>> DON'T LIKE THAT. > >>>>>>>>>> > >>>>>>>>>> _______________________________________________ > >>>>>>>>>> > >>>>>>>>> rsyslog mailing list > >>>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>>> > >>>>>>>> myriad > >>>>> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>>>> > >>>>>>>> you > >>>> > >>>>> DON'T LIKE THAT. > >>>>>>>>> > >>>>>>>>> > >>>>>>>>> > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Performance Monitoring * Log Analytics * Search Analytics > >>>>>>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>>>>>> _______________________________________________ > >>>>>>>> rsyslog mailing list > >>>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>>> > >>>>>>> myriad > >>>>> > >>>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > >>>>>>>> > >>>>>>> you > >>>> > >>>>> DON'T LIKE THAT. > >>>>>>>> > >>>>>>>> _______________________________________________ > >>>>>>>> > >>>>>>> rsyslog mailing list > >>>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>>> http://www.rsyslog.com/professional-services/ > >>>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>>> > >>>>>> myriad > >>>> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>>>> DON'T LIKE THAT. > >>>>>>> > >>>>>>> _______________________________________________ > >>>>>>> > >>>>>> rsyslog mailing list > >>>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>>> http://www.rsyslog.com/professional-services/ > >>>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>>> > >>>>> myriad > >>>> > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>>> DON'T LIKE THAT. > >>>>>> > >>>>>> _______________________________________________ > >>>>> rsyslog mailing list > >>>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>>> http://www.rsyslog.com/professional-services/ > >>>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > >>>>> myriad > >>>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if > you > >>>>> DON'T LIKE THAT. > >>>>> > >>>>> > >>>> > >>>> > >>>> -- > >>>> Performance Monitoring * Log Analytics * Search Analytics > >>>> Solr & Elasticsearch Support * http://sematext.com/ > >>>> _______________________________________________ > >>>> rsyslog mailing list > >>>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>>> http://www.rsyslog.com/professional-services/ > >>>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>>> DON'T LIKE THAT. > >>>> > >>>> _______________________________________________ > >>> rsyslog mailing list > >>> http://lists.adiscon.net/mailman/listinfo/rsyslog > >>> http://www.rsyslog.com/professional-services/ > >>> What's up with rsyslog? Follow https://twitter.com/rgerhards > >>> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a > myriad > >>> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >>> DON'T LIKE THAT. > >>> > >>> _______________________________________________ > >> rsyslog mailing list > >> http://lists.adiscon.net/mailman/listinfo/rsyslog > >> http://www.rsyslog.com/professional-services/ > >> What's up with rsyslog? Follow https://twitter.com/rgerhards > >> NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > >> of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > >> DON'T LIKE THAT. > >> > >> _______________________________________________ > > rsyslog mailing list > > http://lists.adiscon.net/mailman/listinfo/rsyslog > > http://www.rsyslog.com/professional-services/ > > What's up with rsyslog? Follow https://twitter.com/rgerhards > > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > > DON'T LIKE THAT. > > > _______________________________________________ > rsyslog mailing list > http://lists.adiscon.net/mailman/listinfo/rsyslog > http://www.rsyslog.com/professional-services/ > What's up with rsyslog? Follow https://twitter.com/rgerhards > NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad > of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you > DON'T LIKE THAT. > -- Performance Monitoring * Log Analytics * Search Analytics Solr & Elasticsearch Support * http://sematext.com/ _______________________________________________ rsyslog mailing list http://lists.adiscon.net/mailman/listinfo/rsyslog http://www.rsyslog.com/professional-services/ What's up with rsyslog? Follow https://twitter.com/rgerhards NOTE WELL: This is a PUBLIC mailing list, posts are ARCHIVED by a myriad of sites beyond our control. PLEASE UNSUBSCRIBE and DO NOT POST if you DON'T LIKE THAT.
