Hi. When you say "see how the file behaves" I'm not quite sure what you mean by that... As I mentioned earlier, it's not that events do not appear at all but instead, the RATE at which they come decreases, so how can I measure the events rate in a file? I thought that there's another way that I can test this: I'll write a quick-and-dirty program that will send an event to the ELK via TCP every 12ms which should result in events rate of about 5,000 events per minute and I'll let you know if the events rate continues to drop or not...
Thanks, Yuval. On Tuesday, February 10, 2015, Itamar Syn-Hershko <[email protected]> wrote: > I'd start by using logstash with input tcp and output fs and see how the > file behaves. Same for the fs inputs - see how their files behave. And take > it from there. > > -- > > Itamar Syn-Hershko > http://code972.com | @synhershko <https://twitter.com/synhershko> > Freelance Developer & Consultant > Lucene.NET committer and PMC member > > On Tue, Feb 10, 2015 at 7:47 PM, Yuval Khalifa <[email protected] > <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: > >> Great! How can I check that? >> >> >> On Tuesday, February 10, 2015, Itamar Syn-Hershko <[email protected] >> <javascript:_e(%7B%7D,'cvml','[email protected]');>> wrote: >> >>> The graphic you sent suggests the issue is with logstash - since the >>> @timestamp field is being populated by logstash and is the one that is used >>> to display the date histogram graphics in Kibana. I would start there. I.e. >>> maybe SecurityOnion buffers writes etc, and then to check the logstash >>> shipper process stats. >>> >>> -- >>> >>> Itamar Syn-Hershko >>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>> Freelance Developer & Consultant >>> Lucene.NET committer and PMC member >>> >>> On Tue, Feb 10, 2015 at 7:07 PM, Yuval Khalifa <[email protected]> >>> wrote: >>> >>>> Hi. >>>> >>>> Absolutely (but since that in the past I also worked at the helpdesk >>>> dept. I certainly understand why it is important to ask those "Are you sure >>>> it's plugged in?" questions...). One of the logs is comming from >>>> SecurityOnion which logs (via bro-conn) all the connections so it must be >>>> sending data 24x7x365. >>>> >>>> Thanks for the quick reply, >>>> Yuval. >>>> >>>> On Tuesday, February 10, 2015, Itamar Syn-Hershko <[email protected]> >>>> wrote: >>>> >>>>> Are you sure your logs are generated linearly without bursts? >>>>> >>>>> -- >>>>> >>>>> Itamar Syn-Hershko >>>>> http://code972.com | @synhershko <https://twitter.com/synhershko> >>>>> Freelance Developer & Consultant >>>>> Lucene.NET committer and PMC member >>>>> >>>>> On Tue, Feb 10, 2015 at 6:29 PM, Yuval Khalifa <[email protected]> >>>>> wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> We just installed an ELK server and configured the logstash >>>>>> configuration to match the data that we send to it and until last month >>>>>> it >>>>>> seems to be working fine but since then we see very strange behavior in >>>>>> the >>>>>> Kibana, the event over time histogram shows the event rate at the normal >>>>>> level for about a half an hour, then drops to about 20% of the normal >>>>>> rate >>>>>> and then it continues to drop slowly for about two hours and then stops >>>>>> and >>>>>> after a minute or two it returns to normal for the next half an hour or >>>>>> so >>>>>> and the same behavior repeats. Needless to say that both the >>>>>> /var/log/logstash and /var/log/elasticsearch both show nothing since the >>>>>> service started and by using tcpdump we can verify that events keep >>>>>> coming >>>>>> in at the same rate all time. I attached our logstash configuration, the >>>>>> /var/logstash/logstash.log, the /var/log/elasticsearch/clustername.log >>>>>> and >>>>>> a screenshot of our Kibana with no filter applied so that you can see the >>>>>> weird behavior that we see. >>>>>> >>>>>> Is there someone/somewhere that we can turn to to get some help on >>>>>> the subject? >>>>>> >>>>>> >>>>>> Thanks a lot, >>>>>> Yuval. >>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "elasticsearch" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/elasticsearch/c2e5a524-1ba6-4dc9-9fc3-d206d8f82717%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> -- >>>>> You received this message because you are subscribed to a topic in the >>>>> Google Groups "elasticsearch" group. >>>>> To unsubscribe from this topic, visit >>>>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe >>>>> . >>>>> To unsubscribe from this group and all its topics, send an email to >>>>> [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com >>>>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsRoNmJ__QdLnB6NYLhoDVaD9CR1RNkC_9_c%2Boaqccqww%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> -- >>>> >>>> בברכה, >>>> >>>> *יובל כליפא* >>>> >>>> CTO >>>> תחום מערכות מידע | מגדל סוכנויות. >>>> נייד: 052-3336098 >>>> משרד: 03-7966565 >>>> פקס: 03-7976565 >>>> בלוג: http://www.artifex.co.il >>>> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >>>> >>>> *[image: תיאור: תיאור: cid:[email protected]]* >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "elasticsearch" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A9-UtP5GJLORnVW%2BMowbB%2B0ZV%3DeDFMfN5u3xFPD2Zv5FQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "elasticsearch" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. >>> To unsubscribe from this group and all its topics, send an email to >>> [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com >>> <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZsqU9Oimw5g7jEpwOFDPiKB_aNP3hhaWmuFrL1Po_OAZw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> -- >> >> בברכה, >> >> *יובל כליפא* >> >> CTO >> תחום מערכות מידע | מגדל סוכנויות. >> נייד: 052-3336098 >> משרד: 03-7966565 >> פקס: 03-7976565 >> בלוג: http://www.artifex.co.il >> <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> >> >> *[image: תיאור: תיאור: cid:[email protected]]* >> >> -- >> You received this message because you are subscribed to the Google Groups >> "elasticsearch" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] >> <javascript:_e(%7B%7D,'cvml','elasticsearch%[email protected]');> >> . >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com >> <https://groups.google.com/d/msgid/elasticsearch/CADtR2A8nvUiJE40Qssfhu%3DA3zG4bHOPgjL7adM-zr0xJw6R8zA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > You received this message because you are subscribed to a topic in the > Google Groups "elasticsearch" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/elasticsearch/cw7zEVTy09M/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected] > <javascript:_e(%7B%7D,'cvml','elasticsearch%[email protected]');> > . > To view this discussion on the web visit > https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com > <https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZuM4F3ZAaBadQPm8m4DGyZtnzdOOqtSM%3Dq_9BsWrbmPTg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- בברכה, *יובל כליפא* CTO תחום מערכות מידע | מגדל סוכנויות. נייד: 052-3336098 משרד: 03-7966565 פקס: 03-7976565 בלוג: http://www.artifex.co.il <https://owa.mvs.co.il/OWA/redir.aspx?C=2843559e53a94386b1211d26cb20f8ef&URL=http%3a%2f%2fwww.artifex.co.il%2f> *[image: תיאור: תיאור: cid:[email protected]]* -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CADtR2A9JB3FHGHqW%2BuPDzVmuW7vOeRNPb8Bgz%3De4aY%3DXZNDtwg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
