Hi Rob Thanks.. I have posted on that issue
https://github.com/mozilla-services/heka/issues/1452#issuecomment-88241367 Best Regards, On 1 April 2015 at 02:03, Rob Miller <[email protected]> wrote: > Okay... I've opened an issue to track this: https://github.com/mozilla- > services/heka/issues/1452 > > I'd like to move the conversation there, if you don't mind, so we don't > swamp the list with back and forth details. I'd love it if you could > comment on that issue, letting me know how many separate log streams your > Heka instance is handling, and how long it runs before you hit the file > descriptor limit, to get a sense for the scope of the problem. > > Thanks! > > -r > > On 03/31/2015 12:41 AM, rajive kumar wrote: > >> Hi >> >> It seems working but when doing a long run its giving "too many open >> files error" and missing some data too.. :( >> >> Best regards, >> >> On 31 March 2015 at 07:33, rajive kumar <[email protected] >> <mailto:[email protected]>> wrote: >> >> Hi >> >> Sure.. I ll check and let you know >> >> Thank you >> >> On Tue, 31 Mar 2015 at 06:51, Rob Miller <[email protected] >> <mailto:[email protected]>> wrote: >> >> Okay, sorry for the delay... I think I've figured this out. Are >> you able to build Heka from source? If so, maybe you could try >> out the `logstreamer_rotate_fix` branch that I've just pushed >> and see if it resolves your issue? >> >> Thanks, >> >> -r >> >> On 03/29/2015 10:59 AM, rajive kumar wrote: >> > Hi >> > >> > Thank you... >> > >> > Best Regards >> > RajiveKumar >> > >> > On 28 March 2015 at 12:14, Rob Miller <[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> > >> > I'm able to reproduce the issue, I'll work on getting a >> fix on the >> > versions/0.9 branch over the next couple of days. >> > >> > -r >> > >> > >> > On 03/27/2015 09:33 PM, rajive kumar wrote: >> > >> > Hi Guys >> > >> > Any idea on this.. >> > >> > Best Regards, >> > >> > On 25 March 2015 at 11:29, rajive kumar >> <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>>> wrote: >> > >> > Hi >> > >> > Please find the link for the toml file and sh >> script which >> > i have >> > used to test. >> > >> > http://pastebin.ca/2963610 - toml file >> > >> > http://pastebin.ca/2963598 - sh script for simulate the log >> > rotation. >> > >> > Steps I have followed >> > >> > 1. Run the script for some time by increasing >> the no of >> > records >> > 2. Start the heka while script is running. >> > 3. After some time get the line count of >> generated files >> > and heka >> > output files ( Heka out put file have more >> than >> > generated file >> > count i.e its containing duplicate values ) >> > >> > Thank you in advance for the help. >> > >> > Best Regards, >> > >> > >> > On 25 March 2015 at 05:40, Rob Miller >> <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>>> >> > wrote: >> > >> > If you're using the translation map as I've >> specified >> > and you're >> > still seeing the same file (or files) being >> read twice, >> > then >> > there might be a bug. I can't quite tell >> from what >> > you're saying >> > whether or not you've applied the change that >> I >> > recommended. Can >> > you paste your entire Heka config in a >> pastebin >> > somewhere so I >> > can look at it? Or at the very least the >> entire >> > LogstreamerInput >> > section, including the translation map >> subsection if you've >> > applied it. >> > >> > Thanks, >> > >> > -r >> > >> > >> > On 03/24/2015 12:07 PM, rajive kumar wrote: >> > >> > >> > ---------- Forwarded message ---------- >> > From: *rajive kumar* >> <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>__>__> >> >> > Date: 25 March 2015 at 00:35 >> > Subject: Re: [heka] Logstreamer Input >> issue with >> > current log >> > and rolling >> > file >> > To: [email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>> >> > >> > >> > Hi >> > >> > Thank you.. What you have mention is >> correct.I have >> > tested >> > that too. But >> > my problem is when current file rolling >> Heka is >> > reading all >> > the rolled >> > file again. For static file its working >> perfectly. >> > >> > For example >> > >> > In server log files line count is : 3015 >> > But heka file reading log side is : 23065 >> > >> > file format which I have used is >> > >> > [smpp_logs] >> > type = "LogstreamerInput" >> > log_directory = >> > >> "/media/rajive/Office/tmp/________test/sender/smpp" >> > file_match= >> > 'HS_06_SMPP_FULL_CDR(_(?P<__Seq>______\d*))?' >> > priority =["Seq"] >> > [smpp_logs.translation.Seq] >> > missing = 9999999999999 >> > >> > >> > >> > Please find the attachment for more >> details. Also I >> > have >> > started freshly >> > i.e before start i have removed all the >> files. >> > >> > >> > On 25 March 2015 at 00:08, Rob Miller >> > <[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>> <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>__>__> >> >> > wrote: >> > >> > You need to use what we call a >> "translation >> > map" to map >> > a missing >> > `Seq` value to the highest possible >> value. The >> > following is an >> > example of what should work: >> > >> > [smpp_logs] >> > type = "LogstreamerInput" >> > log_directory = "/path/to/smpp/logs" >> > file_match= >> 'smpp_cdr(_(?P<Seq>\d*))?' >> > priority =["Seq"] >> > [smpp_logs.translation.Seq] >> > missing = 9999999999999999 >> > >> > Also be aware that you can test out >> the order that >> > LogstreamerInput >> > will load log files before you >> actually run >> > Heka by >> > pointing the >> > heka-logstreamer utility at your >> configuration: >> > >> hekad.readthedocs.org/en/v0.__9.________0/pluginconfig/____ >> logst__reamer.____html#____verifying-__settings >> <http://hekad.readthedocs.org/en/v0.9.______0/pluginconfig/_ >> _logstreamer.____html#__verifying-settings> >> > >> <http://hekad.readthedocs.__org/__en/v0.9.____0/__ >> pluginconfig/__logstreamer.______html#__verifying-settings < >> http://hekad.readthedocs.org/en/v0.9.____0/pluginconfig/ >> logstreamer.____html#verifying-settings>> >> > >> > >> <http://hekad.readthedocs.__org/____en/v0.9.__0/__ >> pluginconfig/____logstreamer.____html#verifying-____settings < >> http://hekad.readthedocs.org/__en/v0.9.__0/pluginconfig/__ >> logstreamer.__html#verifying-__settings> >> > >> <http://hekad.readthedocs.__org/__en/v0.9.__0/ >> pluginconfig/__l__ogstreamer.__html#verifying-__se__ttings < >> http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/ >> logstreamer.__html#verifying-settings>>> >> > >> > >> > >> <http://hekad.readthedocs.__org/______en/v0.9.0/__ >> pluginconfig/______logstreamer.__html#verifying-______settings < >> http://hekad.readthedocs.org/____en/v0.9.0/pluginconfig/___ >> _logstreamer.html#verifying-____settings> >> > >> <http://hekad.readthedocs.__org/____en/v0.9.0/ >> pluginconfig/______logstreamer.html#verifying-______settings < >> http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ >> logstreamer.html#verifying-__settings>> >> > >> > >> > >> <http://hekad.readthedocs.__org/____en/v0.9.0/ >> pluginconfig/______logstreamer.html#verifying-______settings < >> http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ >> logstreamer.html#verifying-__settings> >> > >> <http://hekad.readthedocs.__org/__en/v0.9.0/pluginconfig/_ >> _log__streamer.html#verifying-__settin__gs <http://hekad.readthedocs.org/ >> en/v0.9.0/pluginconfig/logstreamer.html#verifying-settings>>>> >> >> > >> > -r >> > >> > >> > On 03/24/2015 11:26 AM, rajive >> kumar wrote: >> > >> > Hi Guys >> > >> > Any idea on this >> > >> > I using LogstreamerInput to read >> my >> > ongoing logs. >> > Currently my >> > logs has >> > been rolling for every 15h min. >> When using >> > heka its >> > reading >> > duplicate >> > values. I.e its reading same >> file again >> > and again. >> > >> > Following is the file pattern >> > >> > smpp_cdr (This the current >> log file) >> > smpp_cdr_2015032416150055 >> > smpp_cdr_2015032416300056 >> > smpp_cdr_2015032416450057 >> > >> > I using following pattern to >> identify the >> > file pattern >> > file_match= >> 'smpp_cdr(_(?P<Seq>\d*))?' >> > priority =["Seq"] >> > This is read the rolled files >> > perfectly but its >> > reading the >> > current >> > log file (smpp_cdr) first. Also >> when >> > current file >> > gets rolled heka >> > reading all the file again. >> Please advice >> > / help on >> > this. >> > >> > Thank you >> > >> > On 20 March 2015 at 18:44, >> rajive kumar >> > <[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>__> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>> <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] <mailto: >> [email protected]>> >> > <mailto:[email protected] >> <mailto:[email protected]> >> > <mailto:[email protected] >> <mailto:[email protected]>>>__>__>__> wrote: >> > >> > Hi >> > >> > Currently I am using >> Logstreamer input to >> > process another >> > applications current log >> file. Its >> > working >> > fine but I have >> > an issue >> > when file rolling -> its >> reading the file >> > twice. Please help on >> > this. Below the file >> structure I have >> > in that >> > system >> > >> > *trans_log* <- current log >> > *150320001_trans_log * >> > *150320002_trans_log >> > * >> > *150320003_trans_log >> > * >> > *150320004_trans_log >> > * >> > *150320005_trans_log >> > * >> > *150320006_trans_log* >> > >> > Note my files are rolling >> in every 15 >> > mins and >> > I want to >> > read the >> > previous logs and current >> log while its >> > printing(i.e logging). >> > >> > >> > Best Regards, >> > >> > >> > >> > >> > >> > >> _________________________________________________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> > <mailto:[email protected] >> <mailto:[email protected]> <mailto:[email protected] >> <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>>> >> > https://mail.mozilla.org/__________listinfo/heka >> <https://mail.mozilla.org/______listinfo/heka> >> > <https://mail.mozilla.org/________listinfo/heka >> <https://mail.mozilla.org/____listinfo/heka>> >> > >> <https://mail.mozilla.org/________listinfo/heka >> <https://mail.mozilla.org/____listinfo/heka> >> > <https://mail.mozilla.org/____l__istinfo/heka >> <https://mail.mozilla.org/__listinfo/heka>>> >> > >> > <https://mail.mozilla.org/________listinfo/heka >> <https://mail.mozilla.org/____listinfo/heka> >> > <https://mail.mozilla.org/____l__istinfo/heka >> <https://mail.mozilla.org/__listinfo/heka>> >> > >> <https://mail.mozilla.org/____li__stinfo/heka >> <https://mail.mozilla.org/__listinfo/heka> >> > <https://mail.mozilla.org/__lis__tinfo/heka >> <https://mail.mozilla.org/listinfo/heka>>>> >> > >> > >> > >> > >> > >> > >> > >> _______________________________________________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>>> >> > https://mail.mozilla.org/________listinfo/heka >> <https://mail.mozilla.org/____listinfo/heka> >> > <https://mail.mozilla.org/____l__istinfo/heka >> <https://mail.mozilla.org/__listinfo/heka>> >> > >> <https://mail.mozilla.org/____li__stinfo/heka >> <https://mail.mozilla.org/__listinfo/heka> >> > <https://mail.mozilla.org/__lis__tinfo/heka >> <https://mail.mozilla.org/listinfo/heka>>> >> > >> > >> > >> > >> > >> > >> > _____________________________________________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> <mailto:[email protected] <mailto:[email protected]>> >> > https://mail.mozilla.org/____li__stinfo/heka >> <https://mail.mozilla.org/__listinfo/heka> >> > <https://mail.mozilla.org/__lis__tinfo/heka >> <https://mail.mozilla.org/listinfo/heka>> >> > >> > >> >> >> >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
