Okay... I've opened an issue to track this: https://github.com/mozilla-services/heka/issues/1452
I'd like to move the conversation there, if you don't mind, so we don't swamp the list with back and forth details. I'd love it if you could comment on that issue, letting me know how many separate log streams your Heka instance is handling, and how long it runs before you hit the file descriptor limit, to get a sense for the scope of the problem. Thanks! -r On 03/31/2015 12:41 AM, rajive kumar wrote:
Hi It seems working but when doing a long run its giving "too many open files error" and missing some data too.. :( Best regards, On 31 March 2015 at 07:33, rajive kumar <[email protected] <mailto:[email protected]>> wrote: Hi Sure.. I ll check and let you know Thank you On Tue, 31 Mar 2015 at 06:51, Rob Miller <[email protected] <mailto:[email protected]>> wrote: Okay, sorry for the delay... I think I've figured this out. Are you able to build Heka from source? If so, maybe you could try out the `logstreamer_rotate_fix` branch that I've just pushed and see if it resolves your issue? Thanks, -r On 03/29/2015 10:59 AM, rajive kumar wrote: > Hi > > Thank you... > > Best Regards > RajiveKumar > > On 28 March 2015 at 12:14, Rob Miller <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> wrote: > > I'm able to reproduce the issue, I'll work on getting a fix on the > versions/0.9 branch over the next couple of days. > > -r > > > On 03/27/2015 09:33 PM, rajive kumar wrote: > > Hi Guys > > Any idea on this.. > > Best Regards, > > On 25 March 2015 at 11:29, rajive kumar <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> wrote: > > Hi > > Please find the link for the toml file and sh script which > i have > used to test. > > http://pastebin.ca/2963610 - toml file > > http://pastebin.ca/2963598 - sh script for simulate the log > rotation. > > Steps I have followed > > 1. Run the script for some time by increasing the no of > records > 2. Start the heka while script is running. > 3. After some time get the line count of generated files > and heka > output files ( Heka out put file have more than > generated file > count i.e its containing duplicate values ) > > Thank you in advance for the help. > > Best Regards, > > > On 25 March 2015 at 05:40, Rob Miller <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> > wrote: > > If you're using the translation map as I've specified > and you're > still seeing the same file (or files) being read twice, > then > there might be a bug. I can't quite tell from what > you're saying > whether or not you've applied the change that I > recommended. Can > you paste your entire Heka config in a pastebin > somewhere so I > can look at it? Or at the very least the entire > LogstreamerInput > section, including the translation map subsection if you've > applied it. > > Thanks, > > -r > > > On 03/24/2015 12:07 PM, rajive kumar wrote: > > > ---------- Forwarded message ---------- > From: *rajive kumar* <[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>__>__> > Date: 25 March 2015 at 00:35 > Subject: Re: [heka] Logstreamer Input issue with > current log > and rolling > file > To: [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> > > > Hi > > Thank you.. What you have mention is correct.I have > tested > that too. But > my problem is when current file rolling Heka is > reading all > the rolled > file again. For static file its working perfectly. > > For example > > In server log files line count is : 3015 > But heka file reading log side is : 23065 > > file format which I have used is > > [smpp_logs] > type = "LogstreamerInput" > log_directory = > "/media/rajive/Office/tmp/________test/sender/smpp" > file_match= > 'HS_06_SMPP_FULL_CDR(_(?P<__Seq>______\d*))?' > priority =["Seq"] > [smpp_logs.translation.Seq] > missing = 9999999999999 > > > > Please find the attachment for more details. Also I > have > started freshly > i.e before start i have removed all the files. > > > On 25 March 2015 at 00:08, Rob Miller > <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>__>__> > wrote: > > You need to use what we call a "translation > map" to map > a missing > `Seq` value to the highest possible value. The > following is an > example of what should work: > > [smpp_logs] > type = "LogstreamerInput" > log_directory = "/path/to/smpp/logs" > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' > priority =["Seq"] > [smpp_logs.translation.Seq] > missing = 9999999999999999 > > Also be aware that you can test out the order that > LogstreamerInput > will load log files before you actually run > Heka by > pointing the > heka-logstreamer utility at your configuration: > hekad.readthedocs.org/en/v0.__9.________0/pluginconfig/____logst__reamer.____html#____verifying-__settings <http://hekad.readthedocs.org/en/v0.9.______0/pluginconfig/__logstreamer.____html#__verifying-settings> > <http://hekad.readthedocs.__org/__en/v0.9.____0/__pluginconfig/__logstreamer.______html#__verifying-settings <http://hekad.readthedocs.org/en/v0.9.____0/pluginconfig/logstreamer.____html#verifying-settings>> > > <http://hekad.readthedocs.__org/____en/v0.9.__0/__pluginconfig/____logstreamer.____html#verifying-____settings <http://hekad.readthedocs.org/__en/v0.9.__0/pluginconfig/__logstreamer.__html#verifying-__settings> > <http://hekad.readthedocs.__org/__en/v0.9.__0/pluginconfig/__l__ogstreamer.__html#verifying-__se__ttings <http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/logstreamer.__html#verifying-settings>>> > > > <http://hekad.readthedocs.__org/______en/v0.9.0/__pluginconfig/______logstreamer.__html#verifying-______settings <http://hekad.readthedocs.org/____en/v0.9.0/pluginconfig/____logstreamer.html#verifying-____settings> > <http://hekad.readthedocs.__org/____en/v0.9.0/pluginconfig/______logstreamer.html#verifying-______settings <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__logstreamer.html#verifying-__settings>> > > > <http://hekad.readthedocs.__org/____en/v0.9.0/pluginconfig/______logstreamer.html#verifying-______settings <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__logstreamer.html#verifying-__settings> > <http://hekad.readthedocs.__org/__en/v0.9.0/pluginconfig/__log__streamer.html#verifying-__settin__gs <http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/logstreamer.html#verifying-settings>>>> > > -r > > > On 03/24/2015 11:26 AM, rajive kumar wrote: > > Hi Guys > > Any idea on this > > I using LogstreamerInput to read my > ongoing logs. > Currently my > logs has > been rolling for every 15h min. When using > heka its > reading > duplicate > values. I.e its reading same file again > and again. > > Following is the file pattern > > smpp_cdr (This the current log file) > smpp_cdr_2015032416150055 > smpp_cdr_2015032416300056 > smpp_cdr_2015032416450057 > > I using following pattern to identify the > file pattern > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' > priority =["Seq"] > This is read the rolled files > perfectly but its > reading the > current > log file (smpp_cdr) first. Also when > current file > gets rolled heka > reading all the file again. Please advice > / help on > this. > > Thank you > > On 20 March 2015 at 18:44, rajive kumar > <[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>__> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>> <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> > <mailto:[email protected] <mailto:[email protected]>>>__>__>__> wrote: > > Hi > > Currently I am using Logstreamer input to > process another > applications current log file. Its > working > fine but I have > an issue > when file rolling -> its reading the file > twice. Please help on > this. Below the file structure I have > in that > system > > *trans_log* <- current log > *150320001_trans_log * > *150320002_trans_log > * > *150320003_trans_log > * > *150320004_trans_log > * > *150320005_trans_log > * > *150320006_trans_log* > > Note my files are rolling in every 15 > mins and > I want to > read the > previous logs and current log while its > printing(i.e logging). > > > Best Regards, > > > > > > _________________________________________________________ > Heka mailing list > [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>>> > https://mail.mozilla.org/__________listinfo/heka <https://mail.mozilla.org/______listinfo/heka> > <https://mail.mozilla.org/________listinfo/heka <https://mail.mozilla.org/____listinfo/heka>> > <https://mail.mozilla.org/________listinfo/heka <https://mail.mozilla.org/____listinfo/heka> > <https://mail.mozilla.org/____l__istinfo/heka <https://mail.mozilla.org/__listinfo/heka>>> > > <https://mail.mozilla.org/________listinfo/heka <https://mail.mozilla.org/____listinfo/heka> > <https://mail.mozilla.org/____l__istinfo/heka <https://mail.mozilla.org/__listinfo/heka>> > <https://mail.mozilla.org/____li__stinfo/heka <https://mail.mozilla.org/__listinfo/heka> > <https://mail.mozilla.org/__lis__tinfo/heka <https://mail.mozilla.org/listinfo/heka>>>> > > > > > > > _______________________________________________________ > Heka mailing list > [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > <mailto:[email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>>> > https://mail.mozilla.org/________listinfo/heka <https://mail.mozilla.org/____listinfo/heka> > <https://mail.mozilla.org/____l__istinfo/heka <https://mail.mozilla.org/__listinfo/heka>> > <https://mail.mozilla.org/____li__stinfo/heka <https://mail.mozilla.org/__listinfo/heka> > <https://mail.mozilla.org/__lis__tinfo/heka <https://mail.mozilla.org/listinfo/heka>>> > > > > > > > _____________________________________________________ > Heka mailing list > [email protected] <mailto:[email protected]> <mailto:[email protected] <mailto:[email protected]>> > https://mail.mozilla.org/____li__stinfo/heka <https://mail.mozilla.org/__listinfo/heka> > <https://mail.mozilla.org/__lis__tinfo/heka <https://mail.mozilla.org/listinfo/heka>> > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
