Hi It seems working but when doing a long run its giving "too many open files error" and missing some data too.. :(
Best regards, On 31 March 2015 at 07:33, rajive kumar <[email protected]> wrote: > Hi > > Sure.. I ll check and let you know > > Thank you > > On Tue, 31 Mar 2015 at 06:51, Rob Miller <[email protected]> wrote: > >> Okay, sorry for the delay... I think I've figured this out. Are you able >> to build Heka from source? If so, maybe you could try out the >> `logstreamer_rotate_fix` branch that I've just pushed and see if it >> resolves your issue? >> >> Thanks, >> >> -r >> >> On 03/29/2015 10:59 AM, rajive kumar wrote: >> > Hi >> > >> > Thank you... >> > >> > Best Regards >> > RajiveKumar >> > >> > On 28 March 2015 at 12:14, Rob Miller <[email protected] >> > <mailto:[email protected]>> wrote: >> > >> > I'm able to reproduce the issue, I'll work on getting a fix on the >> > versions/0.9 branch over the next couple of days. >> > >> > -r >> > >> > >> > On 03/27/2015 09:33 PM, rajive kumar wrote: >> > >> > Hi Guys >> > >> > Any idea on this.. >> > >> > Best Regards, >> > >> > On 25 March 2015 at 11:29, rajive kumar <[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> wrote: >> > >> > Hi >> > >> > Please find the link for the toml file and sh script which >> > i have >> > used to test. >> > >> > http://pastebin.ca/2963610 - toml file >> > >> > http://pastebin.ca/2963598 - sh script for simulate the log >> > rotation. >> > >> > Steps I have followed >> > >> > 1. Run the script for some time by increasing the no of >> > records >> > 2. Start the heka while script is running. >> > 3. After some time get the line count of generated files >> > and heka >> > output files ( Heka out put file have more than >> > generated file >> > count i.e its containing duplicate values ) >> > >> > Thank you in advance for the help. >> > >> > Best Regards, >> > >> > >> > On 25 March 2015 at 05:40, Rob Miller <[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> > wrote: >> > >> > If you're using the translation map as I've specified >> > and you're >> > still seeing the same file (or files) being read twice, >> > then >> > there might be a bug. I can't quite tell from what >> > you're saying >> > whether or not you've applied the change that I >> > recommended. Can >> > you paste your entire Heka config in a pastebin >> > somewhere so I >> > can look at it? Or at the very least the entire >> > LogstreamerInput >> > section, including the translation map subsection if >> you've >> > applied it. >> > >> > Thanks, >> > >> > -r >> > >> > >> > On 03/24/2015 12:07 PM, rajive kumar wrote: >> > >> > >> > ---------- Forwarded message ---------- >> > From: *rajive kumar* <[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>> <mailto:[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>>>> >> > Date: 25 March 2015 at 00:35 >> > Subject: Re: [heka] Logstreamer Input issue with >> > current log >> > and rolling >> > file >> > To: [email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> > >> > >> > Hi >> > >> > Thank you.. What you have mention is correct.I have >> > tested >> > that too. But >> > my problem is when current file rolling Heka is >> > reading all >> > the rolled >> > file again. For static file its working perfectly. >> > >> > For example >> > >> > In server log files line count is : 3015 >> > But heka file reading log side is : 23065 >> > >> > file format which I have used is >> > >> > [smpp_logs] >> > type = "LogstreamerInput" >> > log_directory = >> > "/media/rajive/Office/tmp/____test/sender/smpp" >> > file_match= >> > 'HS_06_SMPP_FULL_CDR(_(?P<Seq>____\d*))?' >> > priority =["Seq"] >> > [smpp_logs.translation.Seq] >> > missing = 9999999999999 >> > >> > >> > >> > Please find the attachment for more details. Also I >> > have >> > started freshly >> > i.e before start i have removed all the files. >> > >> > >> > On 25 March 2015 at 00:08, Rob Miller >> > <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>> >> > <mailto:[email protected] >> > <mailto:[email protected]> <mailto:[email protected] >> > <mailto:[email protected]>>>> >> > wrote: >> > >> > You need to use what we call a "translation >> > map" to map >> > a missing >> > `Seq` value to the highest possible value. The >> > following is an >> > example of what should work: >> > >> > [smpp_logs] >> > type = "LogstreamerInput" >> > log_directory = "/path/to/smpp/logs" >> > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >> > priority =["Seq"] >> > [smpp_logs.translation.Seq] >> > missing = 9999999999999999 >> > >> > Also be aware that you can test out the order >> that >> > LogstreamerInput >> > will load log files before you actually run >> > Heka by >> > pointing the >> > heka-logstreamer utility at your >> configuration: >> > hekad.readthedocs.org/en/v0.9.______0/pluginconfig/__logst >> reamer.____html#__verifying-settings >> > <http://hekad.readthedocs.org/en/v0.9.____0/pluginconfig/ >> logstreamer.____html#verifying-settings> >> > >> > <http://hekad.readthedocs.org/__en/v0.9.__0/pluginconfig/__ >> logstreamer.__html#verifying-__settings >> > <http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/l >> ogstreamer.__html#verifying-settings>> >> > >> > >> > <http://hekad.readthedocs.org/____en/v0.9.0/pluginconfig/___ >> _logstreamer.html#verifying-____settings >> > <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ >> logstreamer.html#verifying-__settings> >> > >> > >> > <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ >> logstreamer.html#verifying-__settings >> > <http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/log >> streamer.html#verifying-settings>>> >> > >> > -r >> > >> > >> > On 03/24/2015 11:26 AM, rajive kumar wrote: >> > >> > Hi Guys >> > >> > Any idea on this >> > >> > I using LogstreamerInput to read my >> > ongoing logs. >> > Currently my >> > logs has >> > been rolling for every 15h min. When using >> > heka its >> > reading >> > duplicate >> > values. I.e its reading same file again >> > and again. >> > >> > Following is the file pattern >> > >> > smpp_cdr (This the current log file) >> > smpp_cdr_2015032416150055 >> > smpp_cdr_2015032416300056 >> > smpp_cdr_2015032416450057 >> > >> > I using following pattern to identify the >> > file pattern >> > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' >> > priority =["Seq"] >> > This is read the rolled files >> > perfectly but its >> > reading the >> > current >> > log file (smpp_cdr) first. Also when >> > current file >> > gets rolled heka >> > reading all the file again. Please advice >> > / help on >> > this. >> > >> > Thank you >> > >> > On 20 March 2015 at 18:44, rajive kumar >> > <[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>>> >> > <mailto:[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>> <mailto:[email protected] >> > <mailto:[email protected]> >> > <mailto:[email protected] >> > <mailto:[email protected]>>>>__> wrote: >> > >> > Hi >> > >> > Currently I am using Logstreamer >> input to >> > process another >> > applications current log file. Its >> > working >> > fine but I have >> > an issue >> > when file rolling -> its reading the >> file >> > twice. Please help on >> > this. Below the file structure I have >> > in that >> > system >> > >> > *trans_log* <- current log >> > *150320001_trans_log * >> > *150320002_trans_log >> > * >> > *150320003_trans_log >> > * >> > *150320004_trans_log >> > * >> > *150320005_trans_log >> > * >> > *150320006_trans_log* >> > >> > Note my files are rolling in every 15 >> > mins and >> > I want to >> > read the >> > previous logs and current log while >> its >> > printing(i.e logging). >> > >> > >> > Best Regards, >> > >> > >> > >> > >> > >> > _____________________________________________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> > <mailto:[email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>>> >> > https://mail.mozilla.org/______listinfo/heka >> > <https://mail.mozilla.org/____listinfo/heka> >> > <https://mail.mozilla.org/____listinfo/heka >> > <https://mail.mozilla.org/__listinfo/heka>> >> > >> > <https://mail.mozilla.org/____listinfo/heka >> > <https://mail.mozilla.org/__listinfo/heka> >> > <https://mail.mozilla.org/__listinfo/heka >> > <https://mail.mozilla.org/listinfo/heka>>> >> > >> > >> > >> > >> > >> > >> > ______________________________ >> _____________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> > <mailto:[email protected] <mailto:[email protected]>> >> > https://mail.mozilla.org/____listinfo/heka >> > <https://mail.mozilla.org/__listinfo/heka> >> > <https://mail.mozilla.org/__listinfo/heka >> > <https://mail.mozilla.org/listinfo/heka>> >> > >> > >> > >> > >> > >> > >> > _________________________________________________ >> > Heka mailing list >> > [email protected] <mailto:[email protected]> >> > https://mail.mozilla.org/__listinfo/heka >> > <https://mail.mozilla.org/listinfo/heka> >> > >> > >> >>
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
