Hi Sure.. I ll check and let you know
Thank you On Tue, 31 Mar 2015 at 06:51, Rob Miller <[email protected]> wrote: > Okay, sorry for the delay... I think I've figured this out. Are you able > to build Heka from source? If so, maybe you could try out the > `logstreamer_rotate_fix` branch that I've just pushed and see if it > resolves your issue? > > Thanks, > > -r > > On 03/29/2015 10:59 AM, rajive kumar wrote: > > Hi > > > > Thank you... > > > > Best Regards > > RajiveKumar > > > > On 28 March 2015 at 12:14, Rob Miller <[email protected] > > <mailto:[email protected]>> wrote: > > > > I'm able to reproduce the issue, I'll work on getting a fix on the > > versions/0.9 branch over the next couple of days. > > > > -r > > > > > > On 03/27/2015 09:33 PM, rajive kumar wrote: > > > > Hi Guys > > > > Any idea on this.. > > > > Best Regards, > > > > On 25 March 2015 at 11:29, rajive kumar <[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > wrote: > > > > Hi > > > > Please find the link for the toml file and sh script which > > i have > > used to test. > > > > http://pastebin.ca/2963610 - toml file > > > > http://pastebin.ca/2963598 - sh script for simulate the log > > rotation. > > > > Steps I have followed > > > > 1. Run the script for some time by increasing the no of > > records > > 2. Start the heka while script is running. > > 3. After some time get the line count of generated files > > and heka > > output files ( Heka out put file have more than > > generated file > > count i.e its containing duplicate values ) > > > > Thank you in advance for the help. > > > > Best Regards, > > > > > > On 25 March 2015 at 05:40, Rob Miller <[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > wrote: > > > > If you're using the translation map as I've specified > > and you're > > still seeing the same file (or files) being read twice, > > then > > there might be a bug. I can't quite tell from what > > you're saying > > whether or not you've applied the change that I > > recommended. Can > > you paste your entire Heka config in a pastebin > > somewhere so I > > can look at it? Or at the very least the entire > > LogstreamerInput > > section, including the translation map subsection if > you've > > applied it. > > > > Thanks, > > > > -r > > > > > > On 03/24/2015 12:07 PM, rajive kumar wrote: > > > > > > ---------- Forwarded message ---------- > > From: *rajive kumar* <[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>> <mailto:[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>>>> > > Date: 25 March 2015 at 00:35 > > Subject: Re: [heka] Logstreamer Input issue with > > current log > > and rolling > > file > > To: [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > > > > > Hi > > > > Thank you.. What you have mention is correct.I have > > tested > > that too. But > > my problem is when current file rolling Heka is > > reading all > > the rolled > > file again. For static file its working perfectly. > > > > For example > > > > In server log files line count is : 3015 > > But heka file reading log side is : 23065 > > > > file format which I have used is > > > > [smpp_logs] > > type = "LogstreamerInput" > > log_directory = > > "/media/rajive/Office/tmp/____test/sender/smpp" > > file_match= > > 'HS_06_SMPP_FULL_CDR(_(?P<Seq>____\d*))?' > > priority =["Seq"] > > [smpp_logs.translation.Seq] > > missing = 9999999999999 > > > > > > > > Please find the attachment for more details. Also I > > have > > started freshly > > i.e before start i have removed all the files. > > > > > > On 25 March 2015 at 00:08, Rob Miller > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>> > > <mailto:[email protected] > > <mailto:[email protected]> <mailto:[email protected] > > <mailto:[email protected]>>>> > > wrote: > > > > You need to use what we call a "translation > > map" to map > > a missing > > `Seq` value to the highest possible value. The > > following is an > > example of what should work: > > > > [smpp_logs] > > type = "LogstreamerInput" > > log_directory = "/path/to/smpp/logs" > > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' > > priority =["Seq"] > > [smpp_logs.translation.Seq] > > missing = 9999999999999999 > > > > Also be aware that you can test out the order > that > > LogstreamerInput > > will load log files before you actually run > > Heka by > > pointing the > > heka-logstreamer utility at your configuration: > > hekad.readthedocs.org/en/v0.9.______0/pluginconfig/__ > logstreamer.____html#__verifying-settings > > <http://hekad.readthedocs.org/en/v0.9.____0/pluginconfig/ > logstreamer.____html#verifying-settings> > > > > <http://hekad.readthedocs.org/__en/v0.9.__0/pluginconfig/__ > logstreamer.__html#verifying-__settings > > <http://hekad.readthedocs.org/en/v0.9.__0/pluginconfig/ > logstreamer.__html#verifying-settings>> > > > > > > <http://hekad.readthedocs.org/____en/v0.9.0/pluginconfig/___ > _logstreamer.html#verifying-____settings > > <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ > logstreamer.html#verifying-__settings> > > > > > > <http://hekad.readthedocs.org/__en/v0.9.0/pluginconfig/__ > logstreamer.html#verifying-__settings > > <http://hekad.readthedocs.org/en/v0.9.0/pluginconfig/ > logstreamer.html#verifying-settings>>> > > > > -r > > > > > > On 03/24/2015 11:26 AM, rajive kumar wrote: > > > > Hi Guys > > > > Any idea on this > > > > I using LogstreamerInput to read my > > ongoing logs. > > Currently my > > logs has > > been rolling for every 15h min. When using > > heka its > > reading > > duplicate > > values. I.e its reading same file again > > and again. > > > > Following is the file pattern > > > > smpp_cdr (This the current log file) > > smpp_cdr_2015032416150055 > > smpp_cdr_2015032416300056 > > smpp_cdr_2015032416450057 > > > > I using following pattern to identify the > > file pattern > > file_match= 'smpp_cdr(_(?P<Seq>\d*))?' > > priority =["Seq"] > > This is read the rolled files > > perfectly but its > > reading the > > current > > log file (smpp_cdr) first. Also when > > current file > > gets rolled heka > > reading all the file again. Please advice > > / help on > > this. > > > > Thank you > > > > On 20 March 2015 at 18:44, rajive kumar > > <[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>>> > > <mailto:[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>> <mailto:[email protected] > > <mailto:[email protected]> > > <mailto:[email protected] > > <mailto:[email protected]>>>>__> wrote: > > > > Hi > > > > Currently I am using Logstreamer > input to > > process another > > applications current log file. Its > > working > > fine but I have > > an issue > > when file rolling -> its reading the > file > > twice. Please help on > > this. Below the file structure I have > > in that > > system > > > > *trans_log* <- current log > > *150320001_trans_log * > > *150320002_trans_log > > * > > *150320003_trans_log > > * > > *150320004_trans_log > > * > > *150320005_trans_log > > * > > *150320006_trans_log* > > > > Note my files are rolling in every 15 > > mins and > > I want to > > read the > > previous logs and current log while > its > > printing(i.e logging). > > > > > > Best Regards, > > > > > > > > > > > > _____________________________________________________ > > Heka mailing list > > [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > <mailto:[email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>>> > > https://mail.mozilla.org/______listinfo/heka > > <https://mail.mozilla.org/____listinfo/heka> > > <https://mail.mozilla.org/____listinfo/heka > > <https://mail.mozilla.org/__listinfo/heka>> > > > > <https://mail.mozilla.org/____listinfo/heka > > <https://mail.mozilla.org/__listinfo/heka> > > <https://mail.mozilla.org/__listinfo/heka > > <https://mail.mozilla.org/listinfo/heka>>> > > > > > > > > > > > > > > ___________________________________________________ > > Heka mailing list > > [email protected] <mailto:[email protected]> > > <mailto:[email protected] <mailto:[email protected]>> > > https://mail.mozilla.org/____listinfo/heka > > <https://mail.mozilla.org/__listinfo/heka> > > <https://mail.mozilla.org/__listinfo/heka > > <https://mail.mozilla.org/listinfo/heka>> > > > > > > > > > > > > > > _________________________________________________ > > Heka mailing list > > [email protected] <mailto:[email protected]> > > https://mail.mozilla.org/__listinfo/heka > > <https://mail.mozilla.org/listinfo/heka> > > > > > >
_______________________________________________ Heka mailing list [email protected] https://mail.mozilla.org/listinfo/heka
