Re: [heka] question on output buffering

[Timur Batyrshin]

Sure, I’ll try that. This is just not so easy to test as queues grow quite slow.
Although my issue is a bit different to that described — in my case Heka works 
totally fine but doesn’t
truncate/delete the cache file on disk until restart.

Regards,
Timur

On 2 Nov 2015 at 08:59:46, Mathieu Parent (math.par...@gmail.com) wrote:

2015-10-31 18:58 GMT+01:00 Timur Batyrshin :  
> Hi,  
>  
> I’m using TCPOutput to relay data to another Heka instance and I quite often  
> see that a on-disk  
> buffer grows too much.  

Rob Miller has proposed a patch at:  
https://github.com/mozilla-services/heka/issues/1738#issuecomment-150404452  

I've not had time to test it yet. Can you test it?  

> Today’s case:  
>  
> /var/cache/hekad/output_queue/relay_output# ls -l  
> total 9784788  
> -rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log  
> -rw-r--r-- 1 root root 14 Oct 31 17:37 checkpoint.txt  
> root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt  
> 30 10019808619  
> (the difference in numbers could have been caused by me running cat command  
> after some short delay)  
>  
> At the same time I see the metrics emitted from this host by this output as  
> I actually get alerted on them which helped me to find this.  
> After I restart Heka the file shrinks and starts to grow from the very low  
> size.  
>  
>  
> Here is a section from my config file for relay_output:  
>  
> [relay_output]  
> type = "TcpOutput"  
> address = "my.secret.domain:9123"  
> message_matcher = "TRUE"  
>  
> I have no section for buffering for this plugin (nor for any other plugin)  
> and have no other TcpOutputs.  
>  
> The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say  
> that the file should grow  
> no larger that 128Mb by default but in the above case it is already 10Gb and  
> growing.  

This may be a different problem. I had it too, but couldn't reproduce  
it currently.  

> Is it the default different to that specified in docs or do I miss something  
> else in configuration?  
>  
>  
> I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04  
>  
>  
> Thanks,  
> Timur  
>  
> ___  
> Heka mailing list  
> Heka@mozilla.org  
> https://mail.mozilla.org/listinfo/heka  
>  



--  
Mathieu  
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] question on output buffering

[Mathieu Parent]

2015-10-31 18:58 GMT+01:00 Timur Batyrshin :
> Hi,
>
> I’m using TCPOutput to relay data to another Heka instance and I quite often
> see that a on-disk
> buffer grows too much.

Rob Miller has proposed a patch at:
https://github.com/mozilla-services/heka/issues/1738#issuecomment-150404452

I've not had time to test it yet. Can you test it?

> Today’s case:
>
> /var/cache/hekad/output_queue/relay_output# ls -l
> total 9784788
> -rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log
> -rw-r--r-- 1 root root  14 Oct 31 17:37 checkpoint.txt
> root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt
> 30 10019808619
> (the difference in numbers could have been caused by me running cat command
> after some short delay)
>
> At the same time I see the metrics emitted from this host by this output as
> I actually get alerted on them which helped me to find this.
> After I restart Heka the file shrinks and starts to grow from the very low
> size.
>
>
> Here is a section from my config file for relay_output:
>
> [relay_output]
> type = "TcpOutput"
> address = "my.secret.domain:9123"
> message_matcher = "TRUE"
>
> I have no section for buffering for this plugin (nor for any other plugin)
> and have no other TcpOutputs.
>
> The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say
> that the file should grow
> no larger that 128Mb by default but in the above case it is already 10Gb and
> growing.

This may be a different problem. I had it too, but couldn't reproduce
it currently.

> Is it the default different to that specified in docs or do I miss something
> else in configuration?
>
>
> I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04
>
>
> Thanks,
> Timur
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>



-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka