Re: [heka] question on output buffering

2015-11-01 Thread Timur Batyrshin 
Sure, I’ll try that. This is just not so easy to test as queues grow quite slow.
Although my issue is a bit different to that described — in my case Heka works 
totally fine but doesn’t
truncate/delete the cache file on disk until restart.

Regards,
Timur

On 2 Nov 2015 at 08:59:46, Mathieu Parent (math.par...@gmail.com) wrote:

2015-10-31 18:58 GMT+01:00 Timur Batyrshin :  
> Hi,  
>  
> I’m using TCPOutput to relay data to another Heka instance and I quite often  
> see that a on-disk  
> buffer grows too much.  

Rob Miller has proposed a patch at:  
https://github.com/mozilla-services/heka/issues/1738#issuecomment-150404452  

I've not had time to test it yet. Can you test it?  

> Today’s case:  
>  
> /var/cache/hekad/output_queue/relay_output# ls -l  
> total 9784788  
> -rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log  
> -rw-r--r-- 1 root root 14 Oct 31 17:37 checkpoint.txt  
> root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt  
> 30 10019808619  
> (the difference in numbers could have been caused by me running cat command  
> after some short delay)  
>  
> At the same time I see the metrics emitted from this host by this output as  
> I actually get alerted on them which helped me to find this.  
> After I restart Heka the file shrinks and starts to grow from the very low  
> size.  
>  
>  
> Here is a section from my config file for relay_output:  
>  
> [relay_output]  
> type = "TcpOutput"  
> address = "my.secret.domain:9123"  
> message_matcher = "TRUE"  
>  
> I have no section for buffering for this plugin (nor for any other plugin)  
> and have no other TcpOutputs.  
>  
> The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say  
> that the file should grow  
> no larger that 128Mb by default but in the above case it is already 10Gb and  
> growing.  

This may be a different problem. I had it too, but couldn't reproduce  
it currently.  

> Is it the default different to that specified in docs or do I miss something  
> else in configuration?  
>  
>  
> I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04  
>  
>  
> Thanks,  
> Timur  
>  
> ___  
> Heka mailing list  
> Heka@mozilla.org  
> https://mail.mozilla.org/listinfo/heka  
>  



--  
Mathieu  
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

Re: [heka] question on output buffering

2015-11-01 Thread Mathieu Parent 
2015-10-31 18:58 GMT+01:00 Timur Batyrshin :
> Hi,
>
> I’m using TCPOutput to relay data to another Heka instance and I quite often
> see that a on-disk
> buffer grows too much.

Rob Miller has proposed a patch at:
https://github.com/mozilla-services/heka/issues/1738#issuecomment-150404452

I've not had time to test it yet. Can you test it?

> Today’s case:
>
> /var/cache/hekad/output_queue/relay_output# ls -l
> total 9784788
> -rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log
> -rw-r--r-- 1 root root  14 Oct 31 17:37 checkpoint.txt
> root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt
> 30 10019808619
> (the difference in numbers could have been caused by me running cat command
> after some short delay)
>
> At the same time I see the metrics emitted from this host by this output as
> I actually get alerted on them which helped me to find this.
> After I restart Heka the file shrinks and starts to grow from the very low
> size.
>
>
> Here is a section from my config file for relay_output:
>
> [relay_output]
> type = "TcpOutput"
> address = "my.secret.domain:9123"
> message_matcher = "TRUE"
>
> I have no section for buffering for this plugin (nor for any other plugin)
> and have no other TcpOutputs.
>
> The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say
> that the file should grow
> no larger that 128Mb by default but in the above case it is already 10Gb and
> growing.

This may be a different problem. I had it too, but couldn't reproduce
it currently.

> Is it the default different to that specified in docs or do I miss something
> else in configuration?
>
>
> I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04
>
>
> Thanks,
> Timur
>
> ___
> Heka mailing list
> Heka@mozilla.org
> https://mail.mozilla.org/listinfo/heka
>



-- 
Mathieu
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka

[heka] question on output buffering

2015-10-31 Thread Timur Batyrshin 
Hi,

I’m using TCPOutput to relay data to another Heka instance and I quite often 
see that a on-disk
buffer grows too much.

Today’s case:

/var/cache/hekad/output_queue/relay_output# ls -l
total 9784788
-rw-r--r-- 1 root root 10019573004 Oct 31 17:37 30.log
-rw-r--r-- 1 root root          14 Oct 31 17:37 checkpoint.txt
root@r-jp-cms:/var/cache/hekad/output_queue/relay_output# cat checkpoint.txt
30 10019808619
(the difference in numbers could have been caused by me running cat command 
after some short delay)

At the same time I see the metrics emitted from this host by this output as I 
actually get alerted on them which helped me to find this.
After I restart Heka the file shrinks and starts to grow from the very low size.


Here is a section from my config file for relay_output:

[relay_output]
type = "TcpOutput"
address = "my.secret.domain:9123"
message_matcher = "TRUE"

I have no section for buffering for this plugin (nor for any other plugin) and 
have no other TcpOutputs.

The docs at http://hekad.readthedocs.org/en/v0.10.0b1/buffering.html say that 
the file should grow
no larger that 128Mb by default but in the above case it is already 10Gb and 
growing.

Is it the default different to that specified in docs or do I miss something 
else in configuration?


I’m running Heka 0.10.0b from GH releases page on Ubuntu 12.04


Thanks,
Timur
___
Heka mailing list
Heka@mozilla.org
https://mail.mozilla.org/listinfo/heka