> I did some tests, normally collecting and pushing stats takes < 0.02s, but > it includes time needed to send data to carbon. If remote carbon server > has > high load (creating a lot of new metric files for example) than this time > jumps up a lot (up to 0.5s in my case), so we might be sending everything > in dedicated thread, but it seems that the time remote carbon server needs > to receive our data adds a delay to our loop. > > "carbon now ts: N" - timestamp used in metrics pushed to carbon > "push done, took N" - how long carbon_push_stats() call took, in > microseconds > > [DEBUG] carbon now ts: 1362217035 > [DEBUG] push done, took 50693 [1362217035708550] > [DEBUG] carbon now ts: 1362217036 > [DEBUG] push done, took 83977 [1362217036793970] > [DEBUG] carbon now ts: 1362217037 > [DEBUG] push done, took 415337 [1362217038210110] > [DEBUG] carbon now ts: 1362217039 > [DEBUG] push done, took 47645 [1362217039261166] > > So it isn't hard to make carbon "skip a beat" with such low frequency, and > it seems that at least part of reasons are out of uWSGI control (carbon > server response time). > To make it work better I think that we would need to have a thread that > only does one thing - sleeps for $(carbon freq) seconds and than creates a > new thread that will collect current metrics and push data to carbon, > after > it's done or it had any problems, it dies (?). >
makes sense, but only for high-load scenarios. we can add an option for it. Or we can make a queue (managed by a single thread) on which metrics are put and constantly sent. In both cases the number of threads and the size of the queue must have a fixed size (otherwise if the carbon server is stuck we can take a huge amount of memory). Could it be a way to compute the size of the queue to an amount ensuring us to throw away only metrics no-more needed ? -- Roberto De Ioris http://unbit.it _______________________________________________ uWSGI mailing list [email protected] http://lists.unbit.it/cgi-bin/mailman/listinfo/uwsgi
