How often should compaction run after 24 hours and how long should it take?
We are running 1.0.0-beta1 with just over 1 million series. Server has 16 cores and 32GB of RAM. Writing less than 5000 points per second. For the first 24 hours, query response time is less than a second. Level 1/2 compaction takes place every 20 minutes and completes in seconds. Everything is fast. Then, after the 24 hours configured in "compact-full-write-cold-duration", full compaction kicks in and runs continuously and performance takes a noticeable hit. Queries can take several seconds to complete. The compaction appears to be ongoing after 24 hours - running every minute for more than a minute and constantly writing at approx 100MB/s to disk (RAID array not SSD). Is this normal? If so, are there config parameters that can be tweaked? or will a move to SSDs resolve the problem? Here's an example where the influx process was restarted at about 11:20am on 2016/06/14 - after 24 hours, you can see in the log file the switch from infrequent level 1/2 to ongoing full compaction before and after 11:20am. % grep "files into" influxdb.log [tsm1] 2016/06/15 09:10:24 compacted level 1 group 0 of 2 files into 1 files in 12.953107399s [tsm1] 2016/06/15 09:10:44 compacted level 2 group 0 of 2 files into 1 files in 20.321327593s [tsm1] 2016/06/15 09:28:29 compacted level 1 group 0 of 2 files into 1 files in 14.123922529s [tsm1] 2016/06/15 09:47:39 compacted level 1 group 0 of 2 files into 1 files in 13.128642782s [tsm1] 2016/06/15 09:47:57 compacted level 2 group 0 of 2 files into 1 files in 17.759666145s [tsm1] 2016/06/15 10:06:34 compacted level 1 group 0 of 2 files into 1 files in 11.840285989s [tsm1] 2016/06/15 10:26:02 compacted level 1 group 0 of 2 files into 1 files in 12.742921103s [tsm1] 2016/06/15 10:26:21 compacted level 2 group 0 of 2 files into 1 files in 18.407723192s [tsm1] 2016/06/15 10:43:52 compacted level 1 group 0 of 2 files into 1 files in 11.848901735s [tsm1] 2016/06/15 11:03:10 compacted level 1 group 0 of 2 files into 1 files in 13.017833455s [tsm1] 2016/06/15 11:03:28 compacted level 2 group 0 of 2 files into 1 files in 18.295207874s [tsm1] 2016/06/15 11:05:06 compacted level 3 group 0 of 4 files into 1 files in 1m36.54105056s [tsm1] 2016/06/15 11:06:19 compacted full 2 files into 1 files in 1m13.027526301s [tsm1] 2016/06/15 11:22:07 compacted level 1 group 0 of 2 files into 1 files in 12.6243752s [tsm1] 2016/06/15 11:30:51 compacted full 2 files into 1 files in 479.64141ms [tsm1] 2016/06/15 11:32:42 compacted full 2 files into 2 files in 1m52.257509758s [tsm1] 2016/06/15 11:32:55 compacted full 2 files into 2 files in 2m4.496843095s [tsm1] 2016/06/15 11:33:53 compacted full 2 files into 2 files in 1m9.375094851s [tsm1] 2016/06/15 11:34:00 compacted full 2 files into 2 files in 1m3.399309336s [tsm1] 2016/06/15 11:34:56 compacted full 2 files into 2 files in 1m2.215633678s [tsm1] 2016/06/15 11:35:08 compacted full 2 files into 2 files in 1m7.2966692s [tsm1] 2016/06/15 11:36:00 compacted full 2 files into 2 files in 1m2.651181158s [tsm1] 2016/06/15 11:36:15 compacted full 2 files into 2 files in 1m6.527896164s [tsm1] 2016/06/15 11:37:05 compacted full 2 files into 2 files in 1m4.387296019s [tsm1] 2016/06/15 11:37:31 compacted full 2 files into 2 files in 1m14.593487299s -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/5662f7cc-7514-4105-b175-eb110da6dc74%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
