And are "0" values stored in the TSM files with the same "system" (~3bytes at the end) ?
Thank you Le vendredi 21 octobre 2016 11:17:41 UTC+2, Guillaume Berthomé a écrit : > > @Sean, > > Can I use the rule you gave me for integers ? > > Le jeudi 20 octobre 2016 08:53:34 UTC+2, Guillaume Berthomé a écrit : >> >> Thank you everyone ! >> >> Le mercredi 19 octobre 2016 21:30:54 UTC+2, Sean Beckett a écrit : >>> >>> Mathias, I agree that irregular timestamps do lead to more space taken >>> on disk. However, that number still falls under 3 bytes per numeric value, >>> even with irregular nanosecond timestamps. Both what Jason and I are saying >>> is true; they are not mutually exclusive. >>> >>> > The 2 or 3 bytes footprint is usually not achievable if you store >>> your timestamps with milli, micro or nanosecond precision as the delta >>> between consecutive timestamps will vary and will therefore not compress as >>> well. >>> >>> That statement is not true, at least not in any of our testing >>> experience. It would be fair to say that 2-3 bytes per value is not always >>> achievable, as some rare edge cases can lead to larger footprint on disk, >>> but to say that it is *usually* not achievable is simply incorrect. >>> >>> On Wed, Oct 19, 2016 at 1:10 PM, Mathias Herberts <[email protected] >>> > wrote: >>> >>>> It seems Jason just posted the exact same findings as me, maybe check >>>> with him directly, I persist in my saying, if your timestamps are not >>>> regularly spaced then compression efficiency will decrease. >>>> >>>> On Wednesday, October 19, 2016 at 8:33:56 PM UTC+2, Sean Beckett wrote: >>>>> >>>>> Mathias, I question your findings on this. We've run extensive testing >>>>> with nanosecond timestamps and routinely get < 3 bytes per recorded >>>>> numeric >>>>> value. >>>>> >>>>> Please note that this figure is only for fully compacted data, e.g. >>>>> cold shards. Shards currently hot for writes will be much less compact. >>>>> Over time the steady state of the system will approach ~2.5 bytes per >>>>> numeric field. >>>>> >>>>> On Wed, Oct 19, 2016 at 5:45 AM, Mathias Herberts < >>>>> [email protected]> wrote: >>>>> >>>>>> Efficiency of compression depends on different factors, including >>>>>> interval between timestamps, resolution of said timestamps, volatility >>>>>> and >>>>>> type of values. >>>>>> >>>>>> The 2 or 3 bytes footprint is usually not achievable if you store >>>>>> your timestamps with milli, micro or nanosecond precision as the delta >>>>>> between consecutive timestamps will vary and will therefore not compress >>>>>> as >>>>>> well. >>>>>> >>>>>> >>>>>> On Wednesday, October 19, 2016 at 1:04:44 PM UTC+2, Guillaume >>>>>> Berthomé wrote: >>>>>>> >>>>>>> Thank you for your answer. >>>>>>> >>>>>>> So let's admit that I have 2 metrics collected every 15 seconds >>>>>>> during 24h >>>>>>> >>>>>>> - cpu with 20 values >>>>>>> - mem with 10 values >>>>>>> >>>>>>> The total size of stored data will be 3 bytes * 8 metrics per >>>>>>> minutes * 1440 minutes ? >>>>>>> >>>>>>> >>>>>>> >>>>>>> Le mardi 18 octobre 2016 18:57:53 UTC+2, Sean Beckett a écrit : >>>>>>>> >>>>>>>> After final compactions, InfluxDB uses between 2 and 3 bytes per >>>>>>>> field value stored. If your metrics are all numbers, then the total >>>>>>>> steady-state size will be a bit more than 3 bytes * (metrics per >>>>>>>> minute) * >>>>>>>> minutes of data stored. >>>>>>>> >>>>>>>> See >>>>>>>> http://docs.influxdata.com/influxdb/v1.0/concepts/storage_engine/ >>>>>>>> and >>>>>>>> http://docs.influxdata.com/influxdb/v1.0/guides/hardware_sizing/#how-much-storage-do-i-need >>>>>>>> for >>>>>>>> details. >>>>>>>> >>>>>>>> On Tue, Oct 18, 2016 at 2:51 AM, <[email protected]> wrote: >>>>>>>> >>>>>>>>> Hi everybody, >>>>>>>>> >>>>>>>>> Is there any way to calculate the database's size from the number >>>>>>>>> measurements done per minute ? I mean predict the databse behavior >>>>>>>>> (wal >>>>>>>>> files, tsm files ...) >>>>>>>>> >>>>>>>>> Thank you. >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Remember to include the version number! >>>>>>>>> --- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "InfluxData" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/influxdb/ee32b8d0-f9bf-40df-baf1-e77fa93cddf5%40googlegroups.com >>>>>>>>> . >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -- >>>>>>>> Sean Beckett >>>>>>>> Director of Support and Professional Services >>>>>>>> InfluxDB >>>>>>>> >>>>>>> -- >>>>>> Remember to include the version number! >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "InfluxData" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Beckett >>>>> Director of Support and Professional Services >>>>> InfluxDB >>>>> >>>> -- >>>> Remember to include the version number! >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "InfluxData" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/influxdb. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Sean Beckett >>> Director of Support and Professional Services >>> InfluxDB >>> >> -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/d7c8b6e8-e490-4e61-98ac-7e2e1e07f74e%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
