On Friday, October 21, 2016 at 8:17:06 AM UTC-6, Guillaume Berthomé wrote: > > And are "0" values stored in the TSM files with the same "system" (~3bytes > at the end) ? > > If they are integers and just long runs of the same value, (e.g. 0), they can be run length encoded as well. The max block size is 1000 values and we'd use 11 bytes for all those values which is much less than 3 bytes.
> Thank you > > Le vendredi 21 octobre 2016 11:17:41 UTC+2, Guillaume Berthomé a écrit : >> >> @Sean, >> >> Can I use the rule you gave me for integers ? >> >> Le jeudi 20 octobre 2016 08:53:34 UTC+2, Guillaume Berthomé a écrit : >>> >>> Thank you everyone ! >>> >>> Le mercredi 19 octobre 2016 21:30:54 UTC+2, Sean Beckett a écrit : >>>> >>>> Mathias, I agree that irregular timestamps do lead to more space taken >>>> on disk. However, that number still falls under 3 bytes per numeric value, >>>> even with irregular nanosecond timestamps. Both what Jason and I are >>>> saying >>>> is true; they are not mutually exclusive. >>>> >>>> > The 2 or 3 bytes footprint is usually not achievable if you store >>>> your timestamps with milli, micro or nanosecond precision as the delta >>>> between consecutive timestamps will vary and will therefore not compress >>>> as >>>> well. >>>> >>>> That statement is not true, at least not in any of our testing >>>> experience. It would be fair to say that 2-3 bytes per value is not always >>>> achievable, as some rare edge cases can lead to larger footprint on disk, >>>> but to say that it is *usually* not achievable is simply incorrect. >>>> >>>> On Wed, Oct 19, 2016 at 1:10 PM, Mathias Herberts < >>>> [email protected]> wrote: >>>> >>>>> It seems Jason just posted the exact same findings as me, maybe check >>>>> with him directly, I persist in my saying, if your timestamps are not >>>>> regularly spaced then compression efficiency will decrease. >>>>> >>>>> On Wednesday, October 19, 2016 at 8:33:56 PM UTC+2, Sean Beckett wrote: >>>>>> >>>>>> Mathias, I question your findings on this. We've run extensive >>>>>> testing with nanosecond timestamps and routinely get < 3 bytes per >>>>>> recorded >>>>>> numeric value. >>>>>> >>>>>> Please note that this figure is only for fully compacted data, e.g. >>>>>> cold shards. Shards currently hot for writes will be much less compact. >>>>>> Over time the steady state of the system will approach ~2.5 bytes per >>>>>> numeric field. >>>>>> >>>>>> On Wed, Oct 19, 2016 at 5:45 AM, Mathias Herberts < >>>>>> [email protected]> wrote: >>>>>> >>>>>>> Efficiency of compression depends on different factors, including >>>>>>> interval between timestamps, resolution of said timestamps, volatility >>>>>>> and >>>>>>> type of values. >>>>>>> >>>>>>> The 2 or 3 bytes footprint is usually not achievable if you store >>>>>>> your timestamps with milli, micro or nanosecond precision as the delta >>>>>>> between consecutive timestamps will vary and will therefore not >>>>>>> compress as >>>>>>> well. >>>>>>> >>>>>>> >>>>>>> On Wednesday, October 19, 2016 at 1:04:44 PM UTC+2, Guillaume >>>>>>> Berthomé wrote: >>>>>>>> >>>>>>>> Thank you for your answer. >>>>>>>> >>>>>>>> So let's admit that I have 2 metrics collected every 15 seconds >>>>>>>> during 24h >>>>>>>> >>>>>>>> - cpu with 20 values >>>>>>>> - mem with 10 values >>>>>>>> >>>>>>>> The total size of stored data will be 3 bytes * 8 metrics per >>>>>>>> minutes * 1440 minutes ? >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Le mardi 18 octobre 2016 18:57:53 UTC+2, Sean Beckett a écrit : >>>>>>>>> >>>>>>>>> After final compactions, InfluxDB uses between 2 and 3 bytes per >>>>>>>>> field value stored. If your metrics are all numbers, then the total >>>>>>>>> steady-state size will be a bit more than 3 bytes * (metrics per >>>>>>>>> minute) * >>>>>>>>> minutes of data stored. >>>>>>>>> >>>>>>>>> See >>>>>>>>> http://docs.influxdata.com/influxdb/v1.0/concepts/storage_engine/ >>>>>>>>> and >>>>>>>>> http://docs.influxdata.com/influxdb/v1.0/guides/hardware_sizing/#how-much-storage-do-i-need >>>>>>>>> for >>>>>>>>> details. >>>>>>>>> >>>>>>>>> On Tue, Oct 18, 2016 at 2:51 AM, <[email protected]> wrote: >>>>>>>>> >>>>>>>>>> Hi everybody, >>>>>>>>>> >>>>>>>>>> Is there any way to calculate the database's size from the number >>>>>>>>>> measurements done per minute ? I mean predict the databse behavior >>>>>>>>>> (wal >>>>>>>>>> files, tsm files ...) >>>>>>>>>> >>>>>>>>>> Thank you. >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Remember to include the version number! >>>>>>>>>> --- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "InfluxData" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/influxdb/ee32b8d0-f9bf-40df-baf1-e77fa93cddf5%40googlegroups.com >>>>>>>>>> . >>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> Sean Beckett >>>>>>>>> Director of Support and Professional Services >>>>>>>>> InfluxDB >>>>>>>>> >>>>>>>> -- >>>>>>> Remember to include the version number! >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "InfluxData" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sean Beckett >>>>>> Director of Support and Professional Services >>>>>> InfluxDB >>>>>> >>>>> -- >>>>> Remember to include the version number! >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxData" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>> -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/d9d3b396-32d6-4697-b3aa-2c5fee48909c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
