On 21 October 2016 at 17:02, Jason Wilder <[email protected]> wrote:
> > > On Friday, October 21, 2016 at 8:17:06 AM UTC-6, Guillaume Berthomé wrote: >> >> And are "0" values stored in the TSM files with the same "system" >> (~3bytes at the end) ? >> >> > If they are integers and just long runs of the same value, (e.g. 0), they > can be run length encoded as well. The max block size is 1000 values and > we'd use 11 bytes for all those values which is much less than 3 bytes. > That is great! Does it also apply for float values? This is also common to have long runs of the same float value (e.g. 0.0) > > >> Thank you >> >> Le vendredi 21 octobre 2016 11:17:41 UTC+2, Guillaume Berthomé a écrit : >>> >>> @Sean, >>> >>> Can I use the rule you gave me for integers ? >>> >>> Le jeudi 20 octobre 2016 08:53:34 UTC+2, Guillaume Berthomé a écrit : >>>> >>>> Thank you everyone ! >>>> >>>> Le mercredi 19 octobre 2016 21:30:54 UTC+2, Sean Beckett a écrit : >>>>> >>>>> Mathias, I agree that irregular timestamps do lead to more space taken >>>>> on disk. However, that number still falls under 3 bytes per numeric value, >>>>> even with irregular nanosecond timestamps. Both what Jason and I are >>>>> saying >>>>> is true; they are not mutually exclusive. >>>>> >>>>> > The 2 or 3 bytes footprint is usually not achievable if you store >>>>> your timestamps with milli, micro or nanosecond precision as the delta >>>>> between consecutive timestamps will vary and will therefore not compress >>>>> as >>>>> well. >>>>> >>>>> That statement is not true, at least not in any of our testing >>>>> experience. It would be fair to say that 2-3 bytes per value is not always >>>>> achievable, as some rare edge cases can lead to larger footprint on disk, >>>>> but to say that it is *usually* not achievable is simply incorrect. >>>>> >>>>> On Wed, Oct 19, 2016 at 1:10 PM, Mathias Herberts < >>>>> [email protected]> wrote: >>>>> >>>>>> It seems Jason just posted the exact same findings as me, maybe check >>>>>> with him directly, I persist in my saying, if your timestamps are not >>>>>> regularly spaced then compression efficiency will decrease. >>>>>> >>>>>> On Wednesday, October 19, 2016 at 8:33:56 PM UTC+2, Sean Beckett >>>>>> wrote: >>>>>>> >>>>>>> Mathias, I question your findings on this. We've run extensive >>>>>>> testing with nanosecond timestamps and routinely get < 3 bytes per >>>>>>> recorded >>>>>>> numeric value. >>>>>>> >>>>>>> Please note that this figure is only for fully compacted data, e.g. >>>>>>> cold shards. Shards currently hot for writes will be much less compact. >>>>>>> Over time the steady state of the system will approach ~2.5 bytes per >>>>>>> numeric field. >>>>>>> >>>>>>> On Wed, Oct 19, 2016 at 5:45 AM, Mathias Herberts < >>>>>>> [email protected]> wrote: >>>>>>> >>>>>>>> Efficiency of compression depends on different factors, including >>>>>>>> interval between timestamps, resolution of said timestamps, volatility >>>>>>>> and >>>>>>>> type of values. >>>>>>>> >>>>>>>> The 2 or 3 bytes footprint is usually not achievable if you store >>>>>>>> your timestamps with milli, micro or nanosecond precision as the delta >>>>>>>> between consecutive timestamps will vary and will therefore not >>>>>>>> compress as >>>>>>>> well. >>>>>>>> >>>>>>>> >>>>>>>> On Wednesday, October 19, 2016 at 1:04:44 PM UTC+2, Guillaume >>>>>>>> Berthomé wrote: >>>>>>>>> >>>>>>>>> Thank you for your answer. >>>>>>>>> >>>>>>>>> So let's admit that I have 2 metrics collected every 15 seconds >>>>>>>>> during 24h >>>>>>>>> >>>>>>>>> - cpu with 20 values >>>>>>>>> - mem with 10 values >>>>>>>>> >>>>>>>>> The total size of stored data will be 3 bytes * 8 metrics per >>>>>>>>> minutes * 1440 minutes ? >>>>>>>>> >>>>>>>>> >>>>>>>>> >>>>>>>>> Le mardi 18 octobre 2016 18:57:53 UTC+2, Sean Beckett a écrit : >>>>>>>>>> >>>>>>>>>> After final compactions, InfluxDB uses between 2 and 3 bytes per >>>>>>>>>> field value stored. If your metrics are all numbers, then the total >>>>>>>>>> steady-state size will be a bit more than 3 bytes * (metrics per >>>>>>>>>> minute) * >>>>>>>>>> minutes of data stored. >>>>>>>>>> >>>>>>>>>> See http://docs.influxdata.com/influxdb/v1.0/concepts/storag >>>>>>>>>> e_engine/ and http://docs.influxdata.com/inf >>>>>>>>>> luxdb/v1.0/guides/hardware_sizing/#how-much-storage-do-i-need for >>>>>>>>>> details. >>>>>>>>>> >>>>>>>>>> On Tue, Oct 18, 2016 at 2:51 AM, <[email protected]> wrote: >>>>>>>>>> >>>>>>>>>>> Hi everybody, >>>>>>>>>>> >>>>>>>>>>> Is there any way to calculate the database's size from the >>>>>>>>>>> number measurements done per minute ? I mean predict the databse >>>>>>>>>>> behavior >>>>>>>>>>> (wal files, tsm files ...) >>>>>>>>>>> >>>>>>>>>>> Thank you. >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>>> Remember to include the version number! >>>>>>>>>>> --- >>>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>>> Google Groups "InfluxData" group. >>>>>>>>>>> To unsubscribe from this group and stop receiving emails from >>>>>>>>>>> it, send an email to [email protected]. >>>>>>>>>>> To post to this group, send email to [email protected]. >>>>>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>>>>> To view this discussion on the web visit >>>>>>>>>>> https://groups.google.com/d/msgid/influxdb/ee32b8d0-f9bf-40d >>>>>>>>>>> f-baf1-e77fa93cddf5%40googlegroups.com. >>>>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>>> Sean Beckett >>>>>>>>>> Director of Support and Professional Services >>>>>>>>>> InfluxDB >>>>>>>>>> >>>>>>>>> -- >>>>>>>> Remember to include the version number! >>>>>>>> --- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "InfluxData" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To post to this group, send email to [email protected]. >>>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-439 >>>>>>>> 0-a18c-8d6b67da26f0%40googlegroups.com >>>>>>>> <https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>> >>>>>>> >>>>>>> >>>>>>> -- >>>>>>> Sean Beckett >>>>>>> Director of Support and Professional Services >>>>>>> InfluxDB >>>>>>> >>>>>> -- >>>>>> Remember to include the version number! >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "InfluxData" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a >>>>>> 4-8f43-2c360ccafb3b%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Beckett >>>>> Director of Support and Professional Services >>>>> InfluxDB >>>>> >>>> -- > Remember to include the version number! > --- > You received this message because you are subscribed to the Google Groups > "InfluxData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/d9d3b396-32d6-4697-b3aa-2c5fee48909c%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/d9d3b396-32d6-4697-b3aa-2c5fee48909c%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CAHop23do%2BVQT6DC9vGagLr68So_qLAmGgFApBhP%2Bc1e9Q2vmtg%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
