Mathias, I agree that irregular timestamps do lead to more space taken on disk. However, that number still falls under 3 bytes per numeric value, even with irregular nanosecond timestamps. Both what Jason and I are saying is true; they are not mutually exclusive.
> The 2 or 3 bytes footprint is usually not achievable if you store your timestamps with milli, micro or nanosecond precision as the delta between consecutive timestamps will vary and will therefore not compress as well. That statement is not true, at least not in any of our testing experience. It would be fair to say that 2-3 bytes per value is not always achievable, as some rare edge cases can lead to larger footprint on disk, but to say that it is *usually* not achievable is simply incorrect. On Wed, Oct 19, 2016 at 1:10 PM, Mathias Herberts < [email protected]> wrote: > It seems Jason just posted the exact same findings as me, maybe check with > him directly, I persist in my saying, if your timestamps are not regularly > spaced then compression efficiency will decrease. > > On Wednesday, October 19, 2016 at 8:33:56 PM UTC+2, Sean Beckett wrote: >> >> Mathias, I question your findings on this. We've run extensive testing >> with nanosecond timestamps and routinely get < 3 bytes per recorded numeric >> value. >> >> Please note that this figure is only for fully compacted data, e.g. cold >> shards. Shards currently hot for writes will be much less compact. Over >> time the steady state of the system will approach ~2.5 bytes per numeric >> field. >> >> On Wed, Oct 19, 2016 at 5:45 AM, Mathias Herberts <[email protected]> >> wrote: >> >>> Efficiency of compression depends on different factors, including >>> interval between timestamps, resolution of said timestamps, volatility and >>> type of values. >>> >>> The 2 or 3 bytes footprint is usually not achievable if you store your >>> timestamps with milli, micro or nanosecond precision as the delta between >>> consecutive timestamps will vary and will therefore not compress as well. >>> >>> >>> On Wednesday, October 19, 2016 at 1:04:44 PM UTC+2, Guillaume Berthomé >>> wrote: >>>> >>>> Thank you for your answer. >>>> >>>> So let's admit that I have 2 metrics collected every 15 seconds during >>>> 24h >>>> >>>> - cpu with 20 values >>>> - mem with 10 values >>>> >>>> The total size of stored data will be 3 bytes * 8 metrics per minutes * >>>> 1440 minutes ? >>>> >>>> >>>> >>>> Le mardi 18 octobre 2016 18:57:53 UTC+2, Sean Beckett a écrit : >>>>> >>>>> After final compactions, InfluxDB uses between 2 and 3 bytes per field >>>>> value stored. If your metrics are all numbers, then the total steady-state >>>>> size will be a bit more than 3 bytes * (metrics per minute) * minutes of >>>>> data stored. >>>>> >>>>> See http://docs.influxdata.com/influxdb/v1.0/concepts/storage_engine/ >>>>> and http://docs.influxdata.com/influxdb/v1.0/guides/hardware_siz >>>>> ing/#how-much-storage-do-i-need for details. >>>>> >>>>> On Tue, Oct 18, 2016 at 2:51 AM, <[email protected]> wrote: >>>>> >>>>>> Hi everybody, >>>>>> >>>>>> Is there any way to calculate the database's size from the number >>>>>> measurements done per minute ? I mean predict the databse behavior (wal >>>>>> files, tsm files ...) >>>>>> >>>>>> Thank you. >>>>>> >>>>>> -- >>>>>> Remember to include the version number! >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "InfluxData" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/ee32b8d0-f9bf-40d >>>>>> f-baf1-e77fa93cddf5%40googlegroups.com. >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Beckett >>>>> Director of Support and Professional Services >>>>> InfluxDB >>>>> >>>> -- >>> Remember to include the version number! >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxData" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com >>> <https://groups.google.com/d/msgid/influxdb/b08ad8c0-341e-4390-a18c-8d6b67da26f0%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Sean Beckett >> Director of Support and Professional Services >> InfluxDB >> > -- > Remember to include the version number! > --- > You received this message because you are subscribed to the Google Groups > "InfluxData" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com > <https://groups.google.com/d/msgid/influxdb/4ba6c676-d0be-43a4-8f43-2c360ccafb3b%40googlegroups.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the version number! --- You received this message because you are subscribed to the Google Groups "InfluxData" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvNp0Chq8xBmrmN7Y_P%3Dp8LCkKu3jCEjrT%3Dmy%2BN83%2Bax3Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
