You will likely want to use longer shards. TSM is efficient with sparse field sets, but if you have fields that only appear once or twice in a shard that won't be efficient. Ensure that each field has dozens or hundreds of values per field per shard and the compression will be much better. Queries should be largely unaffected in that case.
On Wed, Aug 17, 2016 at 7:48 AM, bruno binet <[email protected]> wrote: > I also want to write many fields to the same measurement (around 5k), > mainly to be able to do math between the fields (since JOIN queries are not > supported anymore). > > But I won't write all these fields for every single point: instead, each > point will contain only a few fields which will likely be differents from > the other points (so the wireline space to write each point through the > line protocol will remain small even if the global fields cardinality is > very high). > > So my question is about the underlying TSM storage: will this use case > have an impact on storage space efficiency? What about the query efficiency > for this special use case? > > Cheers, > Bruno > > On 21 July 2016 at 23:05, Sean Beckett <[email protected]> wrote: > >> The TSM storage engine only reads the requested fields from disk, so >> there shouldn't be any significant read penalty for having that many fields >> per point. That would mean each point is a minimum of ~350KB in >> uncompressed line protocol. That's going to take a lot of wireline space to >> write each point, since there's currently no compression available for the >> write path. >> >> How many points per second do you anticipate recording? If it's any more >> than 20 I suspect your tests will be challenging. >> >> Your use case is interesting, though. Are you storing telemetry? >> >> On Thu, Jul 21, 2016 at 2:11 PM, <[email protected]> wrote: >> >>> Mostly read I think, but we'd never be reading out all fields at once. >>> I guess I'll give it a shot and see what catches fire. >>> >>> Thanks for the reply >>> >>> On Thursday, July 21, 2016 at 1:00:48 PM UTC-7, Sean Beckett wrote: >>>> >>>> The system should still function, but we haven't tested any of the >>>> performance impacts of that many fields. It might complicate file >>>> compactions, for instance. What write/query load are you considering? >>>> >>>> On Thu, Jul 21, 2016 at 11:31 AM, <[email protected]> wrote: >>>> >>>>> Can I have 30k fields per point for instance? >>>>> >>>>> -- >>>>> Remember to include the InfluxDB version number with all issue reports >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44d >>>>> a-a36b-1dc88eeb5f86%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44da-a36b-1dc88eeb5f86%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>> -- >>> Remember to include the InfluxDB version number with all issue reports >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/f871462e-4481-4217-add2-0a497a678dba%40googlegroups.com >>> <https://groups.google.com/d/msgid/influxdb/f871462e-4481-4217-add2-0a497a678dba%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Sean Beckett >> Director of Support and Professional Services >> InfluxDB >> >> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxci >> Ey-unQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxciEy-unQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ms > gid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6xO%3Dv7SmQ% > 40mail.gmail.com > <https://groups.google.com/d/msgid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6xO%3Dv7SmQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvNiYW2sO4fOGOZtEDhQ_V7m2BsH7j_Q6xHZY1detRMBbA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
