On 17 August 2016 at 23:28, Sean Beckett <[email protected]> wrote: > You will likely want to use longer shards. >
What do you mean by wanting to use longer shards? The shard duration? > TSM is efficient with sparse field sets, but if you have fields that only > appear once or twice in a shard that won't be efficient. Ensure that each > field has dozens or hundreds of values per field per shard and the > compression will be much better. Queries should be largely unaffected in > that case. > Thanks! > > On Wed, Aug 17, 2016 at 7:48 AM, bruno binet <[email protected]> > wrote: > >> I also want to write many fields to the same measurement (around 5k), >> mainly to be able to do math between the fields (since JOIN queries are not >> supported anymore). >> >> But I won't write all these fields for every single point: instead, each >> point will contain only a few fields which will likely be differents from >> the other points (so the wireline space to write each point through the >> line protocol will remain small even if the global fields cardinality is >> very high). >> >> So my question is about the underlying TSM storage: will this use case >> have an impact on storage space efficiency? What about the query efficiency >> for this special use case? >> >> Cheers, >> Bruno >> >> On 21 July 2016 at 23:05, Sean Beckett <[email protected]> wrote: >> >>> The TSM storage engine only reads the requested fields from disk, so >>> there shouldn't be any significant read penalty for having that many fields >>> per point. That would mean each point is a minimum of ~350KB in >>> uncompressed line protocol. That's going to take a lot of wireline space to >>> write each point, since there's currently no compression available for the >>> write path. >>> >>> How many points per second do you anticipate recording? If it's any more >>> than 20 I suspect your tests will be challenging. >>> >>> Your use case is interesting, though. Are you storing telemetry? >>> >>> On Thu, Jul 21, 2016 at 2:11 PM, <[email protected]> wrote: >>> >>>> Mostly read I think, but we'd never be reading out all fields at once. >>>> I guess I'll give it a shot and see what catches fire. >>>> >>>> Thanks for the reply >>>> >>>> On Thursday, July 21, 2016 at 1:00:48 PM UTC-7, Sean Beckett wrote: >>>>> >>>>> The system should still function, but we haven't tested any of the >>>>> performance impacts of that many fields. It might complicate file >>>>> compactions, for instance. What write/query load are you considering? >>>>> >>>>> On Thu, Jul 21, 2016 at 11:31 AM, <[email protected]> wrote: >>>>> >>>>>> Can I have 30k fields per point for instance? >>>>>> >>>>>> -- >>>>>> Remember to include the InfluxDB version number with all issue reports >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "InfluxDB" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To post to this group, send email to [email protected]. >>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44d >>>>>> a-a36b-1dc88eeb5f86%40googlegroups.com >>>>>> <https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44da-a36b-1dc88eeb5f86%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> Sean Beckett >>>>> Director of Support and Professional Services >>>>> InfluxDB >>>>> >>>> -- >>>> Remember to include the InfluxDB version number with all issue reports >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "InfluxDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/influxdb. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/influxdb/f871462e-4481-4217-add2-0a497a678dba%40googlegroups.com >>>> <https://groups.google.com/d/msgid/influxdb/f871462e-4481-4217-add2-0a497a678dba%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> >>> >>> -- >>> Sean Beckett >>> Director of Support and Professional Services >>> InfluxDB >>> >>> -- >>> Remember to include the InfluxDB version number with all issue reports >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxci >>> Ey-unQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxciEy-unQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6 >> xO%3Dv7SmQ%40mail.gmail.com >> <https://groups.google.com/d/msgid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6xO%3Dv7SmQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > > > -- > Sean Beckett > Director of Support and Professional Services > InfluxDB > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/CALGqCvNiYW2sO4fOGOZtEDhQ_V7m2BsH7j_Q6xHZY1detRMBbA% > 40mail.gmail.com > <https://groups.google.com/d/msgid/influxdb/CALGqCvNiYW2sO4fOGOZtEDhQ_V7m2BsH7j_Q6xHZY1detRMBbA%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CAHop23eA%3DYnMpHSBGb5XBR0haWZxn8-_yO6eb6QvPRSFHTO0aw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
