Yes, by longer shards I meant increasing the shard duration so that each shard spans a longer range of time, and thus is more likely to have multiple values per field key.
On Thu, Aug 18, 2016 at 1:11 AM, bruno binet <[email protected]> wrote: > > > On 17 August 2016 at 23:28, Sean Beckett <[email protected]> wrote: > >> You will likely want to use longer shards. >> > > What do you mean by wanting to use longer shards? The shard duration? > > >> TSM is efficient with sparse field sets, but if you have fields that only >> appear once or twice in a shard that won't be efficient. Ensure that each >> field has dozens or hundreds of values per field per shard and the >> compression will be much better. Queries should be largely unaffected in >> that case. >> > > Thanks! > > >> >> On Wed, Aug 17, 2016 at 7:48 AM, bruno binet <[email protected]> >> wrote: >> >>> I also want to write many fields to the same measurement (around 5k), >>> mainly to be able to do math between the fields (since JOIN queries are not >>> supported anymore). >>> >>> But I won't write all these fields for every single point: instead, each >>> point will contain only a few fields which will likely be differents from >>> the other points (so the wireline space to write each point through the >>> line protocol will remain small even if the global fields cardinality is >>> very high). >>> >>> So my question is about the underlying TSM storage: will this use case >>> have an impact on storage space efficiency? What about the query efficiency >>> for this special use case? >>> >>> Cheers, >>> Bruno >>> >>> On 21 July 2016 at 23:05, Sean Beckett <[email protected]> wrote: >>> >>>> The TSM storage engine only reads the requested fields from disk, so >>>> there shouldn't be any significant read penalty for having that many fields >>>> per point. That would mean each point is a minimum of ~350KB in >>>> uncompressed line protocol. That's going to take a lot of wireline space to >>>> write each point, since there's currently no compression available for the >>>> write path. >>>> >>>> How many points per second do you anticipate recording? If it's any >>>> more than 20 I suspect your tests will be challenging. >>>> >>>> Your use case is interesting, though. Are you storing telemetry? >>>> >>>> On Thu, Jul 21, 2016 at 2:11 PM, <[email protected]> wrote: >>>> >>>>> Mostly read I think, but we'd never be reading out all fields at >>>>> once. I guess I'll give it a shot and see what catches fire. >>>>> >>>>> Thanks for the reply >>>>> >>>>> On Thursday, July 21, 2016 at 1:00:48 PM UTC-7, Sean Beckett wrote: >>>>>> >>>>>> The system should still function, but we haven't tested any of the >>>>>> performance impacts of that many fields. It might complicate file >>>>>> compactions, for instance. What write/query load are you considering? >>>>>> >>>>>> On Thu, Jul 21, 2016 at 11:31 AM, <[email protected]> wrote: >>>>>> >>>>>>> Can I have 30k fields per point for instance? >>>>>>> >>>>>>> -- >>>>>>> Remember to include the InfluxDB version number with all issue >>>>>>> reports >>>>>>> --- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "InfluxDB" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To post to this group, send email to [email protected]. >>>>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44d >>>>>>> a-a36b-1dc88eeb5f86%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/influxdb/b8f4b08a-3228-44da-a36b-1dc88eeb5f86%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> >>>>>> >>>>>> -- >>>>>> Sean Beckett >>>>>> Director of Support and Professional Services >>>>>> InfluxDB >>>>>> >>>>> -- >>>>> Remember to include the InfluxDB version number with all issue reports >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "InfluxDB" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To post to this group, send email to [email protected]. >>>>> Visit this group at https://groups.google.com/group/influxdb. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/influxdb/f871462e-4481-421 >>>>> 7-add2-0a497a678dba%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/influxdb/f871462e-4481-4217-add2-0a497a678dba%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>>> For more options, visit https://groups.google.com/d/optout. >>>>> >>>> >>>> >>>> >>>> -- >>>> Sean Beckett >>>> Director of Support and Professional Services >>>> InfluxDB >>>> >>>> -- >>>> Remember to include the InfluxDB version number with all issue reports >>>> --- >>>> You received this message because you are subscribed to the Google >>>> Groups "InfluxDB" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To post to this group, send email to [email protected]. >>>> Visit this group at https://groups.google.com/group/influxdb. >>>> To view this discussion on the web visit https://groups.google.com/d/ms >>>> gid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxci >>>> Ey-unQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/influxdb/CALGqCvP7k8XGmHX%3Dh1Rob9FJbRQdpLbJB91bEPoYdxciEy-unQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >>> Remember to include the InfluxDB version number with all issue reports >>> --- >>> You received this message because you are subscribed to the Google >>> Groups "InfluxDB" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> Visit this group at https://groups.google.com/group/influxdb. >>> To view this discussion on the web visit https://groups.google.com/d/ms >>> gid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6 >>> xO%3Dv7SmQ%40mail.gmail.com >>> <https://groups.google.com/d/msgid/influxdb/CAHop23cyR2KRSs%2BD8NQy%2BLv2f-AyeKu3KjVVzvPUz6xO%3Dv7SmQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> For more options, visit https://groups.google.com/d/optout. >>> >> >> >> >> -- >> Sean Beckett >> Director of Support and Professional Services >> InfluxDB >> >> -- >> Remember to include the InfluxDB version number with all issue reports >> --- >> You received this message because you are subscribed to the Google Groups >> "InfluxDB" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To post to this group, send email to [email protected]. >> Visit this group at https://groups.google.com/group/influxdb. >> To view this discussion on the web visit https://groups.google.com/d/ms >> gid/influxdb/CALGqCvNiYW2sO4fOGOZtEDhQ_V7m2BsH7j_ >> Q6xHZY1detRMBbA%40mail.gmail.com >> <https://groups.google.com/d/msgid/influxdb/CALGqCvNiYW2sO4fOGOZtEDhQ_V7m2BsH7j_Q6xHZY1detRMBbA%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> For more options, visit https://groups.google.com/d/optout. >> > > -- > Remember to include the InfluxDB version number with all issue reports > --- > You received this message because you are subscribed to the Google Groups > "InfluxDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > Visit this group at https://groups.google.com/group/influxdb. > To view this discussion on the web visit https://groups.google.com/d/ > msgid/influxdb/CAHop23eA%3DYnMpHSBGb5XBR0haWZxn8-_ > yO6eb6QvPRSFHTO0aw%40mail.gmail.com > <https://groups.google.com/d/msgid/influxdb/CAHop23eA%3DYnMpHSBGb5XBR0haWZxn8-_yO6eb6QvPRSFHTO0aw%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > > For more options, visit https://groups.google.com/d/optout. > -- Sean Beckett Director of Support and Professional Services InfluxDB -- Remember to include the InfluxDB version number with all issue reports --- You received this message because you are subscribed to the Google Groups "InfluxDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at https://groups.google.com/group/influxdb. To view this discussion on the web visit https://groups.google.com/d/msgid/influxdb/CALGqCvNN%2B_hjR6%3DzrvXCtxLYtL22yhLcY6E6zh_LZzJeNzcfhA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
