Don’t go to thousands of fields, it’s usually a bad idea for that many fields.
If all you’re doing is returning the data for display, have you considered indexing, perhaps in a separate field, a string token for each? Something like store1_USD_125.00 ? Then have the UI split that apart in a pleasing manner. Doesn’t work if you want to, say, sort by price though. Erick > On Oct 21, 2019, at 10:19 AM, Alexandre Rafalovitch <arafa...@gmail.com> > wrote: > > I remember several years ago a discussion/blog post about a similar > problem. The author went through a lot of thinking and decided that > the best way to deal with a similar problem was to have Solr documents > represent different level of abstraction, more granular. > > IIRC, the equivalent for your example would be to represent 'pricing' > (or even pricing/availability) as the document, not a 'product'. You > may need to duplicate product field values for that to work. But that > - apparently - allowed them to represent a lot of concepts (like > upcoming discounts, etc) easier by just adding availability dates, etc > into that final lower-level record. And, of course, allowed to update > individual store/product price by changing one record per time without > having to invalidate the cache for all other stores. > > Regards, > Alex. > > > On Mon, 21 Oct 2019 at 09:59, Vincenzo D'Amore <v.dam...@gmail.com> wrote: >> >> Hi Erick, >> >> thanks for getting back to me. We started to use payloads because we have >> the classical per-store pricing problem. >> Thousands of stores across and different prices. >> Then we found the payloads very useful started to use it for many reasons, >> like enabling/disabling the product for such store, save the stock >> availability, or save the other info like buy/sell price, discount rates, >> and so on. >> All those information are numbers, but stores can also be in different >> countries, I mean would be useful also have the currency and other >> attributes related to the store. >> >> Thinking about an alternative for payloads maybe I could use the dynamic >> fields, well, I know it is ugly. >> >> Consider this hypothetical case where I have two field payload : >> >> payloadPrice: [ >> "store1|125.0", >> "store2|220.0", >> "store3|225.0" >> ] >> >> payloadCurrency: [ >> "store1|USD", >> "store2|EUR", >> "store3|GBP" >> ] >> >> with dynamic fields I could have different fields for each document. >> >> currency_store1_s: "USD" >> currency_store2_s: "EUR" >> currency_store3_s: "GBP" >> >> But how many dynamic fields like this can I have? more than thousands? >> >> Again, I've just started to look at solr-ocrhighlighting github project you >> suggested. >> Those seems have written their own payload object type where store ocr >> highlighting information. >> It seems interesting, I'll take a look immediately. >> >> Thanks again for your time. >> >> Best regards, >> Vincenzo >> >> >> On Mon, Oct 21, 2019 at 2:55 PM Erick Erickson <erickerick...@gmail.com> >> wrote: >> >>> This is one of those situations where I know a client did it, but didn’t >>> see the code myself. >>> >>> So I can’t help much. >>> >>> Perhaps a good question at this point, though, is “why do you want to add >>> string payloads anyway”? >>> >>> This isn’t the client, but it might give you some pointers: >>> >>> >>> https://github.com/dbmdz/solr-ocrpayload-plugin/blob/master/src/main/java/de/digitalcollections/solr/plugin/components/ocrhighlighting/OcrHighlighting.java >>> >>> Best, >>> Erick >>> >>>> On Oct 21, 2019, at 6:37 AM, Vincenzo D'Amore <v.dam...@gmail.com> >>> wrote: >>>> >>>> Hi Erick, >>>> >>>> It seems I've reached a dead-point, or at least it seems looking at the >>>> code, it seems I can't easily add a custom decoder: >>>> >>>> Looking at PayloadUtils class there is getPayloadDecoder method invoked >>> to >>>> return the PayloadDecoder : >>>> >>>> public static PayloadDecoder getPayloadDecoder(FieldType fieldType) { >>>> PayloadDecoder decoder = null; >>>> >>>> String encoder = getPayloadEncoder(fieldType); >>>> >>>> if ("integer".equals(encoder)) { >>>> decoder = (BytesRef payload) -> payload == null ? 1 : >>>> PayloadHelper.decodeInt(payload.bytes, payload.offset); >>>> } >>>> if ("float".equals(encoder)) { >>>> decoder = (BytesRef payload) -> payload == null ? 1 : >>>> PayloadHelper.decodeFloat(payload.bytes, payload.offset); >>>> } >>>> // encoder could be "identity" at this point, in the case of >>>> DelimitedTokenFilterFactory encoder="identity" >>>> >>>> // TODO: support pluggable payload decoders? >>>> >>>> return decoder; >>>> } >>>> >>>> Any advice to work around this situation? >>>> >>>> >>>> On Mon, Oct 21, 2019 at 1:51 AM Erick Erickson <erickerick...@gmail.com> >>>> wrote: >>>> >>>>> You’d need to write one. Payloads are generally intended to hold >>> numerics >>>>> you can then use in a function query to factor into the score… >>>>> >>>>> Best, >>>>> Erick >>>>> >>>>>> On Oct 20, 2019, at 4:57 PM, Vincenzo D'Amore <v.dam...@gmail.com> >>>>> wrote: >>>>>> >>>>>> Sorry, I just realized that I was wrong in how I'm using the payload >>>>>> function. >>>>>> Give that the payload function only handles a numeric (integer or >>> float) >>>>>> payload, could you suggest me an alternative function that handles >>>>> strings? >>>>>> If not, should I write one? >>>>>> >>>>>> On Sun, Oct 20, 2019 at 10:43 PM Vincenzo D'Amore <v.dam...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Hi all, >>>>>>> >>>>>>> I'm trying to understand what I did wrong with a payload query that >>>>>>> returns >>>>>>> >>>>>>> error: { >>>>>>> metadata: [ "error-class", "org.apache.solr.common.SolrException", >>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], >>>>>>> msg: "No payload decoder found for field: colorCode", >>>>>>> code: 400 >>>>>>> } >>>>>>> >>>>>>> I have reduced my problem in a little sample to show what happens to >>> me. >>>>>>> Basically I have a document with a couple of payload fields one >>>>>>> delimited_payloads_string and one delimited_payloads_integer >>>>>>> >>>>>>> { >>>>>>> field_dps: "key|data", >>>>>>> field_dpi: "key|1", >>>>>>> } >>>>>>> >>>>>>> When I execute this query solr returns as expected the payload for the >>>>> key >>>>>>> >>>>>>> q=*:*&fl=payload(field_dpi,key) >>>>>>> >>>>>>> { >>>>>>> payload(field_dpi,key): 1 >>>>>>> } >>>>>>> >>>>>>> But for the strings there have to be something of different to do, >>>>> because >>>>>>> I'm unable receive the payload value back. Executing this query, as in >>>>> the >>>>>>> short introduction of this post, I receive an error. >>>>>>> >>>>>>> ?q=*:*&fl=payload(field_dps,key) >>>>>>> >>>>>>> error: { >>>>>>> metadata: [ "error-class", "org.apache.solr.common.SolrException", >>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], >>>>>>> msg: "No payload decoder found for field: colorCode", >>>>>>> code: 400 >>>>>>> } >>>>>>> >>>>>>> Am I doing something wrong? How can I read strings payload data? >>>>>>> >>>>>>> Thanks in advance for your time, >>>>>>> Vincenzo >>>>>>> >>>>>>> -- >>>>>>> Vincenzo D'Amore >>>>>>> >>>>>>> >>>>>> >>>>>> -- >>>>>> Vincenzo D'Amore >>>>> >>>>> >>>> >>>> -- >>>> Vincenzo D'Amore >>> >>> >> >> -- >> Vincenzo D'Amore
