Hi all, thanks for supporting. And many thanks whom have implemented the integration of the github Solr repository with the intellij IDE. To configure the environment and run the debugger I spent less than one hour, (and most of the time I had to wait the compilation). Solr and you guys really rocks together.
What I've done: I was looking at the original payload function is defined into the ValueSourceParser, this function uses a FloatPayloadValueSource to return the value found. As said I wrote a new version of payload function that handles strings, I named it spayload, and basically is able to extract the string value from the payload. Given the former example where I have a multivalue field payloadCurrency payloadCurrency: [ "store1|USD", "store2|EUR", "store3|GBP" ] executing spayload(payloadCurrency,store2) returns "EUR", and so on for the remaining key/value in the field. To implement the spayload function, I've added a new ValueSourceParser instance to the list of defined functions and which returns a StringPayloadValueSource with the value inside (does the same thing of former FloatPayloadValueSource). That's all. As said, always beware of your code when works at first run. And really there was something wrong, initially I messed up in the conversion of the payload into String (bytes, offset, etc). Now it is fixed, or at least it seems to me. I see this function cannot be used in the sort, very likely the simple implementation of the StringPayloadValueSource miss something. As far as I understand I'm scratching the surface of this solution, there are few things I'm worried about. I have a bunch of questions, please be patient. This function returns an empty string "" when does not match any key, or should return an empty value? not sure about, what's the correct way to return an empty value? I wasn't able to find a test unit for the payload function in the tests. Could you give me few suggestion in order to test properly the implementation? In case the spayload is used on a different field type (i.e. the use spayload on a float payload) the behaviour is not handled. Can this function check the type of the payload content? And at last, what do you think, can this simple fix be interesting for the Solr community, may I try to submit a pull request or add a feature to JIRA? Best regards, Vincenzo On Mon, Oct 21, 2019 at 9:12 PM Erik Hatcher <erik.hatc...@gmail.com> wrote: > Yes. The decoding of a payload based on its schema type is what the > payload() function does. Your Payloader won't currently work well/legibly > for fields encoded numerically: > > > https://github.com/o19s/payload-component/blob/master/src/main/java/com/o19s/payloads/Payloader.java#L130 > < > https://github.com/o19s/payload-component/blob/master/src/main/java/com/o19s/payloads/Payloader.java#L130 > > > > I think that code could probably be slightly enhanced to leverage > PayloadUtils.getPayloadDecoder(fieldType) and use bytes if the field type > doesn't have a better decoder. > > Erik > > > > On Oct 21, 2019, at 2:55 PM, Eric Pugh <ep...@opensourceconnections.com> > wrote: > > > > Have you checked out > > https://github.com/o19s/payload-component > > > > On Mon, Oct 21, 2019 at 2:47 PM Erik Hatcher <erik.hatc...@gmail.com> > wrote: > > > >> How about a single field, with terms like: > >> > >> store1_USD|125.0 store2_EUR|220.0 store3_GBP|225.0 > >> > >> Would that do the trick? > >> > >> And yeah, payload decoding is currently limited to float and int with > the > >> built-in payload() function. We'd need a new way to pull out > >> textual/bytes payloads - like maybe a DocTransformer? > >> > >> Erik > >> > >> > >>> On Oct 21, 2019, at 9:59 AM, Vincenzo D'Amore <v.dam...@gmail.com> > >> wrote: > >>> > >>> Hi Erick, > >>> > >>> thanks for getting back to me. We started to use payloads because we > have > >>> the classical per-store pricing problem. > >>> Thousands of stores across and different prices. > >>> Then we found the payloads very useful started to use it for many > >> reasons, > >>> like enabling/disabling the product for such store, save the stock > >>> availability, or save the other info like buy/sell price, discount > rates, > >>> and so on. > >>> All those information are numbers, but stores can also be in different > >>> countries, I mean would be useful also have the currency and other > >>> attributes related to the store. > >>> > >>> Thinking about an alternative for payloads maybe I could use the > dynamic > >>> fields, well, I know it is ugly. > >>> > >>> Consider this hypothetical case where I have two field payload : > >>> > >>> payloadPrice: [ > >>> "store1|125.0", > >>> "store2|220.0", > >>> "store3|225.0" > >>> ] > >>> > >>> payloadCurrency: [ > >>> "store1|USD", > >>> "store2|EUR", > >>> "store3|GBP" > >>> ] > >>> > >>> with dynamic fields I could have different fields for each document. > >>> > >>> currency_store1_s: "USD" > >>> currency_store2_s: "EUR" > >>> currency_store3_s: "GBP" > >>> > >>> But how many dynamic fields like this can I have? more than thousands? > >>> > >>> Again, I've just started to look at solr-ocrhighlighting github project > >> you > >>> suggested. > >>> Those seems have written their own payload object type where store ocr > >>> highlighting information. > >>> It seems interesting, I'll take a look immediately. > >>> > >>> Thanks again for your time. > >>> > >>> Best regards, > >>> Vincenzo > >>> > >>> > >>> On Mon, Oct 21, 2019 at 2:55 PM Erick Erickson < > erickerick...@gmail.com> > >>> wrote: > >>> > >>>> This is one of those situations where I know a client did it, but > didn’t > >>>> see the code myself. > >>>> > >>>> So I can’t help much. > >>>> > >>>> Perhaps a good question at this point, though, is “why do you want to > >> add > >>>> string payloads anyway”? > >>>> > >>>> This isn’t the client, but it might give you some pointers: > >>>> > >>>> > >>>> > >> > https://github.com/dbmdz/solr-ocrpayload-plugin/blob/master/src/main/java/de/digitalcollections/solr/plugin/components/ocrhighlighting/OcrHighlighting.java > >>>> > >>>> Best, > >>>> Erick > >>>> > >>>>> On Oct 21, 2019, at 6:37 AM, Vincenzo D'Amore <v.dam...@gmail.com> > >>>> wrote: > >>>>> > >>>>> Hi Erick, > >>>>> > >>>>> It seems I've reached a dead-point, or at least it seems looking at > the > >>>>> code, it seems I can't easily add a custom decoder: > >>>>> > >>>>> Looking at PayloadUtils class there is getPayloadDecoder method > invoked > >>>> to > >>>>> return the PayloadDecoder : > >>>>> > >>>>> public static PayloadDecoder getPayloadDecoder(FieldType fieldType) { > >>>>> PayloadDecoder decoder = null; > >>>>> > >>>>> String encoder = getPayloadEncoder(fieldType); > >>>>> > >>>>> if ("integer".equals(encoder)) { > >>>>> decoder = (BytesRef payload) -> payload == null ? 1 : > >>>>> PayloadHelper.decodeInt(payload.bytes, payload.offset); > >>>>> } > >>>>> if ("float".equals(encoder)) { > >>>>> decoder = (BytesRef payload) -> payload == null ? 1 : > >>>>> PayloadHelper.decodeFloat(payload.bytes, payload.offset); > >>>>> } > >>>>> // encoder could be "identity" at this point, in the case of > >>>>> DelimitedTokenFilterFactory encoder="identity" > >>>>> > >>>>> // TODO: support pluggable payload decoders? > >>>>> > >>>>> return decoder; > >>>>> } > >>>>> > >>>>> Any advice to work around this situation? > >>>>> > >>>>> > >>>>> On Mon, Oct 21, 2019 at 1:51 AM Erick Erickson < > >> erickerick...@gmail.com> > >>>>> wrote: > >>>>> > >>>>>> You’d need to write one. Payloads are generally intended to hold > >>>> numerics > >>>>>> you can then use in a function query to factor into the score… > >>>>>> > >>>>>> Best, > >>>>>> Erick > >>>>>> > >>>>>>> On Oct 20, 2019, at 4:57 PM, Vincenzo D'Amore <v.dam...@gmail.com> > >>>>>> wrote: > >>>>>>> > >>>>>>> Sorry, I just realized that I was wrong in how I'm using the > payload > >>>>>>> function. > >>>>>>> Give that the payload function only handles a numeric (integer or > >>>> float) > >>>>>>> payload, could you suggest me an alternative function that handles > >>>>>> strings? > >>>>>>> If not, should I write one? > >>>>>>> > >>>>>>> On Sun, Oct 20, 2019 at 10:43 PM Vincenzo D'Amore < > >> v.dam...@gmail.com> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> I'm trying to understand what I did wrong with a payload query > that > >>>>>>>> returns > >>>>>>>> > >>>>>>>> error: { > >>>>>>>> metadata: [ "error-class", "org.apache.solr.common.SolrException", > >>>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], > >>>>>>>> msg: "No payload decoder found for field: colorCode", > >>>>>>>> code: 400 > >>>>>>>> } > >>>>>>>> > >>>>>>>> I have reduced my problem in a little sample to show what happens > to > >>>> me. > >>>>>>>> Basically I have a document with a couple of payload fields one > >>>>>>>> delimited_payloads_string and one delimited_payloads_integer > >>>>>>>> > >>>>>>>> { > >>>>>>>> field_dps: "key|data", > >>>>>>>> field_dpi: "key|1", > >>>>>>>> } > >>>>>>>> > >>>>>>>> When I execute this query solr returns as expected the payload for > >> the > >>>>>> key > >>>>>>>> > >>>>>>>> q=*:*&fl=payload(field_dpi,key) > >>>>>>>> > >>>>>>>> { > >>>>>>>> payload(field_dpi,key): 1 > >>>>>>>> } > >>>>>>>> > >>>>>>>> But for the strings there have to be something of different to do, > >>>>>> because > >>>>>>>> I'm unable receive the payload value back. Executing this query, > as > >> in > >>>>>> the > >>>>>>>> short introduction of this post, I receive an error. > >>>>>>>> > >>>>>>>> ?q=*:*&fl=payload(field_dps,key) > >>>>>>>> > >>>>>>>> error: { > >>>>>>>> metadata: [ "error-class", "org.apache.solr.common.SolrException", > >>>>>>>> "root-error-class", "org.apache.solr.common.SolrException" ], > >>>>>>>> msg: "No payload decoder found for field: colorCode", > >>>>>>>> code: 400 > >>>>>>>> } > >>>>>>>> > >>>>>>>> Am I doing something wrong? How can I read strings payload data? > >>>>>>>> > >>>>>>>> Thanks in advance for your time, > >>>>>>>> Vincenzo > >>>>>>>> > >>>>>>>> -- > >>>>>>>> Vincenzo D'Amore > >>>>>>>> > >>>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Vincenzo D'Amore > >>>>>> > >>>>>> > >>>>> > >>>>> -- > >>>>> Vincenzo D'Amore > >>>> > >>>> > >>> > >>> -- > >>> Vincenzo D'Amore > >> > >> > > -- Vincenzo D'Amore
