@Dan: basically I'm looking for a "update by query" feature where I need to update a single field but in a large portion of the collection. Atomic updates are not fast enough but I didn't give enough thought to in-place update to be honest. I need ability to filter efficiently on the data, thus my first reaction was that the fact that the field must not be indexed is a no-go but I might be wrong here, right? ________________________________ De : Dan Rosher <rosh...@gmail.com> Envoyé : jeudi 28 juillet 2022 16:51 À : users@solr.apache.org <users@solr.apache.org> Objet : Re: ExternalFileField2, massively scalable external file fields
Out of interest, did you also look into in-place updates, so not having to re-index the whole document, just one field, or were the conditions too restrictive? On Thu, 28 Jul 2022 at 14:38, Gael Jourdan-Weil < gael.jourdan-w...@kelkoogroup.com> wrote: > Definitely something that can benefit the community IMHO. > > I'm soon to start evaluating the use of cross-collection join queries vs. > external file field for one of our use cases (similar to inventory/price > list use case). > > Out of curiosity, would your work make it possible to filter on those > external data? > > Gaël > ________________________________ > De : Eric Pugh <ep...@opensourceconnections.com> > Envoyé : jeudi 28 juillet 2022 13:17 > À : users@solr.apache.org <users@solr.apache.org> > Objet : Re: ExternalFileField2, massively scalable external file fields > > More and more advanced search features require some sort of offline > processing step (looking at your LTR ;-) ) that produces something that > THEN gets deployed into Solr, so I don’t see that as a blocker. > > Maybe in the future we end up having a core part of Solr some sort of > offline processing capability so folks don’t have to deploy “yet another > system” ;-) > > Price lists…. Always a pain ;-). > > > On Jul 27, 2022, at 9:30 PM, Matt Kuiper <kuipe...@gmail.com> wrote: > > > > Hey Joel, > > > > It's been awhile! Hope you are doing well. > > > > The feature you are proposing reminded me of the Lucidworks work I was > > involved with for Lowe's Canada and how to deal with changing product > > prices... Whether in or out of Solr it seems like there would be a > number > > of users who may benefit from it. > > > > Matt > > > > On Wed, Jul 27, 2022 at 4:11 PM Joel Bernstein <joels...@gmail.com> > wrote: > > > >> We are in the process of developing large scale support for external > file > >> fields. This is designed to accommodate very large product catalogs > with a > >> very large number of custom price lists. It can also be applied to > >> inventory or other frequently changing values. The question I have is if > >> this is a suitable open source contribution due to certain > implementation > >> details. > >> > >> The approach taken requires the offline processing of the files from the > >> documented external file format into a specific binary format. A command > >> line tool is being developed to process the external files. Since this > >> offline processing is a required data preparation step it may be > outside of > >> the scope of Solr. But it also might fit as a module. > >> > >> I'm curious how the community feels about this as a module. > >> > >> > >> > >> Joel Bernstein > >> http://joelsolr.blogspot.com/ > >> > > _______________________ > Eric Pugh | Founder & CEO | OpenSource Connections, LLC | 434.466.1467 | > http://www.opensourceconnections.com < > http://www.opensourceconnections.com/> | My Free/Busy < > http://tinyurl.com/eric-cal> > Co-Author: Apache Solr Enterprise Search Server, 3rd Ed < > https://www.packtpub.com/big-data-and-business-intelligence/apache-solr-enterprise-search-server-third-edition-raw > > > This e-mail and all contents, including attachments, is considered to be > Company Confidential unless explicitly stated otherwise, regardless of > whether attachments are marked as such. > >
