This is for use in code in a search component, while I could fire up an http client or solr client to get stuff out of a blob store, that will make it hard to ensure that concurrent core startups don't duplicate each other's work and all process the same file into the blob store etc... and the blob is meant to be a map in which the component will look things up, so can one query into the blob in the blob store? I suspect not.
@Erik, This has been discussed as a possible fallback, though it also involves creating a client that makes requests inside the search component while it processes queries. It also creates a need for manual steps to create the index that stands in place of a map and then that has to be scripted for our continuous deployment... I have a patch that I am testing locally, and my unit tests like it so far today I should get it into a running instance for testing. We will at least use it locally. If it goes well I'll probably contribute it back as a patch to SOLR-3443. On Thu, Nov 12, 2015 at 5:07 AM, Ishan Chattopadhyaya < ichattopadhy...@gmail.com> wrote: > > Or in a separate Solr core/collection. :) > That support is available with this, I think: > https://cwiki.apache.org/confluence/display/solr/Blob+Store+API > > On Thu, Nov 12, 2015 at 2:31 PM, Erik Hatcher <erik.hatc...@gmail.com> > wrote: > >> Or in a separate Solr core/collection. :) >> >> On Nov 11, 2015, at 19:05, Walter Underwood <wun...@wunderwood.org> >> wrote: >> >> Depending on how fast the access needs to be, you could put that big map >> in memcache. >> >> wunder >> Walter Underwood >> wun...@wunderwood.org >> http://observer.wunderwood.org/ (my blog) >> >> >> On Nov 11, 2015, at 4:04 PM, Gus Heck <gus.h...@gmail.com> wrote: >> >> P.S. I posted the original message concurrently with the chat session's >> occurance I beleive, certainly before I had read it, so no I haven't >> actually tried what you suggest yet. >> >> On Wed, Nov 11, 2015 at 7:02 PM, Gus Heck <gus.h...@gmail.com> wrote: >> >>> Yes asked by a colleague :). The chat session is now in our jira ticket >>> :). >>> >>> However, my take on it is that this seems like a pretty broad brush to >>> paint with to move *all* our classes up and out of the normal core loading >>> process. I assume there are good reasons for segregating this stuff into >>> separate class loaders to begin with. It would also be fairly burdensom to >>> make a separate jar file to break out this one component... >>> >>> I really just want a way to stash the map in a place where other cores >>> can see it (and thus I can appropriately synchronize things so that the >>> loading only happens once). I'm asking because it seems like surely this >>> must be a solved problem... if not, it might be easiest to just solve it by >>> adding some sort of shared resources facility to CoreContainer? >>> >>> -Gus >>> >>> On Wed, Nov 11, 2015 at 6:54 PM, Shawn Heisey <apa...@elyograg.org> >>> wrote: >>> >>>> On 11/11/2015 4:11 PM, Gus Heck wrote: >>>> > I have a case where a component loads up a large CSV file (2.5 million >>>> > lines) to build a map. This worked ok in a case where we had a single >>>> > core, but it isn't working so well with 40 cores because each core >>>> loads >>>> > a new copy of the component in a new classloader and I get 40 new >>>> > versions of the same class each holding it's own private static final >>>> > map (one for each core). Each line is small, but a billion of anything >>>> > gets kinda heavy. Is this the intended class loading behavior? >>>> > >>>> > Is there some where that one can cause a class to be loaded in a >>>> parent >>>> > classloader above the core so that it's loaded just once? I want to >>>> load >>>> > it in some way that leverages standard solr resource loading, so that >>>> > I'm not hard coding or setting sysprops just to be able to find it. >>>> > >>>> > This is in a copy of trunk from about a month ago... so 6.x stuff is >>>> > mostly available. >>>> >>>> This sounds like a question that I just recently answered on IRC. >>>> >>>> If you remove all <lib> elements from your solrconfig.xml files and >>>> place all extra jars for Solr into ${solr.solr.home}/lib ... Solr will >>>> load those jars before any cores are created and they will be available >>>> to all cores. >>>> >>>> There is a minor bug with this that will be fixed in Solr 5.4.0. It is >>>> unlikely that this will affect third-party components, but be aware that >>>> until 5.4, jars in that lib directory will be loaded twice by older 5.x >>>> versions. >>>> >>>> https://issues.apache.org/jira/browse/SOLR-6188 >>>> >>>> Thanks, >>>> Shawn >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >>>> >>>> >>> >>> >>> -- >>> http://www.the111shift.com >>> >> >> >> >> -- >> http://www.the111shift.com >> >> >> > -- http://www.the111shift.com
