Re: solr dynamic core creation
Hi nizan, I have the same requirement of creating cores on the fly. Was looking for some API provided by http solr server. Currently working around by writing my own shell script on the server (solr server :) ). Any better leads on the same? Thanks, Samarth On Thu, Nov 11, 2010 at 9:27 PM, Robert Sandiford wrote: > > No - in reading what you just wrote, and what you originally wrote, I think > the misunderstanding was mine, based on the architecture of my code. In my > code, it is our 'server' level that does the SolrJ indexing calls, but you > meant 'server' to be the Solr instance, and what you mean by 'client' is > what I was thinking of (without thinking) as the 'server'... > > Sorry about that. Hopefully someone else can chime in on your specific > issue... > -- > View this message in context: > http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1883354.html > Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr dynamic core creation
No - in reading what you just wrote, and what you originally wrote, I think the misunderstanding was mine, based on the architecture of my code. In my code, it is our 'server' level that does the SolrJ indexing calls, but you meant 'server' to be the Solr instance, and what you mean by 'client' is what I was thinking of (without thinking) as the 'server'... Sorry about that. Hopefully someone else can chime in on your specific issue... -- View this message in context: http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1883354.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr dynamic core creation
Hi, Maybe just don't understand all the concept there and I mix up server and client... Client - The place where I make the http calls (for index, search etc.) - where I use the CommonsHttpSolrServer as the solr server. This machine isn't defined as master or slave, it just use solr as search engine Server - The http calls I made in the client, goes to another server, the master solr server (or one of the slaves), where I have embeddedSolrServer, aren't they? thanks, nizan -- View this message in context: http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1883269.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr dynamic core creation
Hmmm. Maybe you need to define what you mean by 'server' and what you mean by 'client'. -- View this message in context: http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1883238.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr dynamic core creation
Hi, Thanks for the offers, I'll take deeper look into them. In the offers you showed me, if I understand correctly, the call for creation is done in the client side. I need the mechanism we'll work in the server side. I know it sounds stupid, but I need the client side wouldn't know about which cores are there or not, and on the server side I (maybe with a handler?), will understand if the core is not created, and create it if needed. Thanks, nizan -- View this message in context: http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1883213.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: solr dynamic core creation
Hi, nizan. I didn't realize that just replying to a thread from my email
client wouldn't get back to you. Here's some info on this thread since your
original post:
On Nov 10, 2010, at 12:30pm, Bob Sandiford wrote:
> Why not use replication? Call it inexperience...
>
> We're really early into working with and fully understanding Solr and
> the best way to approach various issues. I did mention that this was
> a prototype and non-production code, so I'm covered, though :)
>
> We'll take a look at the replication feature...
Replication doesn't replicate the top-level solr.xml file that defines
available cores, so if dynamic cores is a requirement then your custom code
isn't wasted :)
-- Ken
>> -Original Message-
>> From: Jonathan Rochkind [mailto:rochk...@jhu.edu]
>> Sent: Wednesday, November 10, 2010 3:26 PM
>> To: solr-user@lucene.apache.org
>> Subject: Re: Dynamic creating of cores in solr
>>
>> You could use the actual built-in Solr replication feature to
>> accomplish that same function -- complete re-index to a 'master', and
>> then when finished, trigger replication to the 'slave', with the
>> 'slave' being the live index that actually serves your applications.
>>
>> I am curious if there was any reason you chose to roll your own
>> solution using JSolr and dynamic creation of cores, instead of simply
>> using the replication feature. Were there any downsides of using the
>> replication feature for this purpose that you amerliorated through
>> your solution?
>>
>> Jonathan
>>
>> Bob Sandiford wrote:
>>> We also use SolrJ, and have a dynamically created Core capability -
>> where we don't know in advance what the Cores will be that we
>> require.
>>>
>>> We almost always do a complete index build, and if there's a
>>> previous
>> instance of that index, it needs to be available during a complete
>> index build, so we have two cores per index, and switch them as
>> required at the end of an indexing run.
>>>
>>> Here's a summary of how we do it (we're in an early prototype /
>> implementation right now - this isn't production quality code - as
>> you can tell from our voluminous javadocs on the methods...)
>>>
>>> 1) Identify if the core exists, and if not, create it:
>>>
>>> /**
>>> * This method instantiates two SolrServer objects, solr and
>> indexCore. It requires that
>>> * indexName be set before calling.
>>> */
>>>private void initSolrServer() throws IOException
>>>{
>>>String baseUrl = "http://localhost:8983/solr/";;
>>>solr = new CommonsHttpSolrServer(baseUrl);
>>>
>>>String indexCoreName = indexName +
>> SolrConstants.SUFFIX_INDEX; // SUFIX_INDEX = "_INDEX"
>>>String indexCoreUrl = baseUrl + indexCoreName;
>>>
>>>// Here we create two cores for the indexName, if they don't
>> already exist - the live core used
>>>// for searching and a second core used for indexing. After
>> indexing, the two will be switched so the
>>>// just-indexed core will become the live core. The way that
>> core swapping works, the live core will always
>>>// be named [indexName] and the indexing core will always be
>> named [indexname]_INDEX, but the
>>>// dataDir of each core will alternate between [indexName]_1
>> and [indexName]_2.
>>>createCoreIfNeeded(indexName, indexName + "_1", solr);
>>>createCoreIfNeeded(indexCoreName, indexName + "_2", solr);
>>>indexCore = new CommonsHttpSolrServer(indexCoreUrl);
>>>}
>>>
>>>
>>> /**
>>> * Create a core if it does not already exists. Returns true if a
>> new core was created, false otherwise.
>>> */
>>>private boolean createCoreIfNeeded(String coreName, String
>> dataDir, SolrServer server) throws IOException
>>>{
>>>boolean coreExists = true;
>>>try
>>>{
>>>// SolrJ provides no direct method to check if a core
>> exists, but getStatus will
>>>// return an empty list for any core that doesn't.
>>>CoreAdminResponse statusResponse =
>> CoreAdminRequest.getStatus(coreName, server);
>>>coreExists =
>> statusResponse.getCoreStatus(coreName).size() > 0;
>>>if(!coreExists)
>>>{
>>>// Create the core
>>>LOG.info("Creating Solr core: " + coreName);
>>>CoreAdminRequest.Create create = new
>> CoreAdminRequest.Create();
>>>create.setCoreName(coreName);
>>>create.setInstanceDir(".");
>>>create.setDataDir(dataDir);
>>>create.process(server);
>>>}
>>>}
>>>catch (SolrServerException e)
>>>{
>>>e.printStackTrace();
>>>}
>>>return !coreExists;
>>>}
>>>
>>>
>>> 2) Do the index, clearing it first if it's a complete rebuild:
>>>
>>> [snip]
>>>if (fullIndex)
>>>{
>>>try
>>>{
>>>indexCore.delete
Re: solr dynamic core creation
Does anyone has any idea on how to do this? -- View this message in context: http://lucene.472066.n3.nabble.com/solr-dynamic-core-creation-tp1867705p1881374.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: SOLR dynamic core creation
Thanks. Yup, I have reloaded the wiki all morning (GMT+1 time) Cheers //Marcus On Fri, Aug 28, 2009 at 2:06 PM, Shalin Shekhar Mangar < shalinman...@gmail.com> wrote: > On Fri, Aug 28, 2009 at 5:25 PM, Marcus Herou >wrote: > > > That seems sweet! One conf dir, many data-dirs = perfect. > > > > Some short questions > > > > * How and where do one configure each core to have separate data vs > > instancedirs ? Still manually in solr.xml or by using the > coreadminhandler > > ? > > > You can use any of the two. > > > > > > * How do one use the CoreAdminHandler ? (Google gives me only refs to > the > > javadoc or nabble) > > > The wiki is down right now but you can find more info at > http://wiki.apache.org/solr/CoreAdmin > > > > > > * If one perform the changes through CoreAdminHandler then where are the > > changes persisted ? solr.xml ? > > > > Correct. > > -- > Regards, > Shalin Shekhar Mangar. > -- Marcus Herou CTO and co-founder Tailsweep AB +46702561312 marcus.he...@tailsweep.com http://www.tailsweep.com/
Re: SOLR dynamic core creation
On Fri, Aug 28, 2009 at 5:25 PM, Marcus Herou wrote: > That seems sweet! One conf dir, many data-dirs = perfect. > > Some short questions > > * How and where do one configure each core to have separate data vs > instancedirs ? Still manually in solr.xml or by using the coreadminhandler > ? You can use any of the two. > > * How do one use the CoreAdminHandler ? (Google gives me only refs to the > javadoc or nabble) The wiki is down right now but you can find more info at http://wiki.apache.org/solr/CoreAdmin > > * If one perform the changes through CoreAdminHandler then where are the > changes persisted ? solr.xml ? > Correct. -- Regards, Shalin Shekhar Mangar.
Re: SOLR dynamic core creation
That seems sweet! One conf dir, many data-dirs = perfect. Some short questions * How and where do one configure each core to have separate data vs instancedirs ? Still manually in solr.xml or by using the coreadminhandler ? * How do one use the CoreAdminHandler ? (Google gives me only refs to the javadoc or nabble) * If one perform the changes through CoreAdminHandler then where are the changes persisted ? solr.xml ? Cheers mate //Marcus 2009/8/28 Noble Paul നോബിള് नोब्ळ् > have you looked at the coreadminhandler? > > say you always write to "newcore" > > after say august , you cretae andn empty core "core-aug" and swap it > out with "newcore". > > Make every core use the same instancedir with different dataDir > > On Fri, Aug 28, 2009 at 3:08 PM, Marcus Herou > wrote: > > Hi. > > > > We are now rethinking our sharding strategy to shard on the blog entry > > publishdate instead of just a simple hash. This is due to that the index > > size is growing too much to be handled in one index. > > > > Thing is that we want to select a core based on the publishdate of the > > entry. Let's say we have one core per month. > > > > A new core need to be added at the very latest 00:00 the first day each > > month. How would I do that dynamically ? > > > > I am thinking something like this: > > > > *A core dir which serves as template (i.e. contains no data) > > > > * A cronjob which: > > -Copies the tpl-dir -> $corename-$year-$month > > -Updates the solr.xml > > -Restarts solr (can one only reload the solr.xml file ?) > > > > Of course we get issues on the client since we need to figure out which > > cores to search in for each query... > > The benefit though is that it seems alot easier to manage the indexes, > > especially that we are able to choose which indices to search in to > utilize > > the memory better. We probably need to distribute a file to the clients > so > > they know where each index reside (or have the conf in a DB) > > > > Any thoughts ? > > > > Cheers > > > > //Marcus > > > > > > > > > > > > -- > > Marcus Herou CTO and co-founder Tailsweep AB > > +46702561312 > > marcus.he...@tailsweep.com > > http://www.tailsweep.com/ > > > > > > -- > - > Noble Paul | Principal Engineer| AOL | http://aol.com > -- Marcus Herou CTO and co-founder Tailsweep AB +46702561312 marcus.he...@tailsweep.com http://www.tailsweep.com/
Re: SOLR dynamic core creation
have you looked at the coreadminhandler? say you always write to "newcore" after say august , you cretae andn empty core "core-aug" and swap it out with "newcore". Make every core use the same instancedir with different dataDir On Fri, Aug 28, 2009 at 3:08 PM, Marcus Herou wrote: > Hi. > > We are now rethinking our sharding strategy to shard on the blog entry > publishdate instead of just a simple hash. This is due to that the index > size is growing too much to be handled in one index. > > Thing is that we want to select a core based on the publishdate of the > entry. Let's say we have one core per month. > > A new core need to be added at the very latest 00:00 the first day each > month. How would I do that dynamically ? > > I am thinking something like this: > > *A core dir which serves as template (i.e. contains no data) > > * A cronjob which: > -Copies the tpl-dir -> $corename-$year-$month > -Updates the solr.xml > -Restarts solr (can one only reload the solr.xml file ?) > > Of course we get issues on the client since we need to figure out which > cores to search in for each query... > The benefit though is that it seems alot easier to manage the indexes, > especially that we are able to choose which indices to search in to utilize > the memory better. We probably need to distribute a file to the clients so > they know where each index reside (or have the conf in a DB) > > Any thoughts ? > > Cheers > > //Marcus > > > > > > -- > Marcus Herou CTO and co-founder Tailsweep AB > +46702561312 > marcus.he...@tailsweep.com > http://www.tailsweep.com/ > -- - Noble Paul | Principal Engineer| AOL | http://aol.com
