[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12614337#action_12614337 ] Ryan McKinley commented on SOLR-350: it looks like dataDir option was removed from CoreDescriptor. Was there a reason for this? Can multicore.xml manage the data directories? http://wiki.apache.org/solr/MultiCore#head-2696b6ae9766aa312580b5014f6c8f659a2c1bea I think we should return that configuration. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Assignee: Ryan McKinley Fix For: 1.3 Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, SOLR-350-RemoveStatic.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12614422#action_12614422 ] Henri Biestro commented on SOLR-350: Looks like this was removed around 02/Feb/08 from one of your comments :-) ; the dataDir can be set in solrconfig.xml so configuring it through multicore.xml was considered a dangerous feature. And I agree we should enhance the configuration behaviors. Since we are in the functional vicinity, the 2008-01-23 03:09 AM version of the patch allowed (at least MulitCore.create(...)) the following: Make the instanceDir relative to the multicore instanceDir if not absolute Make the dataDir relative to the multicore dataDir if not absolute Just in case... Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Assignee: Ryan McKinley Fix For: 1.3 Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, SOLR-350-RemoveStatic.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12609944#action_12609944 ] Markus Mautner commented on SOLR-350: - MultiCore persistence is broken. multicore/@sharedLib gets written as multicore/@libDir, so loading the multicore configuration after saving will fail. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Assignee: Ryan McKinley Fix For: 1.3 Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, SOLR-350-RemoveStatic.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12609968#action_12609968 ] Ryan McKinley commented on SOLR-350: thanks for finding this Markus! fixed in rev 673430 Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Assignee: Ryan McKinley Fix For: 1.3 Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, SOLR-350-RemoveStatic.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12581972#action_12581972 ] Erik Hatcher commented on SOLR-350: --- The RemoveStatic patch looks good, Ryan. +1 Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, SOLR-350-RemoveStatic.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12577421#action_12577421 ] Walter Ferrara commented on SOLR-350: - It's been a while since I had a look at this patch, and things seems to have changed a bit meanwhile -- but it looks strange that the only way to access the cores registry inside a solr istance relay on a deprecated class, org.apache.solr.core.SolrMultiCore. I noticed Henri mention that the SolrMultiCore singleton is added to allow a smooth transition, but... If there is no another way to achieve the same result bypassing org.apache.solr.core.SolrMultiCore, that class should not be marked as deprecated. Or that deprecation has to be read as in the final solr 1.3 just use SolrMultiCore and ignore the warning, but remember that in the next version, the 2.0, things will change? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12577439#action_12577439 ] Yonik Seeley commented on SOLR-350: --- Thanks Shalin, I just committed your JSP fixes (after converting the patch from UTF-16 to UTF-8 :-) Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12577434#action_12577434 ] Henri Biestro commented on SOLR-350: Ryan: thanks for the commit. Shalin: thanks a lot for the JSP fix, my bad. Thinking of it, it might be possible to put the Multicore instance as a request attribute from the filter code let JSP consume it this way rather than using SolrMultiCore. I'll look into it. Walter: yes, you are correct, things will most likely change in 2.0. We want MultiCore to be derivable and we dont want core core to consider MultiCore to be a singleton; however, we do not feel current needs require the class to be configurable (yet). May be o.a.s.servlet. would be/have been a better package for SolrMultiCore to make this easier. Sorry for the confusion. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-jsp-fixes.patch, SOLR-350-jsp-fixes.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12574658#action_12574658 ] Yonik Seeley commented on SOLR-350: --- I think this is ready to commit - we can address the variable/config/data issue in a different issue or smaller patches. +1 Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12574839#action_12574839 ] Ryan McKinley commented on SOLR-350: Henri -- i just the previous patch. If you make another smaller one, i'll review and commit quickly. Thanks for all your work on this! Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12572080#action_12572080 ] Henri Biestro commented on SOLR-350: Otis, reading your requirements, I'd be considering using a Solr core to handle an indexed version of multicore.xml; if you have a few thousands indices, it might be convenient to use queries in some occasions to select/retrieve one/many of them. The xml version of the multicore persistent file could be written at application/multicore shutdown and the Lucene based one could be recreated at application/multicore startup; creating a new index would just induce creating a new document in the multicore core (and in fact all CRUD operations could be handled that way) and we'd benefit from Solr autocommit feature al, tackling your functional requirements reusing well-known capabilities code. For ~10 indices/cores, this seems like overkill though... On configuring easily the data/index dir from multicore.xml, it seems we all agree that variables definitions should be able to allow just that; the non-extensible version of the feature (see previous comment)- where we dont allow the user to augment the environment but only expose 'solr.multicore.*'- did not trigger any comment yet, Otis/Hoss/Ryan what do you think of it ? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12571282#action_12571282 ] Otis Gospodnetic commented on SOLR-350: --- I haven't followed the patches, and I quickly read through the last month's worth of comments here. One thing that Hoss said caught my attention: ...easy way to reuse the same solrconfig.xml for multiple cores and still get subtle changes in behavior - all while making it transparent what any one solrconfig.xml will do... Please count this as my +1 for this. Yes, one use case is that each core is unique and thus needs unique configs, but I also have a concrete use case where all cores are identical as far as the configs go, all that needs to be different is the data directory where the index lives. In this case, it would be ideal if one could have a single copy of the schema.xml and solrconfig.xml, and specify core-specific settings (e.g. data/index dir) in multicore.xml. It would be even better if configs for cores were not all in a single/monolithic file - imagine a situation where you have thousands or even tends of thousands of indices and you add a few hundred or a few thousand new ones every day, throughout the day. You could certainly regenerate the whole multicore.xml file every time a new index is added, but it would be much more efficient to generate just the descriptor for that single new index that was just created, and tell Solr - hey, look here, there is a new core/index you need to be aware of. Perhaps one way to deal with this is to expose an API (URL) to send such a hey, look here message to Solr, and let Solr periodically write out multicore.xml to disk. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12568992#action_12568992 ] Henri Biestro commented on SOLR-350: Regarding c/variables/properties, imho we can definitely tackle the bulk of it in here, no need for another issue yet. On that topic, one small nag regarding multicore.xml serialization; do we want multicore.xml serialization to retain expressions if any (ie serailize them back as expressions) or not? Seems like it would be convenient to be able to distribute the same multicore.xml across several hosts - which may have different envs. As of now, we do expand all expressions before parsing resource files; if multicore.xml uses expressions based on environment variables, these will be expanded before we even have a chance to see them which precludes being able to write them back. Since we will have to serialize variables in multicore.xml, one workaround would be for users to declare local variables for each env based expressions (as multicore global properties) and only use those locals (keeping those definitions before expansion that is). Parsing multicore.xml would make one pass before expansion to extract the 'multicore/property' 'core/property' raw expressions, then expand the whole. (implementation/self note: MultiCore CoreDescriptor need to be able to define/serialize properties). Would this be ok / needed? Thoughts ? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[
https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12569018#action_12569018
]
Ryan McKinley commented on SOLR-350:
Regarding c/variables/properties, imho we can definitely tackle the bulk of
it in here, no need for another issue yet.
I think we should try to wrap up thins without properties, then open a new
issue for them. They are functionally different enough. As a note, I'm using
this multicore patch with system variables for the data path in each
solrconfig.xml -- this gives the same behavior you were looking for.
dataDir${solr.data}/corename//dataDir
In my view the one thing we need to fix before getting this patch commited is
the returning results for unloaded cores...
http://www.nabble.com/Multicore---Querying-unloaded-core-returns-results-from-default-td15469303.html
Manage Multiple SolrCores
-
Key: SOLR-350
URL: https://issues.apache.org/jira/browse/SOLR-350
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Ryan McKinley
Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch,
SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch
In SOLR-215, we enabled support for more then one SolrCore - but there is no
way to use them yet.
We need to make some interface to manage, register, modify avaliable SolrCores
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[
https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12569146#action_12569146
]
Henri Biestro commented on SOLR-350:
We need a way to define a global data root without having to define a system
env variable; can't we at least reintroduce the dataDir as a multicore
attribute?
The previous patch version went too far and was ignoring the solrconfig.xml
dataDir specification, but having no way to describe where all data go easily
is really too inconvenient.
Can't we find something acceptable in between ?
Strawman solution would be, if dataDir is *not* specified in solrconfig.xml,
use the previous patch code ?
Hopefully more acceptable, only provide a minimum set of variables with no
possibility to define any for now ? The env would only contain
'solr.multicore.{home,data}' and for each core,'solr.multicore.core.instance'
(I'm reluctant to expose 'sol.multicore.core.name', explanation follows...)
This would not preclude extending variables later and would not delay solr-350
by much now.
We used {{dataDir${solr.data}/corename//dataDir}} to illustrate the
variable solution but I grow feeling uneasy seeing the core *name* as a
variable part of a path (explicit or implicit): if we issue a SWAP command, how
do we end up in a proper state when we stop/start the container without
swapping the directory contents as well ?
My rationale is that the *instanceDir* is really what physically identifies a
core in a persistent manner wrt SWAP/stop/start; when we specify a data root,
the data directory should somehow depend on the instanceDir as well.
For instance, with {{core name=books instanceDir=books,0'.../}} and
{{core name=books-dev instanceDir=books,1.../}} ; even if both use the
same data root _'/solr/data'_, the 'books' core will use
_'/solr/data/books,0/'_ as dataDir and 'books-dev' will use
_'/solr/data/books,1'_.
When we swap('books', 'books-dev') , everything is still ok; 'books' now refers
to_'/solr/data/books,1'_ and books-dev refers to _'/solr/data/books,0/'_ . If
we stop/start the container, since nothing physically persistent depended on
the name, variable substitution (or implicit expansion) can not interfere.
If we are using the core *name* to build data directories, issuing swap is
likely to break something...
Please correct me if I'm deeply misunderstanding something...
Manage Multiple SolrCores
-
Key: SOLR-350
URL: https://issues.apache.org/jira/browse/SOLR-350
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Ryan McKinley
Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch,
SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch
In SOLR-215, we enabled support for more then one SolrCore - but there is no
way to use them yet.
We need to make some interface to manage, register, modify avaliable SolrCores
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12566421#action_12566421 ] Hoss Man commented on SOLR-350: --- just to clarify, i still haven't looked at the patch closely (I trust Ryan/Henri's judgment on bulk of the multicore implementation ... i mainly just want to sanity cehck the concepts and configs) ... but I have just a few follow up questions/clarifications about some of the issues i mentioned before... a) by requestParser configuration back within each core you mean all of the requestDispatcher configuration, correct? (currently requestParser and handleSelect ... likely to be httpCaching as well) i mainly just want to be sure that moving forward we think it makes sense for each solrconfig.xml to have it's own requestDispatcher section containing info on how the SolrDispatchFilter should deal with requests for the core using that config. b) (constructing a) SolrRequestParsers instance seems pretty lightweight to me ... is there any think specific you're worried about that i'm not noticing? c) should i open a separate issue for dealing with generalizing variables (and note that corename and dataDir are two prime use cases) ? it seems like that can definitely be dealt with *after* the bulk of the stuff in this issue is committed. d) anyone have any thoughts regarding my question about abortOnConfigurationError and what it *should* mean when dealing with dynamically loaded cores (i'm pretty sure right now it's ignored for any dynamically loaded cores ... i'm just wondering if that's what we want it to do) Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12564457#action_12564457 ] Henri Biestro commented on SOLR-350: I'm confused and dont see the dataDir element parsing you are referring to in solrconfig.xml; my current understanding is that the dataDir is deduced from the instance dir if not specified explicitly at core construction time. Are you proposing to add it (and/or instanceDir) to solrconfig.xml? Anyway, the current patch code allows both dataDir instanceDir to be specified as multicore core attributes (and everything related to file/directory locations is contained within multicore.xml); it treats absolute directory specifications (ie starting with '/') as such, core specification having precedence over multicore. If the core specified instanceDir is absolute, it is used as is and the dataDir is made relative to it if not absolute. Otherwise, the instanceDir is relative to the multicore instanceDir; If the core specified dataDir is absolute, it is used as such otherwise the core dataDir is relative to the multicore dataDir. When left unspecified, everything behaves relative to the multicore implied instanceDir or as current defaults. If you still find this is a bad solution, I'm confident you Ryan will agree on the good one; just let me know, I'll (try to) code it (if you want). Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[
https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12564212#action_12564212
]
Hoss Man commented on SOLR-350:
---
I agree generalizing variables is somewhat significant, and a larger scope then
just what's being talked about here -- perhaps that's part of the disconnect
... I'm taking it as a given that it's a problem that needs to be solved before
multicores can really be useful -- so if we have to solve that problem, and
that solution can also solve the common dataDir problem, let's not have an
alternate solution to hte dataDir problem that is non transparent to people
reading the configs.
(my assumption being based on the impression that we can't really support a lot
of the use cases people have talked about without having at a minimum a way to
know use the name of the current core as a variable in the configs --
postCommit hooks being one example of a place where this info will be crucial)
In a nutshell: if we know we are going to need variables, then instead of
introducing a new multicore dataDir=... option now (which if used changes
the meaning of the dataDir/) let's solve the broader problem of passing
arbitrary variables to a SolrCore. we can still commit all of the other stuff
you guys have been working on, lets just set the dataDir issue aside until we
add the variable support.
*BUT!!!* part of your comment has me worried that i'm misunderstanding how
multicore dataDir=... works, you just said...
bq. The multicore dataDir attributes is a default directory/roots that can be
overriden by core definitions
...how can it be overridden? My understanding based on your early comment was
that multicore dataDir=... was the directory that the
dataDir.../dataDir options in each solrconfig.xml would be relative to
...do you mean that in the multicore.xml file, each core/ can have a dataDir
option? ... if so that doesn't really solve the concern I have: people should
be able to read a solrconfig.xml and understand when there are outside
inflluences on that config...
bq. Plus it could be argued that it increases the element of surprise or at
least the potential for side effects.
bq. If a solrconfig/schema refers to a variable that can be superseded in a
multicore.xml, the behavior of a core is explictly dependant on whether it is
loaded in a multicore configuration of not. I agree that being explicit rather
than implicit is better but this does modify behavior even deeper nevertheless.
I disagree ... it's true that using a variables approach the evaluation of a
solrconfig.xml would be dependent on the environment it's run in (ie: is there
a multicore.xml? are variables set in it? are any system properties set?) but
the evaluation of solrconfog.xml is already dependent on it's environment (ie:
what is the solr home? are any system properties set?) ... my point is that
when a human is *reading* a config with variables in it, it is crystal clear
that there is an environmental factor that will affect the behavior. If a
person reads a solrconfig.xml that contains this line...
{code}
dataDir${my.special.dir}/data/dataDir
{code}
...then it's very obvious that the location of the data will depends on the
environment the core is run in (in which my.special.dir must be set, either
as a system property or as a multicore.xml variable -- the point being it's an
*known* external factor). The approach you guys have been talking about though
(assuming i'm understanding it correctly) would take away that transparency --
people could look at a solrconfig.xml that looks like this...
dataDirdata/dataDir
...and that that could mean anything depending on whether or not this
solrconfig.xml is running in a multicore setup or not.
Manage Multiple SolrCores
-
Key: SOLR-350
URL: https://issues.apache.org/jira/browse/SOLR-350
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Ryan McKinley
Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch,
SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch
In SOLR-215, we enabled support for more then one SolrCore - but there is no
way to use them yet.
We need to make some interface to manage, register, modify avaliable SolrCores
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12563346#action_12563346 ] Henri Biestro commented on SOLR-350: Regarding introducing variables, this is tempting but this looks like a rather important feature for a rather limited need. Plus it could be argued that it increases the element of surprise or at least the potential for side effects. If a solrconfig/schema refers to a variable that can be superseded in a multicore.xml, the behavior of a core is explictly dependant on whether it is loaded in a multicore configuration of not. I agree that being explicit rather than implicit is better but this does modify behavior even deeper nevertheless. The door that variable introduction would open seems much wider than the functional hole is; the original breach was needed for the shared class loader, a common dataDir root is adressing the good practise to segregate data from configuration. We could introduce a configDir/schemaDir at multicore level to adress sharing config/schema sharing - although using multiple cores is usually related to different config/schema so reusing/sharing them does not look like a must-have feature. The multicore dataDir attributes is a default directory/roots that can be overriden by core definitions, the current convention is really limited in its effects to what's needed. Variables and the huge functional potential of a whole environment defined within Solr seem way beyond the current use-cases; if we follow the precedent of alias vs swap, we should retain the idea but wait till more needs emerge before implementing it, shouldn't we? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[
https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12562834#action_12562834
]
Hoss Man commented on SOLR-350:
---
hey guys ... i'm catching up on some Jira reading and this comment jumped out
at me...
bq. improved code related to configuration wrt absolute/relative locations:
allows core dataDir/instanceDir to be absolute or relative to multicore
(pseudo) instanceDir/dataDir.
(i'm guessing that's what the dataDir option in the multicore/ tag in Ryan's
example is for?)
this seems like a bad idea to me ... violating the principle of least suprise
and all. it will make the behavior of a solrconfig.xml file dependent on
whether or not it's being used in a multicore context or not.
I'd like to suggest that an aternate approach would be to generalize the
current system property based variable substitution to support arbitrary
key=val pairs specified when the SolrCore is constructed...
* we add new syntax to multicore.xml for declaring global properties
* MultiCore converts these global declarations into a SolrParams instance
* we also add syntax to multicore.xml for declaring properties specific to a
core.
* when MultiCore instantiates a core, it uses DefaultSolrParams to let the
specific properties override the global properties *and* to set a special
property containing the name of the core (ie: solr.core.name)
* if cloning a core is possible (i can't remember) MultiCore would reuse the
SolrParams from the source core, changing only the core name property
(solr.core.name)
* system properties with the same names as properties in multicore.xml would
trump anything from the configs (since they are a run time overrides)
{code}
multicore adminPath=/admin/multicore persistent=true
abortOnConfigurationErrortrue/abortOnConfigurationError
requestDispatcher handleSelect=true
requestParsers enableRemoteStreaming=false
multipartUploadLimitInKB=2048 /
/requestDispatcher
property name=alldata.dir/my/solr/basedir/property
property name=magicnumber32/property
!-- core0 gets props above, any other props in it's configs must come from
system props --
core name=core0 instanceDir=core0 /
core name=core1 instanceDir=core1
property name=dataDirfoo/property
/core
core name=core111 instanceDir=core1!-- note same instanceDir as
above--
!-- can reuse exact same instance dir as another core ${solr.core.name}
will be differnet --
property name=dataDirbar/property
!-- and now ${dataDir} will be different too --
/core
/multicore
{code}
This would not only give us the ability to have a common ${alldata.dir} for all
cores, but also an easy way to reuse the same solrconfig.xml for multiple cores
and still get subtle changes in behavior -- all while making it transparent
what any one solrconfig.xml will do.
Super powerful -- and (i think) pretty easy to implement... a new optional
SolrParams arg to the SolrCore, SolrConfig, and Config constructors, and
DOMUtil.substituteSystemProperties plus some code in MultiCore to create the
SolrParams (hmm, DOMUtil doesn't have a very friendly method for that yet, not
that big a deal though)
what do you think?
Manage Multiple SolrCores
-
Key: SOLR-350
URL: https://issues.apache.org/jira/browse/SOLR-350
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Ryan McKinley
Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch,
SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch
In SOLR-215, we enabled support for more then one SolrCore - but there is no
way to use them yet.
We need to make some interface to manage, register, modify avaliable SolrCores
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12562835#action_12562835 ] Hoss Man commented on SOLR-350: --- Actually, one more unrelated comment... bq. Looking good. I took your patch and removed all the 'default' stuff to make it in line with Hoss' observations in: ...i know i suggested moving anything related to the entire solr server into multicore.xml, but i've been looking at SolrDispatchFilter lately because of SOLR-127 and i'm starting to wonder if the requestDispatcher/ config options really need to be webapp wide. They are (currently) only used to construct a protected instance of SolrRequestParsers in SolrDispatchFilter.init, but that SolrRequestParsers is only needed in the doFilter method once we've already figured out what core we're using ... it's a fairly light weight class, so why not construct a new one in each call to doFilter (after we've determined the correct core) and leave those options core specific? (not to mention the HTTP caching options SOLR-127 is probably going to add to requestDispatcher/) ... And while i'm thinking about it ... what does abortOnConfigurationError=true mean in a multicore world when someone attempts to dynamicly load a core with a config error? Currently SolrDispatchFilter only looks at that setting on init ... is MultiCore goung to start checking it after each LOAD core action? will it cause the whole server to stop accepting requests or just do something special for that one core? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=12560234#action_12560234 ] Ryan McKinley commented on SOLR-350: Hi Henri- We're getting there but I had trouble applying this patch, can you post a new one with a few changes? 1. can you change your editor settings to use two spaces rather then tabs? In general, solr code should have two spaces rather then tabs or 4 spaces. 2. To avoid confusion with o.a.s.request.XMLWriter, can we call XmlWriter something else? XmlWriterHelper? XmlWriterUtils? 3. Can we make XmlWriter a package protected class in o.a.s.core? This way we don't have to make it part of the public API. If there is a need for it later, we can easily move it. Also, if it can be replaced with an off the shelf library, we can do that later without mucking anyone up. Thanks for your work and patience with this! Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553922 ] Henri Biestro commented on SOLR-350: RequestHandlers do not today know the path that requested them;I was merely proposing a possible functional extension through usage of aliases. As for core names, being able to carry which version/revision of the config/schema is in use is imho not complex and useful to many (using svn/cvs/webdav to store config/schema) Anyway, the 'aliases' idea is definitely not something you did find useful enough from the beginning and I'm obviously failing to make the case for it. Alas. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553983 ] Ryan McKinley commented on SOLR-350: RequestHandlers do not today know the path that requested them; aaah -- so if we need it later, we could add aliasing then? is imho not complex and useful to many (using svn/cvs/webdav to store config/schema) How does aliasing change this. What can you do that you could not do without it? I store my config/schema in svn and don't have any problems. Anyway, the 'aliases' idea is definitely not something you did find useful enough from the beginning If I understood what you gain, I could be convinced. Right now I just see it as the need to manage and maintain multiple names+one immutable name without any reason. Perhaps we can move forward without aliasing, and add it later if we find (and implement) a solid use case for it. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554045 ] Henri Biestro commented on SOLR-350: SWAP is an important feature to exploit multicore persistence is not production ready yet, so committing feels like the next logical step . Ryan, if possible, I'd appreciate and would greatly benefit from a quick/early review of the solr-315.patch peristence core creation code (XmWriter, CoreDescriptor; keep them or loose them?). As an upside on the ALIAS discussion, if when a use case shows up, I guess we will be ready! Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554053 ] Ryan McKinley commented on SOLR-350: just committed SOLR-350-Naming.patch Ryan, if possible, I'd appreciate and would greatly benefit from a quick/early review of the solr-315.patch peristence core creation code (XmWriter, CoreDescriptor; keep them or loose them?). I gave it a quick look this morning, but did not look too closely because all the 'alias' stuff ;) XmWriter and CoreDescriptor seem reasonable to me. The CoreDescriptor could be used to move both Config and Schema away from knowing what file opened them. Check SOLR-427 Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[
https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12554085
]
Henri Biestro commented on SOLR-350:
On aliases - for completeness - , I had this nagging thought I was missing
something...
Re-reading Hoss's proposal and crossing that with the 1000 unique names point
you made, there is in any case 1000 unique 'instanceDir' that need to be
provided; Hoss proposed to use the 'instanceDir' instead of a name and alias
that if I'm not mistaken.
I got side tracked by the fact that the instanceDir could be absolute which
would have introduced a deployment host 'hard' dependency and lost the
equivalence.
If we define an 'instanceRoot' (at the multicore level or at the core level)
and make the (core) instanceDir = instanceRoot + '/' + name, the uniqueness of
the core name would be put to its initial intended use (instead of just being a
by-product of the alias feature). In that case, at least one alias is
convenient so we can keep the 'url' constant across index revisions.
For instance, if you are using svn, you could have you instanceDir/{schema,
conf} versioned; when you have a new revision ready to go, you copy these over
using the instanceDir+,+revision-number and use that as a name (which isn't
too bad of a convention).
And then, there are maybe future features that could be added to use aliases
for other purpose...
Oh well...
Manage Multiple SolrCores
-
Key: SOLR-350
URL: https://issues.apache.org/jira/browse/SOLR-350
Project: Solr
Issue Type: Improvement
Affects Versions: 1.3
Reporter: Ryan McKinley
Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch,
SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, SOLR-350-Naming.patch,
solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch
In SOLR-215, we enabled support for more then one SolrCore - but there is no
way to use them yet.
We need to make some interface to manage, register, modify avaliable SolrCores
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553754 ] Ryan McKinley commented on SOLR-350: I have not looked at the recent patches yet... but I'm still wondering if there is any value to alias if we have a SWAP command? http://www.nabble.com/purpose-of-MultiCore--22default-22---to14268755.html#a14427376 Aliasing has me nervous about the maintining a unique ID and a name - it seems to just lead to a management/clarity problem. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553809 ] Ryan McKinley commented on SOLR-350: (be it, filtered queries, query parsers, etc) would open to easy ways to fit per-user/usage profiles behaviors. Are you saying there is a big win if you can get stats on: http://host/henri/select vs http://host/ryan/select when 'henri' and 'ryan' are both aliased to 'core1'? Perhaps? but mod_rewrite can do that and much much more (if you really wanted to). With the alias model, how would you reindex a running core and end up with an identical setup at the end? Unless I'm missing something, the new core would need a different name (id), and there would be a brief moment where the main core was not avaliable consider: core name=core0 alias=main ... / and all queries come to solr as: http://host/solr/main/... I would have to run: 1. LOAD core1 using same config as core0 2. send add commands to core1 3. UNALIAS main from core0 (now nothing is available at /main) 4. ALIAS main to core1 (not the persisted configuration is different then when we started) Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553882 ] Henri Biestro commented on SOLR-350: If we are making a new index - a new index version-, it can mean the schema and the config can change; I may change my analysis chain or schema but also warming queries, cache set up, etc. The config is thus not necessarily the same. I may also want to have the new setup tested by a group of users before I make it available to the whole population; http://host/productionl versus http://host/stage. I might even have automated tests that verify that some queries do return some expected documents. If we were to use the 'alias' to map behaviors, it seems more convenient to declare those within Solr than anywhere else; describing that http://host/ryan/select queries on core main with an automated fq author='ryan' should not force mod-rewrite usage imho. Finally, the 'alias' command as it stands, allows to redefine an alias (without havng to unalias first) so the sequence would be: (considering core name=core,0 alias=main ... /) LOAD core,1 // which could even be aliased as 'stage' at this time send adds to core,1 // when done, could run verifications on 'stage' ALIAS core,1 main // 'swap' so to speak, overwrites previous 'main' alias UNLOAD core,0 Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12553885 ] Ryan McKinley commented on SOLR-350: If we were to use the 'alias' to map behaviors, how would an alias map different behaviors? Alias just offer multiple ways to access the *same* core and the same behavior. RequestHandlers don't know what path requested them. My point about mod_rewrite was referring to the use case you referred to: making the log files easier to parse per user. Re production and stage, why do you need aliasing for that? each core has name - when 'stage' is ready -- it can swap with 'production' Finally, the 'alias' command as it stands, allows to redefine an alias (without havng to unalias first) so the sequence would be: (considering core name=core,0 alias=main ... /) LOAD core,1 // which could even be aliased as 'stage' at this time send adds to core,1 // when done, could run verifications on 'stage' ALIAS core,1 main // 'swap' so to speak, overwrites previous 'main' alias UNLOAD core,0 so if you serialize at the beginning, you have: core name=core,0 alias=main ... / at the end you have: core name=core,1 alias=main ... / if you run that every hour, do you end up with core,1000 or switch between them? This would require you ask MultiCore, what i the 'id' for the core sitting at 'main' before you can operate on it. Why add this complexity? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-Naming.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch, solr-350.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12542520 ] Henri Biestro commented on SOLR-350: Walter's suggestion is already in solr-409 (with libDir attribute name). I could not verify everything and wanted to be safe so I loaded an updated version of solr-350_409.patch in solr-409. There are some improvements in the admin webapp that is now multi core aware. (ie: you can switch from core to core). I also made a small change in Config.java; locateInstanceDir seems to look for sol.solr.home as an environment variable. I've quickly checked the deployment against the example starting with: java -Dsolr.home=`pwd`/multicore -jar start.jar . As soon as I'm more confident, I'll push the patch over solr-350. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12539352 ] Doug Steigerwald commented on SOLR-350: --- Any chance there's going to be support to view the admin interface for each core? Doesn't seem like it's possible currently. Also, the admin interface you do see is for the last core loaded and not the default core in the configuration. Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch, SOLR-350-MultiCore.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12529628 ] Stu Hood commented on SOLR-350: --- I feel like the suggested implementation is a re-imagining of the Tomcat Manager REST api (http://tomcat.apache.org/tomcat-6.0-doc/manager-howto.html). The main reason I like the idea of multiple cores in the same instance is to provide tighter integration between them: more like a conventional relational database, with multiple tables that have independent schemas (where Solr core == SQL table). Otherwise, having your servlet container managing the contexts just makes more sense, since that is what it is built for. Also, I think the core should be a parameter of the query, so that there is the possibility of querying multiple cores simultaneously. Having a top-level controller managing dispatch to the cores opens up all kinds of possibilities for future expansion, (such as joins between indexes?) and it would make things like federated search much more elegant. SOLR-303 already has a shards parameter with the same idea behind it: just prefix local cores with the @ symbol, and you are good to go. Loving the potential here! Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526207 ] Yonik Seeley commented on SOLR-350: --- I assume core management stuff needs to be persistent if you add a core via the REST api, and the server restarts, you want it to still be there. So should multicore.xml be changed and written back in this case? Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (SOLR-350) Manage Multiple SolrCores
[ https://issues.apache.org/jira/browse/SOLR-350?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12526210 ] Ryan McKinley commented on SOLR-350: Yes, persistence seems like a good option. For the case where you are updating a live schema it may not make sense though. tempCore = load new core defaultCore = tempCore (close old core when all requests have finished) Manage Multiple SolrCores - Key: SOLR-350 URL: https://issues.apache.org/jira/browse/SOLR-350 Project: Solr Issue Type: Improvement Affects Versions: 1.3 Reporter: Ryan McKinley Attachments: SOLR-350-MultiCore.patch In SOLR-215, we enabled support for more then one SolrCore - but there is no way to use them yet. We need to make some interface to manage, register, modify avaliable SolrCores -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
