[
https://issues.apache.org/jira/browse/SOLR-5316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
olivier soyez updated SOLR-5316:
--------------------------------
Description:
The SOLR-5316.patch is for the lotsofcores use case: a large number of
homogeneous cores with frequent loading/unloading. The new Cores options are:
- defaultLoadOnStartup : true by default
- defaultTransient : false by default
In this way, you can change the default attributes value for all cores.
The lotsofcores use case requires to have defaultLoadOnStartup = false and
defaultTransient = true
- "numBuckets" to create a subdirectory based on a hash on the corename %
numBuckets in the core Datadir, because all cores cannot live in the same
directory
- "auto" with 2 differents values :
1) createLoad: create, if not exist, and load the core on the fly on
the first incoming request (update, select).
2) onlyLoad: load the core on the fly on the first incoming request
(update, select), if exist on disk
The auto option uses an additional cores option named baseDataDir to
automatically generate the dataDir of each core and uses the "numBucket" option
if exists. All the cores have the same solr config defined in the
solr_home/conf.
- discoverOnDemand : false by default. If enabled, the discovery and loading of
cores at the solr startup are disabled and a new option named coreDiscovery in
the STATUS action is available to discover on demand all the cores in the
coreRootDirectory (only add the cores descriptions)
- noCorePropertiesFile : false by default. If enabled, don't use anymore the
core.properties file, a core will be detected on disk based on the file
segments.gen
The unload command was also modified in order to handle the non loaded
transient cores, to be able to apply for example the deleteIndex, deleteDataDir
options.
I added some Junit tests.
An example of solr.xml:
<solr>
<str name="shareSchema">schema.xml</str>
<int name="transientCacheSize">5000</int>
<str name="coreRootDirectory">solr_home/data</str>
<str name="baseDataDir">solr_home/data</str>
<int name="numBuckets">100</int>
<str name="auto">createLoad</str>
<str name="defaultLoadOnStartup">false</str>
<str name="defaultTransient">true</str>
<bool name="discoverOnDemand">true</bool>
<bool name="noCorePropertiesFile">true</bool>
</solr>
The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision
number 1556554)
---------------
The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case,
including some modification :
- by default, all cores have loadOnStartup="false" and transient="true"
attributes
- the create admin command can register a lazy core (to take into account the
transientCacheSize option)
- add transient cores persistency
- handle unload admin command for never launched transient cores (non active
cores)
To improve performance, we use this Solr patched version with the persistence
disabled.
In this way, Solr is working with a solr.xml file without any core entries,
because it's useless in our use case (with the new Auto option for the cores)
The new Cores options :
- "numBuckets" to create a subdirectory based on a hash on the corename
% numBuckets in the core Datadir, because all cores cannot live in the
same directory
- "Auto" with 3 differents values :
1) false : default behaviour
2) createLoad : create, if not exist, and load the core on the fly on
the first incoming request (update, select).
3) onlyLoad : load the core on the fly on the first incoming request
(update, select), if exist on disk
The Auto option uses an additional cores option named baseDataDir to
automatically generate the dataDir of each core and uses the "numBucket" option
if exists.
was:
The SOLR-5316.patch is for the lotsofcores use case: a large number of
homogeneous cores with frequent loading/unloading. The new Cores options are:
- defaultLoadOnStartup : true by default
- defaultTransient : false by default
In this way, you can change the default attributes value for all cores.
The lotsofcores use case requires to have defaultLoadOnStartup = false and
defaultTransient = true
- "numBuckets" to create a subdirectory based on a hash on the corename %
numBuckets in the core Datadir, because all cores cannot live in the same
directory
- "auto" with 2 differents values :
1) createLoad: create, if not exist, and load the core on the fly on
the first incoming request (update, select).
2) onlyLoad: load the core on the fly on the first incoming request
(update, select), if exist on disk
The auto option uses an additional cores option named baseDataDir to
automatically generate the dataDir of each core and uses the "numBucket" option
if exists. All the cores have the same solr config defined in the
solr_home/conf.
- discoverOnDemand : false by default. If enabled, the discovery and loading of
cores at the solr startup are disabled and a new option named coreDiscovery in
the STATUS action is available to discover on demand all the cores in the
coreRootDirectory (only add the cores descriptions)
- noCorePropertiesFile : false by default. If enabled, don't use anymore the
core.properties file, a core will be detected on disk based on the file
segments.gen
The unload command was also modified in order to handle the non loaded
transient cores, to be able to apply for example the deleteIndex, deleteDataDir
options.
I added some Junit tests.
An example of solr.xml:
<solr>
<str name="shareSchema">schema.xml</str>
<int name="transientCacheSize">5000</int>
<str name="coreRootDirectory">solr_home/data</str>
<str name="baseDataDir">solr_home/data</str>
<int name="numBuckets">100</int>
<str name="auto">createLoad</str>
<str name="defaultLoadOnStartup">false</str>
<str name="defaultTransient">true</str>
<bool name="discoverOnDemand">true</bool>
<bool name="noCorePropertiesFile">true</bool>
</solr>
The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision
number 1556554)
---------------
This patch is for the lotsofcores use case, including some modification :
- by default, all cores have loadOnStartup="false" and transient="true"
attributes
- the create admin command can register a lazy core (to take into account the
transientCacheSize option)
- add transient cores persistency
- handle unload admin command for never launched transient cores (non active
cores)
To improve performance, we use this Solr patched version with the persistence
disabled.
In this way, Solr is working with a solr.xml file without any core entries,
because it's useless in our use case (with the new Auto option for the cores)
The new Cores options :
- "numBuckets" to create a subdirectory based on a hash on the corename
% numBuckets in the core Datadir, because all cores cannot live in the
same directory
- "Auto" with 3 differents values :
1) false : default behaviour
2) createLoad : create, if not exist, and load the core on the fly on
the first incoming request (update, select).
3) onlyLoad : load the core on the fly on the first incoming request
(update, select), if exist on disk
The Auto option uses an additional cores option named baseDataDir to
automatically generate the dataDir of each core and uses the "numBucket" option
if exists.
> Solr 4.2.1 LotsOfCores new options
> ----------------------------------
>
> Key: SOLR-5316
> URL: https://issues.apache.org/jira/browse/SOLR-5316
> Project: Solr
> Issue Type: Improvement
> Components: multicore
> Affects Versions: 4.2.1, 4.7
> Reporter: olivier soyez
> Assignee: Erick Erickson
> Priority: Minor
> Labels: patch
> Fix For: 4.2.1
>
> Attachments: solr-4.2.1.patchLOTSOFCORES
>
>
> The SOLR-5316.patch is for the lotsofcores use case: a large number of
> homogeneous cores with frequent loading/unloading. The new Cores options are:
> - defaultLoadOnStartup : true by default
> - defaultTransient : false by default
> In this way, you can change the default attributes value for all cores.
> The lotsofcores use case requires to have defaultLoadOnStartup = false and
> defaultTransient = true
> - "numBuckets" to create a subdirectory based on a hash on the corename %
> numBuckets in the core Datadir, because all cores cannot live in the same
> directory
> - "auto" with 2 differents values :
> 1) createLoad: create, if not exist, and load the core on the fly on
> the first incoming request (update, select).
> 2) onlyLoad: load the core on the fly on the first incoming request
> (update, select), if exist on disk
> The auto option uses an additional cores option named baseDataDir to
> automatically generate the dataDir of each core and uses the "numBucket"
> option if exists. All the cores have the same solr config defined in the
> solr_home/conf.
> - discoverOnDemand : false by default. If enabled, the discovery and loading
> of cores at the solr startup are disabled and a new option named
> coreDiscovery in the STATUS action is available to discover on demand all the
> cores in the coreRootDirectory (only add the cores descriptions)
> - noCorePropertiesFile : false by default. If enabled, don't use anymore the
> core.properties file, a core will be detected on disk based on the file
> segments.gen
> The unload command was also modified in order to handle the non loaded
> transient cores, to be able to apply for example the deleteIndex,
> deleteDataDir options.
> I added some Junit tests.
> An example of solr.xml:
> <solr>
> <str name="shareSchema">schema.xml</str>
> <int name="transientCacheSize">5000</int>
> <str name="coreRootDirectory">solr_home/data</str>
> <str name="baseDataDir">solr_home/data</str>
> <int name="numBuckets">100</int>
> <str name="auto">createLoad</str>
> <str name="defaultLoadOnStartup">false</str>
> <str name="defaultTransient">true</str>
> <bool name="discoverOnDemand">true</bool>
> <bool name="noCorePropertiesFile">true</bool>
> </solr>
> The patch SOLR-5316.patch in attachment is for svn solr branch_4X (revision
> number 1556554)
> ---------------
> The patch solr-4.2.1.patchLOTSOFCORES is for the lotsofcores use case,
> including some modification :
> - by default, all cores have loadOnStartup="false" and transient="true"
> attributes
> - the create admin command can register a lazy core (to take into account the
> transientCacheSize option)
> - add transient cores persistency
> - handle unload admin command for never launched transient cores (non active
> cores)
> To improve performance, we use this Solr patched version with the persistence
> disabled.
> In this way, Solr is working with a solr.xml file without any core entries,
> because it's useless in our use case (with the new Auto option for the cores)
> The new Cores options :
> - "numBuckets" to create a subdirectory based on a hash on the corename
> % numBuckets in the core Datadir, because all cores cannot live in the
> same directory
> - "Auto" with 3 differents values :
> 1) false : default behaviour
> 2) createLoad : create, if not exist, and load the core on the fly on
> the first incoming request (update, select).
> 3) onlyLoad : load the core on the fly on the first incoming request
> (update, select), if exist on disk
> The Auto option uses an additional cores option named baseDataDir to
> automatically generate the dataDir of each core and uses the "numBucket"
> option if exists.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
