Re: Queries to help warm up (mmap)
One of the tricks I had read somewhere was to cat all files in the index directory and OS will have file in the disk cache. On Thu, Oct 6, 2016 at 11:55 AM, Rallavagu <rallav...@gmail.com> wrote: > Looking for clues/recommendations to help warm up during startup. Not > necessarily Solr caches but mmap as well. I have used some like "q= name>:[* TO *]" for various fields and it seems to help with mmap > population around 40-50%. Is there anything else that could help achieve > 90% or more? Thanks. >
Re: Queries to help warm up (mmap)
I use the schema browser to find the 20 most common words. I use those, assuming that they’ll be the most common in queries. Those are static warming queries in solrconfig.xml. This works fairly well for book or movie titles. Not so well for free text. You could do the same thing with query log analysis. Use your most frequent queries. wunder Walter Underwood wun...@wunderwood.org http://observer.wunderwood.org/ (my blog) > On Oct 6, 2016, at 8:55 AM, Rallavagu <rallav...@gmail.com> wrote: > > Looking for clues/recommendations to help warm up during startup. Not > necessarily Solr caches but mmap as well. I have used some like "q= name>:[* TO *]" for various fields and it seems to help with mmap population > around 40-50%. Is there anything else that could help achieve 90% or more? > Thanks.
Queries to help warm up (mmap)
Looking for clues/recommendations to help warm up during startup. Not necessarily Solr caches but mmap as well. I have used some like "q=:[* TO *]" for various fields and it seems to help with mmap population around 40-50%. Is there anything else that could help achieve 90% or more? Thanks.
Does MMap works on the Virtual Box?
Hi All, I have a big index of 256 GB .Right now it is on one physical box of 256 GB RAM . I am planning to virtualize it to the size of 32 GB Ram*8 boxes.Whether the MMap will work regardless in this condition ? Vibhor Jaiswal -- View this message in context: http://lucene.472066.n3.nabble.com/Does-MMap-works-on-the-Virtual-Box-tp4085154.html Sent from the Solr - User mailing list archive at Nabble.com.
Re: Does MMap works on the Virtual Box?
On 8/16/2013 1:02 PM, vibhoreng04 wrote: I have a big index of 256 GB .Right now it is on one physical box of 256 GB RAM . I am planning to virtualize it to the size of 32 GB Ram*8 boxes.Whether the MMap will work regardless in this condition ? As far as MMap goes, if the operating system you are running is 64-bit, your Java is 64-bit, and the OS supports MMap (which almost every operating system does, including Linux and Windows), then you'd be fine. If you have the option of running Solr on bare metal vs. running on the same hardware in a virtualized environment, you should always choose the bare metal. I had a Solr installation with a sharded index. When I first set it up, I used virtual machines, one Solr instance and shard per VM. Half the VMs were running on one physical box, half on another. For redundancy, I had a second pair of physical servers doing the same thing, each with VMs representing half the index. That same setup now runs on bare metal -- the exact same physical machines, in fact. The index arrangement is nearly the same as before, except it uses multicore Solr, one instance per machine. Removing the virtualization layer helped performance quite a bit. Average QTimes went way down and it took less time to do a full index rebuild. Thanks, Shawn
Re: Does MMap works on the Virtual Box?
Hi, You can MMAP a size bigger than your memory without having any problem. Part of your file will just not be loaded into RAM, because you don't access it too often. If you are short in memory, consider deactivating page Host IO Caching, as it will be only redundant with your guest OS page cache. Regards, Paul On Fri, Aug 16, 2013 at 10:26 PM, Shawn Heisey s...@elyograg.org wrote: On 8/16/2013 1:02 PM, vibhoreng04 wrote: I have a big index of 256 GB .Right now it is on one physical box of 256 GB RAM . I am planning to virtualize it to the size of 32 GB Ram*8 boxes.Whether the MMap will work regardless in this condition ? As far as MMap goes, if the operating system you are running is 64-bit, your Java is 64-bit, and the OS supports MMap (which almost every operating system does, including Linux and Windows), then you'd be fine. If you have the option of running Solr on bare metal vs. running on the same hardware in a virtualized environment, you should always choose the bare metal. I had a Solr installation with a sharded index. When I first set it up, I used virtual machines, one Solr instance and shard per VM. Half the VMs were running on one physical box, half on another. For redundancy, I had a second pair of physical servers doing the same thing, each with VMs representing half the index. That same setup now runs on bare metal -- the exact same physical machines, in fact. The index arrangement is nearly the same as before, except it uses multicore Solr, one instance per machine. Removing the virtualization layer helped performance quite a bit. Average QTimes went way down and it took less time to do a full index rebuild. Thanks, Shawn -- __ Masurel Paul e-mail: paul.masu...@gmail.com
Why solr 4.0 use FSIndexOutput to write file, otherwise MMap/NIO
I have checked the FSDirectory, it will create MMapDirectory or NIOFSDirectory for Directory. This two directory only supply IndexInput extend for read file (MMapIndexInput extends ByteBufferIndexInput), why not there is not MMap/NIO IndexOutput extend for file write. It only use FSIndexOutput for file write(FSIndexOutput extends BufferedIndexOutput). Does FSIndexOutput wirte file very slow than MMap/NIO? How to improve the IO write performance. Thanks, __ Jeffery Wang Application Service - Backend Morningstar (Shenzhen) Ltd. Morningstar. Illuminating investing worldwide. +86 755 3311 0220 Office +86 130 7782 2813 Mobile jeffery.w...@morningstar.commailto:jeffery.w...@morningstar.com This e-mail contains privileged and confidential information and is intended only for the use of the person(s) named above. Any dissemination, distribution or duplication of this communication without prior written consent from Morningstar is strictly prohibited. If you received this message in error please contact the sender immediately and delete the materials from any computer.
Re: Why solr 4.0 use FSIndexOutput to write file, otherwise MMap/NIO
Output is quite a bit simpler than input because all we do is write a single stream of bytes with no seeking (append only), and it's done with only one thread, so I don't think there'd be much to gain by using the newer IO APIs for writing... Mike McCandless http://blog.mikemccandless.com On Fri, Jun 28, 2013 at 2:23 AM, Jeffery Wang jeffery.w...@morningstar.com wrote: I have checked the FSDirectory, it will create MMapDirectory or NIOFSDirectory for Directory. This two directory only supply IndexInput extend for read file (MMapIndexInput extends ByteBufferIndexInput), why not there is not MMap/NIO IndexOutput extend for file write. It only use FSIndexOutput for file write(FSIndexOutput extends BufferedIndexOutput). Does FSIndexOutput wirte file very slow than MMap/NIO? How to improve the IO write performance. Thanks, __ Jeffery Wang Application Service - Backend Morningstar (Shenzhen) Ltd. Morningstar. Illuminating investing worldwide. +86 755 3311 0220 Office +86 130 7782 2813 Mobile jeffery.w...@morningstar.commailto:jeffery.w...@morningstar.com This e-mail contains privileged and confidential information and is intended only for the use of the person(s) named above. Any dissemination, distribution or duplication of this communication without prior written consent from Morningstar is strictly prohibited. If you received this message in error please contact the sender immediately and delete the materials from any computer.
Re: Mmap
Any thought on this? Is the default Mmap?
Sent from my mobile device
720-256-8076
On Feb 14, 2012, at 7:16 AM, Bill Bell billnb...@gmail.com wrote:
Does someone have an example of using unmap in 3.5 and chunksize?
I am using Solr 3.5.
I noticed in solrconfig.xml:
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.StandardDirectoryFactory}/
I don't see this parameter taking.. When I set
-Dsolr.directoryFactory=solr.MMapDirectoryFactory
How do I see the setting in the log or in stats.jsp ? I cannot find a place
that indicates it is set or not.
I would assume StandardDirectoryFactory is being used but I do see (when I
set it or NOT set it)
Bill Bell
Sent from mobile
Re: Mmap
Yep.
-Dsolr.directoryFactory=solr.SimpleFSDirectoryFactory
or
-Dsolr.directoryFactory=solr.MMapDirectoryFactory
works great.
On Mon, Jul 16, 2012 at 7:55 PM, Michael Della Bitta
michael.della.bi...@appinions.com wrote:
Hi Bill,
Standard picks one for you. Otherwise, you can hardcode the
DirectoryFactory in your config file, or I believe if you specify
-Dsolr.solr.directoryFactory=solr.MMapDirectoryFactory
That will get you what you want.
Michael Della Bitta
Appinions, Inc. -- Where Influence Isn’t a Game.
http://www.appinions.com
On Mon, Jul 16, 2012 at 9:32 PM, Bill Bell billnb...@gmail.com wrote:
Any thought on this? Is the default Mmap?
Sent from my mobile device
720-256-8076
On Feb 14, 2012, at 7:16 AM, Bill Bell billnb...@gmail.com wrote:
Does someone have an example of using unmap in 3.5 and chunksize?
I am using Solr 3.5.
I noticed in solrconfig.xml:
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.StandardDirectoryFactory}/
I don't see this parameter taking.. When I set
-Dsolr.directoryFactory=solr.MMapDirectoryFactory
How do I see the setting in the log or in stats.jsp ? I cannot find a place
that indicates it is set or not.
I would assume StandardDirectoryFactory is being used but I do see (when I
set it or NOT set it)
Bill Bell
Sent from mobile
--
Bill Bell
billnb...@gmail.com
cell 720-256-8076
Mmap
Does someone have an example of using unmap in 3.5 and chunksize?
I am using Solr 3.5.
I noticed in solrconfig.xml:
directoryFactory name=DirectoryFactory
class=${solr.directoryFactory:solr.StandardDirectoryFactory}/
I don't see this parameter taking.. When I set
-Dsolr.directoryFactory=solr.MMapDirectoryFactory
How do I see the setting in the log or in stats.jsp ? I cannot find a place
that indicates it is set or not.
I would assume StandardDirectoryFactory is being used but I do see (when I set
it or NOT set it)
Bill Bell
Sent from mobile
Fwd: Lucene MMAP Usage with Solr
Folks,
I am trying to get Lucene MMAP to work in solr.
I am assuming that when I configure MMAP the entire index will be loaded
into RAM.
Is that the right assumption ?
I have tried the following ways for using MMAP:
Option 1. Using the solr config below for MMAP configuration
-Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
With this config, when I start solr with a 30G index, I expected that the
RAM usage should go up, but it did not.
Option 2. By Code Change
I made the following code change :
Changed org.apache.solr.core.StandardDirectoryFactory to use MMAP instead
of FSDirectory.
Code snippet pasted below.
Could you help me to understand if these are the right way to use MMAP?
Thanks much
/ST.
Code SNippet for Option 2:
package org.apache.solr.core;
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the License); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
*http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import java.io.File;
import java.io.IOException;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
/**
* Directory provider which mimics original Solr FSDirectory based behavior.
*
*/
public class StandardDirectoryFactory extends DirectoryFactory {
public Directory open(String path) throws IOException {
return MMapDirectory.open(new File(path));
}
}
Re: Fwd: Lucene MMAP Usage with Solr
I thought that was the way to use it (but I've never had to use it myself) and
that it means memory through the roof, yes.
If you look at the Solr Admin statistics page, does it show you which Directory
you are using?
For example, on 1 Solr instance I'm looking at I see:
readerDir : org.apache.lucene.store.NIOFSDirectory@/mnt/
Otis
--
Sematext is hiring -- http://sematext.com/about/jobs.html?mls
Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR
- Original Message
From: ST ST stst2...@gmail.com
To: solr-user@lucene.apache.org
Sent: Fri, November 13, 2009 6:03:57 PM
Subject: Fwd: Lucene MMAP Usage with Solr
Folks,
I am trying to get Lucene MMAP to work in solr.
I am assuming that when I configure MMAP the entire index will be loaded
into RAM.
Is that the right assumption ?
I have tried the following ways for using MMAP:
Option 1. Using the solr config below for MMAP configuration
-Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory
With this config, when I start solr with a 30G index, I expected that the
RAM usage should go up, but it did not.
Option 2. By Code Change
I made the following code change :
Changed org.apache.solr.core.StandardDirectoryFactory to use MMAP instead
of FSDirectory.
Code snippet pasted below.
Could you help me to understand if these are the right way to use MMAP?
Thanks much
/ST.
Code SNippet for Option 2:
package org.apache.solr.core;
/**
* Licensed to the Apache Software Foundation (ASF) under one or more
* contributor license agreements. See the NOTICE file distributed with
* this work for additional information regarding copyright ownership.
* The ASF licenses this file to You under the Apache License, Version 2.0
* (the License); you may not use this file except in compliance with
* the License. You may obtain a copy of the License at
*
*http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
import java.io.File;
import java.io.IOException;
import org.apache.lucene.store.Directory;
import org.apache.lucene.store.MMapDirectory;
/**
* Directory provider which mimics original Solr FSDirectory based behavior.
*
*/
public class StandardDirectoryFactory extends DirectoryFactory {
public Directory open(String path) throws IOException {
return MMapDirectory.open(new File(path));
}
}
Set MMap in Solr
Hi, I'm trying to set my default directory to MMap. I saw that this is done by specifying here A DirectoryProvider plugin can be configured in solrconfig.xml with the following XML: directoryProvider class=class.name !-- Parameters as required by the implementation -- /directoryProvider in solrconfig.xml. This did not work for me when I put in the MMapDirectory class name. I got this information from here http://issues.apache.org/jira/browse/SOLR-465?focusedCommentId=12715282#action_12715282 I'm using the latest nightly build. If anyone knows how to configure solr to use MMap, please let me know. I would greatly appreciate it. Thanks.
Re: Set MMap in Solr
Thanks for the help. -Brad Anderson 2009/11/5 Otis Gospodnetic otis_gospodne...@yahoo.com To use MMapDirectory, invoke Java with the System property org.apache.lucene.FSDirectory.class set to org.apache.lucene.store.MMapDirectory. This will cause FSDirectory.getDirectory(File,boolean) to return instances of this class. So, start your servlet container with -Dorg.apache.lucene.FSDirectory.class=org.apache.lucene.store.MMapDirectory Otis -- Sematext is hiring -- http://sematext.com/about/jobs.html?mls Lucene, Solr, Nutch, Katta, Hadoop, HBase, UIMA, NLP, NER, IR - Original Message From: ba ba solrm...@gmail.com To: solr-user@lucene.apache.org Sent: Thu, November 5, 2009 2:55:42 PM Subject: Set MMap in Solr Hi, I'm trying to set my default directory to MMap. I saw that this is done by specifying here A DirectoryProvider plugin can be configured in solrconfig.xml with the following XML: in solrconfig.xml. This did not work for me when I put in the MMapDirectory class name. I got this information from here http://issues.apache.org/jira/browse/SOLR-465?focusedCommentId=12715282#action_12715282 I'm using the latest nightly build. If anyone knows how to configure solr to use MMap, please let me know. I would greatly appreciate it. Thanks.
