Hi,
so, the app already has a database connection because it updates the
READ flag when the user clicks an entry, right? If you only need the
flag for display purposes, it sounds like it would make sense to also
fetch it directly from the database when displaying the listing. Of
course if you also need to search for READ/UNREAD you need to index the
change, but perhaps you could get away with it taking longer.
--Ere
20.12.2016, 4.12, Lasitha Wattaladeniya kirjoitti:
Hi Shawn,
Thanks for your well detailed explanation. Now I understand, I won't be
able to achieve the 100ms softcommit timeout with my hardware setup.
However let's say someone has a requirement as below (quoted from my
previous mail)
*Requirement *is, we are showing a list of entries on a page. For each
user there's a read / unread flag. The data for listing is fetched from
solr. And you can see the entry was previously read or not. So when a user
views an entry by clicking. We are updating the database flag to READ and
use real time indexing to update solr index. So when the user close the
full view of the entry and go back to entry listing page, the data fetched
from solr should be updated to READ.
Can't we achieve a requirement as described above using solr ? (without
manipulating the previously fetched results list from solr, because at some
point we'll have to go back to search results from solr and at that time it
should be updated).
Regards,
Lasitha
Lasitha Wattaladeniya
Software Engineer
Mobile : +6593896893
Blog : techreadme.blogspot.com
On Mon, Dec 19, 2016 at 6:37 PM, Shawn Heisey <apa...@elyograg.org> wrote:
On 12/18/2016 7:09 PM, Lasitha Wattaladeniya wrote:
@eric : thanks for the lengthy reply. So let's say I increase the
autosoftcommit time out to may be 100 ms. In that case do I have to
wait much that time from client side before calling search ?. What's
the correct way of achieving this?
Some of the following is covered by the links you've already received.
Some of it may be new information.
Before you can see a change you've just made, you will need to wait for
the commit to be fired (in this case, the autoSoftCommit time) plus
however long it actually takes to complete the commit and open a new
searcher. Opening the searcher is the expensive part.
What I typically recommend that people do is have the autoSoftCommit
time as long as they can stand, with 60-300 seconds as a "typical"
value. That's a setting of 60000 to 300000. What you are trying to
achieve is much faster, and much more difficult.
100 milliseconds will typically be far too small a value unless your
index is extremely small or your hardware is incredibly fast and has a
lot of memory. With a value of 100, you'll want each of those soft
commits (which do open a new searcher) to take FAR less than 100
milliseconds to complete. This kind of speed can be difficult to
achieve, especially if the index is large.
To have any hope of fast commit times, you will need to set
autowarmCount on all Solr caches to zero. If you are indexing
frequently enough, you might even want to completely disable Solr's
internal caches, because they may be providing no benefit.
You will want to have enough extra memory that your operating system can
cache the vast majority (or even maybe all) of your index.
https://wiki.apache.org/solr/SolrPerformanceProblems
Some other info that's helpful for understanding why plenty of *spare*
memory (not allocated by programs) is necessary for good performance:
https://en.wikipedia.org/wiki/Page_cache
http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-64bit.html
The reason in a nutshell: Disks are EXTREMELY slow. Memory is very fast.
Thanks,
Shawn
--
Ere Maijala
Kansalliskirjasto / The National Library of Finland
