Re: {soft}Commit and cache flusing
Tim,
my suggestion was very concise, sorry for that. But not at all rude or
anything. Instead, tried to help you.
Dmitry
On Wed, Oct 9, 2013 at 9:28 PM, Tim Vaillancourt t...@elementspace.comwrote:
Apologies all. I think the suggestion that I was replying to get noticed
is what erked me, otherwise I would have moved on. I'll follow this advice.
Cheers,
Tim
On 9 October 2013 05:20, Erick Erickson erickerick...@gmail.com wrote:
Tim:
I think you're mis-interpreting. By replying to a post with the subject:
{soft}Commit and cache flushing
but going in a different direction, it's easy for people to think I'm
not interested in that
thread, I'll ignore it, thereby missing the fact that you're asking a
somewhat different
question that they might have information about. It's not about whether
you're
doing anything particularly wrong with the question. It's about making
it easy for
people to help.
See http://people.apache.org/~hossman/#threadhijack
Best,
Erick
On Tue, Oct 8, 2013 at 6:23 PM, Tim Vaillancourt t...@elementspace.com
wrote:
I have a genuine question with substance here. If anything this
nonconstructive, rude response was to get noticed. Thanks for
contributing to the discussion.
Tim
On 8 October 2013 05:31, Dmitry Kan solrexp...@gmail.com wrote:
Tim,
I suggest you open a new thread and not reply to this one to get
noticed.
Dmitry
On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt
t...@elementspace.com
wrote:
Is there a way to make autoCommit only commit if there are pending
changes,
ie: if there are 0 adds pending commit, don't autoCommit
(open-a-searcher
and wipe the caches)?
Cheers,
Tim
On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote:
right. We've got the autoHard commit configured only atm. The
soft-commits
are controlled on the client. It was just easier to implement the
first
version of our internal commit policy that will commit to all solr
instances at once. This is where we have noticed the reported
behavior.
On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam
bram.van...@intix.eu
wrote:
if there are no modifications to an index and a softCommit or
hardCommit
issued, then solr flushes the cache.
Indeed. The easiest way to work around this is by disabling auto
commits
and only commit when you have to.
Re: {soft}Commit and cache flusing
Tim:
I think you're mis-interpreting. By replying to a post with the subject:
{soft}Commit and cache flushing
but going in a different direction, it's easy for people to think I'm
not interested in that
thread, I'll ignore it, thereby missing the fact that you're asking a
somewhat different
question that they might have information about. It's not about whether you're
doing anything particularly wrong with the question. It's about making
it easy for
people to help.
See http://people.apache.org/~hossman/#threadhijack
Best,
Erick
On Tue, Oct 8, 2013 at 6:23 PM, Tim Vaillancourt t...@elementspace.com wrote:
I have a genuine question with substance here. If anything this
nonconstructive, rude response was to get noticed. Thanks for
contributing to the discussion.
Tim
On 8 October 2013 05:31, Dmitry Kan solrexp...@gmail.com wrote:
Tim,
I suggest you open a new thread and not reply to this one to get noticed.
Dmitry
On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt t...@elementspace.com
wrote:
Is there a way to make autoCommit only commit if there are pending
changes,
ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher
and wipe the caches)?
Cheers,
Tim
On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote:
right. We've got the autoHard commit configured only atm. The
soft-commits
are controlled on the client. It was just easier to implement the first
version of our internal commit policy that will commit to all solr
instances at once. This is where we have noticed the reported behavior.
On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu
wrote:
if there are no modifications to an index and a softCommit or
hardCommit
issued, then solr flushes the cache.
Indeed. The easiest way to work around this is by disabling auto
commits
and only commit when you have to.
Re: {soft}Commit and cache flusing
Apologies all. I think the suggestion that I was replying to get noticed
is what erked me, otherwise I would have moved on. I'll follow this advice.
Cheers,
Tim
On 9 October 2013 05:20, Erick Erickson erickerick...@gmail.com wrote:
Tim:
I think you're mis-interpreting. By replying to a post with the subject:
{soft}Commit and cache flushing
but going in a different direction, it's easy for people to think I'm
not interested in that
thread, I'll ignore it, thereby missing the fact that you're asking a
somewhat different
question that they might have information about. It's not about whether
you're
doing anything particularly wrong with the question. It's about making
it easy for
people to help.
See http://people.apache.org/~hossman/#threadhijack
Best,
Erick
On Tue, Oct 8, 2013 at 6:23 PM, Tim Vaillancourt t...@elementspace.com
wrote:
I have a genuine question with substance here. If anything this
nonconstructive, rude response was to get noticed. Thanks for
contributing to the discussion.
Tim
On 8 October 2013 05:31, Dmitry Kan solrexp...@gmail.com wrote:
Tim,
I suggest you open a new thread and not reply to this one to get
noticed.
Dmitry
On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt t...@elementspace.com
wrote:
Is there a way to make autoCommit only commit if there are pending
changes,
ie: if there are 0 adds pending commit, don't autoCommit
(open-a-searcher
and wipe the caches)?
Cheers,
Tim
On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote:
right. We've got the autoHard commit configured only atm. The
soft-commits
are controlled on the client. It was just easier to implement the
first
version of our internal commit policy that will commit to all solr
instances at once. This is where we have noticed the reported
behavior.
On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu
wrote:
if there are no modifications to an index and a softCommit or
hardCommit
issued, then solr flushes the cache.
Indeed. The easiest way to work around this is by disabling auto
commits
and only commit when you have to.
Re: {soft}Commit and cache flusing
Tim, I suggest you open a new thread and not reply to this one to get noticed. Dmitry On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt t...@elementspace.comwrote: Is there a way to make autoCommit only commit if there are pending changes, ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher and wipe the caches)? Cheers, Tim On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote: right. We've got the autoHard commit configured only atm. The soft-commits are controlled on the client. It was just easier to implement the first version of our internal commit policy that will commit to all solr instances at once. This is where we have noticed the reported behavior. On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu wrote: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Indeed. The easiest way to work around this is by disabling auto commits and only commit when you have to.
Re: {soft}Commit and cache flusing
I have a genuine question with substance here. If anything this nonconstructive, rude response was to get noticed. Thanks for contributing to the discussion. Tim On 8 October 2013 05:31, Dmitry Kan solrexp...@gmail.com wrote: Tim, I suggest you open a new thread and not reply to this one to get noticed. Dmitry On Mon, Oct 7, 2013 at 9:44 PM, Tim Vaillancourt t...@elementspace.com wrote: Is there a way to make autoCommit only commit if there are pending changes, ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher and wipe the caches)? Cheers, Tim On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote: right. We've got the autoHard commit configured only atm. The soft-commits are controlled on the client. It was just easier to implement the first version of our internal commit policy that will commit to all solr instances at once. This is where we have noticed the reported behavior. On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu wrote: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Indeed. The easiest way to work around this is by disabling auto commits and only commit when you have to.
Re: {soft}Commit and cache flusing
Is there a way to make autoCommit only commit if there are pending changes, ie: if there are 0 adds pending commit, don't autoCommit (open-a-searcher and wipe the caches)? Cheers, Tim On 2 October 2013 00:52, Dmitry Kan solrexp...@gmail.com wrote: right. We've got the autoHard commit configured only atm. The soft-commits are controlled on the client. It was just easier to implement the first version of our internal commit policy that will commit to all solr instances at once. This is where we have noticed the reported behavior. On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu wrote: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Indeed. The easiest way to work around this is by disabling auto commits and only commit when you have to.
Re: {soft}Commit and cache flusing
if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Indeed. The easiest way to work around this is by disabling auto commits and only commit when you have to.
Re: {soft}Commit and cache flusing
right. We've got the autoHard commit configured only atm. The soft-commits are controlled on the client. It was just easier to implement the first version of our internal commit policy that will commit to all solr instances at once. This is where we have noticed the reported behavior. On Wed, Oct 2, 2013 at 9:32 AM, Bram Van Dam bram.van...@intix.eu wrote: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Indeed. The easiest way to work around this is by disabling auto commits and only commit when you have to.
{soft}Commit and cache flusing
Hello! This is a minor thing, perhaps, but thought to ask / share: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Is this designed on purpose? Regards, Dmitry
Re: {soft}Commit and cache flusing
On 10/1/2013 2:48 AM, Dmitry Kan wrote: This is a minor thing, perhaps, but thought to ask / share: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Any time you do a commit that opens a new Searcher object (openSearcher=true, which is required if you want index changes to be visible to people making queries), the caches are invalidated. This is because the layout of the index (and therefore the Lucene internal IDs) can completely change with *any* commit/merge, and there is no easy and reliable way to determine when the those numbers have NOT changed. If you have warming queries configured, those happen on the new searcher, populating the new cache. If you have cache autoWarming configured, then keys from the old caches are re-queried against the new index and used to populate the new cache. I do not understand deep Lucene internals, but what I've seen come through Jira activity and commits over the last year or two has been a strong move towards per-segment thinking instead of whole-index thinking. If this idea becomes applicable to all aspects of Lucene, then perhaps Solr caches can also become per-segment, and will not need to be completely invalidated except in the case of a major merge or forceMerge. Thanks, Shawn
Re: {soft}Commit and cache flusing
Thanks a lot Shawn for an exhaustive reply! Regards, Dmitry On Tue, Oct 1, 2013 at 5:37 PM, Shawn Heisey s...@elyograg.org wrote: On 10/1/2013 2:48 AM, Dmitry Kan wrote: This is a minor thing, perhaps, but thought to ask / share: if there are no modifications to an index and a softCommit or hardCommit issued, then solr flushes the cache. Any time you do a commit that opens a new Searcher object (openSearcher=true, which is required if you want index changes to be visible to people making queries), the caches are invalidated. This is because the layout of the index (and therefore the Lucene internal IDs) can completely change with *any* commit/merge, and there is no easy and reliable way to determine when the those numbers have NOT changed. If you have warming queries configured, those happen on the new searcher, populating the new cache. If you have cache autoWarming configured, then keys from the old caches are re-queried against the new index and used to populate the new cache. I do not understand deep Lucene internals, but what I've seen come through Jira activity and commits over the last year or two has been a strong move towards per-segment thinking instead of whole-index thinking. If this idea becomes applicable to all aspects of Lucene, then perhaps Solr caches can also become per-segment, and will not need to be completely invalidated except in the case of a major merge or forceMerge. Thanks, Shawn
