Re: openSearcher, default commit settings
Thanks for clearing this up. The wiki, being an authoritative reference, needs
to be corrected.
Re. default commit settings. I agree educating developers is very essential.
But in reality, you can't rely on this as the sole mechanism for ensuring
proper usage of the update API, especially for calls such as commit,
optimize, expungeDeletes which can be very expensive for large indexes on a
shared infrastructure.
The issue is, there's no control mechanism in Solr for update calls (cf.
rewriting calls via load-balancer). Once you expose the update handler to the
developers, they could send 10 commit/optimise op per min, opening new
searchers for each of those calls (openSearcher is only configurable for
autocommit). And there is nothing you can do about it in Solr, even as a
immediate stopgap while a fix is being implemented for the next sprint.
It's be good to have some consistency in terms of configuring handlers, i.e.
having default/invariant settings for both the search and update handlers.
Thanks,
Boon
-
Boon Low
Search Engineer, DCT Family History
On 29 May 2014, at 18:03, Shawn Heisey
s...@elyograg.orgmailto:s...@elyograg.org wrote:
On 5/29/2014 9:21 AM, Boon Low wrote:
1. openSearcher (autoCommit)
According to the Apache Solr reference, autoCommit/openSearcher is set to
false by default.
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
But on Solr v4.8.1, if openSearcher is omitted from the autoCommit config,
new searchers are opened and warmed post auto-commits. Is this behaviour
intended or the wiki wrong?
I am reasonably certain that the default for openSearcher if it is not
specified will always be true. My understanding and your actual
experience says that the documentation is wrong. Additional note: The
docs for autoSoftCommit are basically a footnote on autoCommit, which I
think is a mistake -- it should have its own section, and the docs
should mention that openSearcher does not apply.
I think the code confirms this. From SolrConfig.java:
protected UpdateHandlerInfo loadUpdatehandlerInfo() {
return new UpdateHandlerInfo(get(updateHandler/@class,null),
getInt(updateHandler/autoCommit/maxDocs,-1),
getInt(updateHandler/autoCommit/maxTime,-1),
getBool(updateHandler/autoCommit/openSearcher,true),
getInt(updateHandler/commitIntervalLowerBound,-1),
getInt(updateHandler/autoSoftCommit/maxDocs,-1),
getInt(updateHandler/autoSoftCommit/maxTime,-1),
getBool(updateHandler/commitWithin/softCommit,true));
}
2. openSearcher and other default commit settings
From previous posts, I know it's not possible to disable commits completely in
Solr config (without coding). But is there a way to configure the default
settings of hard/explicit commits for the update handler? If not it makes sense
to have a configuration mechanism. Currently, a simple commit call seems to be
hard-wired with the following options:
..
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
There's no server-side option, e.g. to set openSearcher=false as default or
invariant (cf. searchHandler) to prevent new searchers from opening.
I found that at times it is necessary to have better server- or
infrastructure-side controls for update/commits, especially in agile teams.
Client/UI developers do not necessarily have complete Solr knowledge.
Unintended commits from misbehaving client-side updates may be norm (e.g. 10
times per minute!).
Since you want to handle commits automatically, you'll want to educate
your developers and tell them that they should never send commits -- let
Solr handle it. If the code that talks to Solr is Java and uses SolrJ,
you might want to consider using forbidden-apis in your project so that
a build will fail if the commit method gets used.
https://code.google.com/p/forbidden-apis/
Thanks,
Shawn
__
brightsolid is used in this email to mean brightsolid online technology
limited.
Email Disclaimer
This message is confidential and may contain privileged information. You should
not disclose its contents to any other person. If you are not the intended
recipient, please notify the sender named above immediately. It is expressly
declared that this e-mail does not constitute nor form part of a contract or
unilateral obligation. Opinions, conclusions and other information in this
message that do not relate to the official business of brightsolid shall be
understood as neither given nor endorsed by it.
__
This email has been scanned by the brightsolid Email Security System. Powered by
McAfee.
__
Re: openSearcher, default commit settings
Boon,
I expect you will find many definitions of “proper usage” depending upon
context and expected results. Personally, don’t believe this is Solr’s job to
enforce, and there are many ways through the use of directives in the servlet
container layer that can allow restrictions if you feel this is required.
I would recommend considering an abstraction layer if you feel your development
team may (accidentally) abuse the system they are permitted to use. I’ve seen
this employed very well with minimal latency and cost in extremely large
corporations that have many multiple development teams using the same search
infrastructure.
Jason
On Jun 2, 2014, at 3:53 AM, Boon Low boon@dctfh.com wrote:
Thanks for clearing this up. The wiki, being an authoritative reference,
needs to be corrected.
Re. default commit settings. I agree educating developers is very essential.
But in reality, you can't rely on this as the sole mechanism for ensuring
proper usage of the update API, especially for calls such as commit,
optimize, expungeDeletes which can be very expensive for large indexes on
a shared infrastructure.
The issue is, there's no control mechanism in Solr for update calls (cf.
rewriting calls via load-balancer). Once you expose the update handler to the
developers, they could send 10 commit/optimise op per min, opening new
searchers for each of those calls (openSearcher is only configurable for
autocommit). And there is nothing you can do about it in Solr, even as a
immediate stopgap while a fix is being implemented for the next sprint.
It's be good to have some consistency in terms of configuring handlers, i.e.
having default/invariant settings for both the search and update handlers.
Thanks,
Boon
-
Boon Low
Search Engineer, DCT Family History
On 29 May 2014, at 18:03, Shawn Heisey
s...@elyograg.orgmailto:s...@elyograg.org wrote:
On 5/29/2014 9:21 AM, Boon Low wrote:
1. openSearcher (autoCommit)
According to the Apache Solr reference, autoCommit/openSearcher is set to
false by default.
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
But on Solr v4.8.1, if openSearcher is omitted from the autoCommit config,
new searchers are opened and warmed post auto-commits. Is this behaviour
intended or the wiki wrong?
I am reasonably certain that the default for openSearcher if it is not
specified will always be true. My understanding and your actual
experience says that the documentation is wrong. Additional note: The
docs for autoSoftCommit are basically a footnote on autoCommit, which I
think is a mistake -- it should have its own section, and the docs
should mention that openSearcher does not apply.
I think the code confirms this. From SolrConfig.java:
protected UpdateHandlerInfo loadUpdatehandlerInfo() {
return new UpdateHandlerInfo(get(updateHandler/@class,null),
getInt(updateHandler/autoCommit/maxDocs,-1),
getInt(updateHandler/autoCommit/maxTime,-1),
getBool(updateHandler/autoCommit/openSearcher,true),
getInt(updateHandler/commitIntervalLowerBound,-1),
getInt(updateHandler/autoSoftCommit/maxDocs,-1),
getInt(updateHandler/autoSoftCommit/maxTime,-1),
getBool(updateHandler/commitWithin/softCommit,true));
}
2. openSearcher and other default commit settings
From previous posts, I know it's not possible to disable commits completely
in Solr config (without coding). But is there a way to configure the default
settings of hard/explicit commits for the update handler? If not it makes
sense to have a configuration mechanism. Currently, a simple commit call
seems to be hard-wired with the following options:
..
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
There's no server-side option, e.g. to set openSearcher=false as default or
invariant (cf. searchHandler) to prevent new searchers from opening.
I found that at times it is necessary to have better server- or
infrastructure-side controls for update/commits, especially in agile teams.
Client/UI developers do not necessarily have complete Solr knowledge.
Unintended commits from misbehaving client-side updates may be norm (e.g. 10
times per minute!).
Since you want to handle commits automatically, you'll want to educate
your developers and tell them that they should never send commits -- let
Solr handle it. If the code that talks to Solr is Java and uses SolrJ,
you might want to consider using forbidden-apis in your project so that
a build will fail if the commit method gets used.
https://code.google.com/p/forbidden-apis/
Thanks,
Shawn
__
brightsolid is used in this email to mean brightsolid online technology
limited.
Email Disclaimer
This message is confidential and
openSearcher, default commit settings
Hi,
1. openSearcher (autoCommit)
According to the Apache Solr reference, autoCommit/openSearcher is set to
false by default.
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
But on Solr v4.8.1, if openSearcher is omitted from the autoCommit config,
new searchers are opened and warmed post auto-commits. Is this behaviour
intended or the wiki wrong?
2. openSearcher and other default commit settings
From previous posts, I know it's not possible to disable commits completely in
Solr config (without coding). But is there a way to configure the default
settings of hard/explicit commits for the update handler? If not it makes sense
to have a configuration mechanism. Currently, a simple commit call seems to be
hard-wired with the following options:
..
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
There's no server-side option, e.g. to set openSearcher=false as default or
invariant (cf. searchHandler) to prevent new searchers from opening.
I found that at times it is necessary to have better server- or
infrastructure-side controls for update/commits, especially in agile teams.
Client/UI developers do not necessarily have complete Solr knowledge.
Unintended commits from misbehaving client-side updates may be norm (e.g. 10
times per minute!).
Regards,
Boon
-
Boon Low
Search Engineer,
DCT Family History
__
brightsolid is used in this email to mean brightsolid online technology
limited.
Email Disclaimer
This message is confidential and may contain privileged information. You should
not disclose its contents to any other person. If you are not the intended
recipient, please notify the sender named above immediately. It is expressly
declared that this e-mail does not constitute nor form part of a contract or
unilateral obligation. Opinions, conclusions and other information in this
message that do not relate to the official business of brightsolid shall be
understood as neither given nor endorsed by it.
__
This email has been scanned by the brightsolid Email Security System. Powered by
McAfee.
__
Re: openSearcher, default commit settings
On 5/29/2014 9:21 AM, Boon Low wrote:
1. openSearcher (autoCommit)
According to the Apache Solr reference, autoCommit/openSearcher is set to
false by default.
https://cwiki.apache.org/confluence/display/solr/UpdateHandlers+in+SolrConfig
But on Solr v4.8.1, if openSearcher is omitted from the autoCommit config,
new searchers are opened and warmed post auto-commits. Is this behaviour
intended or the wiki wrong?
I am reasonably certain that the default for openSearcher if it is not
specified will always be true. My understanding and your actual
experience says that the documentation is wrong. Additional note: The
docs for autoSoftCommit are basically a footnote on autoCommit, which I
think is a mistake -- it should have its own section, and the docs
should mention that openSearcher does not apply.
I think the code confirms this. From SolrConfig.java:
protected UpdateHandlerInfo loadUpdatehandlerInfo() {
return new UpdateHandlerInfo(get(updateHandler/@class,null),
getInt(updateHandler/autoCommit/maxDocs,-1),
getInt(updateHandler/autoCommit/maxTime,-1),
getBool(updateHandler/autoCommit/openSearcher,true),
getInt(updateHandler/commitIntervalLowerBound,-1),
getInt(updateHandler/autoSoftCommit/maxDocs,-1),
getInt(updateHandler/autoSoftCommit/maxTime,-1),
getBool(updateHandler/commitWithin/softCommit,true));
}
2. openSearcher and other default commit settings
From previous posts, I know it's not possible to disable commits completely
in Solr config (without coding). But is there a way to configure the default
settings of hard/explicit commits for the update handler? If not it makes
sense to have a configuration mechanism. Currently, a simple commit call
seems to be hard-wired with the following options:
..
commit{,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=false,prepareCommit=false}
There's no server-side option, e.g. to set openSearcher=false as default or
invariant (cf. searchHandler) to prevent new searchers from opening.
I found that at times it is necessary to have better server- or
infrastructure-side controls for update/commits, especially in agile teams.
Client/UI developers do not necessarily have complete Solr knowledge.
Unintended commits from misbehaving client-side updates may be norm (e.g. 10
times per minute!).
Since you want to handle commits automatically, you'll want to educate
your developers and tell them that they should never send commits -- let
Solr handle it. If the code that talks to Solr is Java and uses SolrJ,
you might want to consider using forbidden-apis in your project so that
a build will fail if the commit method gets used.
https://code.google.com/p/forbidden-apis/
Thanks,
Shawn
