from:"Per Steffensen \(JIRA\)"

[jira] [Commented] (SOLR-3561) Error during deletion of shard/core

2016-10-04 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-3561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15546190#comment-15546190
 ] 

Per Steffensen commented on SOLR-3561:
--

I originally created the ticket. I am not against closing it. I do not know if 
the problem still exists (in some shape), but a lot of things has changed 
since, so someone will have to bring up the problem again if it is still a 
problem.

> Error during deletion of shard/core
> ---
>
> Key: SOLR-3561
> URL: https://issues.apache.org/jira/browse/SOLR-3561
> Project: Solr
>  Issue Type: Bug
>  Components: multicore, replication (java), SolrCloud
>Affects Versions: 4.0-ALPHA
> Environment: Solr trunk (4.0-SNAPSHOT) from 29/2-2012
>Reporter: Per Steffensen
>Assignee: Mark Miller
> Fix For: 4.9, 6.0
>
>
> Running several Solr servers in Cloud-cluster (zkHost set on the Solr 
> servers).
> Several collections with several slices and one replica for each slice (each 
> slice has two shards)
> Basically we want let our system delete an entire collection. We do this by 
> trying to delete each and every shard under the collection. Each shard is 
> deleted one by one, by doing CoreAdmin-UNLOAD-requests against the relevant 
> Solr
> {code}
> CoreAdminRequest request = new CoreAdminRequest();
> request.setAction(CoreAdminAction.UNLOAD);
> request.setCoreName(shardName);
> CoreAdminResponse resp = request.process(new CommonsHttpSolrServer(solrUrl));
> {code}
> The delete/unload succeeds, but in like 10% of the cases we get errors on 
> involved Solr servers, right around the time where shard/cores are deleted, 
> and we end up in a situation where ZK still claims (forever) that the deleted 
> shard is still present and active.
> Form here the issue is easilier explained by a more concrete example:
> - 7 Solr servers involved
> - Several collection a.o. one called "collection_2012_04", consisting of 28 
> slices, 56 shards (remember 1 replica for each slice) named 
> "collection_2012_04_sliceX_shardY" for all pairs in {X:1..28}x{Y:1,2}
> - Each Solr server running 8 shards, e.g Solr server #1 is running shard 
> "collection_2012_04_slice1_shard1" and Solr server #7 is running shard 
> "collection_2012_04_slice1_shard2" belonging to the same slice "slice1".
> When we decide to delete the collection "collection_2012_04" we go through 
> all 56 shards and delete/unload them one-by-one - including 
> "collection_2012_04_slice1_shard1" and "collection_2012_04_slice1_shard2". At 
> some point during or shortly after all this deletion we see the following 
> exceptions in solr.log on Solr server #7
> {code}
> Aug 1, 2012 12:02:50 AM org.apache.solr.common.SolrException log
> SEVERE: Error while trying to recover:org.apache.solr.common.SolrException: 
> core not found:collection_2012_04_slice1_shard1
> request: 
> http://solr_server_1:8983/solr/admin/cores?action=PREPRECOVERY=collection_2012_04_slice1_shard1=solr_server_7%3A8983_solr=solr_server_7%3A8983_solr_collection_2012_04_slice1_shard2=recovering=true=6000=javabin=2
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
> at 
> org.apache.solr.common.SolrExceptionPropagationHelper.decodeFromMsg(SolrExceptionPropagationHelper.java:29)
> at 
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:445)
> at 
> org.apache.solr.client.solrj.impl.CommonsHttpSolrServer.request(CommonsHttpSolrServer.java:264)
> at 
> org.apache.solr.cloud.RecoveryStrategy.sendPrepRecoveryCmd(RecoveryStrategy.java:188)
> at 
> org.apache.solr.cloud.RecoveryStrategy.doRecovery(RecoveryStrategy.java:285)
> at org.apache.solr.cloud.RecoveryStrategy.run(RecoveryStrategy.java:206)
> Aug 1, 2012 12:02:50 AM org.apache.solr.common.SolrException log
> SEVERE: Recovery failed - trying again...
> Aug 1, 2012 12:02:51 AM org.apache.solr.cloud.LeaderElector$1 process
> WARNING:
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.get(ArrayList.java:322)
> at org.apache.solr.cloud.LeaderElector.checkIfIamLeader(LeaderElector.java:96)
> at org.apache.solr.cloud.LeaderElector.access$000(LeaderElector.java:57)
> at org.apache.solr.cloud.LeaderElector$1.process(LeaderElector.java:121)
> at 
> org.apache.zookeeper.ClientCnxn$EventThread.processEvent(ClientCnxn.java:531)
> at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:507)
> Aug 1, 2012 12:02:51 AM org.apache.solr.cloud.LeaderElector$1 process
> {code}
> Im

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2016-08-01 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15402138#comment-15402138
 ] 

Per Steffensen commented on SOLR-6810:
--

I have not looked much into SOLR-8220, but from the little reading I have done, 
I guess you would have to doc-value the id-field for SOLR-8220 to help on the 
SOLR-6810 issue.

I guess, most systems do not have id-field doc-valued. I also guess, for most 
systems it is feasible to do reindex it all, so that id's become doc-value. But 
not for all systems - e.g. not for our systems. We have 1000 billion documents 
in one of our systems, and you do not just reindex all of that. It takes weeks 
and is a very complex operation (we have tried it a few times). I do not know 
if such an argument counts here, but anyway...

> Faster searching limited but high rows across many shards all with many hits
> 
>
> Key: SOLR-6810
> URL: https://issues.apache.org/jira/browse/SOLR-6810
> Project: Solr
>  Issue Type: Improvement
>  Components: search
>Reporter: Per Steffensen
>Assignee: Shalin Shekhar Mangar
>  Labels: distributed_search, performance
> Attachments: SOLR-6810-hack-eoe.patch, SOLR-6810-trunk.patch, 
> SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, 
> branch_5x_rev1642874.patch, branch_5x_rev1645549.patch
>
>
> Searching "limited but high rows across many shards all with many hits" is 
> slow
> E.g.
> * Query from outside client: q=something=1000
> * Resulting in sub-requests to each shard something a-la this
> ** 1) q=something=1000=id,score
> ** 2) Request the full documents with ids in the global-top-1000 found among 
> the top-1000 from each shard
> What does the subject mean
> * "limited but high rows" means 1000 in the example above
> * "many shards" means 200-1000 in our case
> * "all with many hits" means that each of the shards have a significant 
> number of hits on the query
> The problem grows on all three factors above
> Doing such a query on our system takes between 5 min to 1 hour - depending on 
> a lot of things. It ought to be much faster, so lets make it.
> Profiling show that the problem is that it takes lots of time to access the 
> store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
> Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
> really no good reason to ever read information from store for more than the 
> overall top-1000 documents, that has to be returned to the client.
> For further detail see mail-thread "Slow searching limited but high rows 
> across many shards all with high hits" started 13/11-2014 on 
> dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-8922) DocSetCollector can allocate massive garbage on large indexes

2016-04-20 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-8922?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15249790#comment-15249790
 ] 

Per Steffensen commented on SOLR-8922:
--

bq. I don't know how well Per Steffensen tested this, although I got the 
impression he was using it in production at the time

Well, I know I am late, but just for the record. It was tested thoroughly and 
the gain was significant. It has been running in several production-systems 
since forever.

> DocSetCollector can allocate massive garbage on large indexes
> -
>
> Key: SOLR-8922
> URL: https://issues.apache.org/jira/browse/SOLR-8922
> Project: Solr
>  Issue Type: Improvement
>Reporter: Jeff Wartes
>Assignee: Yonik Seeley
> Attachments: SOLR-8922.patch, SOLR-8922.patch
>
>
> After reaching a point of diminishing returns tuning the GC collector, I 
> decided to take a look at where the garbage was coming from. To my surprise, 
> it turned out that for my index and query set, almost 60% of the garbage was 
> coming from this single line:
> https://github.com/apache/lucene-solr/blob/94c04237cce44cac1e40e1b8b6ee6a6addc001a5/solr/core/src/java/org/apache/solr/search/DocSetCollector.java#L49
> This is due to the simple fact that I have 86M documents in my shards. 
> Allocating a scratch array big enough to track a result set 1/64th of my 
> index (1.3M) is also almost certainly excessive, considering my 99.9th 
> percentile hit count is less than 56k.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-5670) _version_ either indexed OR docvalue

2016-03-11 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-5670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15190785#comment-15190785
 ] 

Per Steffensen commented on SOLR-5670:
--

Well, no, I have not. 

Our reason for doing this is not necessarily to make things faster. It 
basically boils down to avoiding OOMs. When you want to get a value for a 
particular field on a particular document in Solr/Lucene there are several 
places to find the information. You can find it in the store (if stored=true 
for that field), which is fairly slow. You can also find it in the FieldCache, 
which is an in-memory doc-id to field-value map. Problem with FieldCache is 
that if you load it, you need to load values for ALL the documents in the index 
(or at least the segment). If you have enormous amounts of documents in your 
index this can cause OOM, because you simple cannot have all those values in 
memory. Then doc-value is for the rescue. Doc-value basically holds the same 
doc-id to field-value data-structure as the FieldCache, but doc-values are 
maintained continuously in additional files in your index-folder, so you can 
just go read the particular value you need, hence avoiding OOM. FieldCache is 
something you "calculate" into memory, based on data form store or index.

> _version_ either indexed OR docvalue
> 
>
> Key: SOLR-5670
> URL: https://issues.apache.org/jira/browse/SOLR-5670
> Project: Solr
>  Issue Type: Improvement
>  Components: SolrCloud
>Affects Versions: 4.7
>Reporter: Per Steffensen
>Assignee: Per Steffensen
>  Labels: solr, solrcloud, version
> Fix For: 4.7, master
>
> Attachments: SOLR-5670.patch, SOLR-5670.patch
>
>
> As far as I can see there is no good reason to require that "_version_" field 
> has to be indexed if it is docvalued. So I guess it will be ok with a rule 
> saying "_version_ has to be either indexed or docvalue (allowed to be both)".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7890) By default require admin rights to access /security.json in ZK

2015-08-25 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14710861#comment-14710861
 ] 

Per Steffensen commented on SOLR-7890:
--

In general, looks good!

Like the JavaDoc you added to {{VMParamsAllAndReadonlyDigestZkACLProvider}}

Remember update documentation on cwiki, if you find appropriate.

Maybe {{ZkZnodeProtection}} should be closer connected to 
{{VMParamsAllAndReadonlyDigestZkACLProvider}} (e.g. inner class). It is only 
when you actively decide to use {{VMParamsAllAndReadonlyDigestZkACLProvider}} 
as your ACL-provider that {{ZkZnodeProtection}} is used. Default is still 
{{DefaultZkACLProvider}} which does not take {{ZkZnodeProtection}} into 
consideration. When {{ZkZnodeProtection}} is a separate standalone class, you 
might be surprised that it is actually only used when you actively pick 
{{VMParamsAllAndReadonlyDigestZkACLProvider}}.

I believe a few versions of Solr have already been released including 
SOLR-4580. Therefore you need to think about if Solr behaves appropriate when 
upgrading to a release including SOLR-7890. Lets say you have a Solr (version 
before SOLR-7890) installation where you have turned on 
{{VMParamsAllAndReadonlyDigestZkACLProvider}}, so that all your znodes are 
ACL'ed with all/admin/admin and read-only/client/client.
* If you do nothing, but read the release-notes including SOLR-7890 
description, you might expect that the ACL's of security.json (already exists 
in ZK when you start up your new Solr) is automatically changed to only have 
ACL all/admin/admin (and not read-only/client/client anymore). That will not 
happen.
* Same problem if you upgrade to the new Solr and provide an explicit 
{{-DzkProtectedPaths}} including znodes that already exist
There is a consistency issue here, that we at least need to be explicit about. 
You will not end up with the expected ACL's on protected-paths if those 
znodes already exist when you upgrade to an SOLR-7890 Solr. This problem also 
existed before SOLR-7890, but only if you changed your ACL-provider 
implementation, and in that case you have a better chance of guessing the issue 
as an administrator of the system. So I think this problem becomes more urgent 
to document/solve as we introduce SOLR-7890.
Same kind of problem will occur, if we in a later release add to the 
default-list 
{{VMParamsAllAndReadonlyDigestZkACLProvider.DEFAULT_PROTECTED_PATHS}}

I do know if we should just document this behavior, or make a solution. In 
case we want to make a solution, we might want to have some code that can run 
through the entire znode-structure and ensure that the ACL's are as they are 
supposed to be according to current code and configuration. Then run that code 
when Solr nodes start and/or as part of upgrading Solr-versions.

Sorry, that I do not have time for more than this quick comment.

 By default require admin rights to access /security.json in ZK
 --

 Key: SOLR-7890
 URL: https://issues.apache.org/jira/browse/SOLR-7890
 Project: Solr
  Issue Type: Sub-task
  Components: security
Reporter: Jan Høydahl
Assignee: Jan Høydahl
 Fix For: Trunk

 Attachments: SOLR-7890.patch


 Perhaps {{VMParamsAllAndReadonlyDigestZkACLProvider}} should by default 
 require admin access for read/write of {{/security.json}}, and other 
 sensitive paths. Today this is left to the user to implement.
 Also, perhaps factor out the already-known sensitive paths into a separate 
 class, so that various {{ACLProvider}} implementations can get a list of 
 paths that should be admin-only, read-only etc from one central place. Then 
 3rd party impls pulling ZK creds from elsewhere will still do the right thing 
 in the future if we introduce other sensitive Znodes...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2015-08-07 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14661611#comment-14661611
 ] 

Per Steffensen commented on SOLR-4470:
--

Hopefully I will soon have the time to see if this other implementation will 
fulfill our needs. But closing SOLR-4470 for now is fine

 Support for basic http auth in internal solr requests
 -

 Key: SOLR-4470
 URL: https://issues.apache.org/jira/browse/SOLR-4470
 Project: Solr
  Issue Type: New Feature
  Components: clients - java, multicore, replication (java), SolrCloud
Affects Versions: 4.0
Reporter: Per Steffensen
Assignee: Jan Høydahl
  Labels: authentication, https, solrclient, solrcloud, ssl
 Fix For: Trunk

 Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, 
 SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, 
 SOLR-4470_trunk_r1568857.patch


 We want to protect any HTTP-resource (url). We want to require credentials no 
 matter what kind of HTTP-request you make to a Solr-node.
 It can faily easy be acheived as described on 
 http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
 also make internal request to other Solr-nodes, and for it to work 
 credentials need to be provided here also.
 Ideally we would like to forward credentials from a particular request to 
 all the internal sub-requests it triggers. E.g. for search and update 
 request.
 But there are also internal requests
 * that only indirectly/asynchronously triggered from outside requests (e.g. 
 shard creation/deletion/etc based on calls to the Collection API)
 * that do not in any way have relation to an outside super-request (e.g. 
 replica synching stuff)
 We would like to aim at a solution where original credentials are 
 forwarded when a request directly/synchronously trigger a subrequest, and 
 fallback to a configured internal credentials for the 
 asynchronous/non-rooted requests.
 In our solution we would aim at only supporting basic http auth, but we would 
 like to make a framework around it, so that not to much refactoring is 
 needed if you later want to make support for other kinds of auth (e.g. digest)
 We will work at a solution but create this JIRA issue early in order to get 
 input/comments from the community as early as possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2015-06-08 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576732#comment-14576732
]

Per Steffensen commented on SOLR-4470:
--

Soon I will be looking into whether SOLR-7274 etc. can be used instead of this
SOLR-4470. And to what extend.

Support for basic http auth in internal solr requests
-

Key: SOLR-4470
URL: https://issues.apache.org/jira/browse/SOLR-4470
Project: Solr
Issue Type: New Feature
Components: clients - java, multicore, replication (java), SolrCloud
Affects Versions: 4.0
Reporter: Per Steffensen
Assignee: Jan Høydahl
Labels: authentication, https, solrclient, solrcloud, ssl
Fix For: Trunk

Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch,
SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch,
SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch,
SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch,
SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch,
SOLR-4470_trunk_r1568857.patch

We want to protect any HTTP-resource (url). We want to require credentials no
matter what kind of HTTP-request you make to a Solr-node.
It can faily easy be acheived as described on
http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes
also make internal request to other Solr-nodes, and for it to work
credentials need to be provided here also.
Ideally we would like to forward credentials from a particular request to
all the internal sub-requests it triggers. E.g. for search and update
request.
But there are also internal requests
* that only indirectly/asynchronously triggered from outside requests (e.g.
shard creation/deletion/etc based on calls to the Collection API)
* that do not in any way have relation to an outside super-request (e.g.
replica synching stuff)
We would like to aim at a solution where original credentials are
forwarded when a request directly/synchronously trigger a subrequest, and
fallback to a configured internal credentials for the
asynchronous/non-rooted requests.
In our solution we would aim at only supporting basic http auth, but we would
like to make a framework around it, so that not to much refactoring is
needed if you later want to make support for other kinds of auth (e.g. digest)
We will work at a solution but create this JIRA issue early in order to get
input/comments from the community as early as possible.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-7624) Correct wrong spelling of zkCredentialsProvider

2015-06-02 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-7624:


 Summary: Correct wrong spelling of zkCredentialsProvider
 Key: SOLR-7624
 URL: https://issues.apache.org/jira/browse/SOLR-7624
 Project: Solr
  Issue Type: Bug
  Components: security, SolrCloud
Reporter: Per Steffensen
Priority: Minor


SolrXmlConfig.java contain the string zkCredientialsProvider. It should be 
corrected to zkCredentialsProvider. I believe no other changes are necessary 
- there are no tests covering this AFAIK



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7543) Create GraphQuery that allows graph traversal as a query operator.

2015-05-15 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14545858#comment-14545858
]

Per Steffensen edited comment on SOLR-7543 at 5/15/15 5:52 PM:
---

Sounds interesting. I can't help thinking that this will help users do only one
particular graph'ish search. But there are millions of other graph'ish searches
one might want to do. The solution here might be too specific. A while back I
wrote a Solr indexing-backend for the Titan graph database. We can do a
storage-backend as well. Putting a full blown graph database on top of Solr (by
supporting indexing- and potentially storage-backends for e.g. Titan) might be
the way to go instead, so that we will not end up with lots and lots of very
specific graph-search query-parsers/resolvers. And this way you will get all
the other cool stuff from a full blown graph database - e.g. I liked playing
with Titans REPL. Just a thought

was (Author: steff1193):
Sounds interesting. I can't help thinking that this will help users doing one
particular graph'ish search. But there are millions of other graph'ish searches
one might want to do. The solution here might be too specific. A while back I
wrote a indexing-backend for the Titan graph database. Putting a full blown
graph database on top of Solr might be the way to go instead, so that we will
not end up with lots and lots of very specific graph-search
query-parsers/resolvers. Just a thought

Create GraphQuery that allows graph traversal as a query operator.
--

Key: SOLR-7543
URL: https://issues.apache.org/jira/browse/SOLR-7543
Project: Solr
Issue Type: New Feature
Components: search
Reporter: Kevin Watters
Priority: Minor

I have a GraphQuery that I implemented a long time back that allows a user to
specify a startQuery to identify which documents to start graph traversal
from. It then gathers up the edge ids for those documents , optionally
applies an additional filter. The query is then re-executed continually
until no new edge ids are identified. I am currently hosting this code up at
https://github.com/kwatters/solrgraph and I would like to work with the
community to get some feedback and ultimately get it committed back in as a
lucene query.
Here's a bit more of a description of the parameters for the query / graph
traversal:
q - the initial start query that identifies the universe of documents to
start traversal from.
fromField - the field name that contains the node id
toField - the name of the field that contains the edge id(s).
traversalFilter - this is an additional query that can be supplied to limit
the scope of graph traversal to just the edges that satisfy the
traversalFilter query.
maxDepth - integer specifying how deep the breadth first search should go.
returnStartNodes - boolean to determine if the documents that matched the
original q should be returned as part of the graph.
onlyLeafNodes - boolean that filters the graph query to only return
documents/nodes that have no edges.
We identify a set of documents with q as any arbitrary lucene query. It
will collect the values in the fromField, create an OR query with those
values , optionally apply an additional constraint from the traversalFilter
and walk the result set until no new edges are detected. Traversal can also
be stopped at N hops away as defined with the maxDepth. This is a BFS
(Breadth First Search) algorithm. Cycle detection is done by not revisiting
the same document for edge extraction.
This query operator does not keep track of how you arrived at the document,
but only that the traversal did arrive at the document.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7543) Create GraphQuery that allows graph traversal as a query operator.

2015-05-15 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14545858#comment-14545858
]

Per Steffensen commented on SOLR-7543:
--

Sounds interesting. I can't help thinking that this will help users doing one
particular graph'ish search. But there are millions of other graph'ish searches
one might want to do. The solution here might be too specific. A while back I
wrote a indexing-backend for the Titan graph database. Putting a full blown
graph database on top of Solr might be the way to go instead, so that we will
not end up with lots and lots of very specific graph-search
query-parsers/resolvers. Just a thought

Create GraphQuery that allows graph traversal as a query operator.
--

Key: SOLR-7543
URL: https://issues.apache.org/jira/browse/SOLR-7543
Project: Solr
Issue Type: New Feature
Components: search
Reporter: Kevin Watters
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Closed] (SOLR-3428) SolrCmdDistributor flushAdds/flushDeletes problems

2015-05-06 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-3428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen closed SOLR-3428.

Resolution: Fixed

None of the problems are present in latest code (looking at 5.1.0 release)

 SolrCmdDistributor flushAdds/flushDeletes problems
 --

 Key: SOLR-3428
 URL: https://issues.apache.org/jira/browse/SOLR-3428
 Project: Solr
  Issue Type: Bug
  Components: replication (java), SolrCloud, update
Affects Versions: 4.0-ALPHA
Reporter: Per Steffensen
Assignee: Per Steffensen
  Labels: add, delete, replica, solrcloud, update
   Original Estimate: 24h
  Remaining Estimate: 24h

 A few problems with SolrCmdDistributor.flushAdds/flushDeletes
 * If number of AddRequests/DeleteRequests in alist/dlist is below limit for a 
 specific node the method returns immediately and doesnt flush for subsequent 
 nodes
 * When returning immediately because there is below limit requests for a 
 given node, then previous nodes that have already been flushed/submitted are 
 not removed from adds/deletes maps (causing them to be flushed/submitted 
 again the next time flushAdds/flushDeletes is executed)
 * The idea about just combining params does not work for SEEN_LEADER params 
 (and probably others as well). Since SEEN_LEADER cannot be expressed (unlike 
 commitWithin and overwrite) for individual operations in the request, you 
 need to sent two separate submits. One containing requests with 
 SEEN_LEADER=true and one with SEEN_LEADER=false.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2015-04-29 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14520125#comment-14520125
 ] 

Per Steffensen commented on SOLR-6810:
--

Happy to see the activity here. Currently I am very busy, so I do not have the 
time to follow the work closely. Hope I will soon be able to.

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: SOLR-6810-trunk.patch, SOLR-6810-trunk.patch, 
 SOLR-6810-trunk.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1642874.patch, branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7176) allow zkcli to modify JSON

2015-04-22 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-7176:
-
Attachment: SOLR-7176.patch

Added Try again in error-msg, also on {{NodeExistsException}}

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
 Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch, 
 SOLR-7176.patch, SOLR-7176.patch


 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-22 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14506567#comment-14506567
]

Per Steffensen commented on SOLR-7176:
--

bq. This would also be valid for NodeExistsException

Hmmm, believe you are right

bq. I also thought about it. But it seems like when ZkStateReader does not own
the ZK client (e.g. in CollectionsHandler), it shouldn't close it? Any thoughts?

ZkStateReader does not own the ZK client if you use the
{{ZkStateReader(SolrZkClient zkClient)}} constructor - and we do in all the
mentioned caces. And it does not close the ZK client on close, when it does not
own it. Hence in the mentioned cases calling close really does not do much,
except setting closed=true. I just thought it was the right thing to do - call
close when you create and finished using a closable component.

allow zkcli to modify JSON
--

Key: SOLR-7176
URL: https://issues.apache.org/jira/browse/SOLR-7176
Project: Solr
Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch,
SOLR-7176.patch

To enable SSL, we have instructions like the following:
{code}
server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put
/clusterprops.json '{urlScheme:https}'
{code}
Overwriting the value won't work well when we have more properties to put in
clusterprops. We should be able to change individual values or perhaps merge
values.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-21 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14504498#comment-14504498
]

Per Steffensen commented on SOLR-7176:
--

bq. Hence the fix for this use-case can not really depend upon the overseer lock

Sure you can. You can say that you have to own the Overseer-lock, in order to
be able to perform this kind of admin-tasks. The CLI job can try to take the
lock and then perform the operation. If it cannot get the lock (maybe retry for
a period of time), ask the Overseer to do it (either by doing the corresponding
HTTP request or by leaving it directly on the Overseer-queue) and wait
synchronously for the Overseer to handle it (it must be running since it the
lock is taken).
But optimistic locking (compare-and-swap) is probably the best in this case.
Only thing I fear is that if that approach is established as the way it is
done it will be repeated in upcoming cases where it might not be sufficient.
Sometime it is worth the effort to establish the best platform to build upon
from the beginning.

allow zkcli to modify JSON
--

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-7176) allow zkcli to modify JSON

2015-04-21 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-7176:
-
Attachment: SOLR-7176.patch

New patch

bq. I would prefer not to retry automatically
Adding Try again to error-msg from ZkCLI if 
KeeperException.BadVersionException (to hint the user that it will probably go 
well if he just tries again).

Besides that
* Also // Don't update ZK unless absolutely necessary in the remove-case
* Sharing constants instead of introducing new similar ones. Because IMHO we 
shouldnt keep adding the same constants again and again, but more important: If 
HTTP is 
{{/admin/collections?action=CLUSTERPROPname=prop-nameval=prop-value}}, 
then the command-line should be {{zkcli.sh … -cmd CLUSTERPROP -name prop-name 
-val prop-value}}, AND if the HTTP ever changes (e.g. using “clusterprop.set” 
instead of “CLUSTERPROP”) so should the command-line
* Adding test of remove
* Closing ZkStateReader in CollectionsHandler, ZkCLI and ZkCLITest

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
 Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch, 
 SOLR-7176.patch


 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-20 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14502904#comment-14502904
 ] 

Per Steffensen commented on SOLR-7176:
--

bq. Why can't we just eliminate the overseer from the picture completely?

Not that it is very important in this case, but there is a problem in general 
with having several threads doing fetch - update-locally - store on state 
concurrently without locking (pessimistically or optimistically). Example, two 
threads running concurrently
* Thread#1 wants to do the task of setting urlScheme to http: 
** fetches {urlScheme:https, autoAddReplicas: true}
** changes it to {urlScheme:http, autoAddReplicas: true} and stores it
* Thread#2 wants to do the task of setting autoAddReplicas to false: 
** fetches {urlScheme:https, autoAddReplicas: true}
** changes it to {urlScheme:https, autoAddReplicas: false} and stores it

Without locking they can run concurrently and you will end up with a wrong state
* {urlScheme:http, autoAddReplicas: true}
* or {urlScheme:https, autoAddReplicas: false}

But you actually expected {urlScheme:http, autoAddReplicas: false}
I do not know what the initial thought was about the Overseer, but I think of 
it as a simple way to get around this locking - making sure that there is never 
more than one thread updating state.

When that is said, if the above was the intention with the Overseer, it does 
not work today, because CollectionsHandler.handleProp is doing the fetch and 
update, and only leaves the storing to Overseer. I would like to see the entire 
job handed over to the Overseer, so that it does both fetch, update and store - 
so that you can avoid the concurrency scenario above. In general Overseer 
should execute entire admin-jobs and not only parts of them.

Anyway, it is a reason not to do this kind of updates without taking locks, and 
Overseer is a primitive way of taking lock, and maybe therefore do not 
eliminate the Overseer. I am not sure it is especially important here.


 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
 Attachments: SOLR-7176.patch, SOLR-7176.patch


 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-20 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503430#comment-14503430
 ] 

Per Steffensen commented on SOLR-7176:
--

Believe it is referred to as compare-and-swap, but anyway 
{{Overseer.handleProp}} does not use that feature (it always uses version=-1 
when calling zk.setData). But actually I just realized that it is not true that 
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update 
to Overseer (as I claimed above) - it actually does forward the entire 
operation to Overseer so that the Overseer does fetch, update and store (just 
at I wanted it to). So you are right, that the problem I sketched above is not 
a problem in todays code, but it is not due to usage of compare-and-swap - it 
is because two threads will never try to do updates at the same time - they 
will ask the Overseer to do the updates (single-threaded = primitive 
pessimistic lock).

So we cannot just eliminate the overseer from the picture completely, if we 
still want to avoid the (more or less theoretical) problem I sketched above.

So the bullet-safe command-line solution should take this into consideration - 
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I would prefer 1) because it is the most generally useable solution to the 
problem. Compare-and-swap (even combined with ZK multi-op feature) will not 
always be sufficient for operations that want to update several znodes 
atomically - and who knows, maybe some day we also want to that kind of stuff 
using command-line. Taking a pessimistic lock (like the Overseer-lock) always 
will be sufficient.

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
 Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch


 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

2015-04-20 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14503430#comment-14503430
 ] 

Per Steffensen edited comment on SOLR-7176 at 4/20/15 7:02 PM:
---

Believe it is referred to as compare-and-swap, but anyway 
{{Overseer.handleProp}} does not use that feature (it always uses version=-1 
when calling zk.setData). But actually I just realized that it is not true that 
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update 
to Overseer (as I claimed above) - it actually does forward the entire 
operation to Overseer so that the Overseer does fetch, update and store (just 
at I wanted it to). So you are right, that the problem I sketched above is not 
a problem in todays code, but it is not due to usage of compare-and-swap - it 
is because two threads will never try to do updates at the same time - they 
will ask the Overseer to do the updates (single-threaded = primitive 
pessimistic lock).

So we cannot just eliminate the overseer from the picture completely, if we 
still want to avoid the (more or less theoretical) problem I sketched above.

So the bullet-safe command-line solution should take this into consideration - 
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)

I believe I would prefer 1) because it is the most generally useable solution 
to the problem. Compare-and-swap (even combined with ZK multi-op feature) will 
not always be sufficient for operations that want to update several znodes 
atomically - and who knows, maybe some day we also want to that kind of stuff 
using command-line. Taking a pessimistic lock (like the Overseer-lock) always 
will be sufficient.


was (Author: steff1193):
Believe it is referred to as compare-and-swap, but anyway 
{{Overseer.handleProp}} does not use that feature (it always uses version=-1 
when calling zk.setData). But actually I just realized that it is not true that 
{{CollectionsHandler.handleProp}} does the fetch-update and only leaves update 
to Overseer (as I claimed above) - it actually does forward the entire 
operation to Overseer so that the Overseer does fetch, update and store (just 
at I wanted it to). So you are right, that the problem I sketched above is not 
a problem in todays code, but it is not due to usage of compare-and-swap - it 
is because two threads will never try to do updates at the same time - they 
will ask the Overseer to do the updates (single-threaded = primitive 
pessimistic lock).

So we cannot just eliminate the overseer from the picture completely, if we 
still want to avoid the (more or less theoretical) problem I sketched above.

So the bullet-safe command-line solution should take this into consideration - 
e.g.
* 1) Take the overseer-lock before performing the operation
* 2) Use compare-and-swap (and make sure Overseer also does)
I would prefer 1) because it is the most generally useable solution to the 
problem. Compare-and-swap (even combined with ZK multi-op feature) will not 
always be sufficient for operations that want to update several znodes 
atomically - and who knows, maybe some day we also want to that kind of stuff 
using command-line. Taking a pessimistic lock (like the Overseer-lock) always 
will be sufficient.

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Assignee: Noble Paul
Priority: Minor
 Attachments: SOLR-7176.patch, SOLR-7176.patch, SOLR-7176.patch


 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-17 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14499632#comment-14499632
 ] 

Per Steffensen commented on SOLR-7176:
--

{quote}
{code}
zkcli.sh -zkhost 127.0.0.1:9983 -collection-action CLUSTERPROP -name urlScheme 
-val https
{code}
{quote}
I agree, except that it should not be the zkcli.sh tool that is extended. Since 
it is the collections API you make a CLI for, so to speak, make a 
collectionscli.sh script
{code}
collectionscli.sh -zkhost 127.0.0.1:9983 -action CLUSTERPROP -name urlScheme 
-val https
{code}
And later maybe
{code}
collectionscli.sh -zkhost 127.0.0.1:9983 -action ADDROLE -role overseer -val ...
{code}
etc

It think also, that it needs to be considered how and if this is an 
extension/modification to the SolrCLI-tool (used from solr/bin/solr and 
solr/bin/solr.cmd)
{code}
solr.sh CLUSTERPROP -zkhost 127.0.0.1:9983 -name urlScheme -val https
{code}
Just saying, even though I do not like the current state of it, because of the 
enormous amounts of redundant code. But we do not want to end up with a million 
different cli-tools either.
BTW, I think solr/bin/solr should be renamed to solr.sh, so I pretended above

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor

 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-17 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500028#comment-14500028
 ] 

Per Steffensen commented on SOLR-7176:
--

I agree, but from time to time I want to add a (async) command to the overseer 
while the cluster is not running, expecting the overseer to pick it up and 
execute it when I start my cluster. If you would enable this tool to do this 
kind of stuff, then suddenly most of the cluster-commands become relevant for 
this tool - if it is able to both execute the command directly (if supported - 
e.g. by {{CLUSTERPROP}} command) or to leave the command for execution by the 
overseer.
And, if you have numerous machines that might or might not currently run a 
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}} 
command as a command-line just to get an not running response.

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor

 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

2015-04-17 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500028#comment-14500028
 ] 

Per Steffensen edited comment on SOLR-7176 at 4/17/15 3:27 PM:
---

I agree, but from time to time I want to add a (async) command to the overseer 
while the cluster is not running, expecting the overseer to pick it up and 
execute it when I start my cluster. If you would enable this tool to do this 
kind of stuff, then suddenly most of the cluster-commands become relevant for 
this tool - if it is able to both execute the command directly (if supported - 
e.g. the {{CLUSTERPROP}} command) or to leave the command for execution by the 
overseer.
And, if you have numerous machines that might or might not currently run a 
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}} 
command as a command-line just to get a not running response.


was (Author: steff1193):
I agree, but from time to time I want to add a (async) command to the overseer 
while the cluster is not running, expecting the overseer to pick it up and 
execute it when I start my cluster. If you would enable this tool to do this 
kind of stuff, then suddenly most of the cluster-commands become relevant for 
this tool - if it is able to both execute the command directly (if supported - 
e.g. by {{CLUSTERPROP}} command) or to leave the command for execution by the 
overseer.
And, if you have numerous machines that might or might not currently run a 
Solr-node, maybe you actually want to be able to run the {{OVERSEERSTATUS}} 
command as a command-line just to get an not running response.

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor

 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-04-17 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14500039#comment-14500039
 ] 

Per Steffensen commented on SOLR-7176:
--

bq. With the current setup, you can use bin/solr at the commandline on *NIX 
and bin\solr on Windows, the only difference is the path separator, which 
will not be a surprise to most admins

Well I think it might come as a surprise to most *NIX admins that the script is 
just called solr - and not e.g. solr.sh. But never mind, this JIRA is not 
about that. I just had a hard time writing {{solr CLUSTERPROP ...}}, because I 
would have to think twice to understand it myself

bq. and it should be handled in a separate issue

Yes, definitely, no one talked about doing the renaming in this issue

 allow zkcli to modify JSON
 --

 Key: SOLR-7176
 URL: https://issues.apache.org/jira/browse/SOLR-7176
 Project: Solr
  Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor

 To enable SSL, we have instructions like the following:
 {code}
 server/scripts/cloud-scripts/zkcli.sh -zkhost localhost:2181 -cmd put 
 /clusterprops.json '{urlScheme:https}'
 {code}
 Overwriting the value won't work well when we have more properties to put in 
 clusterprops.  We should be able to change individual values or perhaps merge 
 values.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)

2015-03-31 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388230#comment-14388230
]

Per Steffensen commented on SOLR-7236:
--

Sorry [~janhoy] - will stop now. The discussion is related to security, but
probably not enough to be discussed here. Guess this JIRA just deals with the
fact that Solr-node is not necessarily web-container in the future - then what
to do about security. Great initiative!

Securing Solr (umbrella issue)
--

Key: SOLR-7236
URL: https://issues.apache.org/jira/browse/SOLR-7236
Project: Solr
Issue Type: New Feature
Reporter: Jan Høydahl
Labels: Security

This is an umbrella issue for adding security to Solr. The discussion here
should discuss real user needs and high-level strategy, before deciding on
implementation details. All work will be done in sub tasks and linked issues.
Solr has not traditionally concerned itself with security. And It has been a
general view among the committers that it may be better to stay out of it to
avoid blood on our hands in this mine-field. Still, Solr has lately seen
SSL support, securing of ZK, and signing of jars, and discussions have begun
about securing operations in Solr.
Some of the topics to address are
* User management (flat file, AD/LDAP etc)
* Authentication (Admin UI, Admin and data/query operations. Tons of auth
protocols: basic, digest, oauth, pki..)
* Authorization (who can do what with what API, collection, doc)
* Pluggability (no user's needs are equal)
* And we could go on and on but this is what we've seen the most demand for

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)

2015-03-31 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388227#comment-14388227
]

Per Steffensen commented on SOLR-7236:
--

I am ok with embedding Jetty, and you are right, that there are probably lots
of things that would be easier. Just make sure that you can still participate
in configuring it from the outside - jetty.xml and web.xml. At least until an
alternative solution gives the same flexibility. What I fear is that we remove
all the flexibility of web-container - because we are using - including its
ability to handle security.

I checked out 5.0.0 code, but I am not able to see that Solr-node is not still
just Jetty on top-level, and that Solr does not control anything before
web.xml/SolrDispatchFilter. Can you please point me to some of the more
important JIRAs around this hiding/removing web-container initiative. Thanks!
Just want to understand what has been done/achieved until now.

Securing Solr (umbrella issue)
--

Key: SOLR-7236
URL: https://issues.apache.org/jira/browse/SOLR-7236
Project: Solr
Issue Type: New Feature
Reporter: Jan Høydahl
Labels: Security

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.

2015-03-30 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386432#comment-14386432
]

Per Steffensen edited comment on SOLR-6816 at 3/30/15 9:22 AM:
---

bq. I think you mis-understood my point. I wasn't talking about retrying
documents in the same UpdateRequest. If a Map/Reduce task fails, the HDFS block
is retried entirely, meaning a Hadoop-based indexing job may send the same docs
that have already been added so using overwite=false is dangerous when doing
this type of bulk indexing. The solution proposed in SOLR-3382 would be great
to have as well though.

Well, we might be mis-understanding each other. Im am not talking about
retrying documents in the same UpdateRequest either. What we have:
Our indexing client (something not in Solr - think of it as the Map/Reduce job)
decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and
delete-doc commands), by sending one bulk-job containing all of those to
Solr-node S1. S1 handles some of the Us itself and forwards other Us to the
other Solr-nodes - depending or routing. For simplicity lets say that we have
three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards
U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500,
U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets
that information back from S2 and S3 (including reasons for each U that
failed), and is able to send a response to our indexing client saying that all
was a success, except that U100, U200, U400, U500, U700 and U800 failed, and
why they failed. Some might fail due to DocumentAlreadyExists (if U was about
creating a new document, assuming that it does not already exist), others might
fail due to VersionConflict (if U was about updating an existing document and
includes its last known (to the client) version, but the document at server has
a higher version-number), others again might fail due to DocumentDoesNotExist
(if U was about updating an existing document, but that document does not exist
(anylonger) at server). Our indexing client takes note of that combined
response from S1, performs the appropriate actions (e.g. version-lookups) and
sends a new request to the Solr-cluster now only including U100', U200', U400',
U500', U700' and U800'.
We have done it like that for a long time, using our solution to EDR-3382 (and
our solution to SOLR-3178). I would expect a Map/Reduce-job could do the same,
playing the role as the indexing client. Essentially only resending (maybe by
issuing a new Map/Reduce-job from the reduce-phase of the first
Map/Reduce-job) the (modified versions of) update-commands that failed the
first time.

was (Author: steff1193):
bq. I think you mis-understood my point. I wasn't talking about retrying
documents in the same UpdateRequest. If a Map/Reduce task fails, the HDFS block
is retried entirely, meaning a Hadoop-based indexing job may send the same docs
that have already been added so using overwite=false is dangerous when doing
this type of bulk indexing. The solution proposed in SOLR-3382 would be great
to have as well though.

Well, we might be mis-understanding each other. Im am not talking about
retrying documents in the same UpdateRequest either. What we have:
Our indexing client (something not in Solr - think of it as the Map/Reduce job)
decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and
delete-doc commands), by sending one bulk-job containing all of those to
Solr-node S1. S1 handles some of the Us itself and forwards other Us to the
other Solr-nodes - depending or routing. For simplicity lets say that we have
three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards
U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500,
U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets
that information back from S2 and S3 (including reasons for each U that
failed), and is able to send a response to our indexing client saying that all
was a success, except that U100, U200, U400, U500, U700 and U800 failed, and
why they failed. Some might fail due to DocumentDoNotExist (if U was about
creating a new document, assuming that it does not already exist), others might
fail due to VersionConflict (if U was about updating an existing document and
includes its last known (to the client) version, but the document at server has
a higher version-number), other might fail due to DocumentDoesNotExist (if U
was about updating an existing document, but that document does not exist
(anylonger) at server). Our indexing client takes note of that combined
response from S1, perform the appropriate actions (e.g. version-lookups) and
sends a new request to the Solr-cluster now only including U100', U200', U400',
U500', U700' and U800'.

[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)

2015-03-30 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386392#comment-14386392
]

Per Steffensen commented on SOLR-7236:
--

Thanks for a constructive answer!

bq. Because as long as the network and HTTP are handled by software that is
outside of Solr, Solr has absolutely no ability to control it.

First of all, that is not true. Second, I am pretty sure that Solr does not
have any ambition to deal with all the hard low-level network stuff itself. We
will always use a 3rd party component for that - Netty, Spray or whatever. So
it will always be a 3rd party component that connects with the network,
receives requests, parses them etc before handing over control to Solr. So why
not let Jetty (or any other web-container) do that - they do the job pretty
well.

bq. Ideally, you should be able to drop a configuration right in a handler
definition (such as the one for /select) found in solrconfig.xml, listing
security credentials (username/password, IP address, perhaps even certificate
information) that you are willing to accept for that handler, along with
exceptions or credentials that will allow SolrCloud inter-node communication to
work.

You can do that with a web-container, and I do believe that the way you would
do it will not change much whether you want to do it with Jetty or Netty. The
HttpServletRequest handed over by the web-container contains everything you
need (maybe together with the web-container context), just as it probably would
with any other network component. You can plug things into the web-container
just as you probably can with any other network component.
If you give me an more exact example of what you want to achieve, that you
believe cannot be achieved with a web-container, but you believe can be
achieved with the other approach, I would love to make the code showing that
you are wrong. If I can’t, I will forever keep quiet - and that in itself is
quit an achievement.

bq. Bringing the servlet container under our control as we did with 5.0 (with
initial work in 4.10) allows us to tell people how to configure the servlet
container for security in a predictable manner

Yes, and if it was a web-container that had control, your could point to
web-container documentation for info about how to configure.
Even though I think it is an unnecessary move, it is an ok idea to say that
Solr IS running in Jetty, making us able to tell exactly how to configure
whatever you need to. If you want to use any other web-container, you are on
your own. But leaving it a web-app behind the scenes would be great, so that
advanced users can still take advantage of that. The problem, though, is that
you lock with Jetty, and Jetty becomes an “implementation detail” of Solr. Do
not do that if it is not necessary, and I still have not seen very good reasons
to do it. But at least I recognize that there might be good reasons.

I am not sure about what you actually did in 5.0 (with initial work in 4.10).
As far as I can see Solr still starts out by starting Jetty. Can you point me
to some of the most important JIRAs for the remove-web-container initiative.
Thanks!

bq. but it is still not *Solr* and its configuration that's controlling it.

But it can be, without removing the web-container.
The thing I fear is spending a lot of resources and time removing Jetty and
replacing it with lots of other 3rd party components (e.g. including Netty),
and at best just reach status quo after a year or two. This entire
umbrella-issue (SOLR-7236) is only/mainly necessary because of the
moving-out-of-web-container initiative.
The fact that Solr is running in a web-container makes it very flexible - e.g.
my projects have significant changes to both web.xml and jetty.xml. Other
people might have similar setups just with Tomcat or whatever. Establishing the
same kind of flexibility without a web-container will take years.
In my organization we started out using ElasticSearch, but for political
reasons we moved to SolrCloud. The first thing that made me happy about that,
was the fact that Solr(Cloud) is a web-app, because I know exactly how they
work, they are standardized and flexible - believe a lot of people feel the
same way

At least, if you insist on removing the web-container, make sure not to do it
before all the flexibility it gives can somehow be achieved in another way. If
you really wanted to do cool stuff in this area, making Solr(Cloud) based on
dependency injection (configuration and/or annotation) would be great - e.g.
using Spring or Guice. Both top-level Solr, but also sub-parts of Solr. E.g.
the fact that solrconfig.xml is a self-invented configuration-structure that
screams to be replaced by (de-facto) standardized dependency injection is a
major problem.

Sorry for partly highjacking the issue, [~janhoy] - I did not manage to keep
this

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2015-03-30 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14386432#comment-14386432
]

Per Steffensen commented on SOLR-6816:
--

Well, we might be mis-understanding each other. Im am not talking about
retrying documents in the same UpdateRequest either. What we have:
Our indexing client (something not in Solr - think of it as the Map/Reduce job)
decides to do 1000 update-doc-commands U1, U2, ... , U1000 (add-doc and
delete-doc commands), by sending one bulk-job containing all of those to
Solr-node S1. S1 handles some of the Us itself and forwards other Us to the
other Solr-nodes - depending or routing. For simplicity lets say that we have
three Solr-nodes S1, S2 and S3 and that S1 handles U1-U333 itself, forwards
U334-U666 to S2 and U667-U1000 to S3. Now lets say that U100, U200, U400, U500,
U700 and U800 fails (two on each Solr-node), and the rest succeeds. S1 gets
that information back from S2 and S3 (including reasons for each U that
failed), and is able to send a response to our indexing client saying that all
was a success, except that U100, U200, U400, U500, U700 and U800 failed, and
why they failed. Some might fail due to DocumentDoNotExist (if U was about
creating a new document, assuming that it does not already exist), others might
fail due to VersionConflict (if U was about updating an existing document and
includes its last known (to the client) version, but the document at server has
a higher version-number), other might fail due to DocumentDoesNotExist (if U
was about updating an existing document, but that document does not exist
(anylonger) at server). Our indexing client takes note of that combined
response from S1, perform the appropriate actions (e.g. version-lookups) and
sends a new request to the Solr-cluster now only including U100', U200', U400',
U500', U700' and U800'.
We have done it like that for a long time, using our solution to EDR-3382 (and
our solution to SOLR-3178). I would expect a Map/Reduce-job could do the same,
playing the role as the indexing client. Essentially only resending (maybe by
issuing a new Map/Reduce-job from the reduce-phase of the first
Map/Reduce-job) the (modified) update-commands that failed the first time.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

We have never really focused on indexing performance, just correctness and
low hanging fruit. We need to vet the performance and try to address any
holes.
Note: A common report is that adding any replication is very slow.

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7236) Securing Solr (umbrella issue)

2015-03-27 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7236?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383624#comment-14383624
]

Per Steffensen commented on SOLR-7236:
--

I am trying to understand why Solr wants to move away from being a web-app
running in a servlet-container. A servlet-container can deal with all of the
things you mention in a standardized way

bq. User management (flat file, AD/LDAP etc)

That should really not be a Solr-concern. If you want your users/credentials in
LDAP manage it through an LDAP management-interface, if you want them in flat
files, let the admin change the files, etc

bq. Authentication (Admin UI, Admin and data/query operations. Tons of auth
protocols: basic, digest, oauth, pki..)

Implement your own realm authenticating against whatever credentials-store you
want with whatever protocol you want - and plug it into your servlet-container.
Or use a 3party realm that authenticates against your preferred
credentials-store and supporting your auth protocol. It is easy to find realms
for Jetty (or Tomcat or ...) authenticating against flat files, LDAP or
whatever. We wrote our own realm, authenticating against credentials-store in
ZK - easy.

bq. Authorization (who can do what with what API, collection, doc)

Let your realm list roles that an authenticated user is allowed to play. Setup
web.xml mapping roles to the url-paths they are allowed use. If limited
flexibility (url-mapping with stupid limitations) of web.xml authentication is
not good enough for you, implement it as a Filter yourself, or use some of the
Filters already out there. E.g. I provided a reg-exp authorization filter with
SOLR-4470 - it is very powerful.

bq. Pluggability (no user's needs are equal)

It is all pluggable - realms, web.xml, filters etc

Securing Solr (umbrella issue)
--

Key: SOLR-7236
URL: https://issues.apache.org/jira/browse/SOLR-7236
Project: Solr
Issue Type: New Feature
Reporter: Jan Høydahl
Labels: Security

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2015-03-27 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383556#comment-14383556
]

Per Steffensen commented on SOLR-6816:
--

Just to clarify. We did our own implementation of bloom-filter, and did not
build on the existing feature, because we did not find it satisfactory. I do
not remember exactly why, but maybe it was because the current solution builds
a bloom-filter on each segment, and bloom-filters are not really mergeable
unless they have the same size and used the same hashing-algorithm (and number
of hashes performed). To be efficient you want your bloom-filter to be bigger
(and potentially do another number of hashes) the bigger your segment is. We
decided to do a one-bloom-filter-per-index/core/replica (or whatever you like
to call it) solution, so that you do not need to merge bloom-filters. Besides
that you can tell Solr about the maximum expected amount of documents in the
index and the FPP you want at that number of documents, and it will calculate
the size of the bloom-filter to use. Because such a one-bloom-filter-per-index
does not go very well hand-in-hand with Lucenes segmenting strategy, we
implemented it as a Solr-feature, that is, it is Solr that maintains the
bloom-filter and the feature is not available for people who just uses Lucene.
It is not a simple task to do it right, so if the lower-hanging fruits are
satisfactory, you should pick them first.
To recap, we have recently-updated-or-lookedup cache to be able to efficiently
find the version of a document that does actually exist (and was touched
recently), and bloom-filter to be able to efficiently find out if a document
does not exist.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2015-03-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14383567#comment-14383567
 ] 

Per Steffensen commented on SOLR-6816:
--

bq. if a task fails, then Hadoop usually re-tries that task a couple of times, 
meaning all docs in the block that failed will be sent again

We do not send all documents again if just a few in a batch (bulk) fails. Lets 
say you send a batch of 1000 docs for indexing and only 2 fails due to e.g. 
version-control, we only do another round on those 2 documents - SOLR-3382

 Review SolrCloud Indexing Performance.
 --

 Key: SOLR-6816
 URL: https://issues.apache.org/jira/browse/SOLR-6816
 Project: Solr
  Issue Type: Task
  Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
 Attachments: SolrBench.pdf


 We have never really focused on indexing performance, just correctness and 
 low hanging fruit. We need to vet the performance and try to address any 
 holes.
 Note: A common report is that adding any replication is very slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-7176) allow zkcli to modify JSON

2015-03-02 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]

Per Steffensen commented on SOLR-7176:
--

I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader, ClusterState
etc. Then it is always those classes that are used to operate on a specific
type of json, and we only have to build consistency, integrity, validity etc
into those classes (separation of concern). The new thing should be that those
classes can be used via an external tool also when no Solr nodes are running.

At least make sure to use those existing classes for actually
reading/writing/verifying the jsons, and make separate tool-classes. Changing
ZkCLI to be able to trigger operations in those tool-classes is less important,
but personally I do not like - actually, if it has to be, I would rather see
e.g. an ClusterPropCLI-tool support a do-in-zk-flag making it use ZkCLI tool
to download first, do its manipulations and then use ZkCLI to upload again.
That is, new tools that use ZkCLI (if you are too lazy doing the
download/upload using ZkCLI yourself), instead of the other way around, where
ZkCLI uses other tools or even just does json-manipulation itself. Please do
not try to implement rules about what can and cannot be in a specific type of
json, what operations you can do on it etc two places in the code-base - use
what we already have.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?

allow zkcli to modify JSON
--

Key: SOLR-7176
URL: https://issues.apache.org/jira/browse/SOLR-7176
Project: Solr
Issue Type: New Feature
Reporter: Yonik Seeley
Priority: Minor

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

2015-03-02 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.

Would like to see clean definitions of which classes own responsibilities. E.g.
* ClusterState.java own the responsibility of a clusterstate.json always being
consistent, valid etc. You never do changes to clusterstate.json with using
ClusterState.java
* ClusterStateUpdater own the set of allowed operations on clusterstate.json.
You never manipulate a clusterstate.json without going through
ClusterStateUpdater. The new off-line ClusterStateCLI-tool should use
ClusterStateUpdater to do its operations - just change ClusterStateUpdater, so
that it support receiving jobs not coming from overseer queue.
Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?

was (Author: steff1193):
I think we should not make ZkCLI a Swiss Army Knife. ZkCLI should deal with ZK
operations, like uploading and downloading. Changing then content of a
json-file is not a ZK operation. Why not see it as different things. So what
you need to do if you want to make arbitrary changes to json-content of a znode
in ZK is
{code}
zkcli getfile thefile.json
use another tool for modifying thefile.json
zkcli putfile thefile.json
{code}

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

2015-03-02 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:48 PM:
--

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.

Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handle cluster-props.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.

[jira] [Comment Edited] (SOLR-7176) allow zkcli to modify JSON

2015-03-02 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-7176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343271#comment-14343271
]

Per Steffensen edited comment on SOLR-7176 at 3/2/15 3:45 PM:
--

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.

Yes, I know you primarily talk about ClusterProps, but the principle will be
the same. It is just that I know more about which classes handle cluster-state
than which handles cluster-props.

That said, maybe you can live with using e.g. http://trentm.com/json/ (or one
of the million other command-line json tools available) in use another tool
for modifying thefile.json above?

It would be nice to make different tools that can make operations on the
different types of jsons we have in a safe way - ensuring e.g. consistency,
integrity, validity etc. But I think those tools should use the classes already
handling those jsons, and not have anything to do with ZkCLI. E.g. a tool for
manipulating clusterstate.json should use classes from
org.apache.solr.common.cloud in solrj project, like ZkStateReader,
ClusterState, ClusterStateUpdater etc. Then it is always those classes that are
used to operate on a specific type of json, and we only have to build
consistency, integrity, validity etc into those classes (separation of
concern). The new thing should be that those classes can be used via an
external tool also when no Solr nodes are running.

[jira] [Comment Edited] (SOLR-6625) HttpClient callback in HttpSolrServer

2015-01-23 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289100#comment-14289100
]

Per Steffensen edited comment on SOLR-6625 at 1/23/15 11:38 AM:

Updated patch to HEAD of branch_x5. Not a completely trivial task. Did not try
to make any of the changes I suggested in previous comments. Just tried to make
a 1-to-1 upgrade to HEAD of branch_5x. Please someone verify that I did not
miss anything.

was (Author: steff1193):
Updated patch to HEAD of branch_x5. Not a completely trivial task.

HttpClient callback in HttpSolrServer
-

Key: SOLR-6625
URL: https://issues.apache.org/jira/browse/SOLR-6625
Project: Solr
Issue Type: Improvement
Components: SolrJ
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch,
SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch

Some of our setups use Solr in a SPNego/kerberos setup (we've done this by
adding our own filters to the web.xml). We have an issue in that SPNego
requires a negotiation step, but some HttpSolrServer requests are not
repeatable, notably the PUT/POST requests. So, what happens is,
HttpSolrServer sends the requests, the server responds with a negotiation
request, and the request fails because the request is not repeatable. We've
modified our code to send a repeatable request beforehand in these cases.
It would be nicer if HttpSolrServer provided a pre/post callback when it was
making an httpclient request. This would allow administrators to make
changes to the request for authentication purposes, and would allow users to
make per-request changes to the httpclient calls (i.e. modify httpclient
requestconfig to modify the timeout on a per-request basis).

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6625) HttpClient callback in HttpSolrServer

2015-01-23 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Per Steffensen updated SOLR-6625:
-
Attachment: SOLR-6625_r1654079.patch

Updated patch to HEAD of branch_x5. Not a completely trivial task.

HttpClient callback in HttpSolrServer
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6625) HttpClient callback in HttpSolrServer

2015-01-23 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6625:
-
Attachment: SOLR-6625_r1654079.patch

New patch
* Support for multiple callbacks (for nicer separation of concerns) - both for 
sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks 
build on contexts from each other - a logical consequence of introducing 
support for multiple callbacks
* Sold-nodes now use both sol.xml-callbacks AND sys-prop-callbacks
* SolrCLI.getJson attempts-variant now actually uses the callbacks given to it 
(via system param) 
* Cleaning up some of the code-duplication. Separating more of the code related 
to this issue in HttpClientRequestCallbackUtil


 HttpClient callback in HttpSolrServer
 -

 Key: SOLR-6625
 URL: https://issues.apache.org/jira/browse/SOLR-6625
 Project: Solr
  Issue Type: Improvement
  Components: SolrJ
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, 
 SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch, 
 SOLR-6625_r1654079.patch


 Some of our setups use Solr in a SPNego/kerberos setup (we've done this by 
 adding our own filters to the web.xml).  We have an issue in that SPNego 
 requires a negotiation step, but some HttpSolrServer requests are not 
 repeatable, notably the PUT/POST requests.  So, what happens is, 
 HttpSolrServer sends the requests, the server responds with a negotiation 
 request, and the request fails because the request is not repeatable.  We've 
 modified our code to send a repeatable request beforehand in these cases.
 It would be nicer if HttpSolrServer provided a pre/post callback when it was 
 making an httpclient request.  This would allow administrators to make 
 changes to the request for authentication purposes, and would allow users to 
 make per-request changes to the httpclient calls (i.e. modify httpclient 
 requestconfig to modify the timeout on a per-request basis).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6625) HttpClient callback in HttpSolrServer

2015-01-23 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14289383#comment-14289383
 ] 

Per Steffensen edited comment on SOLR-6625 at 1/23/15 3:29 PM:
---

New patch
* Support for multiple callbacks (for nicer separation of concerns) - both for 
sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks 
build on contexts from each other - a logical consequence of introducing 
support for multiple callbacks
* Solr-nodes now use both sol.xml-callbacks AND sys-prop-callbacks
* SolrCLI.getJson attempts-variant now actually uses the callbacks given to it 
(via sys-prop) 
* Cleaning up some of the code-duplication. Separating more of the code related 
to this issue in HttpClientRequestCallbackUtil



was (Author: steff1193):
New patch
* Support for multiple callbacks (for nicer separation of concerns) - both for 
sys-prop-callbacks and for solr.xml-callbacks. preExecuteRequest lets callbacks 
build on contexts from each other - a logical consequence of introducing 
support for multiple callbacks
* Sold-nodes now use both sol.xml-callbacks AND sys-prop-callbacks
* SolrCLI.getJson attempts-variant now actually uses the callbacks given to it 
(via system param) 
* Cleaning up some of the code-duplication. Separating more of the code related 
to this issue in HttpClientRequestCallbackUtil


 HttpClient callback in HttpSolrServer
 -

 Key: SOLR-6625
 URL: https://issues.apache.org/jira/browse/SOLR-6625
 Project: Solr
  Issue Type: Improvement
  Components: SolrJ
Reporter: Gregory Chanan
Assignee: Gregory Chanan
Priority: Minor
 Attachments: SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, 
 SOLR-6625.patch, SOLR-6625.patch, SOLR-6625.patch, SOLR-6625_r1654079.patch, 
 SOLR-6625_r1654079.patch


 Some of our setups use Solr in a SPNego/kerberos setup (we've done this by 
 adding our own filters to the web.xml).  We have an issue in that SPNego 
 requires a negotiation step, but some HttpSolrServer requests are not 
 repeatable, notably the PUT/POST requests.  So, what happens is, 
 HttpSolrServer sends the requests, the server responds with a negotiation 
 request, and the request fails because the request is not repeatable.  We've 
 modified our code to send a repeatable request beforehand in these cases.
 It would be nicer if HttpSolrServer provided a pre/post callback when it was 
 making an httpclient request.  This would allow administrators to make 
 changes to the request for authentication purposes, and would allow users to 
 make per-request changes to the httpclient calls (i.e. modify httpclient 
 requestconfig to modify the timeout on a per-request basis).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2015-01-05 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264463#comment-14264463
]

Per Steffensen commented on SOLR-4470:
--

There is a variant of 2nd suggestion, that I just want to mention briefly (I am
not at being brief) - something that we do of other reasons. You might want to
divide your set of Solr-nodes into two sub-sets
* Search-Solrs: Receiving incoming search requests from outside clients. Not
containing any data themselves but orchestrating the distributed searches
against the Data-Solrs
* Data-Solrs: Not directly accessible (network-wise) to outside clients
Setting up web-container security on Search-Solrs only might work, because they
do not have to communicate with each other. And it might be enough for you if
Data-Solrs are not accessible from outside.
We have such a setup but not for security reasons - we have web-container
security on both Search-Solrs and Data-Solrs. Just wanted to mention it because
it also might make things easier for you, and since I know it is possible this
is also an option for you. We do the separation because of different hardware
requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but
almost no disk-space, Data-Solrs need more disk-space but less memory - in our
setup, with the amounts and distribution of data we have and for the searches
we perform against the Sor-cluster.

Technically what we do is that we make a search-collection containing no
data. search-collection only has shards on the Search-Solrs. Then we have
several data-collections (containing the data) only having shards on the
Data-Solrs. We have custom (not much code) SearchHandler/SearchComponents (Solr
concepts) that basically just calculates the data-collection you have to
search and forwards the search from outside clients to search-collection to
those data-collections. We have an advanced algorithm for calculating the
data-collections that has to be searches given the concrete search to
search-collection, but you can just always calculate all data-collections.
This way Search-Solrs handle the distributed search, while Data-Solrs handle
searches against local shard only. This is kinda the same as putting your own
custom gateway in front of your Solr-nodes, and make the Solrs-nodes
inaccessible directly from outside clients. We just use Solr itself as this
front gateway, taking advantage of all the nice features - high availability
(with more than one Search-Solr) etc.

Clients uses CloudSolrServer, but always make searches against the
search-collection.

Support for basic http auth in internal solr requests
-

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests

2015-01-05 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264463#comment-14264463
]

Per Steffensen edited comment on SOLR-4470 at 1/5/15 10:28 AM:
---

There is a variant of 2nd suggestion, that I just want to mention briefly (I am
not good at being brief) - something that we do of other reasons. You might
want to divide your set of Solr-nodes into two sub-sets
* Search-Solrs: Receiving incoming search requests from outside clients. Not
containing any data themselves but orchestrating the distributed searches
against the Data-Solrs
* Data-Solrs: Not directly accessible (network-wise) to outside clients
Setting up web-container security on Search-Solrs only might work, because they
do not have to communicate with each other. And it might be enough for you if
Data-Solrs are not accessible from outside.
We have such a setup but not for security reasons - we have web-container
security on both Search-Solrs and Data-Solrs. Just wanted to mention it because
it also might make things easier for you, and since I know it is possible this
is also an option for you. We do the separation because of different hardware
requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but
almost no disk-space, Data-Solrs need more disk-space but less memory - in our
setup, with the amounts and distribution of data we have and for the searches
we perform against the Sor-cluster.

Clients uses CloudSolrServer, but always make searches against the
search-collection.

was (Author: steff1193):
There is a variant of 2nd suggestion, that I just want to mention briefly (I am
not at being brief) - something that we do of other reasons. You might want to
divide your set of Solr-nodes into two sub-sets
* Search-Solrs: Receiving incoming search requests from outside clients. Not
containing any data themselves but orchestrating the distributed searches
against the Data-Solrs
* Data-Solrs: Not directly accessible (network-wise) to outside clients
Setting up web-container security on Search-Solrs only might work, because they
do not have to communicate with each other. And it might be enough for you if
Data-Solrs are not accessible from outside.
We have such a setup but not for security reasons - we have web-container
security on both Search-Solrs and Data-Solrs. Just wanted to mention it because
it also might make things easier for you, and since I know it is possible this
is also an option for you. We do the separation because of different hardware
requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but
almost no disk-space, Data-Solrs need more disk-space but less memory - in our
setup, with the amounts and distribution of data we have and for the searches
we perform against the Sor-cluster.

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests

2015-01-05 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264463#comment-14264463
]

Per Steffensen edited comment on SOLR-4470 at 1/5/15 10:29 AM:
---

Setting up web-container security on Search-Solrs only might work, because they
do not have to communicate with each other. And it might be enough for you if
Data-Solrs are not accessible from outside.
We have such a setup but not for security reasons - we have web-container
security on both Search-Solrs and Data-Solrs. Just wanted to mention it because
it also might make things easier for you, and since I know it is possible this
is also an option for you. We do the separation because of different hardware
requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but
almost no disk-space, Data-Solrs need more disk-space but less memory - in our
setup, with the amounts and distribution of data we have and for the searches
we perform against the Sor-cluster.

Clients uses CloudSolrServer, but always make searches against the
search-collection.

was (Author: steff1193):
There is a variant of 2nd suggestion, that I just want to mention briefly (I am
not good at being brief) - something that we do of other reasons. You might
want to divide your set of Solr-nodes into two sub-sets
* Search-Solrs: Receiving incoming search requests from outside clients. Not
containing any data themselves but orchestrating the distributed searches
against the Data-Solrs
* Data-Solrs: Not directly accessible (network-wise) to outside clients
Setting up web-container security on Search-Solrs only might work, because they
do not have to communicate with each other. And it might be enough for you if
Data-Solrs are not accessible from outside.
We have such a setup but not for security reasons - we have web-container
security on both Search-Solrs and Data-Solrs. Just wanted to mention it because
it also might make things easier for you, and since I know it is possible this
is also an option for you. We do the separation because of different hardware
requirements for the two kinds of Solr-nodes. Search-Solrs need more memory but
almost no disk-space, Data-Solrs need more disk-space but less memory - in our
setup, with the amounts and distribution of data we have and for the searches
we perform against the Sor-cluster.

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2015-01-05 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14264372#comment-14264372
]

Per Steffensen commented on SOLR-4470:
--

As you noticed. You can configure standard web-container protection in
web.xml/jetty.xml, which makes Jetty require username/password get access. This
means that the requests going in to Solr-nodes (servers) have username/password
added. As long as you send requests from an outside SolrJ client (where you
control the code) you can add the credentials, and you will get access. But
Solr-nodes themselves send requests to other Solr-nodes, and before SOLR-4470
you have no (easy) way to add credentials to those requests - hence they will
not succeed. I guess what happens in you case it that recovery needs to send
inter-solr-node requests that fail due to no-access. To get a nice and easy way
out of this vote for SOLR-4470 (or some other JIRA issue that will make it
possible to add credentials to inter-solr-node requests - e.g. SOLR-6625 will
add a more general feature (callback) that will enable you to get around it).

It might not be entirely impossible for you to get around the problem. Guess
there several more or less hacky options. E.g.
* Some 3rd-party tool that intercepts requests between solr-nodes and add
credentials
* You probably have a well known set of IPs on which the solr-nodes run. You
cannot do it in standard web-container, but Jetty might support configuring a
set of IPs that do not have to authenticate to get access (if that is
acceptable to you). Or maybe you can implement your own bean and plug it into
Jetty.
* Guess you can discard using standard web-container security, and implement it
all yourself. E.g. in a filter that you add in front of SolrDispatchFilter

But I believe it is not easy before SOLR-4470.

Back when I wrote the code for SOLR-4470, I also modified
https://wiki.apache.org/solr/SolrSecurity to show how things will work after
SOLR-4470. Carefully (trying) to note that this depends on SOLR-4470 being
included in the product.

Support for basic http auth in internal solr requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-29 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14260880#comment-14260880
 ] 

Per Steffensen commented on SOLR-6810:
--

{quote}
I think the strategy that Shalin  I were talking about as a potential default 
was one that never collected IDs separately, hence no extra round-trip.
step 1: retrieve sort field values (then merge and calculate the range of 
ordinals needed for each shard)
step 2: retrieve stored fields by specifying the ordinals from each shard
{quote}

It sounds like third DQA - does not seem to be exactly what my new algorithm 
does. But you suggestion still has 2 round-trips. The old/current default-DQA 
with dqa.forceSkipGetIds/distrib.singlePass=false (default) has 2 round-trips. 
My new DQA with dqa.forceSkipGetIds=true (default) has 2 round-trips, so 
choosing that as the new default-DQA will not introduce an extra round-trip 
compared to todays default-DQA. But we could choose to select old/current 
default-DQA with dqa.forceSkipGetIds/distrib.singlePass=true as the new 
default-DQA. It has only 1 round-trip, so compared to that both your DQA 
(above) and my new DQA has an extra round-trip. My new DQA will never become 
1-round-trip only, because the essence is to make a round-trip first 
(inexpensive because you do not access store - not even for ids), collecting 
information needed to limit rows for the second round-trip where you actually 
retrieve the stored fields.

bq. one that never collected IDs separately

My new DQA does not collect IDs separately (as long as you do not explicitly 
set dqa.forceSkipGetIds=false)

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259438#comment-14259438
 ] 

Per Steffensen commented on SOLR-6810:
--

bq. Since dqa.forceSkipGetIds is always true for this new algorithm then 
computing the set X is not necessary and we can just directly fetch all return 
fields from individual shards and return the response to the user. Is that 
correct?

This is what happens by default with the new algorithm. But dqa.forceSkipGetIds 
is not always true. It is true by default, but you can explicitly set it to 
false by sending dqa.forceSkipGetIds=false in your request. So basically there 
are four options
* old alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=false 
(default before SOLR-6810, and currently also after SOLR-6810)
* old alg with dqa.forceSkipGetIds=true (same as with distrib.singlePass=true 
before SOLR-6810)
* new alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=true (does as 
you describe above)
* new alg with dqa.forceSkipGetIds=false (does as described in the JavaDoc you 
quoted)

The JavaDoc descriptions describe how the alg works WITHOUT dqa.forceSkipGetIds 
switched on. But dqa.forceSkipGetIds is switched on for the new alg by default. 
The JavaDoc for ShardParams.DQA.FORCE_SKIP_GET_IDS_PARAM describes how the two 
algs are altered when running with dqa.forceSkipGetIds=true. The thing is that 
you need to know this part as well to understand how the new alg works by 
default.

bq. I think the DefaultProvider and DefaultDefaultProvider aren't necessary? We 
can just keep a single static ShardParams.getDQA(SolrParams params) method and 
modify it if we ever need to change the default.

Well I would prefer to keep ShardParams.DQA.get(params) instead of having a 
ShardParams.getDQA(params) - it think it is better context'ing. But I will 
survive if you want to change it.
DefaultProvider in supposed to isolate the default decisions. 
DefaultDefaultProvider is an implementation that calculates the out-of-the-box 
defaults. It could be done directly in ShardParams.DQA.get, but I like to 
structure things. But I have to admit that the main reason I added the 
DefaultProvider thing, was that it makes it easier to change the 
default-decisions made when running the test-suite. I would like to randomly 
select the DQA to be used for every single query fired across the entire 
test-suite. This way we will have a very thorough test-coverage of both algs. 
Having the option of changing the DefaultProvider made it very easy to achieve 
this in SolrTestCaseJ4
{code}
private static DQA.DefaultProvider testDQADefaultProvider =
new DQA.DefaultProvider() {
  @Override
  public DQA getDefault(SolrParams params) {
// Select randomly the DQA to use
int algNo = 
Math.abs(random().nextInt()%(ShardParams.DQA.values().length));
return DQA.values()[algNo];
  }
};
{code}
{code}
DQA.setDefaultProvider(testDQADefaultProvider);
{code}

bq. If a user wants to change the default, the dqa can be set in the defaults 
section of the search handler.

I know it is a matter of opinion but in my mind the best place to deal with 
default for DQA is in the code that deals with DQA - not somewhere else. This 
makes a much better isolation and it makes code easier to understand. You can 
essentially navigate to ShardParams.DQA and read the code and JavaDoc and 
understand everything about DQA's. You do not have to know that there is a 
decision about default in the SeachHandler. But if you want to change that, it 
is ok for me.

bq. Why do we need the switchToTestDQADefaultProvider() and 
switchToOriginalDQADefaultProvider() methods? You are already applying the DQA 
for each request so why is the switch necessary?

No I am not applying the DQA for each request. I trust you understand why I 
want to run with randomized DQA across the entire test-suite - this is why I 
invented the testDQADefaultProvider. In tests that explicitly deal with testing 
DQA stuff, in some cases I want to switch on the real DefaultProvider because 
some of those tests are actually testing out-of-the-box default-behaviour. E.g. 
verifyForceSkipGetIds-tests in DistributedQueryComponentOptimizationTest. Also 
need it in DistributedExpandComponentTest until SOLR-6813 has been solved.

bq. There's still the ShardParams.purpose field which you added in SOLR-6812 
but I removed it. I still think it is unnecessary for purpose to be sent to 
shard. Is it necessary for this patch or is it just an artifact from SOLR-6812?

You are right. It is a mistake that I did not remove ShardParams.purpose

bq. Did you benchmark it against the current algorithm for other kinds of 
use-cases as well (3-5 shards, small number of rows)? Not asking for id can 
speed up responses there too I think.

I did not do any concrete benchmarking for other requests. We have changed our 
DQA

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259438#comment-14259438
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/27/14 6:09 PM:


bq. Since dqa.forceSkipGetIds is always true for this new algorithm then 
computing the set X is not necessary and we can just directly fetch all return 
fields from individual shards and return the response to the user. Is that 
correct?

This is what happens by default with the new algorithm. But dqa.forceSkipGetIds 
is not always true. It is true by default, but you can explicitly set it to 
false by sending dqa.forceSkipGetIds=false in your request. So basically there 
are four options
* old alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=false 
(default before SOLR-6810, and currently also after SOLR-6810)
* old alg with dqa.forceSkipGetIds=true (same as with distrib.singlePass=true 
before SOLR-6810)
* new alg without dqa.forceSkipGetIds or with dqa.forceSkipGetIds=true (does as 
you describe above)
* new alg with dqa.forceSkipGetIds=false (does as described in the JavaDoc you 
quoted)

The JavaDoc descriptions describe how the alg works WITHOUT dqa.forceSkipGetIds 
switched on. But dqa.forceSkipGetIds is switched on for the new alg by default. 
The JavaDoc for ShardParams.DQA.FORCE_SKIP_GET_IDS_PARAM describes how the two 
algs are altered when running with dqa.forceSkipGetIds=true. The thing is that 
you need to know this part as well to understand how the new alg works by 
default.

bq. I think the DefaultProvider and DefaultDefaultProvider aren't necessary? We 
can just keep a single static ShardParams.getDQA(SolrParams params) method and 
modify it if we ever need to change the default.

Well I would prefer to keep ShardParams.DQA.get(params) instead of having a 
ShardParams.getDQA(params) - it think it is better context'ing. But I will 
survive if you want to change it.
DefaultProvider in supposed to isolate the default decisions. 
DefaultDefaultProvider is an implementation that calculates the out-of-the-box 
defaults. It could be done directly in ShardParams.DQA.get, but I like to 
structure things. But I have to admit that the main reason I added the 
DefaultProvider thing, was that it makes it easier to change the 
default-decisions made when running the test-suite. I would like to randomly 
select the DQA to be used for every single query fired across the entire 
test-suite. This way we will have a very thorough test-coverage of both algs. 
Having the option of changing the DefaultProvider made it very easy to achieve 
this in SolrTestCaseJ4
{code}
private static DQA.DefaultProvider testDQADefaultProvider =
new DQA.DefaultProvider() {
  @Override
  public DQA getDefault(SolrParams params) {
// Select randomly the DQA to use
int algNo = 
Math.abs(random().nextInt()%(ShardParams.DQA.values().length));
return DQA.values()[algNo];
  }
};
{code}
{code}
DQA.setDefaultProvider(testDQADefaultProvider);
{code}

bq. If a user wants to change the default, the dqa can be set in the defaults 
section of the search handler.

I know it is a matter of opinion but in my mind the best place to deal with 
default for DQA is in the code that deals with DQA - not somewhere else. This 
makes a much better isolation and it makes code easier to understand. You can 
essentially navigate to ShardParams.DQA and read the code and JavaDoc and 
understand everything about DQA's. You do not have to know that there is a 
decision about default in the SeachHandler. But if you want to change that, it 
is ok for me.

bq. Why do we need the switchToTestDQADefaultProvider() and 
switchToOriginalDQADefaultProvider() methods? You are already applying the DQA 
for each request so why is the switch necessary?

No I am not applying the DQA for each request. I trust you understand why I 
want to run with randomized DQA across the entire test-suite - this is why I 
invented the testDQADefaultProvider. In tests that explicitly deal with testing 
DQA stuff, in some cases I want to switch on the real DefaultProvider because 
some of those tests are actually testing out-of-the-box default-behaviour. E.g. 
verifyForceSkipGetIds-tests in DistributedQueryComponentOptimizationTest. Also 
need it in DistributedExpandComponentTest until SOLR-6813 has been solved.

bq. There's still the ShardParams.purpose field which you added in SOLR-6812 
but I removed it. I still think it is unnecessary for purpose to be sent to 
shard. Is it necessary for this patch or is it just an artifact from SOLR-6812?

You are right. It is a mistake that I did not remove ShardParams.purpose

bq. Did you benchmark it against the current algorithm for other kinds of 
use-cases as well (3-5 shards, small number of rows)? Not asking for id can 
speed up responses there too I think.

I did not do any concrete

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259445#comment-14259445
 ] 

Per Steffensen commented on SOLR-6810:
--

bq.  IMO, one shouldn't have to look at the patch to figure out what it's 
trying to do.

Seems reasonable. The way things change is IMHO fairly well documented in 
JavaDocs of ShardParams.DQA so I will just steal from there
* Old DQA (FIND_ID_RELEVANCE_FETCH_BY_IDS)
{code}
   /**
* Algorithm
* - Shard-queries 1) Ask, by forwarding the outer query, each shard for id 
and relevance of the (up to) #rows most relevant matching documents
* - Find among those id/relevances the #rows id's with the highest global 
relevances (lets call this set of id's X)
* - Shard-queries 2) Ask, by sending id's, each shard to return the 
documents from set X that it holds
* - Return the fetched documents to the client
*/
...
   // Default do not force skip get-ids phase
{code}
* New DQA (FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS)
{code}
   /**
* Algorithm
* - Shard-queries 1) Ask, by forwarding the outer query, each shard for 
relevance of the (up to) #rows most relevant matching documents
* - Find among those relevances the #rows highest global relevances
* Note for each shard (S) how many entries (docs_among_most_relevant(S)) it 
has among the #rows globally highest relevances
* - Shard-queries 2) Ask, by forwarding the outer query, each shard S for 
id and relevance of the (up to) #docs_among_most_relevant(S) most relevant 
matching documents
* - Find among those id/relevances the #rows id's with the highest global 
relevances (lets call this set of id's X)
* - Shard-queries 3) Ask, by sending id's, each shard to return the 
documents from set X that it holds
* - Return the fetched documents to the client 
* 
* Advantages
* Asking for data from store (id in shard-queries 1) of 
FIND_ID_RELEVANCE_FETCH_BY_IDS) can be expensive, therefore sometimes you want 
to ask for data
* from as few documents as possible.
* The main purpose of this algorithm it to limit the rows asked for in 
shard-queries 2) compared to shard-queries 1) of FIND_ID_RELEVANCE_FETCH_BY_IDS.
* Lets call the number of rows asked for by the outer request for 
outer-rows
* shard-queries 2) will never ask for data from more than outer-rows 
documents total across all involved shards. shard-queries 1) of 
FIND_ID_RELEVANCE_FETCH_BY_IDS
* will ask each shard for data from outer-rows documents, and in worst 
case if each shard contains outer-rows matching documents you will
* fetch data for number of shards involved * outer-rows.
* Using FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS will become more 
beneficial the more
* - shards are involved
* - and/or the more matching documents each shard holds
*/
...
// Default force skip get-ids phase. In this algorithm there are really 
never any reason not to skip it
{code}
* dqa.forceSkipGetIds
{code}
   /** Request parameter to force skip get-ids phase of the distributed query? 
Value: true or false 
* Even if you do not force it, the system might choose to do it anyway
* Skipping the get-ids phase
* - FIND_ID_RELEVANCE_FETCH_BY_IDS: Fetch entire documents in Shard-queries 
1) and skip Shard-queries 2)
* - FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS: Fetch entire 
documents in Shard-queries 2) and skip Shard-queries 3)
*/
{code}

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259447#comment-14259447
 ] 

Per Steffensen commented on SOLR-6810:
--

TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the 
number of store accesses is reduced
{code}
// Test the number of documents read from store using 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
  for (int startValue = 0; startValue = MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds 
used - default)
// Only reads data (required fields) from store for rows + (#shards * 
start) documents across all shards
// This can be optimized to become only rows 
// Only reads the data once

testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
 startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue * 
jettys.size()));

// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used - 
default)
// Reads data (ids only) from store for (rows + startValue) * #shards 
documents for each shard
// Besides that reads data (required fields) for rows documents across 
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue, 
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) * 
jettys.size()));
  }
}
{code}

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14259447#comment-14259447
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/27/14 6:35 PM:


TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the 
number of store accesses is reduced
{code}
// Test the number of documents read from store using 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
  for (int startValue = 0; startValue = MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds 
used - default)
// Only reads data (required fields) from store for rows + (#shards * 
start) documents across all shards
// This can be optimized to become only rows 
// Only reads the data once

testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
 startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue * 
jettys.size()));

// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used - 
default)
// Reads data (ids only) from store for (rows + startValue) * #shards 
documents for each shard
// Besides that reads data (required fields) for rows documents across 
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue, 
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) * 
jettys.size()));
  }
}
{code}
{code}
testDQADocReads(ShardParams.DQA dqa, int start, int rows, int 
expectedUniqueIdCount, int expectedTotalCount) {
...
}
{code}


was (Author: steff1193):
TestDistributedQueryAlgorithm.testDocReads shows very well exactly how the 
number of store accesses is reduced
{code}
// Test the number of documents read from store using 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// vs FIND_ID_RELEVANCE_FETCH_BY_IDS. This demonstrates the advantage of 
FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS
// over FIND_ID_RELEVANCE_FETCH_BY_IDS (and vice versa)
private void testDocReads() throws Exception {
  for (int startValue = 0; startValue = MAX_START; startValue++) {
// FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS (assuming skipGetIds 
used - default)
// Only reads data (required fields) from store for rows + (#shards * 
start) documents across all shards
// This can be optimized to become only rows 
// Only reads the data once

testDQADocReads(ShardParams.DQA.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS,
 startValue, ROWS, ROWS + (startValue * jettys.size()), ROWS + (startValue * 
jettys.size()));

// DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS (assuming skipGetIds not used - 
default)
// Reads data (ids only) from store for (rows + startValue) * #shards 
documents for each shard
// Besides that reads data (required fields) for rows documents across 
all shards
testDQADocReads(ShardParams.DQA.FIND_ID_RELEVANCE_FETCH_BY_IDS, startValue, 
ROWS, (ROWS + startValue) * jettys.size(), ROWS + ((ROWS + startValue) * 
jettys.size()));
  }
}
{code}

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-17 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249639#comment-14249639
]

Per Steffensen commented on SOLR-6816:
--

I believe today overwrite=false will not prevent neither document-version-check
on leader (it will in the Solr we use in my company, but not in Apache Solr)
nor bucket-version-check on non-leaders. As far as I can see
{{DistributedUpdateProcessor.versionAdd}} will do document-version-check if
versionsStored=true, leaderLogic=true and versionOnUpdate != 0. It will do
bucket-version-check if versionsStored=true and leaderLogic=false. This has
nothing to do with overwrite param. This version-check is not only for
add-commands but also for delete-commands.

The overwrite param controls only (in {{DirectUpdateHandler2}}) if you make
sure to delete an existing document with the same id, before you add the new
document. You do that by default, but if overwrite=false you just add the new
document, allowing duplicates (defined to be documents that have the same
id-value).

So as far as I read the code, document-version-check will only be performed on
leaders. Non-leaders will only do bucket-version-check, and I do not think that
is expensive?
As I said our version of Solr does not do document-version-check if
overwrite=false. I think you should introduce that as well. But besides that,
whats left to do in this area?

What did I not understand?

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-17 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249673#comment-14249673
]

Per Steffensen commented on SOLR-6816:
--

Those of you that have been following my comments on misc issues will know that
I like separation of concerns. So in our version of Solr all this
decision-making on when to do document-version-check, when to delete existing
documents with same id-value etc is isolated in {{enum UpdateSemanticsMode}} -
see
https://issues.apache.org/jira/secure/attachment/12553312/SOLR-3173_3178_3382_3428_plus.patch.
We support different modes that makes slightly different decisions on the
above topics, which is the reason for using an enum. You do not need that,
because you only have one mode, but that should not prevent you from
separating the decision-making concern.

The patch is not entirely up to date with what we do today, but at least it
illustrates the separation of concerns. {{DistributedUpdateHandler}} deals
with a million concerns, so maybe you want to adopt that idea and move the code
making the decisions out of {{DistributedUpdateHandler}}.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-17 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14249673#comment-14249673
]

Per Steffensen edited comment on SOLR-6816 at 12/17/14 10:17 AM:
-

Only mention this because I sense that at least [~shalinmangar] agrees that
some cleanup (a.o. of {{DistributedUpdateHandler}}) is required:
https://twitter.com/shalinmangar/status/543874893549277184

was (Author: steff1193):
Those of you that have been following my comments on misc issues will know that
I like separation of concerns. So in our version of Solr all this
decision-making on when to do document-version-check, when to delete existing
documents with same id-value etc is isolated in {{enum UpdateSemanticsMode}} -
see
https://issues.apache.org/jira/secure/attachment/12553312/SOLR-3173_3178_3382_3428_plus.patch.
We support different modes that makes slightly different decisions on the
above topics, which is the reason for using an enum. You do not need that,
because you only have one mode, but that should not prevent you from
separating the decision-making concern.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included

2014-12-16 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14247963#comment-14247963
 ] 

Per Steffensen commented on SOLR-6813:
--

I would prefer making it work instead of saying that single-pass in not 
supported for expand-queries. But I guess that is easy for me to say :-)

If I understand you explanations correctly, expand-queries ought to (almost) 
always fail for distributed. Also when rows/start is not set (default start=0 
and rows=10). Can you explain why it does not fail in those cases
{code}
query(q, *:*, fq, {!collapse field=+group+}, defType, 
edismax, bf, field(test_ti), expand, true, fl,*,score, 
ShardParams.DISTRIB_SINGLE_PASS, true);
query(q, *:*, fq, {!collapse field=+group+}, defType, 
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl 
desc, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, true);
query(q, *:*, fq, {!collapse field=+group+}, defType, 
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl 
desc, expand.rows, 1, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, 
true);
//Test no expand results
query(q, test_ti:5, fq, {!collapse field=+group+}, defType, 
edismax, bf, field(test_ti), expand, true, expand.sort, test_tl 
desc, expand.rows, 1, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, 
true);
//Test zero results
query(q, test_ti:5434343, fq, {!collapse field=+group+}, 
defType, edismax, bf, field(test_ti), expand, true, expand.sort, 
test_tl desc, expand.rows, 1, fl,*,score, 
ShardParams.DISTRIB_SINGLE_PASS, true);
//Test page 2
query(q, *:*, start,1, rows, 1, fq, {!collapse 
field=+group+}, defType, edismax, bf, field(test_ti), expand, 
true, fl,*,score, ShardParams.DISTRIB_SINGLE_PASS, true);
{code}

Only the last one above fails?

 distrib.singlePass does not work for expand-request - start/rows included
 -

 Key: SOLR-6813
 URL: https://issues.apache.org/jira/browse/SOLR-6813
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
Assignee: Joel Bernstein
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests. Even after the 
 fix provided to SOLR-6812, it does not work for requests where you add start 
 and/or rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-16 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6810:
-
Attachment: branch_5x_rev1645549.patch

New patch where the my solution to SOLR-6812 has been removed. Patch now 
matches revision 1645549 where the final solution to SOLR-6812 is included

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch, 
 branch_5x_rev1645549.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included

2014-12-16 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14248410#comment-14248410
 ] 

Per Steffensen commented on SOLR-6813:
--

bq. The distributed deep paging problem will continue to exist

But isnt that a performance problem only. Functionality-wise it will work doing 
deep paging, right? If it is just a performance problem, basically you should 
not use distrib.singlePass if you are deep paging, I guess we should not 
disallow using it together with deep paging. It is just a bad idea 
performance-wise to use distrib.singlePass together with deep paging.

 distrib.singlePass does not work for expand-request - start/rows included
 -

 Key: SOLR-6813
 URL: https://issues.apache.org/jira/browse/SOLR-6813
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
Assignee: Joel Bernstein
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests. Even after the 
 fix provided to SOLR-6812, it does not work for requests where you add start 
 and/or rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-13 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245264#comment-14245264
 ] 

Per Steffensen commented on SOLR-6810:
--

Now that SOLR-6812 has been solved (in a different way than I suggested), my 
solution to SOLR-6812 should be removed from the patch.

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-13 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14245266#comment-14245266
 ] 

Per Steffensen commented on SOLR-6810:
--

Thanks for taking care of SOLR-6795, SOLR-6796 and SOLR-6812 (and SOLR-6813), 
[~shalinmangar]!!!

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6625) HttpClient callback in HttpSolrServer

2014-12-12 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244185#comment-14244185
 ] 

Per Steffensen commented on SOLR-6625:
--

Had a few hours for a look at the latest patch. Not a lot of time so I might 
very well have overlooked something, so please bear with me. But from what I 
got out of it I have the following comments

It seems you can control the used callbacks in two ways
* 1) Through {{solr.xml}}. 
** Request that {{SolrDispatchFilter}} forwards (using {{remoteQuery}})
** Update requests sent through {{UpdateShardHandler}} - 
{{DistributedUpdateProcessor}}, {{SnapPuller}}, {{PeerSync}}, {{SyncStrategy}} 
...
** {{Overseer}} requests (through {{UpdateShardHandler}} via {{ZkController}})
* 2) Through VM parameter. Used for every request sent - from Solr nodes and 
clients or whatever is running in the JVM sending request using 
{{HttpSolrServer}} (or {{ConcurrentUpdateSolrServer}}). Believe, for the 
requests mentioned under 1) above, both callbacks (the one from {{solr.xml}} 
AND the one from VM parameter) will be triggered - is that intentional? (it is 
fine for me)
I think, that if you want to support setting the callback through {{solr.xml}} 
it should be used for all requests sent out of a Solr node - including e.g. 
search requests to shards issued from {{SearchHandler}} (by code inspection it 
seems to me that it will not currently be included here). Regarding this 
principle of using the {{solr.xml}}-callback for ALL requests going out of a 
Solr node, I would really like to see some testing that makes sure this 
principle is fulfilled now and that it is not broken in the future (the day 
after SOLR-6625 is committed and forgotten about, someone adds a new place in 
the code sending requests going out of Solr nodes and forgets to call the 
callbacks). I wanted to do the same in SOLR-4470 (making sure that no requests 
going out of Solr nodes do not have credentials added - not it the code as it 
is today nor in the code as it changes in the future). I achieved it by adding 
a protecting layer around every Solr node ever started during testing, and 
assuming that the test-suite triggers every type of requests going out of Solr 
nodes, that will basically ensure that no one forgets to propagate credentials 
when extending the code in the future (those requests will fail and the test 
will likely fail). I would really like to see something similar here, verifying 
that {{solr.xml}}-callback is triggered for every request going out of a Solr 
node.

Maybe you should consider the naming of {{distribCount}} and {{forwardCount}} 
in {{BasicDistributedZk2Test.HttpClientRequestCallbackCounter}}. They are very 
confusing to me. {{distribCount}} seem to count update-requests (they are kinda 
forwarded) and {{forwardCount}} seem to count everything else including request 
from e.g. {{Overseer}} (they are among those that can not be characterized as 
forwarded)

I know you are not keen on cleaning up while adding a new feature (I was told 
that we have to do cleanup in different JIRAs). Personally I am very much 
against that, because no one will ever make a JIRA just to clean up (at least, 
looking at the code-base I see that it does not happen enough). Assuming we 
want to do a little cleanup (in that code that is touched anyway in this JIRA) 
I have the following suggestions
* Change {{ConcurrentUpdateSolrServer}} to use {{HttpSolrServer}} or at least 
make them share common code. Then we do not have remember to add 
callback-support in both when we solve this SOLR-6625
* In general we should do some Separation of Concerns here and create a 
single component X (e.g. {{SolrNodeOutgoingRequestSender}}) concerned with 
sending requests out of Solr nodes. Make X use callback from {{solr.xml}}.
** Let {{SolrDispatchFilter}}, {{UpdateShardHandler}} and whatever sends 
requests out of Solr nodes use that component X, or something based on or using 
it
** It is strange that {{Overseer}} uses {{UpdateShardHandler}} - {{Overseer}} 
is not doing updates. Either rename {{UpdateShardHandler}} to something that 
does not signal that it is only used for update-requests, or let {{Overseer}} 
use something else based on component X

Instead of adding callback-calls numerous places in the code not knowing if 
some place was forgotten, polluting the code even more and leaving the same 
kind of problems for others doing similar things in the future, IMHO SOLR-6625 
should be solved by refactoring the code so that the actual feature of 
SOLR-6625 is ridiculously easy to add
* Separating the general concern of sending request from solr-code (server or 
client) to one single component Y. ({{HttpSolrServer}} or a class containing 
the code shared between it and {{ConcurrentUpdateSolrServer}} is fine for now). 
It is easy to make a test that ensures that {{HttpClient.execute}} only occurs 
one single

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-12 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14244392#comment-14244392
]

Per Steffensen commented on SOLR-4470:
--

bq. So maybe one suggestion I would make is - don't give up on this patch. Use
this issue as an umbrella issue for adding basic-auth to Solr. Tell us here's
the entire piece of work that I've done, now I want to do it in steps (baby
steps preferred)

I am not too happy about this approach in general. I like that only fully done
features get committed. If you artificially break a complete feature into small
steps AND COMMIT THEM you will temporarily have partly done features. No one
says that you will ever get the last parts committed. I see way to much code in
Solr that really smells like we will just do this simple hacky thing here
without considering nice code-structure/design and complete features - we will
complete, refactor and make it nice later - but later never showed up. One
example that I stumbled over just recently is SOLR-5768 - I had to clean it up
in SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813. Breaking up a complete
feature including thorough testing etc. into smaller pieces will have to make
us enter that area of partly done stuff. I am afraid that only half of the
subsequent issues under the umbrella will never be committed.

That said, I am willing to try. But I want to wait to see what SOLR-6625
brings. It might very well establish a platform for solving this SOLR-4470 much
more easily.

Support for basic http auth in internal solr requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-10 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241199#comment-14241199
]

Per Steffensen commented on SOLR-6816:
--

Make sure to document when it is safe to turn off the version-check for replica
(overwrite=false). Believe the version-check is also used to detect and prevent
old versions that get to the replica last does not overwrite new versions that
get to the replica first. E.g leader might get an update-request of document D
updating it from state S1 to S2, and send a request U1 to a replica with
overwrite=false saying just write S2 without checking. Immediately after the
update of D from S1 to S2, the leader gets another request updating it from S2
to S3. Leader again reacts by sending a just write S3 without
checking-request U2 to the replica. The replica might receive U2 before U1 (if
replica-request is not sent and waited for response while the leader has the
bucket-lock for the document) and then you are in trouble.

Sure that is an odd case, just do not forget that versions are also used for
that on replica (AFAIK)

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6625) HttpClient callback in HttpSolrServer

2014-12-10 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241254#comment-14241254
]

Per Steffensen commented on SOLR-6625:
--

bq. Maybe Per Steffensen can chime in with the motivation there rather than
always using the static credentials – I'm guessing it is because there is some
organization-specific authorization checking expecting the actual user who made
the original request.

* Every paranoid customers. In general we/they just do not want a user to be
able to indirectly (through distributed sub-requests) get a response to a query
he is not allowed to fire directly. There is a lot of details to it that I will
spare you, but overall we are also 1) using authorization to prevent specific
groups of users to do queries (directly or indirectly) that takes certain
amounts of resources from the system or 2) to fetch certain fields or any
information that can be derived from those fields. Ad 1) By a lot of testing we
know which types of queries really takes a lot of resources out of our system,
but it is hard to be sure that a query that does not seem to be such a query,
does not indirectly fire such queries in distributed sub-requests. Ad 2) Just
because an outer request does not specifically ask for a certain field, it is
hard to know that the indirect sub-request also does not, even though in
practice it is so (except for id an score)
* Distributed audit logging
* Etc.

I realize that the reasons we want the credentials from the outer request
forwarded in sub-requests are very thin. I understand if this is not something
that you feel is necessary to support in Apache Solr. But if it does not make
the solution much more complicated you might just support it. I do not think
SOLR-4470 is a complicated solution - most of it, really, is testing.

bq. Likewise you can see all of the various other authentication schemes
HttpClient supports (SPNego included)

SOLR-4470 does not try to do itself what HttpClient can already do. The
HttpClient capabilities can just be used outside or through the
authentication-framework introduced in SOLR-4470. SOLR-4470 is mostly about
joggling the credentials inside Solr-nodes. When they have to be added to
requests, of course use HttpClient capabilities.

bq. To get around this, we use secure impersonation

Interesting!

HttpClient callback in HttpSolrServer
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-10 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14241267#comment-14241267
 ] 

Per Steffensen commented on SOLR-6816:
--

Thats fine. Just wanted to mention it so that you do not forget when you nail 
down a solution

 Review SolrCloud Indexing Performance.
 --

 Key: SOLR-6816
 URL: https://issues.apache.org/jira/browse/SOLR-6816
 Project: Solr
  Issue Type: Task
  Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
 Attachments: SolrBench.pdf


 We have never really focused on indexing performance, just correctness and 
 low hanging fruit. We need to vet the performance and try to address any 
 holes.
 Note: A common report is that adding any replication is very slow.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-05 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14235323#comment-14235323
 ] 

Per Steffensen commented on SOLR-4470:
--

Potentially relating to SOLR-6625. We might want to use the more generic 
callback functionality for adding credentials (in some cases)?

 Support for basic http auth in internal solr requests
 -

 Key: SOLR-4470
 URL: https://issues.apache.org/jira/browse/SOLR-4470
 Project: Solr
  Issue Type: New Feature
  Components: clients - java, multicore, replication (java), SolrCloud
Affects Versions: 4.0
Reporter: Per Steffensen
Assignee: Jan Høydahl
  Labels: authentication, https, solrclient, solrcloud, ssl
 Fix For: Trunk

 Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, 
 SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, 
 SOLR-4470_trunk_r1568857.patch


 We want to protect any HTTP-resource (url). We want to require credentials no 
 matter what kind of HTTP-request you make to a Solr-node.
 It can faily easy be acheived as described on 
 http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
 also make internal request to other Solr-nodes, and for it to work 
 credentials need to be provided here also.
 Ideally we would like to forward credentials from a particular request to 
 all the internal sub-requests it triggers. E.g. for search and update 
 request.
 But there are also internal requests
 * that only indirectly/asynchronously triggered from outside requests (e.g. 
 shard creation/deletion/etc based on calls to the Collection API)
 * that do not in any way have relation to an outside super-request (e.g. 
 replica synching stuff)
 We would like to aim at a solution where original credentials are 
 forwarded when a request directly/synchronously trigger a subrequest, and 
 fallback to a configured internal credentials for the 
 asynchronous/non-rooted requests.
 In our solution we would aim at only supporting basic http auth, but we would 
 like to make a framework around it, so that not to much refactoring is 
 needed if you later want to make support for other kinds of auth (e.g. digest)
 We will work at a solution but create this JIRA issue early in order to get 
 input/comments from the community as early as possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234089#comment-14234089
]

Per Steffensen commented on SOLR-4470:
--

bq. Does anyone have an idea when this will be released? Thanks!

My best guess is never, unfortunately. But you can build your own version of
Solr including it. We do that. I can give you a few hints on how to build your
own Solr, if you want. Do not have the time to give you a thorough description.
But a few hints I will manage. Let me know. Do not know if it is actually
described in details somewhere?

Support for basic http auth in internal solr requests
-

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-04 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234103#comment-14234103
 ] 

Per Steffensen commented on SOLR-4470:
--

But voting for it might help :-)

 Support for basic http auth in internal solr requests
 -

 Key: SOLR-4470
 URL: https://issues.apache.org/jira/browse/SOLR-4470
 Project: Solr
  Issue Type: New Feature
  Components: clients - java, multicore, replication (java), SolrCloud
Affects Versions: 4.0
Reporter: Per Steffensen
Assignee: Jan Høydahl
  Labels: authentication, https, solrclient, solrcloud, ssl
 Fix For: Trunk

 Attachments: SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, SOLR-4470.patch, 
 SOLR-4470.patch, SOLR-4470_branch_4x_r1452629.patch, 
 SOLR-4470_branch_4x_r1452629.patch, SOLR-4470_branch_4x_r145.patch, 
 SOLR-4470_trunk_r1568857.patch


 We want to protect any HTTP-resource (url). We want to require credentials no 
 matter what kind of HTTP-request you make to a Solr-node.
 It can faily easy be acheived as described on 
 http://wiki.apache.org/solr/SolrSecurity. This problem is that Solr-nodes 
 also make internal request to other Solr-nodes, and for it to work 
 credentials need to be provided here also.
 Ideally we would like to forward credentials from a particular request to 
 all the internal sub-requests it triggers. E.g. for search and update 
 request.
 But there are also internal requests
 * that only indirectly/asynchronously triggered from outside requests (e.g. 
 shard creation/deletion/etc based on calls to the Collection API)
 * that do not in any way have relation to an outside super-request (e.g. 
 replica synching stuff)
 We would like to aim at a solution where original credentials are 
 forwarded when a request directly/synchronously trigger a subrequest, and 
 fallback to a configured internal credentials for the 
 asynchronous/non-rooted requests.
 In our solution we would aim at only supporting basic http auth, but we would 
 like to make a framework around it, so that not to much refactoring is 
 needed if you later want to make support for other kinds of auth (e.g. digest)
 We will work at a solution but create this JIRA issue early in order to get 
 input/comments from the community as early as possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-04 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6810:
-
Attachment: branch_5x_rev1642874.patch

Got in doubt how exactly the store-read-improvements would be when start-param 
is  0. Now {{TestDistributedQueryAlgorithm.testDocReads}} clearly show that.
Besides that improved test-coverage for sort + start  0 queries in 
{{TestDistributedQueryAlgorithm}}. And better assert messages.

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch, branch_5x_rev1642874.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-04 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232772#comment-14232772
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/4/14 10:59 AM:


We have solved the problem (reducing response-time by a factor of 60 on our 
particular system/data/distribution) the following way

Introduced the concept of distributed query algorithm (DQA) controlled by 
request parameter {{dqa}}. Naming the existing (default) distributed query 
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and 
introducing a new alternative distributed query algorithm called 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias 
{{frfilrfbi}}) 
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of 
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different 
way  - see JavaDoc of 
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}

Believe “distributed query algorithm” is a pendant to elasticsearch's “search 
type”, but just with much better naming that say something about what it is 
actually controlling :-)

Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to 
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}} 
that becomes single-pass (going from 2 to 1 pass) with this flag. 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes. 
{{dqa.forceSkipGetIds=true}} is default for 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}.

Attaching patch corresponding to our solution - going into production as we 
speak to reduce our response-times by a factor of 60. You do not necessarily 
need to just adopt it. But lets at least consider it a starting-point for a 
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods 
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score. 
This is important for the optimization 
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of 
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s. 
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test 
asserts that you only go to store X times across the cluster and not (up to) 
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help 
collecting information about how {{LeafReader}} s are used in different 
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds 
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes 
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a 
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around 
DQA’s. And made more systematic in the way the tests are performed. Do not want 
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for 
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired 
running the test-suite - when you do not specify which DQA you want explicitly 
in the request. With helper-methods for fixing the DQA for tests that focus on 
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the 
test-suite green. But should probably be committed as part of SOLR-6812, and 
left out of this SOLR-6810. New DQA 
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has 
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default. 
And since we run tests randomly selecting the DQA for every query, we are also 
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will 
likely fail if skip-get-ids does not work for all kinds of requests. This is 
actually also a good way to have {{dqa.forceSkipGetIds}} (old 
{{distrib.singlePass}}) tested, so that we will not have a partially-working 
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to 
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have 
been removed again, because the problems (along with any other problems with 
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of 
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it 
will not make the test-suite fail, by forcing the particular query in 
{{DistributedExpandComponentTest}} to use

[jira] [Commented] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234145#comment-14234145
]

Per Steffensen commented on SOLR-6816:
--

Just want to add my 5 cents on this one. It is only regarding indexing when you
do version-check/optimistic-locking (SOLR-3178). We have a very different
implementation of SOLR-3178, but the performance problems will be the same for
your implementation.

Doing optimistic-locking you typically do a lot of this
* 1) real-time-get document D from Solr
* 2) update D to D' locally on client
* 3) try to replace D with D' in Solr. In case of version-conflict-error go to
1)

In step 1) you get-by-id document D, and I step 3) you UpdateLog.lookupVersion
on the same id.
In our system it is most likely that two processes, both wanting to update
document D, run at the same time or fairly shortly after each other. It is rare
that the same document gets updated a long time apart.
In order to speed up on those aspects, we have introduced a recently looked-up
or updated cache, where we store documents that has recently been fetch by
real-time-get or updated. It has improved our indexing speed significantly. We
have a mature solution that is running in production.

In the scenarios above you most often discover that the document you try to
real-time-get or lookup-version for does NOT exist, but it is relatively
time-consuming to realize that (looking in index). We have a PoC of introducing
a bloom-filter that can help say one of document definitely does not exist
(you do not have to search the index) or document may exist (you will have to
search the index to see if it exists). Our PoC shows that this will speed up
our indexing-speed tremendously (like 60-80% reduction), but we havnt
prioritized to mature and put it into production yet. The PoC was using a
modified version of Guava bloom-filter - modified to be able work in a
memory-mapped file, so that we do not lose bloom-filter information when
shutting down Solr (it will take some time building it from scratch every time
you start Solr). Guava bloom-filter currently is memory only - you can save it
to file and load it again, but it will not go on continuously, and it is not
efficient to store it completely to disk at every update :-) Hence the work in
memory-mapped file modification.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type: Task
Components: SolrCloud
Reporter: Mark Miller
Priority: Critical
Attachments: SolrBench.pdf

--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6816) Review SolrCloud Indexing Performance.

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-6816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234145#comment-14234145
]

Per Steffensen edited comment on SOLR-6816 at 12/4/14 12:11 PM:

Of course, let me know if any of this sounds interesting to you.

was (Author: steff1193):
Just want to add my 5 cents on this one. It is only regarding indexing when you
do version-check/optimistic-locking (SOLR-3178). We have a very different
implementation of SOLR-3178, but the performance problems will be the same for
your implementation.

Review SolrCloud Indexing Performance.
--

Key: SOLR-6816
URL: https://issues.apache.org/jira/browse/SOLR-6816
Project: Solr
Issue Type:

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234178#comment-14234178
]

Per Steffensen commented on SOLR-4470:
--

bq. I noticed that some of the comments were around the size of the patch

Yes, unfortunately I have provided a significant number of features/patches
that are fairly big that never got committed.
It is not that I really want to do it this way. I would much rather, as I sense
its most often done, start out with rough half-working patches, and design the
solution by adding to and discussing the patches as they evolve.
Unfortunately that just does not fit very well with the way my customer expects
me to work. He tells me (in this case) I want to protect my Solrs with
username/password on HTTP level. You have two weeks to do it. I'd better get
started. I make a solution the way I think it should be, and do not have the
time to discuss it to much with the rest of you. When I have made a working
solution to the customer, I feel obligated to at least try to give it back to
Solr - if you want to use Open Source, make sure to (try to) give back whenever
you can.

This way I end up providing a fully functional complete solution. I understand
that this is hard to grasp for you guys. But I also believe that I am often
mistaken. I am not saying that you just have to commit this without any change,
dialog or whatever. I just want you to pretend that we are in the beginning of
solving this ticket, and just look at the full solution from me as a suggestion
on how to solve it, that at least can be used starting point for a discussion
and design process. After I have made the complete solution my customer is
happy, and then I have the time to discuss/design the real solution with you
guys. But it usually never gets that far - no committer really want to jump
into it. Then I just say to myself that I did my duty and forwarded my solution
to the community. If no one wants to participate I cannot do much more.

The thing about the discussion/design process is that it is very CALENDAR-time
consuming. It is not that I will spend that much actual time on it - usually
you just add a few comments, change the patch a little and wait for comments,
while you do other stuff (e.g. implementing the next feature for the customer
:-) ). So I will have the time to participate in this after full
solution-phase of making the right solution for Solr.

bq. Is it correct that every SolrServer can handle authCredentials? And not
e.g. only HttpSolrServer

Yes

bq. Is it mandatory to nearly double the API with this extra parameter?

No. Actually I thought [~janhoy] removed the added API methods

bq. UpdateRequest.add().doc(doc).commitWithin(-1).authCredentials(creds)

Yes the builder-approach is IMHO in general a much better approach compared to
having numerous methods with all combinations of parameters or having one
methods taking all parameters where special values (like null, -1 or
something) means that you just want default on that one. But I dont see that
done in Solr. When in Rome, I guess...

bq. Then we don't put the authCredentials in the face of all users

You are probably right

bq. I think we ought to be able to break it down into smaller manageable
commits

Yes, I am sure that can be done. Seems you have a lot of good ideas on how to
cut the cake. In regards to this particular SOLR-4470, I think you are focusing
too much on the API changes, though. It is a fairly small part of the patch. So
if we want to cut SOLR-4470 into smaller patches we need more cutting that just
API stuff and the rest. But in general I like all of your ideas in this
area.

Support for basic http auth in internal solr requests
-

[jira] [Commented] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234548#comment-14234548
]

Per Steffensen commented on SOLR-4470:
--

Thanks for participating [~zoomieia1980]!!!

I am not sure you really understand what my patch is about. It ONLY deals with
adding credentials to outgoing requests in order to be able to penetrate a
security-layer on the server-side. This is necessary if you choose to somehow
put a security-layer between the Solr client and the Solr server. This patch
has NOTHING to do with setting up that security-layer around the server.
Webcontainers do that very well for you. And if Solr really want to move away
from being a webapplication that has to run in a webcontainer, there are
numerous other frameworks out there that you can use to set up a security-layer
around your server. But this patch has nothing to do with how you add that
security-layer around your server. You keep talking about JAAS LoginModules
etc. and they really are concepts belonging in this security-layer around the
server - thats why I believe you do not completely follow my patch.

Changing the SolrJ client so that it adds hardcoded credentials to all
outgoing requests is really just a few lines around the HTTPClient-usage that
you talk about. In this area, my patch really is not more than those few lines,
except that I support doing it at several levels
* Setting up the credentials once and for all on HTTPClient level, so that it
will add those credentials to all requests sent through it
* Doing the same on SolrJ XXXSolrServer (clients) level - basically doing it on
the HTTPClients of the HTTPSolrServers used directly or underneath more
complicated XXXSolrServers like CloudSolrServer
* It also supports setting credentials on request-level, allowing you to use
different credentials for different requests sent through the same HTTPClient,
without having to change the global credentials on HTTPClient level
But all of this is not that many lines of code

The thing is, that in a Cloud setup, the Solr server itself is a Solr client,
sending requests to other Solr servers. If there is a security-layer between
the Solr servers (and there will be if you use e.g. webcontainer managed
security) you need to somehow tell the Solr servers which credentials to add to
its outgoing requests targeting other Solr servers. My patch is a lot about
this aspect. You can fairly easily make in a way so that it just always uses
the same hardcoded credentials on those outgoing requests. But you might want
to configure those fixed credentials from the outside. What if you actually
want to use different credentials depending on the kind of the outgoing
request. What if the outgoing request is actually triggered from an incoming
request, do you want to use the credentials from the ingoing request on the
outgoing request? E.g. if an outside client sends a distributed query R1 to
Solr server A, Solr server A has to send sub-requests R2 to Solr server B -
should it copy the credentials from R1 to R2 or should it use
configured/hardcoded credentials from Q2?
You might have very different algorithms for Solr servers to decide on
credentials for the requests it sends to other Solr servers. In some cases the
Solr server might need to fetch the credentials from an outside agency, or you
might have a handshake/negotiating-step (like SOLR-6625) or something. Most of
the non-test-code in my patch is about making the component that calculates
credentials for requests going out of Solr servers pluggable.

But most of the patch is actually test-code. I tend to want to test my stuff
thoroughly. In Solr tests we start actual Solr servers sending real requests to
each other. So why not test this entire thing by setting up webcontainer
managed security in the embedded jetty-containers running the Solr servers in
test. Some of the code joggles those things.

Support for basic http auth in internal solr requests
-

We want to protect any

[jira] [Comment Edited] (SOLR-4470) Support for basic http auth in internal solr requests

2014-12-04 Thread Per Steffensen (JIRA)

[
https://issues.apache.org/jira/browse/SOLR-4470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14234548#comment-14234548
]

Per Steffensen edited comment on SOLR-4470 at 12/4/14 7:56 PM:
---

Thanks for participating [~zoomieia1980]!!!

But all of this is not that many lines of code

The thing is, that in a Cloud setup, the Solr server itself is a Solr client,
sending requests to other Solr servers. If there is a security-layer between
the Solr servers (and there will be if you use e.g. webcontainer managed
security) you need to somehow tell the Solr servers which credentials to add to
its outgoing requests targeting other Solr servers. My patch is a lot about
this aspect. You can fairly easily make in a way so that it just always uses
the same hardcoded credentials on those outgoing requests. But you might want
to configure those fixed credentials from the outside. What if you actually
want to use different credentials depending on the kind of the outgoing
request. What if the outgoing request is actually triggered from an incoming
request, do you want to use the credentials from the ingoing request on the
outgoing request? E.g. if an outside client sends a distributed query R1 to
Solr server A, Solr server A has to send sub-requests R2 to Solr server B -
should it copy the credentials from R1 to R2 or should it use
configured/hardcoded credentials for R2?
You might have very different algorithms for Solr servers to decide on
credentials for the requests it sends to other Solr servers. In some cases the
Solr server might need to fetch the credentials from an outside agency, or you
might have a handshake/negotiating-step (like SOLR-6625) or something. Most of
the non-test-code in my patch is about making the component that calculates
credentials for requests going out of Solr servers pluggable.

was (Author: steff1193):
Thanks for participating [~zoomieia1980]!!!

[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6810:
-
Description: 
Searching limited but high rows across many shards all with many hits is slow
E.g.
* Query from outside client: q=somethingrows=1000
* Resulting in sub-requests to each shard something a-la this
** 1) q=somethingrows=1000fl=id,score
** 2) Request the full documents with ids in the global-top-1000 found among 
the top-1000 from each shard

What does the subject mean
* limited but high rows means 1000 in the example above
* many shards means 200-1000 in our case
* all with many hits means that each of the shards have a significant number 
of hits on the query
The problem grows on all three factors above

Doing such a query on our system takes between 5 min to 1 hour - depending on a 
lot of things. It ought to be much faster, so lets make it.

Profiling show that the problem is that it takes lots of time to access the 
store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
Having 1000 shards its up to 1 mio ids that has to be fetched. There is really 
no good reason to ever read information from store for more than the overall 
top-1000 documents, that has to be returned to the client.

For further detail see mail-thread Slow searching limited but high rows across 
many shards all with high hits started 13/11-2014 on dev@lucene.apache.org

  was:
Searching limited but high rows across many shards all with many hits is slow
E.g.
* Query from outside client: q=somethingrows=1000
* Resulting in sub-requests to each shard something a-la this
** 1) q=somethingrows=1000fl=id,score
** 2) Request the full documents with ids in the global-top-1000 found among 
the top-1000 from each shard

What does the subject mean
* limited but high rows means 1000 in the example above
* many shards means 200-1000 in our case
* all with many hits means that each of the shards have a significant number 
of hits on the query
The problem grows on all three factors above

Doing such a query on our system takes between 5 min to 1 hour - depending on a 
lot of things. It ought to be much faster, so lets make it.

For further detail see mail-thread Slow searching limited but high rows across 
many shards all with high hits started 13/11-2014 on dev@lucene.apache.org


 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
  Labels: distributed_search, performance

 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232772#comment-14232772
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/3/14 8:50 AM:
---

We have solved the problem (reducing response-time by a factor of 60 on our 
particular system/data/distribution) the following way

Introduced the concept of distributed query algorithm (DQA) controlled by 
request parameter {{dqa}}. Naming the existing (default) distributed query 
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and 
introducing a new alternative distributed query algorithm called 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias 
{{frfilrfbi}}) 
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of 
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids does}} it in a different 
way  - see JavaDoc of 
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}

Believe “distributed query algorithm” is a pendant to elasticsearch's “search 
type”, but just with much better naming that say something about what it is 
actually controlling :-)

Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to 
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}} 
that becomes single-pass (going from 2 to 1 pass) with this flag. 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes. 
{{dqa.forceSkipGetIds=true}} is default for 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need 
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is 
supported.

Attaching patch corresponding to our solution - going into production as we 
speak to reduce our response-times by a factor of 60. You do not necessarily 
need to just adopt it. But lets at least consider it a starting-point for a 
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods 
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score. 
This is important for the optimization 
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of 
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s. 
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test 
asserts that you only go to store X times across the cluster and not (up to) 
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help 
collecting information about how {{LeafReader}} s are used in different 
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds 
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes 
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a 
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around 
DQA’s. And made more systematic in the way the tests are performed. Do not want 
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for 
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired 
running the test-suite - when you do not specify which DQA you want explicitly 
in the request. With helper-methods for fixing the DQA for tests that focus on 
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the 
test-suite green. But should probably be committed as part of SOLR-6812, and 
left out of this SOLR-6810. New DQA 
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has 
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default. 
And since we run tests randomly selecting the DQA for every query, we are also 
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will 
likely fail if skip-get-ids does not work for all kinds of requests. This is 
actually also a good way to have {{dqa.forceSkipGetIds}} (old 
{{distrib.singlePass}}) tested, so that we will not have a partially-working 
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to 
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have 
been removed again, because the problems (along with any other problems with 
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of 
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it 
will not make the test-suite

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232781#comment-14232781
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/3/14 8:44 AM:
---

SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 where all just preparation for 
this patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work 
for all queries (ever fired during the test-suite). So we have a complete 
feature now (when those issues are corrected).


was (Author: steff1193):
SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 was all just preparation for this 
patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work for 
all queries (ever fires during the test-suite). So we have a complete feature 
now (when those issues are corrected).

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6810:
-
Attachment: branch_5x_rev1642874.patch

We have solved the problem (reducing response-time by a factor of 60 on our 
particular system/data/distribution) the following way

Introduced the concept of distributed query algorithm (DQA) controlled by 
request parameter {{dqa}}. Naming the existing (default) distributed query 
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and 
introducing a new alternative distributed query algorithm called 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias 
{{frfilrfbi}}) 
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of 
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids does}} it in a different 
way  - see JavaDoc of 
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}

Believe “distributed query algorithm” is a pendant to elasticsearch's “search 
type”, but just with much better naming that say something about what it is 
actually controlling :-)

Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to 
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}} 
that becomes single-pass (going from 2 to 1 pass) with this flag. 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes. 
{{dqa.forceSkipGetIds=true}} is default for 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need 
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is 
supported.

Attaching patch corresponding to our solution - going into production as we 
speak to reduce our response-times by a factor of 60. You do not necessarily 
need to just adopt it. But lets at least consider it a starting-point for a 
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods 
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score. 
This is important for the optimization 
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of 
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s. 
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test 
asserts that you only go to store X times across the cluster and not (up to) 
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help 
collecting information about how {{LeafReader}} s are used in different 
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds 
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes 
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a 
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around 
DQA’s. And made more systematic in the way the tests are performed. Do not want 
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for 
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired 
running the test-suite - when you do not specify which DQA you want explicitly 
in the request. With helper-methods for fixing the DQA for tests that focus on 
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the 
test-suite green. But should probably be committed as part of SOLR-6812, and 
left out of this SOLR-6810. New DQA 
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has 
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default. 
And since we run tests randomly selecting the DQA for every query, we are also 
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will 
likely fail if skip-get-ids does not work for all kinds of requests. This is 
actually also a good way to have {{dqa.forceSkipGetIds}} (old 
{{distrib.singlePass}}) tested, so that we will not have a partially-working 
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to 
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have 
been removed again, because the problems (along with any other problems with 
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of 
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it 
will not make the test-suite fail, by forcing the particular query in

[jira] [Commented] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232781#comment-14232781
 ] 

Per Steffensen commented on SOLR-6810:
--

SOLR-6795, SOLR-6796, SOLR-6812 and SOLR-6813 was all just preparation for this 
patch. It makes {{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) work for 
all queries (ever fires during the test-suite). So we have a complete feature 
now (when those issues are corrected).

 Faster searching limited but high rows across many shards all with many hits
 

 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen
  Labels: distributed_search, performance
 Attachments: branch_5x_rev1642874.patch


 Searching limited but high rows across many shards all with many hits is 
 slow
 E.g.
 * Query from outside client: q=somethingrows=1000
 * Resulting in sub-requests to each shard something a-la this
 ** 1) q=somethingrows=1000fl=id,score
 ** 2) Request the full documents with ids in the global-top-1000 found among 
 the top-1000 from each shard
 What does the subject mean
 * limited but high rows means 1000 in the example above
 * many shards means 200-1000 in our case
 * all with many hits means that each of the shards have a significant 
 number of hits on the query
 The problem grows on all three factors above
 Doing such a query on our system takes between 5 min to 1 hour - depending on 
 a lot of things. It ought to be much faster, so lets make it.
 Profiling show that the problem is that it takes lots of time to access the 
 store to get id’s for (up to) 1000 docs (value of rows parameter) per shard. 
 Having 1000 shards its up to 1 mio ids that has to be fetched. There is 
 really no good reason to ever read information from store for more than the 
 overall top-1000 documents, that has to be returned to the client.
 For further detail see mail-thread Slow searching limited but high rows 
 across many shards all with high hits started 13/11-2014 on 
 dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232772#comment-14232772
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/3/14 3:48 PM:
---

We have solved the problem (reducing response-time by a factor of 60 on our 
particular system/data/distribution) the following way

Introduced the concept of distributed query algorithm (DQA) controlled by 
request parameter {{dqa}}. Naming the existing (default) distributed query 
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and 
introducing a new alternative distributed query algorithm called 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias 
{{frfilrfbi}}) 
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of 
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different 
way  - see JavaDoc of 
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}

Believe “distributed query algorithm” is a pendant to elasticsearch's “search 
type”, but just with much better naming that say something about what it is 
actually controlling :-)

Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to 
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}} 
that becomes single-pass (going from 2 to 1 pass) with this flag. 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes. 
{{dqa.forceSkipGetIds=true}} is default for 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need 
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is 
supported.

Attaching patch corresponding to our solution - going into production as we 
speak to reduce our response-times by a factor of 60. You do not necessarily 
need to just adopt it. But lets at least consider it a starting-point for a 
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods 
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score. 
This is important for the optimization 
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of 
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s. 
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test 
asserts that you only go to store X times across the cluster and not (up to) 
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help 
collecting information about how {{LeafReader}} s are used in different 
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds 
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes 
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a 
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around 
DQA’s. And made more systematic in the way the tests are performed. Do not want 
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for 
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired 
running the test-suite - when you do not specify which DQA you want explicitly 
in the request. With helper-methods for fixing the DQA for tests that focus on 
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the 
test-suite green. But should probably be committed as part of SOLR-6812, and 
left out of this SOLR-6810. New DQA 
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has 
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default. 
And since we run tests randomly selecting the DQA for every query, we are also 
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will 
likely fail if skip-get-ids does not work for all kinds of requests. This is 
actually also a good way to have {{dqa.forceSkipGetIds}} (old 
{{distrib.singlePass}}) tested, so that we will not have a partially-working 
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to 
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have 
been removed again, because the problems (along with any other problems with 
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of 
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it 
will not make the test-suite

[jira] [Comment Edited] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-03 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6810?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14232772#comment-14232772
 ] 

Per Steffensen edited comment on SOLR-6810 at 12/3/14 3:53 PM:
---

We have solved the problem (reducing response-time by a factor of 60 on our 
particular system/data/distribution) the following way

Introduced the concept of distributed query algorithm (DQA) controlled by 
request parameter {{dqa}}. Naming the existing (default) distributed query 
algorithm {{find-id-relevance_fetch-by-ids}} (short-alias {{firfbi}}) and 
introducing a new alternative distributed query algorithm called 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} (short-alias 
{{frfilrfbi}}) 
* {{find-id-relevance_fetch-by-ids}} does as always - see JavaDoc of 
{{ShardParams.FIND_ID_RELEVANCE_FETCH_BY_IDS}}
* {{find-relevance_find-ids-limited-rows_fetch-by-ids}} does it in a different 
way  - see JavaDoc of 
{{ShardParams.FIND_RELEVANCE_FIND_IDS_LIMITED_ROWS_FETCH_BY_IDS}}

Believe “distributed query algorithm” is a pendant to elasticsearch's “search 
type”, but just with much better naming that say something about what it is 
actually controlling :-)

Both DQAs support the {{disturb.singlePass}} flag. I have *renamed* it to 
{{dqa.forceSkipGetIds}} because it is only {{find-id-relevance_fetch-by-ids}} 
that becomes single-pass (going from 2 to 1 pass) with this flag. 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}} goes from 3 to 2 passes. 
{{dqa.forceSkipGetIds=true}} is default for 
{{find-relevance_find-ids-limited-rows_fetch-by-ids}}. There are really no need 
to ever run with {{dqa.forceSkipGetIds=false}} for this DQA, but it is 
supported.

Attaching patch corresponding to our solution - going into production as we 
speak to reduce our response-times by a factor of 60. You do not necessarily 
need to just adopt it. But lets at least consider it a starting-point for a 
discussion. Details about the patch
* {{ShardParams.DQA}}: Enum of the DQA’s, including different helper methods 
that IMHO belongs here
* {{QueryComponent}}/{{ResponseBuilder}}: Changed to implement both DQA’s now
* {{SolrIndexSearcher.doc}}: Does not go to store, if only asking for score. 
This is important for the optimization 
* {{TestIndexSearcher}}: Added a test to test this particular new aspect of 
{{SolrIndexSearcher}}
* {{TestDistributedQueryAlgorithm}}: A new test-class dedicated tests of DQA’s. 
{{testDocReads}}-test really shows exactly what this new DQA does for you. Test 
asserts that you only go to store X times across the cluster and not (up to) 
#shards * X times (X = rows in outer query)
* {{LeafReaderTestWrappers}}: Test-wrappers for {{LeafReader}} s. Can help 
collecting information about how {{LeafReader}} s are used in different 
test-scenarios. Used by {{TestIndexSearcher}}. Can be extended with other kinds 
of wrappers that collect different kinds of information.
* {{SolrIndexSearcherTestWrapper}} and {{SolrCoreTestWrapper}}. Generic classes 
that can help wrapping all {{LeafReader}} s under a {{SolrIndexSearcher}} or a 
{{SolrCore}} respectively. Used by {{TestDistributedQueryAlgorithm}}
* {{DistributedQueryComponentOptimizationTest}}: Updated with new tests around 
DQA’s. And made more systematic in the way the tests are performed. Do not want 
to add hundreds of almost similar code-lines
* {{ShardRoutingTest}}: Same comments as for 
{{DistributedQueryComponentOptimizationTest}} above
* {{SolrTestCaseJ4}}: Randomly selecting a DQA for each individual query fired 
running the test-suite - when you do not specify which DQA you want explicitly 
in the request. With helper-methods for fixing the DQA for tests that focus on 
DQA testing
* Fix for SOLR-6812 is included in the patch because it is need to keep the 
test-suite green. But should probably be committed as part of SOLR-6812, and 
left out of this SOLR-6810. New DQA 
({{find-relevance_find-ids-limited-rows_fetch-by-ids}}) has 
{{dqa.forceSkipGetIds}} (old {{disturb.singlePass}}) set to true by default. 
And since we run tests randomly selecting the DQA for every query, we are also 
indirectly randoming {{dqa.forceSkipGetIds}}. Therefore the test-suite will 
likely fail if skip-get-ids does not work for all kinds of requests. This is 
actually also a good way to have {{dqa.forceSkipGetIds}} (old 
{{distrib.singlePass}}) tested, so that we will not have a partially-working 
feature (as before SOLR-6795/SOLR-6796/SOLR-6812/SOLR-6813). The tests added to 
{{DistributedQueryComponentOptimizationTest}} in SOLR-6795 and SOLR-6796 have 
been removed again, because the problems (along with any other problems with 
{{dqa.forceSkipGetIds}}) will now (potentially) be revealed anyway because of 
indirect randomized testing of {{dqa.forceSkipGetIds}}
* I do not have a solution to SOLR-6813, so temporarily making sure that it 
will not make the test-suite

[jira] [Created] (SOLR-6812) distrib.singlePass does not work for expand-request

2014-12-02 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-6812:


 Summary: distrib.singlePass does not work for expand-request
 Key: SOLR-6812
 URL: https://issues.apache.org/jira/browse/SOLR-6812
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen


Using distrib.singlePass does not work for expand-requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6812) distrib.singlePass does not work for expand-request

2014-12-02 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6812:
-
Attachment: test_that_reveals_the_problem.patch

Attached patch (branch_5x) that change an existing test to reveal the problem. 
Apply the patch and run {{DistributedExpandComponentTest}}
Probably not just want to commit the changed test after a fix of the problem, 
but maybe we want to add the test somewhere else.

 distrib.singlePass does not work for expand-request
 ---

 Key: SOLR-6812
 URL: https://issues.apache.org/jira/browse/SOLR-6812
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6812) distrib.singlePass does not work for expand-request

2014-12-02 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6812:
-
Attachment: fix.patch

Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda 
detects that it is supposed to do its thing for shard-searches, by seeing if 
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the 
things in {{QueryComponent}}. So not purpose is distributed in the 
shard-requests.

 distrib.singlePass does not work for expand-request
 ---

 Key: SOLR-6812
 URL: https://issues.apache.org/jira/browse/SOLR-6812
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: fix.patch, test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-12-02 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231325#comment-14231325
 ] 

Per Steffensen commented on SOLR-6796:
--

Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]?

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included

2014-12-02 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-6813:


 Summary: distrib.singlePass does not work for expand-request - 
start/rows included
 Key: SOLR-6813
 URL: https://issues.apache.org/jira/browse/SOLR-6813
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen


Using distrib.singlePass does not work for expand-requests. Even after the fix 
provided to SOLR-6812, it does not work for requests where you add start and/or 
rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included

2014-12-02 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6813:
-
Attachment: test_that_reveals_the_problem.patch

Attached patch (branch_5x) that change an existing test to reveal the problem. 
Apply the patch and run DistributedExpandComponentTest
Probably not just want to commit the changed test after a fix of the problem, 
but maybe we want to add the test somewhere else.

 distrib.singlePass does not work for expand-request - start/rows included
 -

 Key: SOLR-6813
 URL: https://issues.apache.org/jira/browse/SOLR-6813
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests. Even after the 
 fix provided to SOLR-6812, it does not work for requests where you add start 
 and/or rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6813) distrib.singlePass does not work for expand-request - start/rows included

2014-12-02 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231428#comment-14231428
 ] 

Per Steffensen commented on SOLR-6813:
--

Unlike for SOLR-6812 (and SOLR-6795 and SOLR-6796) I am not going to provide a 
fix here, because I really do not know enough about expand-requests to be able 
to say which response is actually the correct one - {{controlRsp}} or {{rsp}} 
in {{BaseDistributedSearchTestCase.query}} when doing the modified request from 
{{test_that_reveals_the_problem.patch}}

 distrib.singlePass does not work for expand-request - start/rows included
 -

 Key: SOLR-6813
 URL: https://issues.apache.org/jira/browse/SOLR-6813
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests. Even after the 
 fix provided to SOLR-6812, it does not work for requests where you add start 
 and/or rows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-12-02 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231325#comment-14231325
 ] 

Per Steffensen edited comment on SOLR-6796 at 12/2/14 12:55 PM:


Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]?
And maybe SOLR-6813? :-) I know I am pushing it now :-)


was (Author: steff1193):
Maybe I can convince you to take a look at SOLR-6812 also, [~shalinmangar]?

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6812) distrib.singlePass does not work for expand-request

2014-12-02 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14231323#comment-14231323
 ] 

Per Steffensen edited comment on SOLR-6812 at 12/2/14 3:09 PM:
---

Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda 
detects that it is supposed to do its thing for shard-searches, by seeing if 
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the 
things in {{QueryComponent}}. So now purpose is distributed in the 
shard-requests.


was (Author: steff1193):
Attached a fix to the problem. Problem is that {{ExpandComponent}} kinda 
detects that it is supposed to do its thing for shard-searches, by seeing if 
ids-param is there. It is not in case of {{distrib.singlePass}}.
Patch works on purpose instead, because that is really whats controlling the 
things in {{QueryComponent}}. So not purpose is distributed in the 
shard-requests.

 distrib.singlePass does not work for expand-request
 ---

 Key: SOLR-6812
 URL: https://issues.apache.org/jira/browse/SOLR-6812
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: fix.patch, test_that_reveals_the_problem.patch


 Using distrib.singlePass does not work for expand-requests.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6810) Faster searching limited but high rows across many shards all with many hits

2014-12-01 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-6810:


 Summary: Faster searching limited but high rows across many shards 
all with many hits
 Key: SOLR-6810
 URL: https://issues.apache.org/jira/browse/SOLR-6810
 Project: Solr
  Issue Type: Improvement
  Components: search
Reporter: Per Steffensen


Searching limited but high rows across many shards all with many hits is slow
E.g.
* Query from outside client: q=somethingrows=1000
* Resulting in sub-requests to each shard something a-la this
** 1) q=somethingrows=1000fl=id,score
** 2) Request the full documents with ids in the global-top-1000 found among 
the top-1000 from each shard

What does the subject mean
* limited but high rows means 1000 in the example above
* many shards means 200-1000 in our case
* all with many hits means that each of the shards have a significant number 
of hits on the query
The problem grows on all three factors above

Doing such a query on our system takes between 5 min to 1 hour - depending on a 
lot of things. It ought to be much faster, so lets make it.

For further detail see mail-thread Slow searching limited but high rows across 
many shards all with high hits started 13/11-2014 on dev@lucene.apache.org



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-27 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6796:
-
Attachment: fix.patch

But ok, here is a fix that cleans up a little bit. Essentially using the FL's 
of the outer request. But we have to add id-field and score if not already in 
there

Also passes 4.4.0+ test-suite running with distrib.singlePass all over

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-27 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227498#comment-14227498
 ] 

Per Steffensen edited comment on SOLR-6796 at 11/27/14 11:07 AM:
-

But ok, here is a fix that cleans up a little bit. Essentially using the FL's 
of the outer request. But we have to add id-field and score if not already in 
there
Besides the patch you should be able to delete import java.util.regex.Pattern 
as well

Also passes 4.4.0+ test-suite running with distrib.singlePass all over


was (Author: steff1193):
But ok, here is a fix that cleans up a little bit. Essentially using the FL's 
of the outer request. But we have to add id-field and score if not already in 
there

Also passes 4.4.0+ test-suite running with distrib.singlePass all over

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-26 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6796:
-
Attachment: fix.patch

I believe this fix will make {{distrib.singlePass=true}} work for any query. 
That is, you can turn on distrib.singlePass whenever you like, and have a 
response similar to a non-distributed request to a single core/shard containing 
the same data.
We have a version of Solr based on 4.4.0, with a lot of our own changes, and 
with SOLR-5768, SOLR-1880 and parts of SOLR-5399 merged. With this fix, then 
entire 4.4.0 test-suite is green, when we make all queries issued across the 
test-suite run with {{distrib.singlePass=true}}.
At least it fixes the particular problem that this SOLR-6796 is about.

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Comment Edited] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-26 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14225979#comment-14225979
 ] 

Per Steffensen edited comment on SOLR-6796 at 11/26/14 10:11 AM:
-

I believe this fix will make {{distrib.singlePass=true}} work for any query 
(together with SOLR-6795). That is, you can turn on distrib.singlePass whenever 
you like, and have a response similar to a non-distributed request to a single 
core/shard containing the same data.
We have a version of Solr based on 4.4.0, with a lot of our own changes, and 
with SOLR-5768, SOLR-1880 and parts of SOLR-5399 merged. With this fix, then 
entire 4.4.0 test-suite is green, when we make all queries issued across the 
test-suite run with {{distrib.singlePass=true}}.
At least it fixes the particular problem that this SOLR-6796 is about.


was (Author: steff1193):
I believe this fix will make {{distrib.singlePass=true}} work for any query. 
That is, you can turn on distrib.singlePass whenever you like, and have a 
response similar to a non-distributed request to a single core/shard containing 
the same data.
We have a version of Solr based on 4.4.0, with a lot of our own changes, and 
with SOLR-5768, SOLR-1880 and parts of SOLR-5399 merged. With this fix, then 
entire 4.4.0 test-suite is green, when we make all queries issued across the 
test-suite run with {{distrib.singlePass=true}}.
At least it fixes the particular problem that this SOLR-6796 is about.

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Commented] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-26 Thread Per Steffensen (JIRA)


[ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14227348#comment-14227348
 ] 

Per Steffensen commented on SOLR-6796:
--

bq. Both the original code and the patches seem more complex than needed

I agree. I didnt want to spend much time making it less complex. Just wanted it 
to work. When I have brought cleaned up code in the past, I hasnt been 
appreciated much. Some times it has been rejected with an argument about 
creating another issue for cleanup. 

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
Assignee: Shalin Shekhar Mangar
  Labels: distributed_search, search
 Attachments: fix.patch, fix.patch, fix.patch, 
 test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6795) distrib.singlePass returns score even though not asked for

2014-11-25 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-6795:


 Summary: distrib.singlePass returns score even though not asked for
 Key: SOLR-6795
 URL: https://issues.apache.org/jira/browse/SOLR-6795
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen


If I pass distrib.singlePass in a request and do not ask for score back (fl 
does not include score) it will return the score back anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6795) distrib.singlePass returns score even though not asked for

2014-11-25 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6795:
-
Attachment: test_that_reveals_the_problem.patch

Attached patch (branch_5x) that change an existing test to reveal the problem. 
Apply the patch and run e.g. {{BasicDistributedZk2Test}}
Probably not just want to commit the changed test after a fix of the problem, 
but maybe we want to add the test somewhere else. In my company we have an 
agreement that whenever we fix a bug, we also create a test that reveals it

 distrib.singlePass returns score even though not asked for
 --

 Key: SOLR-6795
 URL: https://issues.apache.org/jira/browse/SOLR-6795
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request and do not ask for score back (fl 
 does not include score) it will return the score back anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6795) distrib.singlePass returns score even though not asked for

2014-11-25 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6795:
-
Attachment: fix.patch

Attached a fix to the problem. I believe it will not ruin other functionality - 
but I havnt run the entire test-suite with this patch

 distrib.singlePass returns score even though not asked for
 --

 Key: SOLR-6795
 URL: https://issues.apache.org/jira/browse/SOLR-6795
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: fix.patch, test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request and do not ask for score back (fl 
 does not include score) it will return the score back anyway.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Created] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-25 Thread Per Steffensen (JIRA)

Per Steffensen created SOLR-6796:


 Summary: distrib.singlePass does not return correct set of fields 
for multi-fl-parameter requests
 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen


If I pass distrib.singlePass in a request that also has two fl-parameters, in 
some cases, I will not get the expected set of fields back for the returned 
documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-25 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6796:
-
Attachment: test_that_reveals_the_problem.patch

Attached patch (branch_5x) that change an existing test to reveal the problem. 
Apply the patch and run {{TestDistributedSearch}}
Probably not just want to commit the changed test after a fix of the problem, 
but maybe we want to add the test somewhere else. In my company we have an 
agreement that whenever we fix a bug, we also create a test that reveals it

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

[jira] [Updated] (SOLR-6796) distrib.singlePass does not return correct set of fields for multi-fl-parameter requests

2014-11-25 Thread Per Steffensen (JIRA)


 [ 
https://issues.apache.org/jira/browse/SOLR-6796?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Per Steffensen updated SOLR-6796:
-
Attachment: fix.patch

Attached an ugly fix to the problem. I believe it will not ruin other 
functionality - but I havnt run the entire test-suite with this patch

 distrib.singlePass does not return correct set of fields for 
 multi-fl-parameter requests
 

 Key: SOLR-6796
 URL: https://issues.apache.org/jira/browse/SOLR-6796
 Project: Solr
  Issue Type: Bug
  Components: multicore, search
Affects Versions: 5.0
Reporter: Per Steffensen
  Labels: distributed_search, search
 Attachments: fix.patch, test_that_reveals_the_problem.patch


 If I pass distrib.singlePass in a request that also has two fl-parameters, in 
 some cases, I will not get the expected set of fields back for the returned 
 documents



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org

1 2 3 4 >

1 - 100 of 392 matches

Mail list logo