Hello,
In our scale environment, we see that the deep paging queries using
cursormark are running really slow. When we traced out the calls, we see
that the second query which queries the individual id's of matched pages is
sending the q param that is already sent by the first query again. If we
remove the q param and directly query for ids, the query runs really fast.
For example, the initial pagination query is like this with q param on
timestamp field -
2017-10-14 12:20:51.647 UTC INFO (qtp331844619-393343)
[core='x:c6e422fc3054c475-core-1']
org.apache.solr.core.SolrCore.Request@2304 [c6e422fc3054c475-core-1]
webapp=/solr path=/select
params={distrib=false&df=text&paginatedQuery=true&fl=id&shards.purpose=4&start=0&fsv=true&sort=timestamp+desc+,id+asc&shard.url=
http://ops-data-solr-svc-1.rattle.svc.cluster.local:80/solr/c6e422fc3054c475-core-1&;
*rows=50*&version=2&
*q=(timestamp:["2017-10-13T18:42:36Z"+TO+"2017-10-13T21:09:00Z"])*
&shards.tolerant=true&*cursorMark=**&NOW=1507928978918&isShard=true&timeAllowed=-1&wt=javabin&trackingId=d5eff5476247487555b7413214648}
hits=40294067 status=0 QTime=12727
This query results in a second query due to solr implementation of deep
paging like below. In this query, we already know the ids to be matched.
So, there is no reason to pass the q param again. We tried manually
executing the below query without the q param and just passing the ids
alone and that executes in 50ms. So, this looks like a bug that Solr is
passing in the q param again. Any ideas if there is workaround for this
problem we can use?
2017-10-14 12:21:09.193 UTC INFO (qtp331844619-742579)
[core='x:6d63f95961c46475-core-1']
org.apache.solr.core.SolrCore.Request@2304 [6d63f95961c46475-core-1]
webapp=/solr path=/select
params={distrib=false&df=text&paginatedQuery=true&fl=*,[removedocvaluesuffix]&shards.purpose=64&shard.url=
http://ops-data-solr-svc-1.rattle.svc.cluster.local:80/solr/6d63f95961c46475-core-1&rows=50&version=2&;
*q=(timestamp:["2017-10-14T08:50:16.340Z"+TO+"2017-10-14T19:19:50Z"])*&shards.tolerant=true&NOW=1507983581099&ids=00f037832e571941ed46ddd195920502,145c82e3eaa7678564b9e520822a3de1,09633cfabc6c830dfb44e04c313ba6b4,0032a76ed4ea01207c2891070348ea39,1b5179ee23fe3e17236da37d6b8d991f,04ee42e481b2a657bd3bb3c9f91b5ed5,2a910cf8a259925046a0c9fb5ee013c3,1d1d607b03c18ec59c14c2f9ca0ab47f,034e775c96633dae7e629a1d37da86e6,2759ca26d449d5df9f41689aa8ed3bac,16995a57699a7bb56d5018fe145028ce,0509d16399e679470ffc07a8af22a918,1797ab6e0174c65bf2f6b650b3538407,11c804ec4ae153a31929abe8613de336,11d20ed5dc0cf3d71f57aefc4e4b3ee2,0135baecd2d3ae819909a0c021bbd48b,224b0671196fd141196b15add2e49b91,271088227cf81e3641130d3bd5de8cc6,01f266b9c130239a06b00e45bda277a0,1438bed6ffd956f1c49d765b942f1988,2fc9fef6500124b1b48218169a7cf974,2d85d00593847398bf09e168bb3a190c,10e1c2803df1db3d47e525d3bd8a1868,28b6d72729e79da3ad65ac3879740133,14be34af9995721b358b3fdb0bcb18d7,1f2e0867bd495b8a332b8c8bd8ce2508,12cf1a1c07d9b9550ece4079b15f7583,022cd0b3eef93cd9a6389c0394cf3899,11aa3132e00a96df6a49540612b91c8f,0ff348e0433c9e751f1475db6dcab213,2b48279c9ff833f43a910edfa455a31d,241e002d744ff0215155f214166fdd49,0fee30860c82d9a24bb8389317cd772c,07f04d380832f514b0575866958eebaa,20b0efa5d88e2a9950fa4fd8ba930455,14a9cadb7c75274bfc028bb9ae31236b,1829730aa4ee4750eb242266830b576b,1ad5012e83bd271cf00b0c70ea86a856,0af4247d057bd833753e1f7bef959fc4,0a09767d81cb351ab1598987022b6955,2f166fae9ca809642b8e20cea3020c24,2c4d900575d8594a040c94751af59cb5,03f1c46a004a4e3b995295b512e1e324,2c2aae83afc7426424c7de5301f8c692,034baf21ac1db436a7f3f2cf2cc668b0,1dda29d03fb8611f8de80b90685fd9ee,0632292ab704dcaa606440cb1fee017b,0fbd68f293c6964458a93f3034348625,2cdff46ab2e4d44b42f3381d5e3250b7,1b2c90dce4a51b5e5c344fc2f9ab431d&isShard=true&timeAllowed=-1&wt=javabin&trackingId=d5eff5476247487555b80c9ac7b82}
status=0 QTime=18136
Thanks
Sundeep
