solo source build in local error
solr-repo/lucene-solr/build.xml:21: The following error occurred while executing this line:/solr-repo/lucene-solr/lucene/common-build.xml:623: java.lang.NullPointerException at java.util.Arrays.stream(Arrays.java:5004) at java.util.stream.Stream.of(Stream.java:1000) at java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:267) at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)I am trying to build and debug SOLR . seems build throwing error. anything to set before run build ? -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
solr 7.5.0 custom search component.
We are migrating solr5 to solr7.5. we have custom search component. the prepare() method called multiple times ( single node(one box) two shard) in solr 7.5. we assume that each shard it called one time. Due to that response builder are changed in prepare method so causing issue. But the same was working fine solr 5.x with single call prepare method eventhough we have same collection setup ( single node(one box) two shard). Any guidance could help. Please advice. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
SOLR 7.5.0 (Migrate from 5.3.1 to 7.5.0)
We are migrating solr 5.3 to solr 7.5 after done, 2/12/2019, 11:22:37 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=4 2/12/2019, 11:22:37 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=4 2/12/2019, 11:22:37 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=5 2/12/2019, 11:22:37 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=5 2/12/2019, 11:22:38 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=5 2/12/2019, 11:22:38 AM WARN false x:test_shard20_replica_n38 SolrCore [test_shard20_replica_n38] PERFORMANCE WARNING: Overlapping onDeckSearchers=6 2/12/2019, 1:45:39 PM WARN true x:test_shard20_replica_n38 DirectUpdateHandler2 Starting optimize... Reading and rewriting the entire index! Use with care. Eventhough index is optimized and segment xount is 1. Keep triggering "DirectUpdateHandler2 Starting optimize... Reading and rewriting the entire index! Use with care." What these warning signifies and how to resolve those ? we configured ZK host (3 hosts) created CNAME and use that to SOLR configuration zkHost=http://<>/ -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Migrate from sol 5.3.1 to 7.5.0
We are migrating SOLR version, We used 3 ZK hosts that configured to SOLR as ZK connection string: zookeeper.solrtest.net:2181/test-config Ensemble size: 1 Ensemble mode: ensemble zookeeper.solrtest.net:2181 ok true clientPort 2181 zk_server_state follower zk_version 3.4.5 zk_approximate_data_size9902464 zk_znode_count 1734 zk_num_alive_connections43 serverId4 electionPort3888 quorumPort 2888 zookeeper.solrtest.net . is nothing but CNAME of all three ZKHost together. In solr admin console we see Errors: We do not have a leader And we are using custom component before query goes to "query" component in proper order execution but pagination is not working after some limit for example :start:599 results comes but start:600 docs[] empty in the response. Results counts are reduced for some queries fields(combined) if we use our custom component order but works if we use /select. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Migrate from sol 5.3.1 to 7.5.0
Thanks Erick, I was waiting your Day time appears to get suggestion because
you are very spontaneous for this great open source community.
1. Did you recompile and redistriburte your custom component?
* Yes*
2. Did you take the solrconfig that came with 7.5 and modify it rather
than copy your 5.3 solrconfig?
No, We have adjusted existing 5.3.1 config when we used for couple years to
SOLR 7.5.0.
3. Did you reindex from scratch with 7x? Lucene/Solr guarantees only
one major version back-compat.
Yes, We have done Full Reindexing. No issues noticed with reindex.
All works as expected in the flow of /select request handler. Noticed one
stuff below changed in SOLR 7.5.0
All Shard Requests are distributed back to custom component flow instead
default components.
// for distributed queries that don’t include shards.qt, use the original
path
// as the default but operators need to update their
luceneMatchVersion to enable
// this behavior since it did not work this way prior to 5.1
if
(req.getCore().getSolrConfig().luceneMatchVersion.onOrAfter(Version.LUCENE_5_1_0))
{
String reqPath = (String) req.getContext().get(PATH);
if (!“/select”.equals(reqPath)) {
params.set(CommonParams.QT, reqPath);
} // else if path is /select, then the qt gets passed thru
if set
} else {
// this is the pre-5.1 behavior, which translates to
sending the shard request to /select
params.remove(CommonParams.QT);
}
Below some portion of debug log.
Enabled Debug and noticed that for the custom component handler flow for all
shard request,
"PARSE_QUERY":{
"http://<>_shard26_replica_n9/":{
"QTime":"27",
"ElapsedTime":"28",
"RequestPurpose":"GET_TERM_STATS",
"NumFound":"<>",
"Response":"{responseHeader={zkConnected=true,xqs=best,info={},status=0,QTime=27},response={numFound=
"EXECUTE_QUERY":{
"http://<>_shard3_replica_n27/":{
"QTime":"26",
"ElapsedTime":"26",
"RequestPurpose":"GET_TOP_IDS,GET_FIELDS,GET_DEBUG,SET_TERM_STATS",
"NumFound":"<>",
"Response":"{responseHeader={zkConnected=true,xqs=best,info={},status=0,QTime=26},response={numFound=
*But when Default(/select) flow goes,*
"PARSE_QUERY":{
"http://<>_shard24_replica_n6/":{
"QTime":"0",
"ElapsedTime":"1",
"RequestPurpose":"GET_TERM_STATS",
"Response":"{responseHeader={zkConnected=true,status=0,QTime=0,params={df=all,distrib=false,debug=[false,
timing, track],shards.purpose=3276
}
"EXECUTE_QUERY":{
"http://<>_shard30_replica_n1/":{
"QTime":"17",
"ElapsedTime":"19",
"RequestPurpose":"GET_TOP_IDS,SET_TERM_STATS",
"NumFound":"316605",
"Response":"{responseHeader={zkConnected=true,status=0,QTime=17,params={df=all,distrib=false,fl=[id,
score],shards.purpose=16388
is there any documentation about to read all these flow purpose and
understand better
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Migrate from sol 5.3.1 to 7.5.0
Thanks Jan, I am unaware how our Devops team decided this But this was working well without any issues with SOLR 5.3.1 for couple of years. Just wanted to make any changes in SOLR7 mandates . -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Migrate from sol 5.3.1 to 7.5.0
Hi Eric, Found the reason and all shard request to go /select flow solves the problem. Regarding In SOLR 7 when relevancy added for the search it is not working (not expected results) for the above fieldtype but same works fine for /select because it uses Lucene Parser but our flow uses EDISMAX. But 5.3.1 works fine both cases. As soon as we change as below, synonymQueryStyle="as_same_term" Both cases works in 7.5.0 flow. Is there any intentional changes for the above scenario in SOLR 7.5.0 where I can refer ? -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Migrate from sol 5.3.1 to 7.5.0
Do we need to reindex if we change synonymQueryStyle values for a fieldType ? I hope not. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
COLLECTION CREATE and CLUSTERSTATUS changes in SOLR 7.5.0
SOLR 7.5.0
Created collection as below:
/admin/collections?action=CREATE=test_collection=1=1=replica:<2,node:*=test_collection_config
Created Successfully.
After that when I try to see CLUSTERSTATUS, it is giving "replicas": {}
empty. But IN SOLR 5.3.1 it was giving the response but SOLR 7.5.0 it is
not.
Please help out what is missing how to fix.
/solr/admin/collections?action=CLUSTERSTATUS=json
{
"responseHeader": {
"status": 0,
"QTime": 1
},
"cluster": {
"collections": {
"test_collection": {
"pullReplicas": "0",
"replicationFactor": "1",
"shards": {
"shard1": {
"range": "8000-7fff",
"state": "active",
"replicas": {}
}
},
"router": {
"name": "compositeId"
},
"maxShardsPerNode": "1",
"autoAddReplicas": "false",
"nrtReplicas": "1",
"tlogReplicas": "0",
"rule": [
{
"replica": "<2",
"node": "*"
}
],
"znodeVersion": 6,
"configName": "test_collection_config"
}
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: COLLECTION CREATE and CLUSTERSTATUS changes in SOLR 7.5.0
I found the reason, =true when I create a collection with this parameter I could find that replicas data in CLUSTERSTATUS api request,. is there anything wrong if I use this in SOLR 7.5.0 when create a collection ? Please advice. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: COLLECTION CREATE and CLUSTERSTATUS changes in SOLR 7.5.0
Thanks a lot Erick, Shawn and Walter. As Shawn mentioned I should not use core admin API when in solrcloud, Just reloading collection solved the problem. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: COLLECTION CREATE and CLUSTERSTATUS changes in SOLR 7.5.0
Thanks Eric, After we create a collection and copy the index from one place
new place, we are doing UNLOAD core and CREATE core as below, is it wrong
and we have alternative to do that ?
r = requests.get('{}/admin/cores'.format(base_url), params={
'action': 'UNLOAD',
'core': core_name
})
r.raise_for_status()
time.sleep(10)
r = requests.get('{}/admin/cores'.format(base_url), params={
'action': 'CREATE',
'name': core_name,
'shard': shard_name,
'collection': collection_name,
'collection.configName': config_name
})
r.raise_for_status()
this is why legacyCloud=false is not supporting ?
--
Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: COLLECTION CREATE and CLUSTERSTATUS changes in SOLR 7.5.0
What is legacyCloud properties does. Is there a document I could refer. Why it impacts collection creation with rule based and index copy from one to another. Replicas not registered how to fix without legacyCloud=true settings. -- Sent from: http://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Suggestion or recommendation for NRT
Even though same document indexed over and over again due to incremental update. Index size is being increased. Do I miss any configuration to make optimization occur by internally ? -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Suggestion or recommendation for NRT
Hi Team, Any suggestion or recommendation for the above approach which we are doing to have better search performance. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Suggestion or recommendation for NRT
Thanks a lot for your time to respond for my clarifications. We are having two environment, ENV A and ENV B ( Both same capacity of RAM ( r5.2xlarge and same number of shards and replicas type (NRT) for the collection) ENV A - it is having a collection which is optimized ( segment count 1 and numdocs = maxdocs ) it is used only for Search request. No delta updates are being triggerred. ENV B - It is having same collection copied from "ENV A" and continues DELTA updates in progress so it is used for Indexing and search request. Indexing using KAFKA connect plugin that uses SOLRJ with solr.commit.within=30 ( milli seconds ) We are comparing performance between those environments for search request using automation test running with bunch of queries. Regarding search warmup: 1 true 20 200 *:* true *:* true false 24 -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Re: Suggestion or recommendation for NRT
Thanks Erick for the details and reference to understand better about merging segment stuff. When I compare performance of uninterrupted/optimized ( segment count 1) collection for search request vs (indexing + search) in parallel going on collection performance is 3 times higher, for example : first one is responding 100ms in average but second one around 400ms. is that expected behaviour like we need to tradeoff if we do Indexing and search in the same collection parallel. or we can still fine tune with some parameters for better performance then please suggest some. -- Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
Suggestion or recommendation for NRT
Hi,
We are using SOLR 7.5.0 version, We are testing one collection for both
Search and Index.
Our collection created with below indexerconfig, We are using indexing
process KAFKA connect plugin with every 5 min commit (cloud SOLRJ) as below
https://github.com/jcustenborder/kafka-connect-solr
Our collection 30 shard and 3 replica with good RAM EC2 nodes ( 90 nodes) .
it is almost 2.5 TB size. We could see the performance impact for search
request when indexing in progress. Any kind of recommendation or fine
tunning steps to be considered , Please provide any references if there is
available that will help.
150
8000
100
10
10
${solr.lock.type:native}
true
--
Sent from: https://lucene.472066.n3.nabble.com/Solr-User-f472068.html
