Concurrent query execution and Solr
Hi, Does anybody know if work is in progress to make Lucene's concurrent query execution accessible through Solr? I am talking about this: http://blog.mikemccandless.com/2019/10/concurrent-query-execution-in-apache.html I find this compelling in particular since the changes in LUCENE-7976 / Solr 7.5 where, even after an optimize, you end up with a number of almost equally sized segments. And for those who would go to Solr Cloud for parallel query execution only because they have other means of redundancy in place, this is a nice way to avoid additional complexity with ZooKeeper. Thanks, André
Inaccuracies sorting by payload value
I have a problem when sorting by payload value ... the resulting sort order
is correct for some documents, but not all.
The field type and field definitions are as follows:
The request parameters are the following:
_exact_utag_primary_id:utag77n5840c6h5v0g9b9ww
_id,_utag_order,sort_val:${COMPUTED_SORT_VAL}
payload(_utag_order,$COLL_ID)
${COMPUTED_SORT_VAL} asc
999
xml
utag77n5840c6h5v0g9b9ww
And here is an excerpt of some of the documents in the result that are not
sorted correctly:
doc77n53r5bag9e3075ikm
utag77n53dda1mf1cda749s1|1571733614
utag77n5840c6h5v0g9b9ww|1571734246
1.57173427E9
doc77n52cnj78nmksuhikl
utag77n520zikegu6yjkfhn|1571733431
utag77n5840c6h5v0g9b9ww|1571734248
1.57173427E9
doc77n52c05tevwo08hikk
utag77n520zikegu6yjkfhn|1571733428
utag77n5840c6h5v0g9b9ww|1571734247
1.57173427E9
Please check the payload value (number after the dash) in the second item
under "_utag_order": the order returned is 1571734246, 1571734248,
1571734247, so obviously wrong.
It looks like the fact the payload function always returns a float results
in a loss of precision the encoded integer payload values.
Is this a known issue? Is there a work-around to correctly sort by the
value that has been encoded?
Thanks for reading,
André
Re: Slow soft-commit
almost forgot to report back, maybe it helps somebody else it turned
out to be caused by a feature in our software being used in a way we did
not anticipate.
That resulted in a lot (> 100.000) of different dynamic fields which
probably is an anti-pattern on its own, but the slow commits where related
to the fact that these fields had DocValues. After some profiling, it
clearly showed a lot of time was spent in FieldInfos' addOrUpdateInternal()
and related code.
André
Am Mi., 22. Mai 2019 um 18:12 Uhr schrieb André Widhani :
> Hi everyone,
>
> I need some advice how to debug slow soft commits.
>
> We use Solr for searches in a DAM system and in similar setups, soft
> commits take about one to two seconds, in this case nearly ten seconds.
> Solr runs on a dedicated VM with eight cores and 64 GB RAM (16G heap),
> which is common scenario with our software and the index holds about 20
> million documents. Queries are as fast as expected.
>
> This is Solr 7.5.0, stand-alone, auto hard-commit set to 60 seconds, no
> explicit soft-commits but documents added with commitWhitin=5000 or 1000
> depending on the use case. No warm-up queries, caches set to zero.
>
> I enabled infostream and debug logging. Here is a little test case where I
> stopped any other requests to Solr and just manually added a single
> document and then posted a soft commit request.
>
> 2019-05-22 17:19:42.160 INFO (qtp26728049-20)
> o.a.s.u.DirectUpdateHandler2 start
> commit{_version_=1634245942610755584,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
> 2019-05-22 17:19:42.160 DEBUG (qtp26728049-20) o.a.s.u.UpdateLog TLOG:
> preSoftCommit: prevMap=1930097580 new map=1023061476
> 2019-05-22 17:19:42.160 DEBUG (qtp26728049-20)
> o.a.s.c.CachingDirectoryFactory Reusing cached directory:
> CachedDir<>
> 2019-05-22 17:19:42.160 DEBUG (qtp26728049-20)
> o.a.s.c.CachingDirectoryFactory Releasing directory:
> /solr/core-tex73oy02hnxgx1dqc14p5o-index1 0 false
> 2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
> [DW][qtp26728049-20]: anyChanges? numDocsInRam=1 deletes=true
> hasTickets:false pendingChangesInFullFlush: false
> 2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
> [IW][qtp26728049-20]: nrtIsCurrent: infoVersion matches: false; DW changes:
> true; BD changes: false
> 2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
> [IW][qtp26728049-20]: flush at getReader
> ... a lot of things logged here (omitted) that happen within milli-seconds
> ...
> 2019-05-22 17:19:42.168 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
> [IW][qtp26728049-20]: getReader took 8 msec
> 2019-05-22 17:19:47.499 INFO (qtp26728049-20) o.a.s.s.SolrIndexSearcher
> Opening [Searcher@6211242f[core-tex73oy02hnxgx1dqc14p5o-index1] main]
> 2019-05-22 17:19:47.499 DEBUG (qtp26728049-20)
> o.a.s.c.CachingDirectoryFactory incRef'ed:
> CachedDir<>
> 2019-05-22 17:19:50.233 DEBUG (qtp26728049-20) o.a.s.s.SolrIndexSearcher
> Closing [Searcher@78d9785[core-tex73oy02hnxgx1dqc14p5o-index1] realtime]
> 2019-05-22 17:19:50.233 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
> [IW][qtp26728049-20]: decRefDeleter for NRT reader version=10560246
> segments=_22lfz(7.5.0):c1033782/56603:delGen=2772
> _22lfp(7.5.0):c1025574/39055:delGen=2113
> _22lfr(7.5.0):c759249/32191:delGen=1386
> _26q49(7.5.0):c923418/29825:delGen=958
> _22lfx(7.5.0):c684064/30952:delGen=1098
> _22lfv(7.5.0):c856317/78777:delGen=928
> _22lg1(7.5.0):c1062384/188447:delGen=1750
> _22lg0(7.5.0):c561881/1480:delGen=386
> _22lg5(7.5.0):c1104218/1004:delGen=139 _22lgh(7.5.0):c1156482/46:delGen=33
> _22lgf(7.5.0):c626273/27:delGen=19 _22lfy(7.5.0):c697224/6:delGen=6
> _22lgd(7.5.0):c399283/6:delGen=3 _22lgg(7.5.0):c482373/3:delGen=3
> _22lg4(7.5.0):c656746/2:delGen=2 _22lgc(7.5.0):c664274/3:delGen=3
> _22lg7(7.5.0):c703377 _22lfu(7.5.0):c700340/3:delGen=3
> _22lg2(7.5.0):c743334 _22lg6(7.5.0):c1091387/44659:delGen=946
> _22lfs(7.5.0):c845018 _22lg8(7.5.0):c675649 _22lg9(7.5.0):c686292
> _22lft(7.5.0):c636751/1004:delGen=300 _22lgb(7.5.0):c664531
> _22lga(7.5.0):c647696/1:delGen=1 _22lge(7.5.0):c659794
> _22lfw(7.5.0):c568537/1:delGen=1 _22lg3(7.5.0):c837568/1426:delGen=423
> _26r20(7.5.0):c63899/10456:delGen=257 _273qn(7.5.0):c39076/8075:delGen=323
> _27q6g(7.5.0):c40830/8111:delGen=195 _27aii(7.5.0):c30182/6777:delGen=287
> _27tkq(7.5.0):c57620/6234:delGen=162 _27jrm(7.5.0):c33298/4797:delGen=202
> _280zq(7.5.0):c60476/2341:delGen=173 _28h7x(7.5.0):c48570/453:delGen=35
> _28c75(7.5.0):c29536/1088:delGen=47 _28h71(7.5.0):c1191/138:delGen=5
> _28idl(7.5.0):c782/57:delGen=6 _28ihc(7.5.0):c5398/348:delGen=9
> _28ig3(7.5.0):c1118/302:delGen=4 _28j1s(7
Slow soft-commit
Hi everyone,
I need some advice how to debug slow soft commits.
We use Solr for searches in a DAM system and in similar setups, soft
commits take about one to two seconds, in this case nearly ten seconds.
Solr runs on a dedicated VM with eight cores and 64 GB RAM (16G heap),
which is common scenario with our software and the index holds about 20
million documents. Queries are as fast as expected.
This is Solr 7.5.0, stand-alone, auto hard-commit set to 60 seconds, no
explicit soft-commits but documents added with commitWhitin=5000 or 1000
depending on the use case. No warm-up queries, caches set to zero.
I enabled infostream and debug logging. Here is a little test case where I
stopped any other requests to Solr and just manually added a single
document and then posted a soft commit request.
2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.DirectUpdateHandler2
start
commit{_version_=1634245942610755584,optimize=false,openSearcher=true,waitSearcher=true,expungeDeletes=false,softCommit=true,prepareCommit=false}
2019-05-22 17:19:42.160 DEBUG (qtp26728049-20) o.a.s.u.UpdateLog TLOG:
preSoftCommit: prevMap=1930097580 new map=1023061476
2019-05-22 17:19:42.160 DEBUG (qtp26728049-20)
o.a.s.c.CachingDirectoryFactory Reusing cached directory:
CachedDir<>
2019-05-22 17:19:42.160 DEBUG (qtp26728049-20)
o.a.s.c.CachingDirectoryFactory Releasing directory:
/solr/core-tex73oy02hnxgx1dqc14p5o-index1 0 false
2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
[DW][qtp26728049-20]: anyChanges? numDocsInRam=1 deletes=true
hasTickets:false pendingChangesInFullFlush: false
2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
[IW][qtp26728049-20]: nrtIsCurrent: infoVersion matches: false; DW changes:
true; BD changes: false
2019-05-22 17:19:42.160 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
[IW][qtp26728049-20]: flush at getReader
... a lot of things logged here (omitted) that happen within milli-seconds
...
2019-05-22 17:19:42.168 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
[IW][qtp26728049-20]: getReader took 8 msec
2019-05-22 17:19:47.499 INFO (qtp26728049-20) o.a.s.s.SolrIndexSearcher
Opening [Searcher@6211242f[core-tex73oy02hnxgx1dqc14p5o-index1] main]
2019-05-22 17:19:47.499 DEBUG (qtp26728049-20)
o.a.s.c.CachingDirectoryFactory incRef'ed:
CachedDir<>
2019-05-22 17:19:50.233 DEBUG (qtp26728049-20) o.a.s.s.SolrIndexSearcher
Closing [Searcher@78d9785[core-tex73oy02hnxgx1dqc14p5o-index1] realtime]
2019-05-22 17:19:50.233 INFO (qtp26728049-20) o.a.s.u.LoggingInfoStream
[IW][qtp26728049-20]: decRefDeleter for NRT reader version=10560246
segments=_22lfz(7.5.0):c1033782/56603:delGen=2772
_22lfp(7.5.0):c1025574/39055:delGen=2113
_22lfr(7.5.0):c759249/32191:delGen=1386
_26q49(7.5.0):c923418/29825:delGen=958
_22lfx(7.5.0):c684064/30952:delGen=1098
_22lfv(7.5.0):c856317/78777:delGen=928
_22lg1(7.5.0):c1062384/188447:delGen=1750
_22lg0(7.5.0):c561881/1480:delGen=386
_22lg5(7.5.0):c1104218/1004:delGen=139 _22lgh(7.5.0):c1156482/46:delGen=33
_22lgf(7.5.0):c626273/27:delGen=19 _22lfy(7.5.0):c697224/6:delGen=6
_22lgd(7.5.0):c399283/6:delGen=3 _22lgg(7.5.0):c482373/3:delGen=3
_22lg4(7.5.0):c656746/2:delGen=2 _22lgc(7.5.0):c664274/3:delGen=3
_22lg7(7.5.0):c703377 _22lfu(7.5.0):c700340/3:delGen=3
_22lg2(7.5.0):c743334 _22lg6(7.5.0):c1091387/44659:delGen=946
_22lfs(7.5.0):c845018 _22lg8(7.5.0):c675649 _22lg9(7.5.0):c686292
_22lft(7.5.0):c636751/1004:delGen=300 _22lgb(7.5.0):c664531
_22lga(7.5.0):c647696/1:delGen=1 _22lge(7.5.0):c659794
_22lfw(7.5.0):c568537/1:delGen=1 _22lg3(7.5.0):c837568/1426:delGen=423
_26r20(7.5.0):c63899/10456:delGen=257 _273qn(7.5.0):c39076/8075:delGen=323
_27q6g(7.5.0):c40830/8111:delGen=195 _27aii(7.5.0):c30182/6777:delGen=287
_27tkq(7.5.0):c57620/6234:delGen=162 _27jrm(7.5.0):c33298/4797:delGen=202
_280zq(7.5.0):c60476/2341:delGen=173 _28h7x(7.5.0):c48570/453:delGen=35
_28c75(7.5.0):c29536/1088:delGen=47 _28h71(7.5.0):c1191/138:delGen=5
_28idl(7.5.0):c782/57:delGen=6 _28ihc(7.5.0):c5398/348:delGen=9
_28ig3(7.5.0):c1118/302:delGen=4 _28j1s(7.5.0):c917/269:delGen=2
_28iu9(7.5.0):c758/129:delGen=4 _28j23(7.5.0):c567/70:delGen=2
_28j3j(7.5.0):c802/11:delGen=1 _28j3u(7.5.0):c697/11:delGen=2
_28iz5(7.5.0):c858/116:delGen=4 _28j2n(7.5.0):c566/78:delGen=1
_28j3t(7.5.0):C20/11:delGen=1 _28j3v(7.5.0):C13/8:delGen=1 _28j3y(7.5.0):C1
2019-05-22 17:19:50.234 DEBUG (qtp26728049-20)
o.a.s.c.CachingDirectoryFactory Releasing directory:
/solr/core-tex73oy02hnxgx1dqc14p5o-index1/index 3 false
2019-05-22 17:19:50.234 INFO (qtp26728049-20) o.a.s.u.DirectUpdateHandler2
end_commit_flush
2019-05-22 17:19:50.234 DEBUG
(searcherExecutor-10-thread-1-processing-x:core-tex73oy02hnxgx1dqc14p5o-index1)
o.a.s.s.SolrIndexSearcher autowarming
[Searcher@6211242f[core-tex73oy02hnxgx1dqc14p5o-index1]
main{ExitableDirectoryReader(UninvertingDirectoryReader(Uninverting(_22lfz(7.5.0):c1033782/56603:delGen=2772)
... ...Uninverting(_28j3z(7.5.0):C1)))}] from
Re: Replacing the entire schema using schema API
Hi Shawn, thanks for confirming. I am not using Solr Cloud (I forgot to mention that), or at least not in all instances where that particular piece of code would be used. I'll think about opening a Jira issue, or just doing it iteratively through the API. Regards, André 2018-01-05 15:05 GMT+01:00 Shawn Heisey <apa...@elyograg.org>: > On 1/5/2018 6:51 AM, André Widhani wrote: > >> I know I can retrieve the entire schema using Schema API and I can also >> use >> it to manipulate the schema by adding fields etc. >> >> I don't see any way to post an entire schema file back to the Schema API >> though ... this is what most REST APIs offer: You retrieve an object, >> modify it and send back the entire object. >> >> I could of course loop through all items in the schema, delete them and >> re-create ... this sounds more complicated than it should be. >> >> Is there anything I might have missed in the documentation? >> > > I'm not aware of any way to do this with the Schema API. It is designed > to make individual changes, not wholesale replacement, so that a user > doesn't need to understand the full XML syntax for the schema - they can > send JSON-formatted change requests that are easier to understand than the > entire schema. > > If you're running SolrCloud, then you can upload an entire schema file to > zookeepeer and reload the collection, which can be done remotely. > > If Solr is not in cloud mode, then the only option is to replace the file > on the filesystem and then reload the core(s). > > The feature you want would open up the possibility of uploading a bad > schema in the same way that manual edits can. You're free to file an > enhancement issue in Jira, though. > > Thanks, > Shawn > -- André Widhani Research & Development *Don’t miss our latest news! * www.digicol.de/news Digital Collections Content Systems GmbH Hindenburgstraße 49, 22297 Hamburg Tel: +49 40 23535-128 Fax: +49 40 23535-180 E-Mail: andre.widh...@digicol.de Internet: www.digicol.de HRB Hamburg 48373, Geschäftsführer: Jörn Olsen Haftungsbeschränkung: Diese Nachricht enthält vertrauliche Informationen und ist ausschließlich für den Adressaten bestimmt. Der Gebrauch durch Dritte ist verboten. Das Unternehmen ist nicht verantwortlich für die ordnungsgemäße, vollständige oder verzögerungsfreie Übertragung dieser Nachricht.
Replacing the entire schema using schema API
Hi, I know I can retrieve the entire schema using Schema API and I can also use it to manipulate the schema by adding fields etc. I don't see any way to post an entire schema file back to the Schema API though ... this is what most REST APIs offer: You retrieve an object, modify it and send back the entire object. I could of course loop through all items in the schema, delete them and re-create ... this sounds more complicated than it should be. Is there anything I might have missed in the documentation? Thanks, André
Re: PHP + Solr
I have no experience with either as we have our own PHP layer to interface with Solr. If I started again from scratch today I would surely consider Solarium. The PHP extension seems outdated and no longer maintained. It says it works with Solr 3.1 and last update is from 2011. André 2014-01-28 Jorge Luis Betancourt Gonzalez jlbetanco...@uci.cu I’ve some experience using Solarium and have been great so far. In particular we use the NelmioSolariumBundle to integrate with Symfony2. Greetings! On Jan 28, 2014, at 1:54 PM, Felipe Dantas de Souza Paiva cad_fpa...@uolinc.com wrote: Hi Folks, I would like to know what is the best way to integrate PHP and Apache Solr. Until now I've found two options: 1) http://www.php.net/manual/en/intro.solr.php 2) http://www.solarium-project.org/ What do you guys say? Cheers, Felipe AVISO: A informaç?o contida neste e-mail, bem como em qualquer de seus anexos, é CONFIDENCIAL e destinada ao uso exclusivo do(s) destinat?rio(s) acima referido(s), podendo conter informaç?es sigilosas e/ou legalmente protegidas. Caso você n?o seja o destinat?rio desta mensagem, informamos que qualquer divulgaç?o, distribuiç?o ou c?pia deste e-mail e/ou de qualquer de seus anexos é absolutamente proibida. Solicitamos que o remetente seja comunicado imediatamente, respondendo esta mensagem, e que o original desta mensagem e de seus anexos, bem como toda e qualquer c?pia e/ou impress?o realizada a partir destes, sejam permanentemente apagados e/ou destru?dos. Informaç?es adicionais sobre nossa empresa podem ser obtidas no site http://sobre.uol.com.br/. NOTICE: The information contained in this e-mail and any attachments thereto is CONFIDENTIAL and is intended only for use by the recipient named herein and may contain legally privileged and/or secret information. If you are not the e-mail´s intended recipient, you are hereby notified that any dissemination, distribution or copy of this e-mail, and/or any attachments thereto, is strictly prohibited. Please immediately notify the sender replying to the above mentioned e-mail address, and permanently delete and/or destroy the original and any copy of this e-mail and/or its attachments, as well as any printout thereof. Additional information about our company may be obtained through the site http://www.uol.com.br/ir/. III Escuela Internacional de Invierno en la UCI del 17 al 28 de febrero del 2014. Ver www.uci.cu -- André Widhani Research Development Digital Collections Verlagsgesellschaft mbH Wendenstrasse 130, 20537 Hamburg Tel: +49 40 23535-0
Lucene/Solr 4.5.1 svn tag
Hi, shouldn't there be a tag for the 4.5.1 release under http://svn.apache.org/repos/asf/lucene/dev/tags/ ? Or am I looking at the wrong place? Regards, André
AW: Lucene/Solr 4.5.1 svn tag
Thanks, Mark!
AW: Need Help in migrating Solr version 1.4 to 4.3
fwiw, I can confirm that Solr 4.x can definitely not read indexes created with 1.4. You'll get an exception like the following: Caused by: org.apache.lucene.index.IndexFormatTooOldException: Format version is not supported (resource: segment _16ofy in resource ChecksumIndexInput(MMapIndexInput(path=/var/opt/dcx/solr2/core-tex60l254lpachcjhtz4se-index2/data/index/segments_1dlof))): 2.x. This version of Lucene only supports indexes created with release 3.0 and later. But as Erick mentioned, you could get away with optimizing the index with 3.x instead of re-indexing from scratch before moving on to 4.x - I think I did that once and it worked. Regards, André Von: Erick Erickson [erickerick...@gmail.com] Gesendet: Dienstag, 25. Juni 2013 19:37 An: solr-user@lucene.apache.org Betreff: Re: Need Help in migrating Solr version 1.4 to 4.3 bq: I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes Solr/Lucene explicitly try to read _one_ major revision backwards. Solr 3.x should be able to read 1.4 indexes. Solr 4.x should be able to read Solr 3.x. No attempt is made to allow Solr 4.x to read Solr 1.4 indexes, so I wouldn't even try. Shalin's comment is best. If at all possible I'd just forget about reading the old index and re-index from scratch. But if you _do_ try upgrading 1.4 - 3.x - 4.x, you probably want to optimize at each step. That'll (I think) rewrite all the segments in the current format. Good luck! Erick On Tue, Jun 25, 2013 at 12:59 AM, Shalin Shekhar Mangar shalinman...@gmail.com wrote: You must carefully go through the upgrade instructions starting from 1.4 upto 4.3. In particular the instructions for 1.4 to 3.1 and from 3.1 to 4.0 should be given special attention. On Tue, Jun 25, 2013 at 11:43 AM, Sandeep Gupta gupta...@gmail.com wrote: Hello All, We are planning to migrate solr 1.4 to Solr 4.3 version. And I am seeking some help in this side. Considering Schema file change: By default there are lots of changes if I compare original Solr 1.4 schema file to Sol 4.3 schema file. And that is the reason we are not copying paste of schema file. In our Solr 1.4 schema implementation, we have some custom fields with type textgen and text So in migration of these custom fields to Solr 4.3, should I use type of text_general as replacement of textgen and text_en as replacement of text? Please confirm the same. Please check the text_general definition in 4.3 against the textgen fieldtype in Solr 1.4 to see if they're equivalent. Same for text_en and text. Considering Solrconfig change: As we didn't have lots of changes in 1.4 solrconfig file except the dataimport request handler. And therefore in migration side, we are simply modifying the Solr 4.3 solrconfig file with his request handler. And you need to add the dataimporthandler jar into Solr's lib directory. DIH is not added automatically anymore. Considering the application development: We used all the queries as BOOLEAN type style (was not good) I mean put all the parameter in query fields i.e *:* AND EntityName: AND fileName:fieldValue AND . I think we should simplify our queries using other fields like df, qf Probably. AND queries are best done by filter queries (fq). We also used to create Solr server object via CommonsHttpSolrServer() so I am planning to use now HttpSolrServer API Yes. Also, there was a compatibility break between Solr 1.4 and 3.1 in the javabin format so old clients using javabin won't be able to communicate with Solr until you upgrade both solr client and solr servers. Please let me know the suggestion for above points also what are the other factors I need to take care while considering the migration. There is no substitute for reading the upgrade sections in the changes.txt. I'm not sure if Solr 4.3 will be able to read Solr 1.4 indexes. You will most likely need to re-index your documents. You should also think about switching to SolrCloud to take advantage of its features. -- Regards, Shalin Shekhar Mangar.
AW: Best way to match umlauts
We configure both baseletter conversion (removing accents and umlauts) and alternate spelling through the mapping file. For baseletter conversion and mostly german content we transform all accents that are not used in german language (like french é, è, ê etc.) to their baseletter. We do not do do this for german umlauts, because the assumption is that a user will know the correct spelling in his or her native language but probably not in foreign languages. For alternate spelling, we use the following mapping: # * Alternate spelling # # Additionally, german umlauts are converted to their base form (ä = ae), # and ß is converted to ss. Which means both spellings can be used to find # either one. # \u00C4 = AE \u00D6 = OE \u00DC = UE \u00E4 = ae \u00F6 = oe \u00DF = ss \u00FC = ue André
AW: Solr 4.3 - Schema Parsing Failed: Invalid field property: compressed
From what version are you upgrading? The compressed attribute is unsupported since the 3.x releases. The change log (CHANGES.txt) has a section Upgrading from Solr 1.4 in the notes for Solr 3.1: Field compression is no longer supported. Fields that were formerly compressed will be uncompressed as index segments are merged. For shorter fields, this may actually be an improvement, as the compression used was not very good for short text. Some indexes may get larger though. Also, indices created with 1.4 cannot be opened with 4.x, only 3.x. Regards, André Von: Uomesh [uom...@gmail.com] Gesendet: Montag, 10. Juni 2013 06:19 An: solr-user@lucene.apache.org Betreff: Solr 4.3 - Schema Parsing Failed: Invalid field property: compressed Hi, I am getting below after upgrading to Solr 4.3. Is compressed attribute no longer supported in Solr 4.3 or it is a bug in 4.3? org.apache.solr.common.SolrException:org.apache.solr.common.SolrException: Schema Parsing Failed: Invalid field property: compressed Thanks, Umesh -- View this message in context: http://lucene.472066.n3.nabble.com/Solr-4-3-Schema-Parsing-Failed-Invalid-field-property-compressed-tp4069254.html Sent from the Solr - User mailing list archive at Nabble.com.
AW: Heap space problem with mlt query
I am just reading through this thread by chance, but doesn't this exception: Caused by: org.apache.solr.common.SolrException: Error in xpath:/config/luceneMatchVersion for solrconfig.xml org.apache.solr.common.SolrException: Error in xpath:/config/luceneMatchVersion for solrconfig.xml at indicate some missing or wrong information in solrconfig.xml, specifically the luceneMatchVersion field? Sorry for the confusion if I am on a wrong track here. André
AW: Core admin action CREATE fails to persist some settings in solr.xml with Solr 4.3
I created SOLR-4862 ... I found no way to assign the ticket to somebody though (I guess it is is under Workflow, but the button is greyed out). Thanks, André
AW: Core admin action CREATE fails for existing core
Added the issue. https://issues.apache.org/jira/browse/SOLR-4857 Core admin action RELOAD lacks requests parameters to point core to another config or schema file or data dir Von: André Widhani [andre.widh...@digicol.de] Gesendet: Donnerstag, 23. Mai 2013 23:13 An: solr-user@lucene.apache.org Betreff: AW: Core admin action CREATE fails for existing core Ok - yes, will do so tomorrow. Thanks, André Von: Mark Miller [markrmil...@gmail.com] Gesendet: Donnerstag, 23. Mai 2013 22:46 An: solr-user@lucene.apache.org Betreff: Re: Core admin action CREATE fails for existing core Your right - that does seem to be a new limitation. Could you create a JIRA issue for it? It would be fairly simple to add another reload method that also took the name of a new solrconfig/schema file. - Mark On May 23, 2013, at 4:11 PM, André Widhani andre.widh...@digicol.de wrote: Mark, Alan, thanks for explaining and updating the wiki. When reloading the core using action=CREATE with Solr 4.1 I could specify the path to schema and config. In fact I used this to reconfigure the core to use a specific one of two prepared config files depending on some external index state (instead of making changes to one and the same config file). action=RELOAD does not understand the corresponding request parameters schema and config (which is why I used CREATE, not RELOAD in the first place). So the functionality to switch to a different config file for an existing core is no longer there, I guess? Thanks, André Von: Alan Woodward [a...@flax.co.uk] Gesendet: Donnerstag, 23. Mai 2013 18:43 An: solr-user@lucene.apache.org Betreff: Re: Core admin action CREATE fails for existing core Thanks! Alan Woodward www.flax.co.uk On 23 May 2013, at 17:38, Steve Rowe wrote: Alan, I've added AlanWoodward to the Solr AdminGroup page. On May 23, 2013, at 12:29 PM, Alan Woodward a...@flax.co.uk wrote: I think the wiki needs to be updated to reflect this? http://wiki.apache.org/solr/CoreAdmin If somebody adds me as an editor (AlanWoodward), I'll do it. Alan Woodward www.flax.co.uk On 23 May 2013, at 16:43, Mark Miller wrote: Yes, this did change - it's actually a protection for a previous change though. There was a time when you did a core reload by just making a new core with the same name and closing the old core - that is no longer really supported though - the proper way to do this is to use SolrCore#reload, and that has been the case for all of 4.x release if I remember right. I supported making this change to force people who might still be doing what is likely quite a buggy operation to switch to the correct code. Sorry about the inconvenience. - Mark On May 23, 2013, at 10:45 AM, André Widhani andre.widh...@digicol.de wrote: It seems to me that the behavior of the Core admin action CREATE has changed when going from Solr 4.1 to 4.3. With 4.1, I could re-configure an existing core (changing path/name to solrconfig.xml for example). In 4.3, I get an error message: SEVERE: org.apache.solr.common.SolrException: Error CREATEing SolrCore 'core-tex69b6iom1djrbzmlmg83-index2': Core with name 'core-tex69b6iom1djrbzmlmg83-index2' already exists. Is this change intended? André
AW: Broken pipe
This usually happens when the client sending the request to Solr has given up
waiting for the response (terminated the connection).
In your example, we see that the Solr query time is 81 seconds. Probably the
client issuing the request has a time-out of maybe 30 or 60 seconds.
André
Von: Arkadi Colson [ark...@smartbit.be]
Gesendet: Donnerstag, 23. Mai 2013 15:40
An: solr-user@lucene.apache.org
Betreff: Broken pipe
Any idea why I got a Broken pipe?
INFO - 2013-05-23 13:37:19.881; org.apache.solr.core.SolrCore;
[messages_shard3_replica1] webapp=/solr path=/select/
params={sort=score+descfl=id,smsc_module,smsc_modulekey,smsc_userid,smsc_ssid,smsc_description,smsc_description_ngram,smsc_content,smsc_content_ngram,smsc_courseid,smsc_lastdate,score,metadata_stream_size,metadata_stream_source_info,metadata_stream_name,metadata_stream_content_type,last_modified,author,title,subjectdebugQuery=truedefaultOperator=ANDindent=onstart=0q=(smsc_content:banaan+||+smsc_content_ngram:banaan+||+smsc_description:banaan+||+smsc_description_ngram:banaan)+%26%26+(smsc_lastdate:[2000-04-23T15:14:40Z+TO+2013-05-23T15:14:40Z])+%26%26+(smsc_ssid:9)collection=messageswt=xmlrows=50version=2.2}
hits=119 status=0 QTime=81108
ERROR - 2013-05-23 13:37:19.892; org.apache.solr.common.SolrException;
null:ClientAbortException: java.net.SocketException: Broken pipe
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:406)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:342)
at
org.apache.catalina.connector.OutputBuffer.writeBytes(OutputBuffer.java:431)
at
org.apache.catalina.connector.OutputBuffer.write(OutputBuffer.java:419)
at
org.apache.catalina.connector.CoyoteOutputStream.write(CoyoteOutputStream.java:91)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:221)
at sun.nio.cs.StreamEncoder.implWrite(StreamEncoder.java:282)
at sun.nio.cs.StreamEncoder.write(StreamEncoder.java:125)
at java.io.OutputStreamWriter.write(OutputStreamWriter.java:207)
at org.apache.solr.util.FastWriter.flush(FastWriter.java:141)
at org.apache.solr.util.FastWriter.flushBuffer(FastWriter.java:155)
at
org.apache.solr.response.TextResponseWriter.close(TextResponseWriter.java:85)
at
org.apache.solr.response.XMLResponseWriter.write(XMLResponseWriter.java:41)
at
org.apache.solr.servlet.SolrDispatchFilter.writeResponse(SolrDispatchFilter.java:644)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:372)
at
org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:155)
at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:953)
at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:408)
at
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1008)
at
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at
org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:310)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.net.SocketException: Broken pipe
at java.net.SocketOutputStream.socketWrite0(Native Method)
at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:109)
at java.net.SocketOutputStream.write(SocketOutputStream.java:153)
at
org.apache.coyote.http11.InternalOutputBuffer.realWriteBytes(InternalOutputBuffer.java:215)
at org.apache.tomcat.util.buf.ByteChunk.flushBuffer(ByteChunk.java:480)
at org.apache.tomcat.util.buf.ByteChunk.append(ByteChunk.java:366)
at
org.apache.coyote.http11.InternalOutputBuffer$OutputStreamOutputBuffer.doWrite(InternalOutputBuffer.java:240)
at
org.apache.coyote.http11.filters.ChunkedOutputFilter.doWrite(ChunkedOutputFilter.java:117)
at
org.apache.coyote.http11.AbstractOutputBuffer.doWrite(AbstractOutputBuffer.java:192)
at org.apache.coyote.Response.doWrite(Response.java:505)
at
org.apache.catalina.connector.OutputBuffer.realWriteBytes(OutputBuffer.java:401)
... 30 more
ERROR -
Core admin action CREATE fails for existing core
It seems to me that the behavior of the Core admin action CREATE has changed when going from Solr 4.1 to 4.3. With 4.1, I could re-configure an existing core (changing path/name to solrconfig.xml for example). In 4.3, I get an error message: SEVERE: org.apache.solr.common.SolrException: Error CREATEing SolrCore 'core-tex69b6iom1djrbzmlmg83-index2': Core with name 'core-tex69b6iom1djrbzmlmg83-index2' already exists. Is this change intended? André
AW: Core admin action CREATE fails for existing core
Mark, Alan, thanks for explaining and updating the wiki. When reloading the core using action=CREATE with Solr 4.1 I could specify the path to schema and config. In fact I used this to reconfigure the core to use a specific one of two prepared config files depending on some external index state (instead of making changes to one and the same config file). action=RELOAD does not understand the corresponding request parameters schema and config (which is why I used CREATE, not RELOAD in the first place). So the functionality to switch to a different config file for an existing core is no longer there, I guess? Thanks, André Von: Alan Woodward [a...@flax.co.uk] Gesendet: Donnerstag, 23. Mai 2013 18:43 An: solr-user@lucene.apache.org Betreff: Re: Core admin action CREATE fails for existing core Thanks! Alan Woodward www.flax.co.uk On 23 May 2013, at 17:38, Steve Rowe wrote: Alan, I've added AlanWoodward to the Solr AdminGroup page. On May 23, 2013, at 12:29 PM, Alan Woodward a...@flax.co.uk wrote: I think the wiki needs to be updated to reflect this? http://wiki.apache.org/solr/CoreAdmin If somebody adds me as an editor (AlanWoodward), I'll do it. Alan Woodward www.flax.co.uk On 23 May 2013, at 16:43, Mark Miller wrote: Yes, this did change - it's actually a protection for a previous change though. There was a time when you did a core reload by just making a new core with the same name and closing the old core - that is no longer really supported though - the proper way to do this is to use SolrCore#reload, and that has been the case for all of 4.x release if I remember right. I supported making this change to force people who might still be doing what is likely quite a buggy operation to switch to the correct code. Sorry about the inconvenience. - Mark On May 23, 2013, at 10:45 AM, André Widhani andre.widh...@digicol.de wrote: It seems to me that the behavior of the Core admin action CREATE has changed when going from Solr 4.1 to 4.3. With 4.1, I could re-configure an existing core (changing path/name to solrconfig.xml for example). In 4.3, I get an error message: SEVERE: org.apache.solr.common.SolrException: Error CREATEing SolrCore 'core-tex69b6iom1djrbzmlmg83-index2': Core with name 'core-tex69b6iom1djrbzmlmg83-index2' already exists. Is this change intended? André
Core admin action CREATE fails to persist some settings in solr.xml with Solr 4.3
When I create a core with Core admin handler using these request parameters: action=CREATE name=core-tex69bbum21ctk1kq6lmkir-index3 schema=/etc/opt/dcx/solr/conf/schema.xml instanceDir=/etc/opt/dcx/solr/ config=/etc/opt/dcx/solr/conf/solrconfig.xml dataDir=/var/opt/dcx/solr/core-tex69bbum21ctk1kq6lmkir-index3 in Solr 4.1, solr.xml would have the following entry: core schema=/etc/opt/dcx/solr/conf/schema.xml loadOnStartup=true instanceDir=/etc/opt/dcx/solr/ transient=false name=core-tex69bbum21ctk1kq6lmkir-index3 config=/etc/opt/dcx/solr/conf/solrconfig.xml dataDir=/var/opt/dcx/solr/core-tex69bbum21ctk1kq6lmkir-index3/ collection=core-tex69bbum21ctk1kq6lmkir-index3/ while in Solr 4.3 schema, config and dataDir will be missing: core loadOnStartup=true instanceDir=/etc/opt/dcx/solr/ transient=false name=core-tex69bbum21ctk1kq6lmkir-index3 collection=core-tex69bbum21ctk1kq6lmkir-index3/ The new core would use the settings specified during CREATE, but after a Solr restart they are lost (fall back to some defaults), as they are not persisted in solr.xml. Is this a bug or am I doing something wrong here? André
AW: Core admin action CREATE fails for existing core
Ok - yes, will do so tomorrow. Thanks, André Von: Mark Miller [markrmil...@gmail.com] Gesendet: Donnerstag, 23. Mai 2013 22:46 An: solr-user@lucene.apache.org Betreff: Re: Core admin action CREATE fails for existing core Your right - that does seem to be a new limitation. Could you create a JIRA issue for it? It would be fairly simple to add another reload method that also took the name of a new solrconfig/schema file. - Mark On May 23, 2013, at 4:11 PM, André Widhani andre.widh...@digicol.de wrote: Mark, Alan, thanks for explaining and updating the wiki. When reloading the core using action=CREATE with Solr 4.1 I could specify the path to schema and config. In fact I used this to reconfigure the core to use a specific one of two prepared config files depending on some external index state (instead of making changes to one and the same config file). action=RELOAD does not understand the corresponding request parameters schema and config (which is why I used CREATE, not RELOAD in the first place). So the functionality to switch to a different config file for an existing core is no longer there, I guess? Thanks, André Von: Alan Woodward [a...@flax.co.uk] Gesendet: Donnerstag, 23. Mai 2013 18:43 An: solr-user@lucene.apache.org Betreff: Re: Core admin action CREATE fails for existing core Thanks! Alan Woodward www.flax.co.uk On 23 May 2013, at 17:38, Steve Rowe wrote: Alan, I've added AlanWoodward to the Solr AdminGroup page. On May 23, 2013, at 12:29 PM, Alan Woodward a...@flax.co.uk wrote: I think the wiki needs to be updated to reflect this? http://wiki.apache.org/solr/CoreAdmin If somebody adds me as an editor (AlanWoodward), I'll do it. Alan Woodward www.flax.co.uk On 23 May 2013, at 16:43, Mark Miller wrote: Yes, this did change - it's actually a protection for a previous change though. There was a time when you did a core reload by just making a new core with the same name and closing the old core - that is no longer really supported though - the proper way to do this is to use SolrCore#reload, and that has been the case for all of 4.x release if I remember right. I supported making this change to force people who might still be doing what is likely quite a buggy operation to switch to the correct code. Sorry about the inconvenience. - Mark On May 23, 2013, at 10:45 AM, André Widhani andre.widh...@digicol.de wrote: It seems to me that the behavior of the Core admin action CREATE has changed when going from Solr 4.1 to 4.3. With 4.1, I could re-configure an existing core (changing path/name to solrconfig.xml for example). In 4.3, I get an error message: SEVERE: org.apache.solr.common.SolrException: Error CREATEing SolrCore 'core-tex69b6iom1djrbzmlmg83-index2': Core with name 'core-tex69b6iom1djrbzmlmg83-index2' already exists. Is this change intended? André
Status of EDisMax
Hi, what is the current status of the Extended DisMax Query Parser? The release notes for Solr 3.1 say it was experimental at that time (two years back). The current wiki page for EDisMax does not contain any such statement. We recently ran into the issue described in SOLR-2649 (using q.op=AND) which I think is a very fundamental defect making it unusable at least in our case. Thanks, André
AW: java.lang.OutOfMemoryError: Map failed
Hi Arkadi, this error usually indicates that virtual memory is not sufficient (should be unlimited). Please see http://comments.gmane.org/gmane.comp.jakarta.lucene.solr.user/69168 Regards, André Von: Arkadi Colson [ark...@smartbit.be] Gesendet: Dienstag, 2. April 2013 10:24 An: solr-user@lucene.apache.org Betreff: java.lang.OutOfMemoryError: Map failed Hi Recently solr crashed. I've found this in the error log. My commit settings are loking like this: autoCommit maxTime1/maxTime openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime2000/maxTime /autoSoftCommit The machine has 10GB of memory. Tomcat is running with -Xms2048m -Xmx6144m Versions Solr: 4.2 Tomcat: 7.0.33 Java: 1.7 Anybody any idea? Thx! Arkadi SEVERE: auto commit error...:org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1415) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1527) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:562) at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:228) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:195) at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:232) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.init(CompressingStoredFieldsReader.java:96) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113) at org.apache.lucene.index.SegmentCoreReaders.init(SegmentCoreReaders.java:147) at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:56) at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:269) at org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2961) at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:2952) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:368) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:270) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:249) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1353) ... 11 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846) ... 28 more SEVERE: auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2661) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:541) at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at
AW: AW: java.lang.OutOfMemoryError: Map failed
The output is from the root user. Are you running Solr as root? If not, please try again using the operating system user that runs Solr. André Von: Arkadi Colson [ark...@smartbit.be] Gesendet: Dienstag, 2. April 2013 11:26 An: solr-user@lucene.apache.org Cc: André Widhani Betreff: Re: AW: java.lang.OutOfMemoryError: Map failed Hmmm I checked it and it seems to be ok: root@solr01-dcg:~# ulimit -v unlimited Any other tips or do you need more debug info? BR On 04/02/2013 11:15 AM, André Widhani wrote: Hi Arkadi, this error usually indicates that virtual memory is not sufficient (should be unlimited). Please see http://comments.gmane.org/gmane.comp.jakarta.lucene.solr.user/69168 Regards, André Von: Arkadi Colson [ark...@smartbit.be] Gesendet: Dienstag, 2. April 2013 10:24 An: solr-user@lucene.apache.org Betreff: java.lang.OutOfMemoryError: Map failed Hi Recently solr crashed. I've found this in the error log. My commit settings are loking like this: autoCommit maxTime1/maxTime openSearcherfalse/openSearcher /autoCommit autoSoftCommit maxTime2000/maxTime /autoSoftCommit The machine has 10GB of memory. Tomcat is running with -Xms2048m -Xmx6144m Versions Solr: 4.2 Tomcat: 7.0.33 Java: 1.7 Anybody any idea? Thx! Arkadi SEVERE: auto commit error...:org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1415) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1527) at org.apache.solr.update.DirectUpdateHandler2.commit(DirectUpdateHandler2.java:562) at org.apache.solr.update.CommitTracker.run(CommitTracker.java:216) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722) Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:228) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:195) at org.apache.lucene.store.NRTCachingDirectory.openInput(NRTCachingDirectory.java:232) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.init(CompressingStoredFieldsReader.java:96) at org.apache.lucene.codecs.compressing.CompressingStoredFieldsFormat.fieldsReader(CompressingStoredFieldsFormat.java:113) at org.apache.lucene.index.SegmentCoreReaders.init(SegmentCoreReaders.java:147) at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:56) at org.apache.lucene.index.ReadersAndLiveDocs.getReader(ReadersAndLiveDocs.java:121) at org.apache.lucene.index.BufferedDeletesStream.applyDeletes(BufferedDeletesStream.java:269) at org.apache.lucene.index.IndexWriter.applyAllDeletes(IndexWriter.java:2961) at org.apache.lucene.index.IndexWriter.maybeApplyDeletes(IndexWriter.java:2952) at org.apache.lucene.index.IndexWriter.getReader(IndexWriter.java:368) at org.apache.lucene.index.StandardDirectoryReader.doOpenFromWriter(StandardDirectoryReader.java:270) at org.apache.lucene.index.StandardDirectoryReader.doOpenIfChanged(StandardDirectoryReader.java:255) at org.apache.lucene.index.DirectoryReader.openIfChanged(DirectoryReader.java:249) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1353) ... 11 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:846) ... 28 more SEVERE: auto commit error...:java.lang.IllegalStateException: this writer hit an OutOfMemoryError; cannot commit at org.apache.lucene.index.IndexWriter.prepareCommitInternal(IndexWriter.java:2661) at org.apache.lucene.index.IndexWriter.commitInternal(IndexWriter.java:2827) at org.apache.lucene.index.IndexWriter.commit(IndexWriter.java:2807
AW: Highlighting problems
Hi Dirk, please check http://wiki.apache.org/solr/HighlightingParameters#hl.requireFieldMatch - this may help you. Regards, André Von: Dirk Wintergruen [dwin...@mpiwg-berlin.mpg.de] Gesendet: Montag, 11. März 2013 13:56 An: solr-user@lucene.apache.org Betreff: Highlighting problems Hi all, I have problems with the higlighting mechanism: The query is: http://127.0.0.1:8983/solr/mpiwgweb/select?facet=truefacet.field=descriptionfacet.field=langfacet.field=main_contentstart=0q=meier+AND+%28description:member+OR+description:project%29 after that: In the field main_content which is the default search field. meier as well as as member and project is highlighted, although im searching for member and project only in the field description. The search results are ok, as far as I can see. my settings requestHandler name=/select class=solr.SearchHandler !-- default values for query parameters can be specified, these will be overridden by parameters in the request -- lst name=defaults str name=echoParamsexplicit/str int name=rows10/int str name=facet.limit300/str str name=hlon/str str name=hl.flmain_content/str str name=hl.encoderhtml/str str name=hl.simple.pre![CDATA[em class=webSearch_hl]]/str str name=hl.fragsize200/str str name=hl.snippets2/str str name=hl.usePhraseHighlightertrue/str /lst arr name=last-components strtvComponent/str /arr /requestHandler Cheers Dirk
AW: 170G index, 1.5 billion documents, out of memory on query
Could you check the virtual memory limit (ulimit -v, check this for the operating system user that runs Solr). It should report unlimited. André Von: zqzuk [ziqizh...@hotmail.co.uk] Gesendet: Dienstag, 26. Februar 2013 13:22 An: solr-user@lucene.apache.org Betreff: Re: 170G index, 1.5 billion documents, out of memory on query Hi, the full stack trace is below. - SEVERE: Unable to create core: collection1 org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.init(SolrCore.java:794) at org.apache.solr.core.SolrCore.init(SolrCore.java:607) at org.apache.solr.core.CoreContainer.createFromLocal(CoreContainer.java:1003) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1033) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1423) at org.apache.solr.core.SolrCore.getSearcher(SolrCore.java:1535) at org.apache.solr.core.SolrCore.init(SolrCore.java:769) ... 13 more Caused by: java.io.IOException: Map failed at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:748) at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:285) at org.apache.lucene.store.MMapDirectory$MMapIndexInput.init(MMapDirectory.java:257) at org.apache.lucene.store.MMapDirectory.openInput(MMapDirectory.java:225) at org.apache.lucene.codecs.lucene41.Lucene41PostingsReader.init(Lucene41PostingsReader.java:72) at org.apache.lucene.codecs.lucene41.Lucene41PostingsFormat.fieldsProducer(Lucene41PostingsFormat.java:430) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader.init(PerFieldPostingsFormat.java:194) at org.apache.lucene.codecs.perfield.PerFieldPostingsFormat.fieldsProducer(PerFieldPostingsFormat.java:233) at org.apache.lucene.index.SegmentCoreReaders.init(SegmentCoreReaders.java:107) at org.apache.lucene.index.SegmentReader.init(SegmentReader.java:57) at org.apache.lucene.index.StandardDirectoryReader$1.doBody(StandardDirectoryReader.java:62) at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:783) at org.apache.lucene.index.StandardDirectoryReader.open(StandardDirectoryReader.java:52) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:87) at org.apache.solr.core.StandardIndexReaderFactory.newReader(StandardIndexReaderFactory.java:34) at org.apache.solr.search.SolrIndexSearcher.init(SolrIndexSearcher.java:123) at org.apache.solr.core.SolrCore.openNewSearcher(SolrCore.java:1399) ... 15 more Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:745) ... 31 more 26-Feb-2013 12:20:46 org.apache.solr.common.SolrException log SEVERE: null:org.apache.solr.common.SolrException: Unable to create core: collection1 at org.apache.solr.core.CoreContainer.recordAndThrow(CoreContainer.java:1654) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:1039) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:629) at org.apache.solr.core.CoreContainer$3.call(CoreContainer.java:624) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) at java.util.concurrent.FutureTask.run(FutureTask.java:138) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Caused by: org.apache.solr.common.SolrException: Error opening new searcher at org.apache.solr.core.SolrCore.init(SolrCore.java:794) at
AW: Does solr 4.1 support field compression?
This is what it listed under the Highlights on the Apache page announcing the Solr 4.1 release: The default codec incorporates an efficient compressed stored fields implementation that compresses chunks of documents together with LZ4. (see http://blog.jpountz.net/post/33247161884/efficient-compressed-stored-fields-with-lucene) André Von: Rafał Kuć [r@solr.pl] Gesendet: Donnerstag, 24. Januar 2013 16:45 An: solr-user@lucene.apache.org Betreff: Re: Does solr 4.1 support field compression? Hello! It should be turned on by default, because the stored fields compression is the behavior of the default Lucene 4.1 codec. -- Regards, Rafał Kuć Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - ElasticSearch Hi everyone, I didn't see any mention of field compression in the release notes for Solr 4.1. Did the ability to automatically compress fields end up getting added to this release? Thanks!, Ken
AW: Does solr 4.1 support field compression?
These are the figures I got after indexing 4 and half million documents with both Solr 3.6.1 and 4.1.0 (and optimizing the index at the end). $ du -h --max-depth=1 67G ./solr410 80G ./solr361 Main contributor to the reduced space consumption is (as expected I guess) the .fdt file: $ ls -lh solr361/*/*/*.fdt 29G solr361/core-tex68bohyrh23qs192adaq-index361/index/_bab.fdt $ ls -lh solr410/*/*/*.fdt 18G solr410/core-tex68bohyz1teef3xsjdaw-index410/index/_23uy.fdt Depends of course on your individual ratio of stored versus indexed-only fields. André Von: Shawn Heisey [s...@elyograg.org] Gesendet: Donnerstag, 24. Januar 2013 16:58 An: solr-user@lucene.apache.org Betreff: Re: Does solr 4.1 support field compression? On 1/24/2013 8:42 AM, Ken Prows wrote: I didn't see any mention of field compression in the release notes for Solr 4.1. Did the ability to automatically compress fields end up getting added to this release? The concept of compressed fields (an option in schema.xml) that existed in the 1.x versions of Solr (based on Lucene 2.9) was removed in Lucene 3.0. Because Lucene and Solr development were combined, the Solr version after 1.4.1 is 3.1.0, there is no 1.5 or 2.x version of Solr. Solr/Lucene 4.1 compresses all stored field data by default. I don't think there's a way to turn it off at the moment, which is causing performance problems for a small subset of Solr users. When it comes out, Solr 4.2 will also have compressed term vectors. The release note contains this text: Stored fields are compressed. (See http://blog.jpountz.net/post/33247161884/efficient-compressed-stored-fields-with-lucene) It looks like the solr CHANGES.txt file fails to specifically mention LUCENE-4226 https://issues.apache.org/jira/browse/LUCENE-4226 which implemented compressed stored fields. Thanks, Shawn
AW: ConcurrentModificationException in Solr 3.6.1
This should be fixed in 3.6.2 which is available since Dec 25. From the release notes: Fixed ConcurrentModificationException during highlighting, if all fields were requested. André Von: mechravi25 [mechrav...@yahoo.co.in] Gesendet: Freitag, 18. Januar 2013 11:10 An: solr-user@lucene.apache.org Betreff: ConcurrentModificationException in Solr 3.6.1 Hi all, I am using Solr 3.6.1 version. I am giving a set of requests to solr simultaneously. When I check the log file, I noticed the below exception stack trace SEVERE: java.util.ConcurrentModificationException at java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:761) at java.util.LinkedList$ListItr.next(LinkedList.java:696) at org.apache.solr.highlight.SolrHighlighter.getHighlightFields(SolrHighlighter.java:106) at org.apache.solr.highlight.DefaultSolrHighlighter.doHighlighting(DefaultSolrHighlighter.java:369) at org.apache.solr.handler.component.HighlightComponent.process(HighlightComponent.java:131) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:186) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:129) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1376) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:365) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:260) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399) at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216) at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182) at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766) at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450) at org.mortbay.jetty.handler.ContextHandlerCollection.handle(ContextHandlerCollection.java:230) at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114) at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152) at org.mortbay.jetty.Server.handle(Server.java:326) at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542) at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945) at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756) at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218) at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404) at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228) at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582) When I searched through the solr issues, I got the following two url's, https://issues.apache.org/jira/browse/SOLR-2684 https://issues.apache.org/jira/browse/SOLR-3790 The stack trace given in the second url coincides with the one given above so I have applied the code change as given in the below link http://svn.apache.org/viewvc/lucene/dev/trunk/solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java?r1=1229401r2=1231606diff_format=h The first url's stack trace seems to be different. I have two questions here. 1.) Please tell me why this exception stack trace occurs 2.) IS there any other patch/solution available to overcome this exception. Please guide me. -- View this message in context: http://lucene.472066.n3.nabble.com/ConcurrentModificationException-in-Solr-3-6-1-tp4034493.html Sent from the Solr - User mailing list archive at Nabble.com.
AW: java.io.IOException: Map failed :: OutOfMemory
I just saw that you are running on SUSE 11 - unlike RHEL for example, it does
not have virtual memory set to unlimited by default.
Please check is the virtual memory limit (ulimit -v, check this for the
operating system user that runs Tomcat /Solr).
Since 3.1, Solr maps the index files to virtual memory. So if the size of your
index files are larger than the allowed virtual memory, it is likely that you
get these kind of exceptions.
Regards,
André
Von: uwe72 [uwe.clem...@exxcellent.de]
Gesendet: Dienstag, 13. November 2012 17:58
An: solr-user@lucene.apache.org
Betreff: Re: java.io.IOException: Map failed :: OutOfMemory
today the same exception:
INFO: [] webapp=/solr path=/update
params={waitSearcher=truecommit=truewt=javabinwaitFlush=trueversion=2}
status=0 QTime=1009
Nov 13, 2012 2:02:27 PM org.apache.solr.core.SolrDeletionPolicy onInit
INFO: SolrDeletionPolicy.onInit: commits:num=1
commit{dir=/net/smtcax0033/connect/Portal/solr-home/data/index,segFN=segments_3gm,version=1352803609067,generation=4486,filenames=[_21c.fdt,
_4mv.tis, _4mh.fnm, _1si.fdt, _4n0.fdx, _4mx.nrm, _1si.fdx, _2n0.nrm,
_2n0.prx, _4mv.tii, _3ii.fnm, _4mz.tvd, _4mv.nrm, _2ie.frq, _1l9.fnm,
_4my.fnm, _21c.fdx, _308.tvd, _4mz.tvf, _308.tvf, _sc.tis, _4mw.tii,
_4n1.fnm, _4mv.fdt, _1o2.nrm, _1si.nrm, _4mw.fdt, _it.tvf, _4mv.fdx,
_sc.tii, _4mw.tis, _4mw.fdx, _37y.tvx, _4mz.tvx, _4mh.nrm, _1si.prx,
_1o2.prx, _it.tvx, _3ii.tis, _3yn.nrm, _43w.tii, _37y.tvd, _3yn.prx,
_308.prx, _cv.nrm, _37y.tvf, _1b9.nrm, _3xp.frq, _43w.tis, _4mf.tvf,
_4mf.tvd, _1b9.fdt, _4ag.fdt, _1b9.fdx, _4mz.frq, _4ag.fdx, _418.tvx,
_4mf.tvx, _418.frq, _473.tis, _3ii.nrm, _4mx.fnm, _cv.frq, _3yn.tvd,
_418.tvd, _3yn.tvf, _418.tvf, _2ie.tvf, _2ie.tvd, _sc.frq, _1b9.frq,
_4ag.nrm, _37y.tii, _cv.prx, _4mx.tis, _4ag.prx, _2ie.tvx, _2n0.fdx,
_4mx.tii, _4mh.prx, _4my.prx, _4mz.nrm, _4lc.prx, _2ie.nrm, _3yn.tis,
_4n0.tii, _4mw.prx, _3yn.tvx, _it.fnm, _2n0.fdt, _4ag.frq, _21c.tvf,
_21c.tvd, _21c.nrm, _43w.prx, _308.fdt, _4my.frq, _1si.tvx, _4n3.prx,
_3yn.tii, _37y.tis, _4dj.fdt, _473.frq, _1l9.prx, _2ie.fnm, _4dj.fdx,
_308.fdx, _473.tvx, _cv.fdx, _4mz.tii, _473.tii, _cv.fdt, _3xp.tii,
_4lc.nrm, _2em.fnm, _it.tis, _418.fdx, _4n3.fdx, _3xp.tis, _418.fdt,
_1ih.fdx, _it.tii, _4n3.fdt, _4ix.tis, _1ih.fdt, _4lc.fdt, _4ix.tii,
_4mz.tis, _1b9.prx, _4n0.tis, _4lc.fdx, _473.tvd, _1ih.nrm, _2n0.frq,
_473.tvf, _4mz.fdx, _sc.fdx, _it.nrm, _4mz.fdt, _4my.tvx, _4mx.tvf,
_3ii.tii, _1b9.tvf, _4mx.tvd, _1b9.tvd, _418.prx, _3ii.tvx, _3xp.fnm,
_4mv.tvx, _sc.fdt, _sc.prx, segments_3gm, _418.fnm, _2n0.tii, _4mf.tis,
_sc.nrm, _4mf.tii, _4dj.nrm, _3ii.tvd, _1ih.frq, _3ii.tvf, _4n1.prx,
_1o2.tii, _37y.frq, _2em.prx, _4n3.frq, _4ix.fdt, _473.fdt, _21c.prx,
_1o2.tvx, _3xp.nrm, _473.fdx, _sc.fnm, _2n0.tis, _43w.fdt, _4mf.fnm,
_4ix.fdx, _43w.fdx, _4dj.tis, _473.nrm, _4my.tvf, _4mx.tvx, _4mv.tvd,
_1o2.tvd, _4my.tvd, _1o2.tvf, _4dj.tii, _4mv.frq, _1si.tvf, _4mv.tvf,
_1si.tvd, _473.fnm, _4ix.frq, _cv.tvx, _4dj.tvd, _21c.tii, _473.prx,
_4n1.tvx, _1ih.tvx, _1si.tis, _cv.tvf, _4ag.fnm, _1b9.tvx, _1ih.tvf,
_1l9.fdx, _4lc.tii, _1ih.tvd, _4n1.fdx, _4lc.tis, _1l9.fdt, _21c.tis,
_4dj.tvf, _1si.tii, _4n1.fdt, _4n0.fnm, _cv.tvd, _it.frq, _4mv.prx,
_4mh.tis, _3xp.tvf, _4n0.tvf, _3xp.tvd, _4n0.tvd, _4mx.fdx, _4my.nrm,
_4dj.frq, _4mx.fdt, _43w.frq, _1o2.frq, _4n0.tvx, _it.tvd, _1si.fnm,
_4n3.tvx, _3xp.tvx, _4mz.prx, _4my.tis, _21c.tvx, _37y.prx, _1ih.tii,
_4ix.prx, _4mh.fdt, _2n0.fnm, _4n3.tvf, _21c.fnm, _4mh.fdx, _2em.tvx,
_1b9.tii, _308.frq, _4mx.prx, _37y.fdx, _3yn.fnm, _4n3.tvd, _4mh.tii,
_4ag.tis, _4my.tii, _1b9.tis, _2ie.prx, _1ih.prx, _4ag.tii, _4n1.tvd,
_1ih.fnm, _3ii.prx, _4ix.nrm, _4n1.tvf, _4n1.nrm, _2em.tvd, _4mv.fnm,
_4mw.fnm, _37y.nrm, _it.fdx, _4mf.frq, _4n0.nrm, _3ii.frq, _it.fdt,
_1o2.tis, _37y.fdt, _4dj.tvx, _4n3.fnm, _4lc.fnm, _4my.fdt, _4lc.frq,
_2em.tvf, _4my.fdx, _37y.fnm, _4n0.prx, _1l9.tvd, _418.nrm, _2em.tis,
_4mw.nrm, _3xp.prx, _2ie.tis, _3xp.fdx, _1l9.frq, _1l9.tvf, _4mf.nrm,
_2em.tii, _4ix.fnm, _3xp.fdt, _4mh.tvd, _4mh.tvf, _2ie.tii, _1o2.fdt,
_4mh.tvx, _4mf.fdt, _4n0.frq, _308.tii, _4mw.tvx, _4ag.tvx, _308.tis,
_4n1.frq, _4mf.fdx, _sc.tvd, _sc.tvf, _3yn.fdt, _4mw.tvf, _4ag.tvf,
_4mw.tvd, _3yn.fdx, _1o2.fdx, _43w.fnm, _1o2.fnm, _4ag.tvd, _1si.frq,
_sc.tvx, _cv.tis, _4dj.fnm, _4mh.frq, _1ih.tis, _4lc.tvf, _2em.fdt,
_4lc.tvd, _2em.frq, _4ix.tvd, _21c.frq, _3ii.fdt, _2em.fdx, _4ix.tvf,
_4n1.tis, _cv.tii, _4mz.fnm, _308.tvx, _4dj.prx, _4lc.tvx, _43w.tvf,
_308.fnm, _3yn.frq, _43w.tvd, _43w.nrm, _it.prx, _4mx.frq, _cv.fnm,
_2n0.tvx, _1l9.tii, _4n0.fdt, _418.tis, _418.tii, _1l9.tis, _4n3.nrm,
_1l9.nrm, _4mw.frq, _4mf.prx, _4ix.tvx, _1l9.tvx, _2ie.fdx, _1b9.fnm,
_43w.tvx, _2n0.tvd, _4n3.tii, _2n0.tvf, _3ii.fdx, _4n1.tii, _2em.nrm,
_4n3.tis, _308.nrm, _2ie.fdt]
Nov 13, 2012 2:02:27 PM org.apache.solr.core.SolrDeletionPolicy
updateCommits
INFO: newest commit = 1352803609067
Nov 13, 2012 2:02:27 PM org.apache.solr.update.processor.LogUpdateProcessor
finish
AW: Stemmer German2
Do you use the LowerCaseFilterFactory filter in your analysis chain? You will probably want to add it and if you aready have, make sure it is _before_ the stemming filter so you get consistent results regardless of lower- or uppercase spelling. You can protect words from being subject to stemming by adding a KeyWordMarkerFilterFactory filter before the stemmer, protected words are in a text file. This should be placed after the lower case filter so you can use lower csase terms in the file. Some stemmer classes like SnowballPorterFilterFactory also allow you to pass a protected attribute (again pointing to a file). All of this is on the Solr wiki (AnalyzersTokenizersTokenFilters, LanguageAnalysis) if you need more details. Regards, André Von: Andreas Niekler [aniek...@informatik.uni-leipzig.de] Gesendet: Mittwoch, 7. November 2012 10:02 An: solr-user@lucene.apache.org Betreff: Stemmer German2 Dear List, i have an unwanted behavior with the German2 Stemmer. For example the river Elbe: If i input elbe - the word gets reduced to elb If i input Elbe - everything is ok and elbe is stored to the index. If i now query for elbe or Elbe i get of course differnt Results allowing the users not either use Elbe or elbe to get the same results. Can i insert an exception list to the Stemmer. Otherwise we will have a very hard time explaining some users why this is happaning for some words. Thank you Andreas -- Andreas Niekler, Dipl. Ing. (FH) NLP Group | Department of Computer Science University of Leipzig Johannisgasse 26 | 04103 Leipzig mail: aniek...@informatik.uni-leipzig.deg.de
AW: Stemmer German2
No - you need to restart Solr to pick up the changes to the schema and you need to re-index the existing documents. Regards, André Von: Andreas Niekler [aniek...@informatik.uni-leipzig.de] Gesendet: Mittwoch, 7. November 2012 16:40 An: solr-user@lucene.apache.org Betreff: Re: Stemmer German2 Hello, thanks for the advice. If i now change the schema that my lowercase factory is before the stemmer. is the index updating itself after the change? How could i achieve this. I stored all values within the index. Thanks andreas Am 07.11.2012 10:47, schrieb André Widhani: Do you use the LowerCaseFilterFactory filter in your analysis chain? You will probably want to add it and if you aready have, make sure it is _before_ the stemming filter so you get consistent results regardless of lower- or uppercase spelling. You can protect words from being subject to stemming by adding a KeyWordMarkerFilterFactory filter before the stemmer, protected words are in a text file. This should be placed after the lower case filter so you can use lower csase terms in the file. Some stemmer classes like SnowballPorterFilterFactory also allow you to pass a protected attribute (again pointing to a file). All of this is on the Solr wiki (AnalyzersTokenizersTokenFilters, LanguageAnalysis) if you need more details. Regards, André Von: Andreas Niekler [aniek...@informatik.uni-leipzig.de] Gesendet: Mittwoch, 7. November 2012 10:02 An: solr-user@lucene.apache.org Betreff: Stemmer German2 Dear List, i have an unwanted behavior with the German2 Stemmer. For example the river Elbe: If i input elbe - the word gets reduced to elb If i input Elbe - everything is ok and elbe is stored to the index. If i now query for elbe or Elbe i get of course differnt Results allowing the users not either use Elbe or elbe to get the same results. Can i insert an exception list to the Stemmer. Otherwise we will have a very hard time explaining some users why this is happaning for some words. Thank you Andreas -- Andreas Niekler, Dipl. Ing. (FH) NLP Group | Department of Computer Science University of Leipzig Johannisgasse 26 | 04103 Leipzig mail: aniek...@informatik.uni-leipzig.deg.de -- Andreas Niekler, Dipl. Ing. (FH) NLP Group | Department of Computer Science University of Leipzig Johannisgasse 26 | 04103 Leipzig mail: aniek...@informatik.uni-leipzig.deg.de
AW: solr multicore problem on SLES 11
The first thing I would check is the virtual memory limit (ulimit -v, check this for the operating system user that runs Tomcat /Solr). It should be set to unlimited, but this is as far as i remember not the default settings on SLES 11. Since 3.1, Solr maps the index files to virtual memory. So if the size of your index files are larger than the allowed virtual memory, it may fail. Regards, André Von: Jochen Lienhard [lienh...@ub.uni-freiburg.de] Gesendet: Montag, 17. September 2012 09:17 An: solr-user@lucene.apache.org Betreff: solr multicore problem on SLES 11 Hello, I have a problem with solr and multicores on SLES 11 SP 2. I have 3 cores, each with more than 20 segments. When I try to start the tomcat6, it can not start the CoreContainer. Caused by: java.lang.OutOfMemoryError: Map failed at sun.nio.ch.FileChannelImpl.map0(Native Method) I read a lot about this problem, but I do not find the solution. The strange problem is now: It works fine under openSuSE 12.x, tomcat6, openjdk. But the virtual maschine with SLES 11 SP 2, tomcat6, openjdk it crashes. Both tomcat/java configurations are the same. Has anyboday a idea, how to solve this problem? I have another SLES maschine with 5 core, but each has only 1 segment (very small index), and this maschine runs fine. Greetings Jochen -- Dr. rer. nat. Jochen Lienhard Dezernat EDV Albert-Ludwigs-Universität Freiburg Universitätsbibliothek Rempartstr. 10-16 | Postfach 1629 79098 Freiburg | 79016 Freiburg Telefon: +49 761 203-3908 E-Mail: lienh...@ub.uni-freiburg.de Internet: www.ub.uni-freiburg.de
AW: IndexOutOfBoundsException
I think we had a similar exception recently when attempting to sort on a multi-valued field ... could that be possible in your case? André -Ursprüngliche Nachricht- Von: Dominik Lange [mailto:dominikla...@searchmetrics.com] Gesendet: Mittwoch, 9. Februar 2011 10:55 An: solr-user@lucene.apache.org Betreff: IndexOutOfBoundsException hi, we have a problem with our solr test instance. This instance is running with 90 cores with about 2 GB of Index-Data per core. This worked fine for a few weeks. Now we get an exception querying data from one core : java.lang.IndexOutOfBoundsException: Index: 104, Size: 11 at java.util.ArrayList.rangeCheck(ArrayList.java:571) at java.util.ArrayList.get(ArrayList.java:349) at org.apache.lucene.index.FieldInfos.fieldInfo(FieldInfos.java:288) at org.apache.lucene.index.FieldInfos.fieldName(FieldInfos.java:277) at org.apache.lucene.index.TermBuffer.read(TermBuffer.java:86) at org.apache.lucene.index.SegmentTermEnum.next(SegmentTermEnum.java:129) at org.apache.lucene.index.SegmentTermEnum.scanTo(SegmentTermEnum.java:160) at org.apache.lucene.index.TermInfosReader.get(TermInfosReader.java:211) at org.apache.lucene.index.TermInfosReader.terms(TermInfosReader.java:277) at org.apache.lucene.index.SegmentReader.terms(SegmentReader.java:961) at org.apache.lucene.index.DirectoryReader$MultiTermEnum.lt;initgt;(DirectoryReader.java:989) at org.apache.lucene.index.DirectoryReader.terms(DirectoryReader.java:626) at org.apache.solr.search.SolrIndexReader.terms(SolrIndexReader.java:302) at org.apache.lucene.search.PrefixTermEnum.lt;initgt;(PrefixTermEnum.java:41) at org.apache.lucene.search.PrefixQuery.getEnum(PrefixQuery.java:45) at org.apache.lucene.search.MultiTermQuery$ConstantScoreAutoRewrite.rewrite(MultiTermQuery.java:227) at org.apache.lucene.search.MultiTermQuery.rewrite(MultiTermQuery.java:382) at org.apache.lucene.search.BooleanQuery.rewrite(BooleanQuery.java:438) at org.apache.lucene.search.IndexSearcher.rewrite(IndexSearcher.java:311) at org.apache.lucene.search.Query.weight(Query.java:98) at org.apache.lucene.search.Searcher.createWeight(Searcher.java:230) at org.apache.lucene.search.Searcher.search(Searcher.java:171) at org.apache.solr.search.SolrIndexSearcher.getDocListNC(SolrIndexSearcher.java:988) at org.apache.solr.search.SolrIndexSearcher.getDocListC(SolrIndexSearcher.java:884) at org.apache.solr.search.SolrIndexSearcher.search(SolrIndexSearcher.java:341) at org.apache.solr.handler.component.QueryComponent.process(QueryComponent.java:182) at org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:195) at org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) at org.apache.solr.core.SolrCore.execute(SolrCore.java:1316) at org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:338) at org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:241) at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089) at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365) ... All other cores are working fine with the same schema. This problem only occurs when querying for specific data like q=fieldA:valueA%20AND%20fieldB:valueB By using the following query data is returned q=*:* Has anybody any suggestions on what is causing this problem? Are 90 cores too much for a single solr instance? Thanks in advance, Dominik
