Use absolute term position for scoring
Hi all, I want to take into account the absolute position of the term for the score calculation. I found many threads that deal with this issue, and the answer is often: "use SpanFirstQuery". The problem with this approach is that it is too "boolean" for me (the document matches the spanfirstquery or it does not, so the boost is either 0 or 1) and you cannot do exciting calculation with log of the position or other funny functions. If the position was not there, we could put it in the payload of the term and then reuse it with a customscorer. But we already have the position in the term, so why can't we use the same approach? Is it a problem of interface? Or maybe it would have a huge impact on performance? Thank you for your advice, Regards, Aurelien - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Indexing a binary field
What's the best way to index binary data in Lucene? I'm adding a Lucene index to a key value store, and I want to be able to delete documents based on a binary key. As far as I can tell the Lucene API does not support indexing binary data, and I was looking into Base64 encoding the key. Is there a better way? -Dan
Re: Indexing a binary field
StringField now also takes a BytesRef value to index, so you can index a single binary token that way. Does that work? Mike McCandless http://blog.mikemccandless.com On Mon, Aug 31, 2015 at 12:19 PM, Dan Smith wrote: > What's the best way to index binary data in Lucene? I'm adding a Lucene > index to a key value store, and I want to be able to delete documents based > on a binary key. > > As far as I can tell the Lucene API does not support indexing binary data, > and I was looking into Base64 encoding the key. Is there a better way? > > -Dan - To unsubscribe, e-mail: java-user-unsubscr...@lucene.apache.org For additional commands, e-mail: java-user-h...@lucene.apache.org
Re: Indexing a binary field
Aha! My version of Lucene was out of date. That should work perfectly. Thanks, -Dan Original message From: Michael McCandless Date:08/31/2015 12:57 PM (GMT-08:00) To: Lucene Users , dsm...@pivotal.io Cc: Subject: Re: Indexing a binary field StringField now also takes a BytesRef value to index, so you can index a single binary token that way. Does that work? Mike McCandless http://blog.mikemccandless.com On Mon, Aug 31, 2015 at 12:19 PM, Dan Smith wrote: > What's the best way to index binary data in Lucene? I'm adding a Lucene > index to a key value store, and I want to be able to delete documents based > on a binary key. > > As far as I can tell the Lucene API does not support indexing binary data, > and I was looking into Base64 encoding the key. Is there a better way? > > -Dan
IndexWriter is not closing the FDs (deleted files)
Hey! :) It seems IndexWriter is not closing the descriptors of the removed files, see the log below. Thanks, Napoli [root@server01 log]# ls -l /proc/59491/fd | grep index l-wx--. 1 wildfly wildfly 64 Ago 31 11:26 429 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/write.lock lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 529 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_4.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 530 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_3.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 531 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_2.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 532 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_1.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 533 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_0.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 535 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_a.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 536 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_9.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 537 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_8.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 538 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_7.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 539 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_6.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 540 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.doc (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 541 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.pos (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 542 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.tim (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 543 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5.nvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 544 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5.fdt (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 545 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene410_0.dvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 619 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_o.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 676 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_k.cfs (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 677 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.doc (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 678 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.pos (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 679 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.tim (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 680 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j.nvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 681 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j.fdt (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 682 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene410_0.dvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 685 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.doc (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 686 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.pos (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 687 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.tim (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 688 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n.nvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 689 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n.fdt (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 690 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene410_0.dvd (deleted) lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 730 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_15.cfs lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 737 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.doc lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 738 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.pos lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 739 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.tim lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 740 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r.nvd lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 741 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r.fdt lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 742 -> /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene410_0.dvd lr-x--. 1 wildfly wildfly 64 Ago 31 20:25
Re: IndexWriter is not closing the FDs (deleted files)
Are you sure you are not holding open readers somewhere? On Mon, Aug 31, 2015 at 7:46 PM, Marcio Napoli wrote: > Hey! :) > > It seems IndexWriter is not closing the descriptors of the removed files, > see the log below. > > Thanks, > Napoli > > [root@server01 log]# ls -l /proc/59491/fd | grep index > l-wx--. 1 wildfly wildfly 64 Ago 31 11:26 429 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/write.lock > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 529 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_4.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 530 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_3.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 531 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_2.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 532 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_1.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 533 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_0.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 535 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_a.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 536 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_9.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 537 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_8.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 538 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_7.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 539 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_6.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 540 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.doc > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 541 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.pos > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 542 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene41_0.tim > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 543 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5.nvd (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 544 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5.fdt (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 11:26 545 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_5_Lucene410_0.dvd > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 619 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_o.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 676 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_k.cfs (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 677 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.doc > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 678 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.pos > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 679 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene41_0.tim > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 680 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j.nvd (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 681 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j.fdt (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 682 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_j_Lucene410_0.dvd > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 685 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.doc > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 686 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.pos > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 687 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene41_0.tim > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 688 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n.nvd (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 689 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n.fdt (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 690 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_n_Lucene410_0.dvd > (deleted) > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 730 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_15.cfs > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 737 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.doc > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 738 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.pos > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 739 -> > /usr/local/wildfly-2.0/standalone/data/index/cidadao/_r_Lucene41_0.tim > lr-x--. 1 wildfly wildfly 64 Ago 31 20:25 740 -> > /usr/local/wildfly-2.0/standalone/data/index/c
