[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549651#comment-13549651 ] Varun Thacker commented on LUCENE-3354: --- Hi, I have a doubt on FieldCache supporting MultiValued fields in general. So FieldCache on a multiValued field works by consuming it from FieldCache.DocTermOrds but, * I was trying out FunctionQuery in Solr and still got a cannot FieldCache on multiValued field error. This is because any impl. of FieldCacheSource for example StrFieldSource#getValues() returns DocTermsIndexDocValues where FieldCache.DocTermsIndex instance loads up. Is this supposed to be consumed like this? * Secondly slightly off topic but I went through the lucene4547 branch where there was a discussion on how to consume DocValues. I'm still trying to figure a lot of stuff around DocValues, FieldCache etc. but do we need to discuss all these issues and it's impact on Solr and ES as a whole? Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Core Issue Type: Improvement Reporter: Bill Bell Fix For: 4.0-ALPHA Attachments: LUCENE-3354.patch, LUCENE-3354.patch, LUCENE-3354_testspeed.patch I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086197#comment-13086197 ] Martijn van Groningen commented on LUCENE-3354: --- I committed a fix. Test pass now on my local box with -Dtests.multiplier=3. If build is successful on Jenkins we can close this issue. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell Fix For: 4.0 Attachments: LUCENE-3354.patch, LUCENE-3354.patch I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[
https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086251#comment-13086251
]
Robert Muir commented on LUCENE-3354:
-
Thanks Martijn: any idea how we can speed this test up? for our 'ant test' runs
with multiplier=3, this takes a significant amount of time (over 15 minutes!),
more than all the other tests combined.
Before the commit my builds were taking about 9 minutes, log here:
http://sierranevada.servebeer.com/
{noformat}
[junit] Testsuite: org.apache.lucene.search.TestFieldCache
[junit] Tests run: 3, Failures: 0, Errors: 0, Time elapsed: 1,062.362 sec
{noformat}
Extend FieldCache architecture to multiple Values
-
Key: LUCENE-3354
URL: https://issues.apache.org/jira/browse/LUCENE-3354
Project: Lucene - Java
Issue Type: Improvement
Reporter: Bill Bell
Fix For: 4.0
Attachments: LUCENE-3354.patch, LUCENE-3354.patch
I would consider this a bug. It appears lots of people are working around
this limitation,
why don't we just change the underlying data structures to natively support
multiValued fields in the FieldCache architecture?
Then functions() will work properly, and we can do things like easily
geodist() on a multiValued field.
Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086266#comment-13086266 ] Martijn van Groningen commented on LUCENE-3354: --- I don't think there is any reason for generating long unicode strings. Only the cache behavior needs to be tested. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell Fix For: 4.0 Attachments: LUCENE-3354.patch, LUCENE-3354.patch, LUCENE-3354_testspeed.patch I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086271#comment-13086271 ] Robert Muir commented on LUCENE-3354: - OK, thanks. I bet this was probably slowing things down for simpletext or something stupid :) Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell Fix For: 4.0 Attachments: LUCENE-3354.patch, LUCENE-3354.patch, LUCENE-3354_testspeed.patch I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[
https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13086088#comment-13086088
]
Robert Muir commented on LUCENE-3354:
-
The new multivalued test in TestFieldCache exhibits some problems if NUM_ORD
2.
This is the case if you e.g. use -Dtests.multiplier=3 (like hudson does)... I
temporarily disabled it and put in a loud system.out.println
{noformat}
-NUM_ORDS = atLeast(2);
+System.out.println(WARNING: NUM_ORDS is wired to 2, test fails
otherwise!);
+NUM_ORDS = 2; //atLeast(2);
{noformat}
Extend FieldCache architecture to multiple Values
-
Key: LUCENE-3354
URL: https://issues.apache.org/jira/browse/LUCENE-3354
Project: Lucene - Java
Issue Type: Improvement
Reporter: Bill Bell
Fix For: 4.0
Attachments: LUCENE-3354.patch, LUCENE-3354.patch
I would consider this a bug. It appears lots of people are working around
this limitation,
why don't we just change the underlying data structures to natively support
multiValued fields in the FieldCache architecture?
Then functions() will work properly, and we can do things like easily
geodist() on a multiValued field.
Thoughts?
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13085032#comment-13085032 ] Michael McCandless commented on LUCENE-3354: Patch looks good Martijn! Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell Attachments: LUCENE-3354.patch, LUCENE-3354.patch I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13079203#comment-13079203 ] Bill Bell commented on LUCENE-3354: --- Lots of activity... Can someone lead this? Bill Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073518#comment-13073518 ] Martijn van Groningen commented on LUCENE-3354: --- +1. If DocTermOrds is available in FieldCache, then Grouping (Term based impl) can also use DocTermOrds. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073522#comment-13073522 ] Ryan McKinley commented on LUCENE-3354: --- What are thoughts on using DocValues rather then FieldCache? If we do choose to extend the FieldCache architecture, it would be so much cleaner if it were a simple MapK,V directly on the Reader rather then a static thing holding a WeakHashMapReader,Cache Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073530#comment-13073530 ] Robert Muir commented on LUCENE-3354: - +1, die insanity, die. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073533#comment-13073533 ] Michael McCandless commented on LUCENE-3354: +1 to moving FC to atomic readers only, and let SlowMultiReaderWrapper absorb the insanity. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073532#comment-13073532 ] Martijn van Groningen commented on LUCENE-3354: --- bq. What are thoughts on using DocValues rather then FieldCache? Maybe both should be available. Not all fields have indexed docvalues. bq. We should start with this in 4.0! For backwards compatibility we could still have the FieldCache class, but just delegating. Changing the architecture seems like a big task to me. Maybe that should be done in a different issue. This issue will then depend on it. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073536#comment-13073536 ] Yonik Seeley commented on LUCENE-3354: -- bq. (icluding the broken Solr parts still using TopLevel FieldCache entries). Some top-level field cache uses are very much by design in Solr. If that ability is removed from Lucene, I guess we could always move some of the old FieldCache logic to Solr though. Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073538#comment-13073538 ] Uwe Schindler commented on LUCENE-3354: --- bq. If that ability is removed from Lucene, I guess we could always move some of the old FieldCache logic to Solr though. Solr can always use SlowMultiReaderWrapper (see above) Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (LUCENE-3354) Extend FieldCache architecture to multiple Values
[ https://issues.apache.org/jira/browse/LUCENE-3354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13073828#comment-13073828 ] Hoss Man commented on LUCENE-3354: -- bq. This would also remove the insanity issues. FWIW: the WeakHashMap isn't the sole source of insanity - that can also come about from inconsistent usage for a single field (ie: asking for string and int caches for the same field) Extend FieldCache architecture to multiple Values - Key: LUCENE-3354 URL: https://issues.apache.org/jira/browse/LUCENE-3354 Project: Lucene - Java Issue Type: Improvement Reporter: Bill Bell I would consider this a bug. It appears lots of people are working around this limitation, why don't we just change the underlying data structures to natively support multiValued fields in the FieldCache architecture? Then functions() will work properly, and we can do things like easily geodist() on a multiValued field. Thoughts? -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
