[jira] [Updated] (LUCENE-3298) FST has hard limit max size of 2.1 GB
[ https://issues.apache.org/jira/browse/LUCENE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-3298: --- Attachment: LUCENE-3298.patch New patch, beefing up the test (it passes: takes 10 GB heap and ~ 2 hours on my machine, making various 3 GB / 3 B node FST and test them), removing nocommits. I think it's ready. FST has hard limit max size of 2.1 GB - Key: LUCENE-3298 URL: https://issues.apache.org/jira/browse/LUCENE-3298 Project: Lucene - Core Issue Type: Improvement Components: core/FSTs Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor Attachments: LUCENE-3298.patch, LUCENE-3298.patch, LUCENE-3298.patch, LUCENE-3298.patch The FST uses a single contiguous byte[] under the hood, which in java is indexed by int so we cannot grow this over Integer.MAX_VALUE. It also internally encodes references to this array as vInt. We could switch this to a paged byte[] and make the far larger. But I think this is low priority... I'm not going to work on it any time soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3298) FST has hard limit max size of 2.1 GB
[ https://issues.apache.org/jira/browse/LUCENE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-3298: --- Attachment: LUCENE-3298.patch Initial patch with int - long in lots of places ... the Test2BFST is still running ... FST has hard limit max size of 2.1 GB - Key: LUCENE-3298 URL: https://issues.apache.org/jira/browse/LUCENE-3298 Project: Lucene - Core Issue Type: Improvement Components: core/FSTs Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor Attachments: LUCENE-3298.patch, LUCENE-3298.patch, LUCENE-3298.patch The FST uses a single contiguous byte[] under the hood, which in java is indexed by int so we cannot grow this over Integer.MAX_VALUE. It also internally encodes references to this array as vInt. We could switch this to a paged byte[] and make the far larger. But I think this is low priority... I'm not going to work on it any time soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3298) FST has hard limit max size of 2.1 GB
[ https://issues.apache.org/jira/browse/LUCENE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael McCandless updated LUCENE-3298: --- Attachment: LUCENE-3298.patch Initial test to confirm FSTs can grow beyond 2GB (it fails today!). FST has hard limit max size of 2.1 GB - Key: LUCENE-3298 URL: https://issues.apache.org/jira/browse/LUCENE-3298 Project: Lucene - Core Issue Type: Improvement Components: core/FSTs Reporter: Michael McCandless Assignee: Michael McCandless Priority: Minor Attachments: LUCENE-3298.patch, LUCENE-3298.patch The FST uses a single contiguous byte[] under the hood, which in java is indexed by int so we cannot grow this over Integer.MAX_VALUE. It also internally encodes references to this array as vInt. We could switch this to a paged byte[] and make the far larger. But I think this is low priority... I'm not going to work on it any time soon. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Updated] (LUCENE-3298) FST has hard limit max size of 2.1 GB
[ https://issues.apache.org/jira/browse/LUCENE-3298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Dyer updated LUCENE-3298: --- Attachment: LUCENE-3298.patch Here's a patch for this. Not fully optimized, but possibly a start. FST has hard limit max size of 2.1 GB - Key: LUCENE-3298 URL: https://issues.apache.org/jira/browse/LUCENE-3298 Project: Lucene - Java Issue Type: Improvement Components: core/FSTs Reporter: Michael McCandless Priority: Minor Attachments: LUCENE-3298.patch The FST uses a single contiguous byte[] under the hood, which in java is indexed by int so we cannot grow this over Integer.MAX_VALUE. It also internally encodes references to this array as vInt. We could switch this to a paged byte[] and make the far larger. But I think this is low priority... I'm not going to work on it any time soon. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
