[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-19 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772167#comment-16772167 ] ASF subversion and git services commented on LUCENE-8635: - Commit

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-19 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772156#comment-16772156 ] ASF subversion and git services commented on LUCENE-8635: - Commit

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-19 Thread ASF subversion and git services (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772162#comment-16772162 ] ASF subversion and git services commented on LUCENE-8635: - Commit

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-19 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16772153#comment-16772153 ] Michael McCandless commented on LUCENE-8635: I ran luceneutil on {{wikimediumall}} with

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-10 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764370#comment-16764370 ] Ankit Jain commented on LUCENE-8635: I added print statements while running the benchmarks, and the

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-08 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16764051#comment-16764051 ] Ankit Jain commented on LUCENE-8635: {quote}Ankit Jain that's strange yeah – this patch was supposed

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-07 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16762692#comment-16762692 ] Mike Sokolov commented on LUCENE-8635: -- [~akjain] that's strange yeah -- this patch was supposed to

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-04 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16760118#comment-16760118 ] Ankit Jain commented on LUCENE-8635: I have created [pull

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-01 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758457#comment-16758457 ] Mike Sokolov commented on LUCENE-8635: -- Yes, [~akjain] that approach sounds good to me; we should

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-02-01 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758431#comment-16758431 ] Michael McCandless commented on LUCENE-8635: {quote}Better would be an attribute of 

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-30 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756963#comment-16756963 ] Ankit Jain commented on LUCENE-8635: Given that reversing the index during write to make it forward

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-30 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16756098#comment-16756098 ] Mike Sokolov commented on LUCENE-8635: -- I agree that would be a good start. Perhaps as a separate

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-30 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755945#comment-16755945 ] Adrien Grand commented on LUCENE-8635: -- bq. Does that exlude autogenerated id fields that are uuid,

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-29 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755391#comment-16755391 ] Mike Sokolov commented on LUCENE-8635: -- I posted my latest patch including off-heap change + FST

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-29 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755389#comment-16755389 ] Ankit Jain commented on LUCENE-8635: {quote}Given that the performance hit is mostly on PK lookups,

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-29 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755374#comment-16755374 ] Michael McCandless commented on LUCENE-8635: Oooh I like that proposal [~jpountz]! > Lazy

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-29 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755352#comment-16755352 ] Adrien Grand commented on LUCENE-8635: -- Given that the performance hit is mostly on PK lookups,

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-29 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16755344#comment-16755344 ] Michael McCandless commented on LUCENE-8635: OK net/net it looks like there is a small

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-27 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753609#comment-16753609 ] Ankit Jain commented on LUCENE-8635: Results for bigger data sets: {code| title=wikimedium10m, java

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-27 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753595#comment-16753595 ] Ankit Jain commented on LUCENE-8635: I also independently tried performance run after removing the

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-27 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16753468#comment-16753468 ] Mike Sokolov commented on LUCENE-8635: -- I tried that [~akjain] and strangely got a big drop in

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-23 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750316#comment-16750316 ] Ankit Jain commented on LUCENE-8635: {quote}Ankit Jain unfortunately RandomAccessInput doesn't offer

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-23 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750135#comment-16750135 ] Mike Sokolov commented on LUCENE-8635: -- {quote}we can simply change readBytes to below: {quote}

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-22 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16749180#comment-16749180 ] Ankit Jain commented on LUCENE-8635: bq. {quote}Technically we could make things work for existing

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-22 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748996#comment-16748996 ] Mike Sokolov commented on LUCENE-8635: -- I uploaded a patch that combines these three things:

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-21 Thread Murali Krishna P (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748461#comment-16748461 ] Murali Krishna P commented on LUCENE-8635: -- Wondering whether avoiding 'array reversal' in the

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-21 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748082#comment-16748082 ] Mike Sokolov commented on LUCENE-8635: -- I opened LUCENE-8653 to explore reversing FSTs; if we can

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-18 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746726#comment-16746726 ] Mike Sokolov commented on LUCENE-8635: -- {quote}you can still end up with a cold FS cache eg. when

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-18 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746730#comment-16746730 ] Mike Sokolov commented on LUCENE-8635: -- For the cold host case, we already have to take measures to

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-18 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16746576#comment-16746576 ] Adrien Grand commented on LUCENE-8635: -- bq. The PK lookup doesn't concern me much since such

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-17 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16745253#comment-16745253 ] Michael McCandless commented on LUCENE-8635: OK thanks [~sokolov].  I'll try to also run

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744621#comment-16744621 ] Mike Sokolov commented on LUCENE-8635: -- I used the wikimedia2m data set for the second set of tests

Re: [jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Michael Sokolov
I used the wikimedia2m data set for the second set of tests (the first test was on a tiny index - 10k docs) -- at least I think I did! I am kind of new to the benchmarking game. I ran the becnhmarks with python src/python/localrun.py -source wikimedium2m, and I can see that the index dir is 861M.

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744538#comment-16744538 ] Michael McCandless commented on LUCENE-8635: Thanks [~sokolov] – those numbers look quite a

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744519#comment-16744519 ] Mike Sokolov commented on LUCENE-8635: -- Right, it seems crazy that makes a difference. I guess

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744419#comment-16744419 ] Ankit Jain commented on LUCENE-8635: Thanks [~sokolov] for updating patch and doing another run. As

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16744344#comment-16744344 ] Mike Sokolov commented on LUCENE-8635: -- Following a suggestion from ~mikemccand I tried a slightly

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-16 Thread Adrien Grand (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743730#comment-16743730 ] Adrien Grand commented on LUCENE-8635: -- This is pretty cool. I'm happily surprised as well of how

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-15 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743577#comment-16743577 ] Ankit Jain commented on LUCENE-8635:  Rally tests use underlying elasticsearch cluster which use

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-15 Thread David Smiley (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743246#comment-16743246 ] David Smiley commented on LUCENE-8635: -- +1 looks valuable, especially for cases where you don't

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-15 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743137#comment-16743137 ] Michael McCandless commented on LUCENE-8635: Thanks for testing [~sokolov] – the results

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-15 Thread Mike Sokolov (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16743067#comment-16743067 ] Mike Sokolov commented on LUCENE-8635: -- This looked interesting to me, too, so I did run the

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-11 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740997#comment-16740997 ] Ankit Jain commented on LUCENE-8635: Thanks for the tip Erick. I ran the failing tests individually

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-11 Thread Erick Erickson (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740932#comment-16740932 ] Erick Erickson commented on LUCENE-8635: Ankit:   The autoscaling tests are have been failing

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-11 Thread Ankit Jain (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740855#comment-16740855 ] Ankit Jain commented on LUCENE-8635: I ran the test suite and couple of tests seem to fail. Though, 

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-11 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740764#comment-16740764 ] Michael McCandless commented on LUCENE-8635: Also, have you confirmed that all tests pass

[jira] [Commented] (LUCENE-8635) Lazy loading Lucene FST offheap using mmap

2019-01-11 Thread Michael McCandless (JIRA)
[ https://issues.apache.org/jira/browse/LUCENE-8635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16740757#comment-16740757 ] Michael McCandless commented on LUCENE-8635: Wow, this is impressive!  Surprising how small