[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-02-10 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490174#comment-17490174 ] Adrien Grand commented on LUCENE-8739: -- My opinion is that there are interesting benefits, but they

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-02-10 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17490161#comment-17490161 ] Praveen Nishchal commented on LUCENE-8739: -- Hi Adrien, Thank you for your feedback! I am a

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-02-03 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486619#comment-17486619 ] Adrien Grand commented on LUCENE-8739: -- Robert disagreed with introducing a requirement on libzstd

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-02-03 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17486614#comment-17486614 ] Praveen Nishchal commented on LUCENE-8739: -- As we observed earlier, Zstd is at par with

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-10 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17472298#comment-17472298 ] Praveen Nishchal commented on LUCENE-8739: -- Hi [~rcmuir]  That is exactly what I am doing :)

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-09 Thread Robert Muir (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17471315#comment-17471315 ] Robert Muir commented on LUCENE-8739: - We already have a compression abstraction in lucene:

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-07 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470999#comment-17470999 ] Praveen Nishchal commented on LUCENE-8739: -- Hi [~rcmuir]  This is why I have created a custom

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-07 Thread Robert Muir (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470624#comment-17470624 ] Robert Muir commented on LUCENE-8739: - > +1 ZSTD is quite great. I wouldn't use it in the Lucene

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-07 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17470506#comment-17470506 ] Praveen Nishchal commented on LUCENE-8739: -- WOW! That's a lot of wonderful feedback here :) I

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-04 Thread Tobias Ibounig (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468610#comment-17468610 ] Tobias Ibounig commented on LUCENE-8739: Ok this all sounds very good. Just one more thing for

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-04 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468498#comment-17468498 ] Adrien Grand commented on LUCENE-8739: -- bq. Would such an increase even make sense or would this

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-04 Thread Tobias Ibounig (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468482#comment-17468482 ] Tobias Ibounig commented on LUCENE-8739: Would it make sense to increase the block size until

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-04 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468417#comment-17468417 ] Adrien Grand commented on LUCENE-8739: -- I updated block sizes so that ZSTD uses the same block

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-03 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468175#comment-17468175 ] Adrien Grand commented on LUCENE-8739: -- I may have found the issue, your codec was using the same

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2022-01-03 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468137#comment-17468137 ] Adrien Grand commented on LUCENE-8739: -- I ran the same benchmark over the above PR with the

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-12-27 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17465718#comment-17465718 ] Praveen Nishchal commented on LUCENE-8739: -- Added dictionary support for Zstandard -

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-11-15 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443976#comment-17443976 ] Adrien Grand commented on LUCENE-8739: -- Side thought: it would be nice to use Project Panama's

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-11-15 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443964#comment-17443964 ] Adrien Grand commented on LUCENE-8739: -- I ran your PR with the new stored fields benchmark to see

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-11-15 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443912#comment-17443912 ] Praveen Nishchal commented on LUCENE-8739: -- I have created a pull request -

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432677#comment-17432677 ] Adrien Grand commented on LUCENE-8739: -- You need to download

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432590#comment-17432590 ] Praveen Nishchal commented on LUCENE-8739: -- Hi Adrien, Can you please help me by stating the

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432589#comment-17432589 ] Praveen Nishchal commented on LUCENE-8739: -- Hi Mike, -Dtests.nightly=true ran successfully ,

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432506#comment-17432506 ] Adrien Grand commented on LUCENE-8739: -- You might be interested in the new simple benchmark for

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Michael McCandless (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432444#comment-17432444 ] Michael McCandless commented on LUCENE-8739: {quote}My codec passed all test cases with test

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-21 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432288#comment-17432288 ] Praveen Nishchal commented on LUCENE-8739: -- Hi Mike, My codec passed all test cases with test

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-10-11 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17427037#comment-17427037 ] Praveen Nishchal commented on LUCENE-8739: -- Hi Mike I see Adrien has used JNA based Zstd

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-09-30 Thread Michael McCandless (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422815#comment-17422815 ] Michael McCandless commented on LUCENE-8739: Wow, these are compelling results! Can you try

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-09-15 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415363#comment-17415363 ] Praveen Nishchal commented on LUCENE-8739: -- _I have developed new custom codec which integrates

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-06-07 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17358574#comment-17358574 ] Adrien Grand commented on LUCENE-8739: -- I opened a PR that uses the exact same approach and block

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-06-04 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17357132#comment-17357132 ] Praveen Nishchal commented on LUCENE-8739: -- Zstd JNI https://github.com/luben/zstd-jni looks

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-19 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324904#comment-17324904 ] Adrien Grand commented on LUCENE-8739: -- Hi [~wicked1099], force-merging wouldn't change anything:

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-18 Thread Sean Torres (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324662#comment-17324662 ] Sean Torres commented on LUCENE-8739: - If the current runtime compression is comparable to DEFLATE,

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-16 Thread Sean Torres (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17324044#comment-17324044 ] Sean Torres commented on LUCENE-8739: - How about the performance and storage cost once a force merge

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-16 Thread Dawid Weiss (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17323762#comment-17323762 ] Dawid Weiss commented on LUCENE-8739: - Because it would make it very difficult to work for everyone

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-16 Thread Praveen Nishchal (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322945#comment-17322945 ] Praveen Nishchal commented on LUCENE-8739: -- Hi [~jpountz], Kindly help us understand why

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2021-04-15 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322015#comment-17322015 ] Adrien Grand commented on LUCENE-8739: -- I forgot to update this issue but I actually played with

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2020-03-26 Thread Michael McCandless (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17067798#comment-17067798 ] Michael McCandless commented on LUCENE-8739: I think this is worth a deep dive, at least to

[jira] [Commented] (LUCENE-8739) ZSTD Compressor support in Lucene

2019-12-16 Thread Adrien Grand (Jira)
[ https://issues.apache.org/jira/browse/LUCENE-8739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16997346#comment-16997346 ] Adrien Grand commented on LUCENE-8739: -- As I expected it needs quite a lot of code, compared to the