[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16125413#comment-16125413
]
Chesnay Schepler commented on FLINK-6333:
-
[~tedyu] The rocksdb issue has been closed for a month now, please stop bumping
this thread.
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16095490#comment-16095490
]
Ted Yu commented on FLINK-6333:
---
Looks like rocksdb issue 1964 can be closed.
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16028212#comment-16028212
]
Stefan Richter commented on FLINK-6333:
---
This issue still exists after some testing and stops us from upgrading:
https://github.com/facebook/rocksdb/issues/1964
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026533#comment-16026533
]
Ted Yu commented on FLINK-6333:
---
Issue #1988 has been closed.
What's the next step ?
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
> Components: State Backends, Checkpointing
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992627#comment-15992627
]
Stefan Richter commented on FLINK-6333:
---
It is mentioned in this issue: https://github.com/facebook/rocksdb/issues/1988
Code in the OP segfaults with newer versions of RocksDB.
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15979076#comment-15979076
]
Ted Yu commented on FLINK-6333:
---
I found this issue which was resolved:
https://github.com/facebook/rocksdb/issues/1267
What is the current issue w.r.t. potential segfault ?
Thanks
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976306#comment-15976306
]
Stefan Richter commented on FLINK-6333:
---
Right now, the biggest problem in updating RockDB above version 4.11.2 are bugs
in rocksjni that will cause some functionality that is used in Flink (e.g.
merge operator) to segfault. We can only move to a higher version once those
problems are fixed in RocksDB.
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (FLINK-6333) Utilize Bloomfilters in RocksDb
[
https://issues.apache.org/jira/browse/FLINK-6333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15976008#comment-15976008
]
Fang Yong commented on FLINK-6333:
--
Now flink uses RocksDb jni 4.11.2, and already support setFilter(new
BloomFilter()) method in BlockBasedTableConfig. Can bloomfilters be used in
rocksdb 4.11.2 directly? Are there any other improvements from current version
to RocksDb 5.2.1+?
> Utilize Bloomfilters in RocksDb
> ---
>
> Key: FLINK-6333
> URL: https://issues.apache.org/jira/browse/FLINK-6333
> Project: Flink
> Issue Type: Improvement
>Reporter: Ted Yu
>
> Bloom Filters would speed up RocksDb lookups.
> When we upgrade to RocksDb 5.2.1+, we would be able to do:
> {code}
> new BlockBasedTableConfig()
> .setBlockCacheSize(blockCacheSize)
> .setBlockSize(blockSize)
> .setFilter(new BloomFilter())
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
