[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[hsaputra]

Github user hsaputra commented on the issue:

https://github.com/apache/flink/pull/3704
  
Yeah totally understand the reason. Just want to make sure we have plan to 
move back as soon as RocksDB usable for Flink.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3704
  
@hsaputra There is a plan to do that and we are in touch with the RocksDB 
folks. The latest RocksDB master does not work for Flink though, currently, so 
we needed a custom backport to an earlier RocksDB version...


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[hsaputra]

Github user hsaputra commented on the issue:

https://github.com/apache/flink/pull/3704
  
I hope there is a plan to contribute back to rocksdb and we could come back 
to bring back dependencies on the fixed version. Relying on modified lib never 
good practice 


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3704
  
I'll merge the change now


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[rmetzger]

Github user rmetzger commented on the issue:

https://github.com/apache/flink/pull/3704
  
@SyinChwunLeo: we'll merge the fix to both master and the release 1.2 
branch.
But you can start using FrocksDB right away, because its available in maven 
central.

You just have to exclude the rocksdb dependency from Flink and include 
frocksdb:
```diff

org.apache.flink

flink-statebackend-rocksdb_${scala.binary.version}
+   
+   
+   org.rocksdb
+   rocksdbjni
+   
+   
+   
+   
+   com.data-artisans
+   frocksdbjni
+   4.11.2-artisans


org.apache.flink

```


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3704
  
We are using the newer StringAppendTESTOperator over the 
StringAppendOperator in rocksdbjni, because the merge algorithm used in 
StringAppendOperator has quadratic complexity in the number of merged elements. 
The modification is just instantiating the other implementation by providing a 
different parameter to a factory method.

The source code is here: 
https://github.com/dataArtisans/frocksdb/tree/master 


> Am 09.04.2017 um 18:51 schrieb Tao Wang :
> 
> Could you tell what modifications are done in "FRocksDB" and post the url 
of source code repository?
> 
> —
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub 
, or mute the 
thread 
.
> 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[SyinChwunLeo]

Github user SyinChwunLeo commented on the issue:

https://github.com/apache/flink/pull/3704
  
OK， FRocksDB is only available in master branch or we can also use it in 
flink-1.2.0?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[WangTaoTheTonic]

Github user WangTaoTheTonic commented on the issue:

https://github.com/apache/flink/pull/3704
  
Could you tell what modifications are done in "FRocksDB" and post the url 
of source code repository?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[StephanEwen]

Github user StephanEwen commented on the issue:

https://github.com/apache/flink/pull/3704
  
@SyinChwunLeo We will try and contribute the patch to RocksDB and will also 
soon try and move to a newer RocksDB version, as soon as its Java API works 
again for the required functions. The RocksDB folks mentioned that the next 
release of RocksDB is quite soon and should fix that.

That will hopefully address the issue. Until then, we cannot upgrade, 
unfortunately :-(


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3704
  
Unfortunately, all newer versions of RocksDB are broken for some functions 
when used through the Java API and will segfault when using merge operators. 
This is the same version as we previously used in Flink, so there is no 
regression. Furthermore, it is possible to use a different compression scheme. 
We might at some point try to build our custom version against zlib-1.2.9.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[SyinChwunLeo]

Github user SyinChwunLeo commented on the issue:

https://github.com/apache/flink/pull/3704
  
I notice that RocksDB in 4.11.2 version is dependent on zlib-1.2.8, 
unfortunately, there is a security leak in zlib-1.2.8. 
https://bugzilla.redhat.com/show_bug.cgi?id=CVE-2016-9840，and fixed in 1.2.9 
version. Only RocksDB's latest version 5.2.1 using zlib-1.2.9. Is it possible 
to update rocksDB to 5.2.1 in flink ?


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] flink issue #3704: [FLINK-5756] Replace RocksDB dependency with FRocksDB

[StefanRRichter]

Github user StefanRRichter commented on the issue:

https://github.com/apache/flink/pull/3704
  
LGTM. +1 for merging this.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---