Re: Cassandra on RocksDB experiment result

2017-04-26 Thread Dikang Gu
@Samba, that's a very good point, I definitely do not expect all storage engines provide exactly same features, and each storage engine should have it's own strength and sweet spots as well. For features not supported by certain storage engine, I think it should throw exceptions and fail the

Re: Cassandra on RocksDB experiment result

2017-04-26 Thread Samba
some features may work with some storage engine but with others; for example, storing large blobs may be efficient in one storage engine while quite worse in another. perhaps some storage engines may want to SKIP some features or add more. if a storage engine skips a feature, how should the query

Re: Cassandra on RocksDB experiment result

2017-04-25 Thread Dikang Gu
I created several tickets to start the discussion, please free feel to comment on the JIRAs. I'm also open for suggestions about other efficient ways to discuss it. https://issues.apache.org/jira/browse/CASSANDRA-13474 https://issues.apache.org/jira/browse/CASSANDRA-13475

Re: Cassandra on RocksDB experiment result

2017-04-24 Thread Dikang Gu
Thanks everyone for the feedback and suggestions! They are all very helpful. I'm looking forward to having more discussions about the implementation details. As the next step, we will be focus on three areas: 1. Pluggable storage engine interface. 2. Wide column support on RocksDB. 3. Streaming

Re: Cassandra on RocksDB experiment result

2017-04-24 Thread Patrick McFadin
Dikang, First I want to thank you and everyone else at Instragram for the engineering talent you have devoted to the Cassandra project. Here's yet another great example. He's going to hate me for dragging him into this, but Vijay Parthasarathy has done some exploratory work before on integrating

Re: Cassandra on RocksDB experiment result

2017-04-24 Thread LuckyBoy
unsubscribe On Sun, Apr 23, 2017 at 4:25 PM, Nate McCall wrote: > > Please take a look and let me know your thoughts. I think the biggest > > latency win comes from we get rid of most Java garbages created by > current > > read/write path and compactions, which reduces the

Re: Cassandra on RocksDB experiment result

2017-04-24 Thread Nate McCall
> Please take a look and let me know your thoughts. I think the biggest > latency win comes from we get rid of most Java garbages created by current > read/write path and compactions, which reduces the JVM overhead and makes > the latency to be more predictable. > I want to put this here for the

Re: Cassandra on RocksDB experiment result

2017-04-22 Thread Eric Stevens
In the spirit of what Eric mentions, as a community member, I'm enthusiastically +1 on the idea. On Fri, Apr 21, 2017 at 9:28 AM, Eric Evans wrote: > On Fri, Apr 21, 2017 at 4:32 AM, benjamin roth wrote: > > I am not a PMC member or sth but just my

Re: Cassandra on RocksDB experiment result

2017-04-21 Thread Eric Evans
On Fri, Apr 21, 2017 at 4:32 AM, benjamin roth wrote: > I am not a PMC member or sth but just my 2 cents: Somewhat off-topic here, but I'd like to start discouraging people from prefacing remarks like this ("not a PMC member", "non-binding +1"). The exchange rate here is 1:1

Re: Cassandra on RocksDB experiment result

2017-04-20 Thread Jason Brown
I'm +1 on the idea of a pluggable storage engine. There's clearly a bandwidth problem for developing/reviewing/maintain multiple storage engines, but I think having the interface is a good thing and can enhance testability. At a minimum I think it's worthwhile to explore the storage engine

Re: Cassandra on RocksDB experiment result

2017-04-20 Thread Jeff Jirsa
Let's try to make this actionable. Long time contributors/committers/members of the PMC (especially you guys who have been working on internals for 4-8 years): Setting aside details of the implementation, does anyone feel that pluggable storage in itself is inherently a bad idea (so much so that

RE: Cassandra on RocksDB experiment result

2017-04-20 Thread Jacques-Henri Berthemet
Subject: Re: Cassandra on RocksDB experiment result > Right now all the compaction strategies share the assumption that the data > structure and layout on disk is fixed. With pluggable storage engine, we > need to special case each compaction strategy (or at least the Abstract > class o

Re: Cassandra on RocksDB experiment result

2017-04-19 Thread Ben Bromhead
This looks super cool would love to see more details. On a general note, a pluggable storage layer allows other storage engines (and possibly datastores) to leverage Cassandras distributed primitives (dynamo, gossip, paxsos?, drivers, cql etc). This could allow Cassandra to fill similar use cases

RE: Cassandra on RocksDB experiment result

2017-04-19 Thread Bob Dourandish
off the ground. Bob -Original Message- From: DuyHai Doan [mailto:doanduy...@gmail.com] Sent: Wednesday, April 19, 2017 3:33 PM To: dev@cassandra.apache.org Subject: Re: Cassandra on RocksDB experiment result "I have no clue what it would take to accomplish a pluggable storage engi

Re: Cassandra on RocksDB experiment result

2017-04-19 Thread DuyHai Doan
"I have no clue what it would take to accomplish a pluggable storage engine, but I love this idea." This was a long and old debate we had several times in the past. One of the difficulty of pluggable storage engine is that we need to manage the differences between the LSMT of native C* and RockDB

Re: Cassandra on RocksDB experiment result

2017-04-19 Thread sankalp kohli
We should definitely evaluate pluggable storage engine...Besides several other advantages, it also helps in adding lot of tests to the storage engine. On Wed, Apr 19, 2017 at 11:22 AM, Jon Haddad wrote: > I have no clue what it would take to accomplish a pluggable

Re: Cassandra on RocksDB experiment result

2017-04-19 Thread Jon Haddad
I have no clue what it would take to accomplish a pluggable storage engine, but I love this idea. Obviously the devil is in the details, & a simple K/V is very different from supporting partitions, collections, etc, but this is very cool & seems crazy not to explore further. Will you be

Cassandra on RocksDB experiment result

2017-04-19 Thread Dikang Gu
Hi Cassandra developers, This is Dikang from Instagram, I'd like to share you some experiment results we did recently, to use RocksDB as Cassandra's storage engine. In the experiment, I built a prototype to integrate Cassandra 3.0.12 and RocksDB on single column (key-value) use case, shadowed one