Hi, As Spark is an example of something I really don't want. It's resource heavy, it involves copying data and it involves managing yet another distributed system. Actually I would also need a distributed system to schedule the spark jobs also.
Sounds like a nightmare to implement a compression method. Might as well run Hadoop. - Micke ----- Original Message ----- From: "DuyHai Doan" <doanduy...@gmail.com> To: user@cassandra.apache.org Sent: Thursday, August 4, 2016 11:26:09 PM Subject: Re: Merging cells in compaction / compression? Look like you're asking for some sort of ETL on your C* data, why not use Spark to compress those data into blobs and use User-Defined-Function to explode them when reading ? On Thu, Aug 4, 2016 at 10:08 PM, Michael Burman <mibur...@redhat.com> wrote: > Hi, > > No, I don't want to lose precision (if that's what you meant), but if you > meant just storing them in a larger bucket (which I could decompress either > on client side or server side). To clarify, it could be like: > > 04082016T230215.1234, value > 04082016T230225.4321, value > 04082016T230235.2563, value > 04082016T230245.1145, value > 04082016T230255.0204, value > > -> > > 04082016T230200 -> blob (that has all the points for this minute stored - > no data is lost to aggregated avgs or sums or anything). > > That's acceptable, of course the prettiest solution would be to keep this > hidden from a client so it would see while decompressing the original rows > (like with byte[] compressors), but this is acceptable for my use-case. If > this is what you meant, then yes. > > - Micke > > ----- Original Message ----- > From: "Eric Stevens" <migh...@gmail.com> > To: user@cassandra.apache.org > Sent: Thursday, August 4, 2016 10:26:30 PM > Subject: Re: Merging cells in compaction / compression? > > When you say merge cells, do you mean re-aggregating the data into courser > time buckets? > > On Thu, Aug 4, 2016 at 5:59 AM Michael Burman <mibur...@redhat.com> wrote: > > > Hi, > > > > Considering the following example structure: > > > > CREATE TABLE data ( > > metric text, > > value double, > > time timestamp, > > PRIMARY KEY((metric), time) > > ) WITH CLUSTERING ORDER BY (time DESC) > > > > The natural inserting order is metric, value, timestamp pairs, one > > metric/value pair per second for example. That means creating more and > more > > cells to the same partition, which creates a large amount of overhead and > > reduces the compression ratio of LZ4 & Deflate (LZ4 reaches ~0.26 and > > Deflate ~0.10 ratios in some of the examples I've run). Now, to improve > > compression ratio, how could I merge the cells on the actual Cassandra > > node? I looked at ICompress and it provides only byte-level compression. > > > > Could I do this on the compaction phase, by extending the > > DateTieredCompaction for example? It has SSTableReader/Writer facilities > > and it seems to be able to see the rows? I'm fine with the fact that > repair > > run might have to do some conflict resolution as the final merged rows > > would be quite "small" (50kB) in size. The naive approach is of course to > > fetch all the rows from Cassandra - merge them on the client and send > back > > to the Cassandra, but this seems very wasteful and has its own problems. > > Compared to table-LZ4 I was able to reduce the required size to 1/20th > > (context-aware compression is sometimes just so much better) so there are > > real benefits to this approach, even if I would probably violate multiple > > design decisions. > > > > One approach is of course to write to another storage first and once the > > blocks are ready, write them to Cassandra. But that again seems idiotic > (I > > know some people are using Kafka in front of Cassandra for example, but > > that means maintaining yet another distributed solution and defeats the > > benefit of Cassandra's easy management & scalability). > > > > Has anyone done something similar? Even planned? If I need to extend > > something in Cassandra I can accept that approach also - but as I'm not > > that familiar with Cassandra source code I could use some hints. > > > > - Micke > > >
