question on the code formatter
Hi all, I was trying to configure the Cassandra code formatter and downloaded IntelliJ-codestyle.jar from this link: https://wiki.apache.org/cassandra/CodeStyle After extracting this JAR, I was able to import codestyle/Default_1_.xml into my project and formatting seemed to work. However, I'm wondering what options/code.style.schemes.xml file is exactly used for? Could anyone please give me an idea if I need to configure this as well? Thanks, Preetika
Re: Cassandra table definition with no clustering key
Hi Michael, This is exactly what I was looking for. Thank you so much! :) Preetika On Fri, Jul 14, 2017 at 8:56 PM, Michael Shuler <mich...@pbandjelly.org> wrote: > I think this note in the partition key section is what you might be > looking for? > > "Note that a table always has a partition key, and that if the table has > no clustering columns, then every partition of that table is only > comprised of a single row (since the primary key uniquely identifies > rows and the primary key is equal to the partition key if there is no > clustering columns)." > > http://cassandra.apache.org/doc/latest/cql/ddl.html#the-partition-key > > -- > Kind regards, > Michael > > On 07/14/2017 06:59 PM, preetika tyagi wrote: > > Hi all, > > > > I'm trying to understand the scenario when no clustering key is > > specified in a table definition. > > > > If a table has only a partition key and no clustering key, what order > > the rows under the same partition are stored in? Is it even allowed to > > have multiple rows under the same partition when no clustering key > > exists? I tried searching for it online but couldn't get a clear > > explanation. > > > > Thanks, > > Preetika > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Cassandra table definition with no clustering key
Hi all, I'm trying to understand the scenario when no clustering key is specified in a table definition. If a table has only a partition key and no clustering key, what order the rows under the same partition are stored in? Is it even allowed to have multiple rows under the same partition when no clustering key exists? I tried searching for it online but couldn't get a clear explanation. Thanks, Preetika
Re: How to avoid flush if the data can fit into memtable
Great explanation and the blog post, Akhil. Sorry for the delayed response (somehow didn't notice the email in my inbox), but this is what I concluded as well. In addition to compression, I believe the sstable is serialized as well and the combination of both results into much smaller sstable size in comparison to in-memory memtable size which holds all the data in java objects. I also did a small experiment for this. When I allocate 4GB of heap (resulting into roughly 981MB for memtable as per your post) and then write approx 920MB of data, it ends up writing some sstables. However, if I increase the heap size to 120GB and write ~920MB of data again, it doesn't write anything to the sstable. Therefore, it clearly indicates that I need bigger heap sizes. One interesting fact though, if I bring heap size down to 64GB which means memtable will roughly be around 16GB and again write ~920MB data, it still writes some sstables. The ratio of 920MB serialized + compressed data and more than 16GB in-memory memtable data looks a bit weird but I don't have a solid explanation for this behavior. However, I'm not going to look into that so we can conclude this post :) Thank you all for your responses! Preetika On Fri, Jun 2, 2017 at 10:56 AM, Jeff Jirsa <jji...@apache.org> wrote: > > > On 2017-05-24 17:42 (-0700), preetika tyagi <preetikaty...@gmail.com> > wrote: > > Hi, > > > > I'm running Cassandra with a very small dataset so that the data can > exist > > on memtable only. Below are my configurations: > > > > In jvm.options: > > > > -Xms4G > > -Xmx4G > > > > In cassandra.yaml, > > > > memtable_cleanup_threshold: 0.50 > > memtable_allocation_type: heap_buffers > > > > As per the documentation in cassandra.yaml, the > *memtable_heap_space_in_mb* > > and *memtable_heap_space_in_mb* will be set of 1/4 of heap size i.e. > 1000MB > > > > According to the documentation here ( > > http://docs.datastax.com/en/cassandra/3.0/cassandra/ > configuration/configCassandra_yaml.html#configCassandra_ > yaml__memtable_cleanup_threshold), > > the memtable flush will trigger if the total size of memtabl(s) goes > beyond > > (1000+1000)*0.50=1000MB. > > 1/4 heap (=1G) * .5 cleanup means cleanup happens at 500MB, or when > commitlog hits its max size. If you disable durable writes (disable the > cleanup), you're flushing at 500MB. > > Recall that your 300MB of data also has associated data with it > (timestamps, ttls, etc) that will increase size beyond your nominal > calculation from the user side. > > If you're sure you want to do this, set durable_writes=false and either > raise the memtable_cleanup_threshold significantly, raise your heap or > memtable size. > > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Re: How to avoid flush if the data can fit into memtable
I agree that for such a small data, Cassandra is obviously not needed. However, this is purely an experimental setup by using which I'm trying to understand how and exactly when memtable flush is triggered. As I mentioned in my post, I read the documentation and tweaked the parameters accordingly so that I never hit memtable flush but it is still doing that. As far the the setup is concerned, I'm just using 1 node and running Cassandra using "cassandra -R" option and then running some queries to insert some dummy data. I use the schema from CASSANDRA_HOME/tools/cqlstress-insanity-example.yaml and add "durable_writes=false" in the keyspace_definition. @Daemeon - The previous post lead to this post but since I was unaware of memtable flush and I assumed memtable flush wasn't happening, the previous post was about something else (throughput/latency etc.). This post is explicitly about exactly when memtable is being dumped to the disk. Didn't want to confuse two different goals that's why posted a new one. On Thu, May 25, 2017 at 10:38 AM, Avi Kivity <a...@scylladb.com> wrote: > It doesn't have to fit in memory. If your key distribution has strong > temporal locality, then a larger memtable that can coalesce overwrites > greatly reduces the disk I/O load for the memtable flush and subsequent > compactions. Of course, I have no idea if the is what the OP had in mind. > > > On 05/25/2017 07:14 PM, Jonathan Haddad wrote: > > Sorry for the confusion. That was for the OP. I wrote it quickly right > after waking up. > > What I'm asking is why does the OP want to keep his data in the memtable > exclusively? If the goal is to "make reads fast", then just turn on row > caching. > > If there's so little data that it fits in memory (300MB), and there aren't > going to be any writes past the initial small dataset, why use Cassandra? > It sounds like the wrong tool for this job. Sounds like something that > could easily be stored in S3 and loaded in memory when the app is fired up. > > > On Thu, May 25, 2017 at 8:06 AM Avi Kivity <a...@scylladb.com> wrote: > >> Not sure whether you're asking me or the original poster, but the more >> times data gets overwritten in a memtable, the less it has to be compacted >> later on (and even without overwrites, larger memtables result in less >> compaction). >> >> On 05/25/2017 05:59 PM, Jonathan Haddad wrote: >> >> Why do you think keeping your data in the memtable is a what you need to >> do? >> On Thu, May 25, 2017 at 7:16 AM Avi Kivity <a...@scylladb.com> wrote: >> >>> Then it doesn't have to (it still may, for other reasons). >>> >>> On 05/25/2017 05:11 PM, preetika tyagi wrote: >>> >>> What if the commit log is disabled? >>> >>> On May 25, 2017 4:31 AM, "Avi Kivity" <a...@scylladb.com> wrote: >>> >>>> Cassandra has to flush the memtable occasionally, or the commit log >>>> grows without bounds. >>>> >>>> On 05/25/2017 03:42 AM, preetika tyagi wrote: >>>> >>>> Hi, >>>> >>>> I'm running Cassandra with a very small dataset so that the data can >>>> exist on memtable only. Below are my configurations: >>>> >>>> In jvm.options: >>>> >>>> -Xms4G >>>> -Xmx4G >>>> >>>> In cassandra.yaml, >>>> >>>> memtable_cleanup_threshold: 0.50 >>>> memtable_allocation_type: heap_buffers >>>> >>>> As per the documentation in cassandra.yaml, the >>>> *memtable_heap_space_in_mb* and *memtable_heap_space_in_mb* will be >>>> set of 1/4 of heap size i.e. 1000MB >>>> >>>> According to the documentation here (http://docs.datastax.com/en/ >>>> cassandra/3.0/cassandra/configuration/configCassandra_ >>>> yaml.html#configCassandra_yaml__memtable_cleanup_threshold), the >>>> memtable flush will trigger if the total size of memtabl(s) goes beyond >>>> (1000+1000)*0.50=1000MB. >>>> >>>> Now if I perform several write requests which results in almost ~300MB >>>> of the data, memtable still gets flushed since I see sstables being created >>>> on file system (Data.db etc.) and I don't understand why. >>>> >>>> Could anyone explain this behavior and point out if I'm missing >>>> something here? >>>> >>>> Thanks, >>>> >>>> Preetika >>>> >>>> >>>> >>> >> >
Re: How to avoid flush if the data can fit into memtable
What if the commit log is disabled? On May 25, 2017 4:31 AM, "Avi Kivity" <a...@scylladb.com> wrote: > Cassandra has to flush the memtable occasionally, or the commit log grows > without bounds. > > On 05/25/2017 03:42 AM, preetika tyagi wrote: > > Hi, > > I'm running Cassandra with a very small dataset so that the data can > exist on memtable only. Below are my configurations: > > In jvm.options: > > -Xms4G > -Xmx4G > > In cassandra.yaml, > > memtable_cleanup_threshold: 0.50 > memtable_allocation_type: heap_buffers > > As per the documentation in cassandra.yaml, the > *memtable_heap_space_in_mb* and *memtable_heap_space_in_mb* will be set > of 1/4 of heap size i.e. 1000MB > > According to the documentation here (http://docs.datastax.com/en/ > cassandra/3.0/cassandra/configuration/configCassandra_ > yaml.html#configCassandra_yaml__memtable_cleanup_threshold), the memtable > flush will trigger if the total size of memtabl(s) goes beyond > (1000+1000)*0.50=1000MB. > > Now if I perform several write requests which results in almost ~300MB of > the data, memtable still gets flushed since I see sstables being created on > file system (Data.db etc.) and I don't understand why. > > Could anyone explain this behavior and point out if I'm missing something > here? > > Thanks, > > Preetika > > >
How to avoid flush if the data can fit into memtable
Hi, I'm running Cassandra with a very small dataset so that the data can exist on memtable only. Below are my configurations: In jvm.options: -Xms4G -Xmx4G In cassandra.yaml, memtable_cleanup_threshold: 0.50 memtable_allocation_type: heap_buffers As per the documentation in cassandra.yaml, the *memtable_heap_space_in_mb* and *memtable_heap_space_in_mb* will be set of 1/4 of heap size i.e. 1000MB According to the documentation here ( http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/configCassandra_yaml.html#configCassandra_yaml__memtable_cleanup_threshold), the memtable flush will trigger if the total size of memtabl(s) goes beyond (1000+1000)*0.50=1000MB. Now if I perform several write requests which results in almost ~300MB of the data, memtable still gets flushed since I see sstables being created on file system (Data.db etc.) and I don't understand why. Could anyone explain this behavior and point out if I'm missing something here? Thanks, Preetika
Re: Impact on latency with larger memtable
Thank you all for the response. I figured out the root cause. I thought all my data was in memtable only but the data was actually being dumped to the disk. That's why I was noticing the drop in throughput. On Wed, May 24, 2017 at 9:42 AM, daemeon reiydelle <daeme...@gmail.com> wrote: > You speak of increase. Please provide your results. Specific examples, Eg > 25% increase results in n% increase. Also please include number of nodes, > size of total keyspace, rep factor, etc. > > Hopefully this is a 6 node cluster with several hundred gig per keyspace, > not some single node free tier box. > > “All men dream, but not equally. Those who dream by night in the dusty > recesses of their minds wake up in the day to find it was vanity, but the > dreamers of the day are dangerous men, for they may act their dreams with > open eyes, to make it possible.” — T.E. Lawrence > > sent from my mobile > Daemeon Reiydelle > skype daemeon.c.m.reiydelle > USA 415.501.0198 <(415)%20501-0198> > > On May 24, 2017 9:32 AM, "preetika tyagi" <preetikaty...@gmail.com> wrote: > >> Hi, >> >> I'm experimenting with memtable/heap size on my Cassandra server to >> understand how it impacts the latency/throughput for read requests. >> >> I vary heap size (Xms and -Xmx) in jvm.options so memtable will be 1/4 of >> this. When I increase the heap size and hence memtable, I notice the drop >> in throughput and increase in latency. I'm also creating the database such >> that its size doesn't exceed the size of memtable. Therefore, all data >> exist in memtable and I'm not able to reason why bigger size of memtable is >> resulting into higher latency/low throughput. >> >> Since everything is DRAM, shouldn't the throughput/latency remain same in >> all the cases? >> >> Thanks, >> Preetika >> >
Impact on latency with larger memtable
Hi, I'm experimenting with memtable/heap size on my Cassandra server to understand how it impacts the latency/throughput for read requests. I vary heap size (Xms and -Xmx) in jvm.options so memtable will be 1/4 of this. When I increase the heap size and hence memtable, I notice the drop in throughput and increase in latency. I'm also creating the database such that its size doesn't exceed the size of memtable. Therefore, all data exist in memtable and I'm not able to reason why bigger size of memtable is resulting into higher latency/low throughput. Since everything is DRAM, shouldn't the throughput/latency remain same in all the cases? Thanks, Preetika
Re: how to recover a dead node using commit log when memtable is lost
Assuming we are using periodic mode for commit log sync. On Wed, Apr 5, 2017 at 3:29 PM, preetika tyagi <preetikaty...@gmail.com> wrote: > Very good explanation. > One follow-up question. If CL is set to 1 and RF to 3, then there are > chances of the data being lost if the machine crashes before replication > happens and the commit log (on the node which processed the data for CF=1) > is not synced yet. Right? > > Thanks, > Preetika > > On Wed, Apr 5, 2017 at 1:17 PM, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: > >> I beg to differ with @Matija here, IMO by default cassandra syncs data >> into commit log in a periodic fashion with a fsync period of 10 sec (Ref - >> https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L361). >> If a write is not written to disk and RF is 1 else CL is Local One & node >> goes down then there could be potential data loss and client would expect >> data to be present. >> >> Therefore a good strategy would be to either have RF 3 and write with >> quorum. Or if thats not a feasible option then use Batch Mode for commitlog >> sync - That could lead to much higher disk io overhead - (say if you fsync >> every 10 ms in batch (write latency in this case will be 10 ms as write >> threads will be blocked for 10ms. assuming continuous writes- you would be >> issuing 1000/10 IO write IO to disk - 100 IOPS. If thats kept to 1ms to >> reduce write latency to 1ms then IOPS becomes 1000)). >> >> So in case of batch mode it get tricky to balance latency & disk >> utilisation. Testing this setting thoroughly on dev env would be >> recommended as it can adversely affect performance. We had done some >> benchmarks and had found 50ms to be ideal for our use case but thats >> subjective as it leads to write latencies in excess of 50ms which could be >> really high for some use cases. Though with modern day ssd's batch option >> can be worthwhile to experiment with. >> >> A good description is also given here - http://stackoverflow.com/a/310 >> 33900/3646120 >> >> On Thu, Apr 6, 2017 at 12:30 AM, Matija Gobec <matija0...@gmail.com> >> wrote: >> >>> Flushes have nothing to do with data persistence and node failure. Each >>> write is acknowledged only when data has been written to the commit log AND >>> memtable. That solves the issues of node failures and data consistency. >>> When the node boots back up it replays commit log files and you don't loose >>> data that was already written to that node. >>> >>> On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi <preetikaty...@gmail.com> >>> wrote: >>> >>>> Hi, >>>> >>>> I read in Cassandra architecture documentation that if a node dies and >>>> there is some data in memtable which hasn't been written to the sstable, >>>> the commit log replay happens (assuming the commit log had been flushed to >>>> disk) when the node restarts and hence the data can be recovered. >>>> >>>> However, I was wondering if a node is fully dead for some reason with >>>> consistency level 1 (replication factor 3 but let's say it dies right after >>>> it finishes a request and hence the data hasn't been replicated to other >>>> nodes yet) and the data is only available in commit log on that node. Is >>>> there a way to recover data from this node which includes both sstable and >>>> commit log so that we can use it to replace it with a new node where we >>>> could replay commit log to recover the data? >>>> >>>> Thanks, >>>> Preetika >>>> >>> >>> >> >
Re: how to recover a dead node using commit log when memtable is lost
Very good explanation. One follow-up question. If CL is set to 1 and RF to 3, then there are chances of the data being lost if the machine crashes before replication happens and the commit log (on the node which processed the data for CF=1) is not synced yet. Right? Thanks, Preetika On Wed, Apr 5, 2017 at 1:17 PM, Bhuvan Rawal <bhu1ra...@gmail.com> wrote: > I beg to differ with @Matija here, IMO by default cassandra syncs data > into commit log in a periodic fashion with a fsync period of 10 sec (Ref - > https://github.com/apache/cassandra/blob/trunk/conf/cassandra.yaml#L361). > If a write is not written to disk and RF is 1 else CL is Local One & node > goes down then there could be potential data loss and client would expect > data to be present. > > Therefore a good strategy would be to either have RF 3 and write with > quorum. Or if thats not a feasible option then use Batch Mode for commitlog > sync - That could lead to much higher disk io overhead - (say if you fsync > every 10 ms in batch (write latency in this case will be 10 ms as write > threads will be blocked for 10ms. assuming continuous writes- you would be > issuing 1000/10 IO write IO to disk - 100 IOPS. If thats kept to 1ms to > reduce write latency to 1ms then IOPS becomes 1000)). > > So in case of batch mode it get tricky to balance latency & disk > utilisation. Testing this setting thoroughly on dev env would be > recommended as it can adversely affect performance. We had done some > benchmarks and had found 50ms to be ideal for our use case but thats > subjective as it leads to write latencies in excess of 50ms which could be > really high for some use cases. Though with modern day ssd's batch option > can be worthwhile to experiment with. > > A good description is also given here - http://stackoverflow.com/a/ > 31033900/3646120 > > On Thu, Apr 6, 2017 at 12:30 AM, Matija Gobec <matija0...@gmail.com> > wrote: > >> Flushes have nothing to do with data persistence and node failure. Each >> write is acknowledged only when data has been written to the commit log AND >> memtable. That solves the issues of node failures and data consistency. >> When the node boots back up it replays commit log files and you don't loose >> data that was already written to that node. >> >> On Wed, Apr 5, 2017 at 6:22 PM, preetika tyagi <preetikaty...@gmail.com> >> wrote: >> >>> Hi, >>> >>> I read in Cassandra architecture documentation that if a node dies and >>> there is some data in memtable which hasn't been written to the sstable, >>> the commit log replay happens (assuming the commit log had been flushed to >>> disk) when the node restarts and hence the data can be recovered. >>> >>> However, I was wondering if a node is fully dead for some reason with >>> consistency level 1 (replication factor 3 but let's say it dies right after >>> it finishes a request and hence the data hasn't been replicated to other >>> nodes yet) and the data is only available in commit log on that node. Is >>> there a way to recover data from this node which includes both sstable and >>> commit log so that we can use it to replace it with a new node where we >>> could replay commit log to recover the data? >>> >>> Thanks, >>> Preetika >>> >> >> >
how to recover a dead node using commit log when memtable is lost
Hi, I read in Cassandra architecture documentation that if a node dies and there is some data in memtable which hasn't been written to the sstable, the commit log replay happens (assuming the commit log had been flushed to disk) when the node restarts and hence the data can be recovered. However, I was wondering if a node is fully dead for some reason with consistency level 1 (replication factor 3 but let's say it dies right after it finishes a request and hence the data hasn't been replicated to other nodes yet) and the data is only available in commit log on that node. Is there a way to recover data from this node which includes both sstable and commit log so that we can use it to replace it with a new node where we could replay commit log to recover the data? Thanks, Preetika
Re: question on maximum disk seeks
Oh I see. I understand it now. Thank you for the clarification! Preetika On Tue, Mar 21, 2017 at 11:07 AM, Jonathan Haddad <j...@jonhaddad.com> wrote: > Each sstable has it's own partition index, therefore it's never updated. > > On Tue, Mar 21, 2017 at 11:04 AM preetika tyagi <preetikaty...@gmail.com> > wrote: > >> Yes, I understand that. However, what I'm trying to understand is the >> internal structure of partition index. When a record associate with the >> same partition key is updated, we have two different records with different >> timestamps. There are chances of these two records being split across two >> different SSTables (of course as long as compaction is not merging them >> into one SSTable eventually). How partition index looks like in such case? >> For the same key, we have two different records in different SSTables. How >> does partition index store such information? Can it have repeated partition >> keys with different disk offsets pointing to different SSTables? >> >> On Tue, Mar 21, 2017 at 10:09 AM, Jonathan Haddad <j...@jonhaddad.com> >> wrote: >> >> The partition index is never updated, as sstables are immutable. >> >> On Tue, Mar 21, 2017 at 9:40 AM preetika tyagi <preetikaty...@gmail.com> >> wrote: >> >> Thank you Jan & Jeff for the responses. That was really useful. >> >> Jan - I have one follow-up question. When the data is spread over more >> than one SSTable in case of updates as you mentioned, we will need two >> seeks per SSTable (one for partition index and another for SSTable itself). >> I'm curious to know how partition index is structured internally. I was >> assuming it to be a table with <key, disk offset> pairs. In case of an >> update to the same key for several times, how it is recorded in the >> partition index? >> >> Thanks, >> Preetika >> >> On Mon, Mar 20, 2017 at 10:37 PM, <j.kes...@enercast.de> wrote: >> >> Hi, >> >> >> >> youre right – one seek with hit in the partition key cache and two if not. >> >> >> >> Thats the theory – but two thinge to mention: >> >> >> >> First, you need two seeks per sstable not per entire read. So if you data >> is spread over multiple sstables on disk you obviously need more then two >> reads. Think of often updated partition keys – in combination with memory >> preassure you can easily end up with maaany sstables (ok they will be >> compacted some time in the future). >> >> >> >> Second, there could be fragmentation on disk which leads to seeks during >> sequential reads. >> >> >> >> Jan >> >> >> >> Gesendet von meinem Windows 10 Phone >> >> >> >> *Von: *preetika tyagi <preetikaty...@gmail.com> >> *Gesendet: *Montag, 20. März 2017 21:18 >> *An: *user@cassandra.apache.org >> *Betreff: *question on maximum disk seeks >> >> >> >> I'm trying to understand the maximum number of disk seeks required in a >> read operation in Cassandra. I looked at several online articles including >> this one: https://docs.datastax.com/en/cassandra/3.0/ >> cassandra/dml/dmlAboutReads.html >> >> As per my understanding, two disk seeks are required in the worst case. >> One is for reading the partition index and another is to read the actual >> data from the compressed partition. The index of the data in compressed >> partitions is obtained from the compression offset tables (which is stored >> in memory). Am I on the right track here? Will there ever be a case when >> more than 1 disk seek is required to read the data? >> >> Thanks, >> >> Preetika >> >> >> >> >> >> >>
Re: question on maximum disk seeks
Yes, I understand that. However, what I'm trying to understand is the internal structure of partition index. When a record associate with the same partition key is updated, we have two different records with different timestamps. There are chances of these two records being split across two different SSTables (of course as long as compaction is not merging them into one SSTable eventually). How partition index looks like in such case? For the same key, we have two different records in different SSTables. How does partition index store such information? Can it have repeated partition keys with different disk offsets pointing to different SSTables? On Tue, Mar 21, 2017 at 10:09 AM, Jonathan Haddad <j...@jonhaddad.com> wrote: > The partition index is never updated, as sstables are immutable. > > On Tue, Mar 21, 2017 at 9:40 AM preetika tyagi <preetikaty...@gmail.com> > wrote: > >> Thank you Jan & Jeff for the responses. That was really useful. >> >> Jan - I have one follow-up question. When the data is spread over more >> than one SSTable in case of updates as you mentioned, we will need two >> seeks per SSTable (one for partition index and another for SSTable itself). >> I'm curious to know how partition index is structured internally. I was >> assuming it to be a table with <key, disk offset> pairs. In case of an >> update to the same key for several times, how it is recorded in the >> partition index? >> >> Thanks, >> Preetika >> >> On Mon, Mar 20, 2017 at 10:37 PM, <j.kes...@enercast.de> wrote: >> >> Hi, >> >> >> >> youre right – one seek with hit in the partition key cache and two if not. >> >> >> >> Thats the theory – but two thinge to mention: >> >> >> >> First, you need two seeks per sstable not per entire read. So if you data >> is spread over multiple sstables on disk you obviously need more then two >> reads. Think of often updated partition keys – in combination with memory >> preassure you can easily end up with maaany sstables (ok they will be >> compacted some time in the future). >> >> >> >> Second, there could be fragmentation on disk which leads to seeks during >> sequential reads. >> >> >> >> Jan >> >> >> >> Gesendet von meinem Windows 10 Phone >> >> >> >> *Von: *preetika tyagi <preetikaty...@gmail.com> >> *Gesendet: *Montag, 20. März 2017 21:18 >> *An: *user@cassandra.apache.org >> *Betreff: *question on maximum disk seeks >> >> >> >> I'm trying to understand the maximum number of disk seeks required in a >> read operation in Cassandra. I looked at several online articles including >> this one: https://docs.datastax.com/en/cassandra/3.0/ >> cassandra/dml/dmlAboutReads.html >> >> As per my understanding, two disk seeks are required in the worst case. >> One is for reading the partition index and another is to read the actual >> data from the compressed partition. The index of the data in compressed >> partitions is obtained from the compression offset tables (which is stored >> in memory). Am I on the right track here? Will there ever be a case when >> more than 1 disk seek is required to read the data? >> >> Thanks, >> >> Preetika >> >> >> >> >> >>
Re: question on maximum disk seeks
Thank you Jan & Jeff for the responses. That was really useful. Jan - I have one follow-up question. When the data is spread over more than one SSTable in case of updates as you mentioned, we will need two seeks per SSTable (one for partition index and another for SSTable itself). I'm curious to know how partition index is structured internally. I was assuming it to be a table with <key, disk offset> pairs. In case of an update to the same key for several times, how it is recorded in the partition index? Thanks, Preetika On Mon, Mar 20, 2017 at 10:37 PM, <j.kes...@enercast.de> wrote: > Hi, > > > > youre right – one seek with hit in the partition key cache and two if not. > > > > Thats the theory – but two thinge to mention: > > > > First, you need two seeks per sstable not per entire read. So if you data > is spread over multiple sstables on disk you obviously need more then two > reads. Think of often updated partition keys – in combination with memory > preassure you can easily end up with maaany sstables (ok they will be > compacted some time in the future). > > > > Second, there could be fragmentation on disk which leads to seeks during > sequential reads. > > > > Jan > > > > Gesendet von meinem Windows 10 Phone > > > > *Von: *preetika tyagi <preetikaty...@gmail.com> > *Gesendet: *Montag, 20. März 2017 21:18 > *An: *user@cassandra.apache.org > *Betreff: *question on maximum disk seeks > > > > I'm trying to understand the maximum number of disk seeks required in a > read operation in Cassandra. I looked at several online articles including > this one: https://docs.datastax.com/en/cassandra/3.0/ > cassandra/dml/dmlAboutReads.html > > As per my understanding, two disk seeks are required in the worst case. > One is for reading the partition index and another is to read the actual > data from the compressed partition. The index of the data in compressed > partitions is obtained from the compression offset tables (which is stored > in memory). Am I on the right track here? Will there ever be a case when > more than 1 disk seek is required to read the data? > > Thanks, > > Preetika > > > > >
question on maximum disk seeks
I'm trying to understand the maximum number of disk seeks required in a read operation in Cassandra. I looked at several online articles including this one: https://docs.datastax.com/en/cassandra/3.0/cassandra/dml/dmlAboutReads.html As per my understanding, two disk seeks are required in the worst case. One is for reading the partition index and another is to read the actual data from the compressed partition. The index of the data in compressed partitions is obtained from the compression offset tables (which is stored in memory). Am I on the right track here? Will there ever be a case when more than 1 disk seek is required to read the data? Thanks, Preetika
Re: Row cache tuning
I see. Thanks, Arvydas! In terms of eviction policy in the row cache, does a write operation invalidates only the row(s) which are going be modified or the whole partition? In older version of Cassandra, I believe the whole partition gets invalidated even if only one row is modified. Is that still true for the latest release (3.10). I browsed through many online articles and tutorials but cannot find information on this. On Sun, Mar 12, 2017 at 2:25 PM, Arvydas Jonusonis < arvydas.jonuso...@gmail.com> wrote: > You can experiment quite easily without even needing to restart the > Cassandra service. > > The caches (row and key) can be enabled on a table-by-table basis via a > schema directive. But the cache capacity (which is the one that you > referred to in your original post, set to 0 in cassandra.yaml) is a global > setting and can be manipulated via JMX or nodetool (nodetool > setcachecapacity > <https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsSetCacheCapacity.html> > ). > > Arvydas > > On Sun, Mar 12, 2017 at 9:46 AM, preetika tyagi <preetikaty...@gmail.com> > wrote: > >> Thanks, Matija! That was insightful. >> >> I don't really have a use case in particular, however, what I'm trying to >> do is to figure out how the Cassandra performance can be leveraged by using >> different caching mechanisms, such as row cache, key cache, partition >> summary etc. Of course, it will also heavily depend on the type of workload >> but I'm trying to gain more understanding of what's available in the >> Cassandra framework. >> >> Also, I read somewhere that either row cache or key cache can be turned >> on for a specific table, not both. Based on your comment, I guess the >> combination of page cache and key cache is used widely for tuning the >> performance. >> >> Thanks, >> Preetika >> >> On Sat, Mar 11, 2017 at 2:01 PM, Matija Gobec <matija0...@gmail.com> >> wrote: >> >>> Hi, >>> >>> In 99% of use cases Cassandra's row cache is not something you should >>> look into. Leveraging page cache yields good results and if accounted for >>> can provide you with performance increase on read side. >>> I'm not a fan of a default row cache implementation and its invalidation >>> mechanism on updates so you really need to be careful when and how you use >>> it. There isn't much to configuration as there is to your use case. Maybe >>> explain what are you trying to solve with row cache and people can get into >>> discussion with more context. >>> >>> Regards, >>> Matija >>> >>> On Sat, Mar 11, 2017 at 9:15 PM, preetika tyagi <preetikaty...@gmail.com >>> > wrote: >>> >>>> Hi, >>>> >>>> I'm new to Cassandra and trying to get a better understanding on how >>>> the row cache can be tuned to optimize the performance. >>>> >>>> I came across think this article: https://docs.datastax >>>> .com/en/cassandra/3.0/cassandra/operations/opsConfiguringCaches.html >>>> >>>> And it suggests not to even touch row cache unless read workload is > >>>> 95% and mostly rely on machine's default cache mechanism which comes with >>>> OS. >>>> >>>> The default row cache size is 0 in cassandra.yaml file so the row cache >>>> won't be utilized at all. >>>> >>>> Therefore, I'm wondering how exactly I can decide to chose to tweak row >>>> cache if needed. Are there any good pointers one can provide on this? >>>> >>>> Thanks, >>>> Preetika >>>> >>> >>> >> >
Re: Row cache tuning
Thanks, Matija! That was insightful. I don't really have a use case in particular, however, what I'm trying to do is to figure out how the Cassandra performance can be leveraged by using different caching mechanisms, such as row cache, key cache, partition summary etc. Of course, it will also heavily depend on the type of workload but I'm trying to gain more understanding of what's available in the Cassandra framework. Also, I read somewhere that either row cache or key cache can be turned on for a specific table, not both. Based on your comment, I guess the combination of page cache and key cache is used widely for tuning the performance. Thanks, Preetika On Sat, Mar 11, 2017 at 2:01 PM, Matija Gobec <matija0...@gmail.com> wrote: > Hi, > > In 99% of use cases Cassandra's row cache is not something you should look > into. Leveraging page cache yields good results and if accounted for can > provide you with performance increase on read side. > I'm not a fan of a default row cache implementation and its invalidation > mechanism on updates so you really need to be careful when and how you use > it. There isn't much to configuration as there is to your use case. Maybe > explain what are you trying to solve with row cache and people can get into > discussion with more context. > > Regards, > Matija > > On Sat, Mar 11, 2017 at 9:15 PM, preetika tyagi <preetikaty...@gmail.com> > wrote: > >> Hi, >> >> I'm new to Cassandra and trying to get a better understanding on how the >> row cache can be tuned to optimize the performance. >> >> I came across think this article: https://docs.datastax >> .com/en/cassandra/3.0/cassandra/operations/opsConfiguringCaches.html >> >> And it suggests not to even touch row cache unless read workload is > 95% >> and mostly rely on machine's default cache mechanism which comes with OS. >> >> The default row cache size is 0 in cassandra.yaml file so the row cache >> won't be utilized at all. >> >> Therefore, I'm wondering how exactly I can decide to chose to tweak row >> cache if needed. Are there any good pointers one can provide on this? >> >> Thanks, >> Preetika >> > >
Row cache tuning
Hi, I'm new to Cassandra and trying to get a better understanding on how the row cache can be tuned to optimize the performance. I came across think this article: https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsConfiguringCaches.html And it suggests not to even touch row cache unless read workload is > 95% and mostly rely on machine's default cache mechanism which comes with OS. The default row cache size is 0 in cassandra.yaml file so the row cache won't be utilized at all. Therefore, I'm wondering how exactly I can decide to chose to tweak row cache if needed. Are there any good pointers one can provide on this? Thanks, Preetika
