Thanks for the insights. I may first try disabling hinted handoff for one run of our data pipeline and see if it exhibits the same behavior. Will post back if I see anything enlightening there.
On Sep 10, 2011, at 5:04 PM, Chris Goffinet wrote: > You could tail the commit log with `strings` to see what keys are being > inserted. > > On Sat, Sep 10, 2011 at 2:24 PM, Jonathan Ellis <jbel...@gmail.com> wrote: > Two possibilities: > > 1) Hinted handoff (this will show up in the logs on the sending > machine, on the receiving one it will just look like any other write) > > 2) You have something doing writes that you're not aware of, I guess > you could track that down using wireshark to see where the write > messages are coming from > > On Sat, Sep 10, 2011 at 3:56 PM, Jeremy Hanna > <jeremy.hanna1...@gmail.com> wrote: > > Oh and we're running 0.8.4 and the RF is 3. > > > > On Sep 10, 2011, at 3:49 PM, Jeremy Hanna wrote: > > > >> In addition, the mutation stage and the read stage are backed up like: > >> > >> Pool Name Active Pending Blocked > >> ReadStage 32 773 0 > >> RequestResponseStage 0 0 0 > >> ReadRepairStage 0 0 0 > >> MutationStage 158 525918 0 > >> ReplicateOnWriteStage 0 0 0 > >> GossipStage 0 0 0 > >> AntiEntropyStage 0 0 0 > >> MigrationStage 0 0 0 > >> StreamStage 0 0 0 > >> MemtablePostFlusher 1 5 0 > >> FILEUTILS-DELETE-POOL 0 0 0 > >> FlushWriter 2 5 0 > >> MiscStage 0 0 0 > >> FlushSorter 0 0 0 > >> InternalResponseStage 0 0 0 > >> HintedHandoff 0 0 0 > >> CompactionManager n/a 29 > >> MessagingService n/a 0,34 > >> > >> On Sep 10, 2011, at 3:38 PM, Jeremy Hanna wrote: > >> > >>> We are experiencing massive writes to column families when only doing > >>> reads from Cassandra. A set of 5 hadoop jobs are reading from Cassandra > >>> and then writing out to hdfs. That is the only thing operating on the > >>> cluster. We are reading at CL.QUORUM with hadoop and have written with > >>> CL.QUORUM. Read repair chance is set to 0.0 on all column families. > >>> However, in the logs, I'm seeing flush after flush of memtables and > >>> compactions taking place. Is there something else that would be writing > >>> based on the above description? > >>> > >>> Jeremy > >> > > > > > > > > -- > Jonathan Ellis > Project Chair, Apache Cassandra > co-founder of DataStax, the source for professional Cassandra support > http://www.datastax.com >
