those will be removed on GC or restart, and are harmless
On Thu, Dec 3, 2009 at 6:20 PM, B. Todd Burruss <[email protected]> wrote: > another note on this, i stopped my client and after about 35 minutes the > compaction did complete, no more pending in compaction-pool. however > the Index, Data, and Filter files still exist with lots of data in them. > "Compact" files exist for all but 4 of the Data files - these "compact" > files are zero length. > > thx! > > > On Thu, 2009-12-03 at 15:40 -0800, B. Todd Burruss wrote: >> in my situation it seems like the compaction process is being starved. >> i'm hitting the server hard for the last 45 minutes and the compaction >> pool is sitting at 1 active, 25 pending, and 7 completed. it has been >> at 1 active and 7 completed for about 20 minutes. the pending have been >> growing steadily since then. and as i was typing it finally finished >> another compaction, so they must be just taking forever. >> >> snapshots of nodeprobe and iostats follow: >> >> Pool Name Active Pending Completed >> FILEUTILS-DELETE-POOL 0 0 116 >> MESSAGING-SERVICE-POOL 0 0 0 >> STREAM-STAGE 0 0 0 >> RESPONSE-STAGE 0 0 0 >> ROW-READ-STAGE 1 4 8652560 >> LB-OPERATIONS 0 0 0 >> COMMITLOG 1 0 14695623 >> MESSAGE-DESERIALIZER-POOL 0 0 0 >> GMFD 0 0 0 >> LB-TARGET 0 0 0 >> CONSISTENCY-MANAGER 0 0 0 >> ROW-MUTATION-STAGE 1 1 14692604 >> MESSAGE-STREAMING-POOL 0 0 0 >> LOAD-BALANCER-STAGE 0 0 0 >> FLUSH-SORTER-POOL 0 0 28 >> MEMTABLE-POST-FLUSHER 0 0 28 >> COMPACTION-POOL 1 25 7 >> FLUSH-WRITER-POOL 0 0 28 >> HINTED-HANDOFF-POOL 0 0 0 >> >> >> avg-cpu: %user %nice %system %iowait %steal %idle >> 61.85 0.00 26.68 7.73 0.00 3.74 >> >> Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn >> sda 246.00 11456.00 18528.00 11456 18528 >> sda2 23074.00 20.50 1854.00 20 1854 >> sda1 0.00 0.00 0.00 0 0 >> >> >> >> >> On Thu, 2009-12-03 at 17:05 -0600, Jonathan Ellis wrote: >> > Thanks for looking into this. Doesn't seem like there's much >> > low-hanging fruit to make compaction faster but I'll keep that in the >> > back of my mind. >> > >> > -Jonathan >> > >> > On Thu, Dec 3, 2009 at 4:58 PM, Freeman, Tim <[email protected]> wrote: >> > >>So this is working as designed, but the design is poor because it >> > >>causes confusion. If you can open a ticket for this that would be >> > >>great. >> > > >> > > Done, see: >> > > >> > > https://issues.apache.org/jira/browse/CASSANDRA-599 >> > > >> > >>What does iostat -x 10 (for instance) say about the disk activity? >> > > >> > > rkB/s is consistently high, and wkB/s varies. This is a typical entry >> > > with wkB/s at the high end of its range: >> > > >> > >>avg-cpu: %user %nice %sys %iowait %idle >> > >> 1.52 0.00 1.70 27.49 69.28 >> > >> >> > >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> > >>avgrq-sz avgqu-sz await svctm %util >> > >>sda 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 >> > >>13144.06 342.04 17.75 92.25 5.98 91.92 >> > >>sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> > >> 0.00 0.00 0.00 0.00 0.00 >> > >>sda2 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 >> > >>13144.06 342.04 17.75 92.25 5.98 91.92 >> > >>sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> > >> 0.00 0.00 0.00 0.00 0.00 >> > > >> > > and at the low end: >> > > >> > >>avg-cpu: %user %nice %sys %iowait %idle >> > >> 1.50 0.00 1.77 25.80 70.93 >> > >> >> > >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s >> > >>avgrq-sz avgqu-sz await svctm %util >> > >>sda 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00 >> > >> 235.33 6.13 56.63 6.21 90.81 >> > >>sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> > >> 0.00 0.00 0.00 0.00 0.00 >> > >>sda2 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00 >> > >> 235.33 6.13 56.63 6.21 90.81 >> > >>sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 >> > >> 0.00 0.00 0.00 0.00 0.00 >> > > >> > > Tim Freeman >> > > Email: [email protected] >> > > Desk in Palo Alto: (650) 857-2581 >> > > Home: (408) 774-1298 >> > > Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and >> > > Thursday; call my desk instead.) >> > > >> > > >> > > -----Original Message----- >> > > From: Jonathan Ellis [mailto:[email protected]] >> > > Sent: Thursday, December 03, 2009 2:45 PM >> > > To: [email protected] >> > > Subject: Re: Persistently increasing read latency >> > > >> > > On Thu, Dec 3, 2009 at 4:34 PM, Freeman, Tim <[email protected]> wrote: >> > >>>Can you tell if the system is i/o or cpu bound during compaction? >> > >> >> > >> It's I/O bound. It's using ~9% of 1 of 4 cores as I watch it, and all >> > >> it's doing right now is compactions. >> > > >> > > What does iostat -x 10 (for instance) say about the disk activity? >> > > >> >> > > >
