in my situation it seems like the compaction process is being starved.
i'm hitting the server hard for the last 45 minutes and the compaction
pool is sitting at 1 active, 25 pending, and 7 completed. it has been
at 1 active and 7 completed for about 20 minutes. the pending have been
growing steadily since then. and as i was typing it finally finished
another compaction, so they must be just taking forever.
snapshots of nodeprobe and iostats follow:
Pool Name Active Pending Completed
FILEUTILS-DELETE-POOL 0 0 116
MESSAGING-SERVICE-POOL 0 0 0
STREAM-STAGE 0 0 0
RESPONSE-STAGE 0 0 0
ROW-READ-STAGE 1 4 8652560
LB-OPERATIONS 0 0 0
COMMITLOG 1 0 14695623
MESSAGE-DESERIALIZER-POOL 0 0 0
GMFD 0 0 0
LB-TARGET 0 0 0
CONSISTENCY-MANAGER 0 0 0
ROW-MUTATION-STAGE 1 1 14692604
MESSAGE-STREAMING-POOL 0 0 0
LOAD-BALANCER-STAGE 0 0 0
FLUSH-SORTER-POOL 0 0 28
MEMTABLE-POST-FLUSHER 0 0 28
COMPACTION-POOL 1 25 7
FLUSH-WRITER-POOL 0 0 28
HINTED-HANDOFF-POOL 0 0 0
avg-cpu: %user %nice %system %iowait %steal %idle
61.85 0.00 26.68 7.73 0.00 3.74
Device: tps kB_read/s kB_wrtn/s kB_read kB_wrtn
sda 246.00 11456.00 18528.00 11456 18528
sda2 23074.00 20.50 1854.00 20 1854
sda1 0.00 0.00 0.00 0 0
On Thu, 2009-12-03 at 17:05 -0600, Jonathan Ellis wrote:
> Thanks for looking into this. Doesn't seem like there's much
> low-hanging fruit to make compaction faster but I'll keep that in the
> back of my mind.
>
> -Jonathan
>
> On Thu, Dec 3, 2009 at 4:58 PM, Freeman, Tim <[email protected]> wrote:
> >>So this is working as designed, but the design is poor because it
> >>causes confusion. If you can open a ticket for this that would be
> >>great.
> >
> > Done, see:
> >
> > https://issues.apache.org/jira/browse/CASSANDRA-599
> >
> >>What does iostat -x 10 (for instance) say about the disk activity?
> >
> > rkB/s is consistently high, and wkB/s varies. This is a typical entry with
> > wkB/s at the high end of its range:
> >
> >>avg-cpu: %user %nice %sys %iowait %idle
> >> 1.52 0.00 1.70 27.49 69.28
> >>
> >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> >>avgrq-sz avgqu-sz await svctm %util
> >>sda 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 13144.06
> >> 342.04 17.75 92.25 5.98 91.92
> >>sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>0.00 0.00 0.00 0.00 0.00
> >>sda2 3.10 3249.25 124.08 29.67 26299.30 26288.11 13149.65 13144.06
> >> 342.04 17.75 92.25 5.98 91.92
> >>sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>0.00 0.00 0.00 0.00 0.00
> >
> > and at the low end:
> >
> >>avg-cpu: %user %nice %sys %iowait %idle
> >> 1.50 0.00 1.77 25.80 70.93
> >>
> >>Device: rrqm/s wrqm/s r/s w/s rsec/s wsec/s rkB/s wkB/s
> >>avgrq-sz avgqu-sz await svctm %util
> >>sda 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00
> >>235.33 6.13 56.63 6.21 90.81
> >>sda1 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>0.00 0.00 0.00 0.00 0.00
> >>sda2 3.40 817.10 128.60 17.70 27828.80 6600.00 13914.40 3300.00
> >>235.33 6.13 56.63 6.21 90.81
> >>sda3 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
> >>0.00 0.00 0.00 0.00 0.00
> >
> > Tim Freeman
> > Email: [email protected]
> > Desk in Palo Alto: (650) 857-2581
> > Home: (408) 774-1298
> > Cell: (408) 348-7536 (No reception business hours Monday, Tuesday, and
> > Thursday; call my desk instead.)
> >
> >
> > -----Original Message-----
> > From: Jonathan Ellis [mailto:[email protected]]
> > Sent: Thursday, December 03, 2009 2:45 PM
> > To: [email protected]
> > Subject: Re: Persistently increasing read latency
> >
> > On Thu, Dec 3, 2009 at 4:34 PM, Freeman, Tim <[email protected]> wrote:
> >>>Can you tell if the system is i/o or cpu bound during compaction?
> >>
> >> It's I/O bound. It's using ~9% of 1 of 4 cores as I watch it, and all
> >> it's doing right now is compactions.
> >
> > What does iostat -x 10 (for instance) say about the disk activity?
> >
