Wei Deng created CASSANDRA-11714:
------------------------------------
Summary: shutdown script should wait more than 30 seconds before
force kill
Key: CASSANDRA-11714
URL: https://issues.apache.org/jira/browse/CASSANDRA-11714
Project: Cassandra
Issue Type: Improvement
Components: Tools
Reporter: Wei Deng
Priority: Minor
I'm running with 3.0.6 and it appears that if I let it run for multiple days
(ingesting a never-ending streaming data from Kafka), then the next time I'm
restarting the JVM by "service cassandra restart", it will take 9 minutes to
replay the commit log segments (the commitlog_total_space_in_mb is using the
default 8GB and I observed close to 8GB commitlog segment files before the
restart). Here is a fragment of the system.log showing how long it takes:
{noformat}
INFO [main] 2016-05-04 19:06:54,356 CommitLog.java:168 - Replaying
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248425.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248429.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248433.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248437.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248441.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248445.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248449.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248453.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248457.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248461.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248466.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248470.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248474.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248478.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248482.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248486.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248490.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248494.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248498.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248502.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248506.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248510.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248514.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248518.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248522.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248526.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248528.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248529.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248530.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248531.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248532.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248533.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248534.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248535.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248536.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248537.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248538.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248539.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248540.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248541.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248542.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248543.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248544.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248545.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248546.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248547.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248548.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248549.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248550.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248551.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248552.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248553.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248554.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248555.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248556.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248557.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248558.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248559.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248560.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248561.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248562.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248563.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248564.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248565.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248566.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248567.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248568.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248569.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248570.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248571.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248572.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248573.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248574.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248575.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248576.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248577.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248578.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248579.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248580.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248581.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248582.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248583.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248584.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248585.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248586.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248587.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248588.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248589.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248590.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248591.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248592.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248593.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248594.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248595.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248596.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248597.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248598.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248599.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248600.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248601.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248602.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248603.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248604.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248605.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248606.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248607.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248608.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248609.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248610.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248611.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248612.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248613.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248614.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248615.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248616.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248617.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248618.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248619.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248620.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248621.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248622.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248623.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248624.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248625.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248626.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248627.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248628.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248629.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248630.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248631.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248632.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248633.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248634.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248635.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248636.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248637.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248638.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248639.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248640.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248641.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248642.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248643.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248644.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248645.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248646.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248647.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248648.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248649.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248650.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248651.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248652.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248653.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248654.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248655.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248656.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248657.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248658.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248659.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248660.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248661.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248662.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248663.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248664.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248665.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248666.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248667.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248668.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248669.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248670.log,
/mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248671.log
INFO [main] 2016-05-04 19:15:44,172 CommitLog.java:170 - Log replay complete,
331463 replayed mutations
{noformat}
It appears that the init script here only gives 30-second grace period after
SIGTERM before forcefully killing it.
https://github.com/apache/cassandra/blob/cassandra-3.0/debian/init/#L101
If we could increase this 30-second to a higher wait time like 1 or 2 minutes
we could make the startup delay much shorter.
Keep in mind that we don't always have to wait for this long during the
shutdown so we're not really losing anything with this improvement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
