[ 
https://issues.apache.org/jira/browse/CASSANDRA-11714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jeremy Hanna updated CASSANDRA-11714:
-------------------------------------
    Complexity: Low Hanging Fruit

> shutdown script should wait more than 30 seconds before force kill
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-11714
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11714
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Legacy/Tools
>            Reporter: Wei Deng
>            Assignee: Wei Deng
>            Priority: Low
>              Labels: lhf
>
> I'm running with 3.0.6 and it appears that if I let it run for multiple days 
> (ingesting a never-ending streaming data from Kafka), then the next time I'm 
> restarting the JVM by "service cassandra restart", it will take 9 minutes to 
> replay the commit log segments (the commitlog_total_space_in_mb is using the 
> default 8GB and I observed close to 8GB commitlog segment files before the 
> restart). Here is a fragment of the system.log showing how long it takes:
> {noformat}
> INFO  [main] 2016-05-04 19:06:54,356  CommitLog.java:168 - Replaying 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248187.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248191.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248195.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248198.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248201.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248205.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248209.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248213.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248217.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248221.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248225.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248230.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248234.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248238.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248242.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248246.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248250.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248254.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248258.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248262.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248266.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248270.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248274.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248278.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248282.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248286.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248290.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248294.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248299.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248303.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248307.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248311.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248315.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248319.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248323.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248327.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248331.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248335.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248340.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248344.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248348.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248352.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248356.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248360.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248364.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248368.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248372.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248376.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248380.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248384.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248388.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248392.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248396.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248397.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248401.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248405.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248409.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248413.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248417.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248421.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248425.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248429.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248433.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248437.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248441.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248445.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248449.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248453.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248457.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248461.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248466.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248470.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248474.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248478.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248482.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248486.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248490.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248494.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248498.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248502.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248506.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248510.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248514.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248518.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248522.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248526.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248528.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248529.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248530.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248531.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248532.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248533.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248534.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248535.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248536.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248537.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248538.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248539.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248540.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248541.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248542.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248543.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248544.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248545.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248546.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248547.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248548.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248549.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248550.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248551.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248552.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248553.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248554.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248555.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248556.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248557.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248558.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248559.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248560.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248561.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248562.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248563.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248564.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248565.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248566.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248567.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248568.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248569.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248570.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248571.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248572.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248573.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248574.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248575.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248576.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248577.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248578.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248579.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248580.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248581.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248582.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248583.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248584.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248585.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248586.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248587.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248588.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248589.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248590.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248591.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248592.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248593.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248594.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248595.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248596.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248597.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248598.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248599.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248600.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248601.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248602.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248603.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248604.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248605.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248606.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248607.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248608.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248609.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248610.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248611.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248612.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248613.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248614.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248615.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248616.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248617.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248618.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248619.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248620.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248621.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248622.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248623.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248624.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248625.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248626.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248627.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248628.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248629.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248630.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248631.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248632.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248633.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248634.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248635.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248636.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248637.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248638.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248639.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248640.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248641.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248642.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248643.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248644.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248645.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248646.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248647.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248648.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248649.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248650.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248651.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248652.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248653.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248654.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248655.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248656.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248657.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248658.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248659.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248660.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248661.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248662.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248663.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248664.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248665.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248666.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248667.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248668.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248669.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248670.log, 
> /mnt/ephemeral/cassandra/commitlog/CommitLog-6-1462146248671.log
> INFO  [main] 2016-05-04 19:15:44,172  CommitLog.java:170 - Log replay 
> complete, 331463 replayed mutations
> {noformat}
> It appears that the init script here only gives 30-second grace period after 
> SIGTERM before forcefully killing it.
> https://github.com/apache/cassandra/blob/cassandra-3.0/debian/init/#L101
> If we could increase this 30-second to a higher wait time like 1 or 2 minutes 
> we could make the startup delay much shorter.
> Keep in mind that we don't always have to wait for this long during the 
> shutdown so we're not really losing anything with this improvement. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org

Reply via email to