[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228557#comment-17228557
]
Aleksey Yeschenko commented on CASSANDRA-13019:
---
Sanity checked.
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17225253#comment-17225253
]
Marcus Eriksson commented on CASSANDRA-13019:
-
rebased on current trunk:
https://github.com/krummas/cassandra/commits/jeff/13019
cci run:
https://app.circleci.com/pipelines/github/krummas/cassandra?branch=jeff%2F13019
could someone sanity check before I commit?
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x, 4.0-beta
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17006592#comment-17006592
]
Jeremy Hanna commented on CASSANDRA-13019:
--
I like the idea to reduce the effect on the regular server operations when
performing the snapshot, especially if there is a coordinated snapshot across
the cluster.
Because it may affect time it takes for operations that call `snapshot`
indirectly, should we make a note of this in the NEWS.txt - both the
availability of the throttle and that it may affect time to run things like
`truncate` and `drop`?
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16976959#comment-16976959
]
Jeff Jirsa commented on CASSANDRA-13019:
Patch is approved by 3 people in GH PR (Aleksey, Chris, Maxwell)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 2h 10m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971282#comment-16971282
]
maxwellguo commented on CASSANDRA-13019:
I have made another comment on the code, just a very small point . Then +1 on
this patch. :)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971223#comment-16971223
]
Jeff Jirsa commented on CASSANDRA-13019:
All good suggestions. Here's how the nodetool command is setup:
{code}
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool | grep -i snapshot
clearsnapshotRemove the snapshot with the given name from
the given keyspaces. If no snapshotName is specified we will remove all
snapshots
getsnapshotthrottle Print the hardlink-per-second throttle for
snapshot/clearsnapshot
listsnapshotsLists all the snapshots along with the size on
disk and true size.
setsnapshotthrottle Set the hardlinks-per-second cap for snapshot
and clearsnapshot throttling
snapshot Take a snapshot of specified keyspaces or a
snapshot of the specified table
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool getsnapshotthrottle
Snapshot throttle is disabled
(oss-ccm) jjirsa:cassandra jjirsa$ time ccm node1 nodetool snapshot
Requested creating snapshot(s) for [all keyspaces] with snapshot name
[1573418615757] and options {skipFlush=false}
Snapshot directory: 1573418615757
real0m1.593s
user0m1.791s
sys 0m0.308s
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool setsnapshotthrottle 2
(oss-ccm) jjirsa:cassandra jjirsa$ ccm node1 nodetool getsnapshotthrottle
Current snapshot throttle: 2 links/s
(oss-ccm) jjirsa:cassandra jjirsa$ time ccm node1 nodetool snapshot
Requested creating snapshot(s) for [all keyspaces] with snapshot name
[1573418645021] and options {skipFlush=false}
Snapshot directory: 1573418645021
real1m53.393s
user0m1.750s
sys 0m0.371s
{code}
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13019) Improve clearsnapshot to delete the snapshot files slowly
[
https://issues.apache.org/jira/browse/CASSANDRA-13019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16971135#comment-16971135
]
maxwellguo commented on CASSANDRA-13019:
[~jjirsa] I have just review the code ,and left some comment. Looking forward
to your feedback. :)
If we can make a nodetool command , that we can set the rate dynamically. Once
we want to change the rate of making snapshot or delete file rate , restart the
node is too expensive .
I think we can open a new issue for this , for it is a new kind of problem.And
this issue should be fix after this issue. I have saw you have made two method
interface in StorageServiceMbean.:)
> Improve clearsnapshot to delete the snapshot files slowly
> --
>
> Key: CASSANDRA-13019
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13019
> Project: Cassandra
> Issue Type: Improvement
> Components: Legacy/Core
>Reporter: Dikang Gu
>Assignee: Jeff Jirsa
>Priority: Normal
> Labels: pull-request-available
> Fix For: 4.x
>
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In our environment, we are creating snapshots for backup, after we finish the
> backup, we are running {{clearsnapshot}} to delete the snapshot files. At
> that time we may have thousands of files to delete, and it's causing sudden
> disk usage spike. As a result, we are experiencing a spike of drop messages
> from Cassandra.
> I think we should implement something like {{slowrm}} to delete the snapshot
> files slowly, avoid the sudden disk usage spike.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
