[
https://issues.apache.org/jira/browse/CASSANDRA-17619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17536341#comment-17536341
]
Paulo Motta commented on CASSANDRA-17619:
-----------------------------------------
Addressed review comments on github
[PR|https://github.com/apache/cassandra/pull/1615] and fixed two extra nits
found during testing:
* [fix
log|https://github.com/apache/cassandra/pull/1615/commits/333f813f6872b87f7e0f79f71a4f9688a9cc3193]
* [Add verbose log to
trace|https://github.com/apache/cassandra/pull/1615/commits/c2d7c679c3492b66d41f8c60ce29d38efdf1c88e]
This is what the output of nodetool listsnapshots looks like after doing the
following operations:
* global snapshot of all keyspaces
* drop keyspace ks2 with auto_snapshot_ttl unset
* drop keyspace ks1 with auto_snapshot_ttl=1 minute
* nodetool snapshots
{noformat}
grep -v "all_keyspaces_snapshot" snapshot.txt
Snapshot Details:
Snapshot name Keyspace name Column family name
True size Size on disk Creation time Expiration time
dropped-1652386867075-my_table ks2 my_table
5.81 KiB 5.81 KiB 2022-05-12T20:21:07.075Z
dropped-1652388565491-my_table ks my_table
5.83 KiB 5.83 KiB 2022-05-12T20:49:25.491Z 2022-05-12T20:50:25.491Z
dropped-1652388565604-my_table2 ks my_table2
5.83 KiB 5.83 KiB 2022-05-12T20:49:25.604Z 2022-05-12T20:50:25.604Z
dropped-1652386867016-my_table2 ks2 my_table2
5.81 KiB 5.81 KiB 2022-05-12T20:21:07.016Z
Total TrueDiskSpaceUsed: 0 byte
pmottagomes@C02FD3GTMD6R Documents %
{noformat}
After this I restarted the node and verified the snapshot from ks1 was removed
after 1 minute:
{noformat}
Snapshot Details:
Snapshot name Keyspace name Column family name
True size Size on disk Creation time Expiration time
dropped-1652386867075-my_table ks2 my_table
5.81 KiB 5.81 KiB 2022-05-12T20:21:07.075Z
dropped-1652386867016-my_table2 ks2 my_table2
5.81 KiB 5.81 KiB 2022-05-12T20:21:07.016Z
Total TrueDiskSpaceUsed: 0 bytes
{noformat}
This is the debug logs of the manual test above:
{noformat}
DEBUG [main] 2022-05-12 17:51:07,265 SnapshotManager.java:119 - Adding
snapshots:
system:peers:37f71aca-7dc2-383b-a706-72528af04d4f:all_keyspaces_snapshot,
system:transferred_ranges:6cad20f7-d4f5-3af2-b6e2-0da33c6c1f83:all_keyspaces_snapshot,
system_traces:sessions:c5e99f16-8677-3914-b17e-960613512345:all_keyspaces_snapshot,
system_schema:dropped_columns:5e7583b5-f3f4-3af1-9a39-b7e1d6f5f11f:all_keyspaces_snapshot,
system_auth:role_members:0ecdaa87-f8fb-3e60-88d1-74fb36fe5c0d:all_keyspaces_snapshot,
ks:my_table2:74a02c00-d230-11ec-b12d-d9d82874084f:dropped-1652388565604-my_table2,
system:repairs:a3d277d1-cfaf-36f5-a2a7-38d5eea9ad6a:all_keyspaces_snapshot,
ks2:my_table2:846d1fd0-d230-11ec-b12d-d9d82874084f:all_keyspaces_snapshot,
system:IndexInfo:9f5c6374-d485-3229-9a0a-5094af9ad1e3:all_keyspaces_snapshot,
system_schema:functions:96489b79-80be-3e14-a701-66a0b9159450:all_keyspaces_snapshot,
system:top_partitions:7e5a361c-317c-351f-b15f-ffd8afd3dd4b:all_keyspaces_snapshot,
system:peers_v2:c4325fbb-8e5e-3baf-bd07-0f9250ed818e:all_keyspaces_snapshot,
system_distributed:partition_denylist:d6123acc-8649-3496-9d4e-f3fe39a6018b:all_keyspaces_snapshot,
system_schema:views:9786ac1c-dd58-3201-a7cd-ad556410c985:all_keyspaces_snapshot,
system_auth:resource_role_permissons_index:5f2fbdad-91f1-3946-bd25-d5da3a5c35ec:all_keyspaces_snapshot,
system_schema:columns:24101c25-a2ae-3af7-87c1-b40ee1aca33f:all_keyspaces_snapshot,
system:transferred_ranges_v2:1ff78f1a-7df1-3a2a-a998-6f4932270af5:all_keyspaces_snapshot,
system:prepared_statements:18a9c257-6a0c-3841-ba71-8cd529849fef:all_keyspaces_snapshot,
system:local:7ad54392-bcdd-35a6-8417-4e047860b377:all_keyspaces_snapshot,
system_schema:types:5a8b1ca8-6602-3f77-a045-9273d308917a:all_keyspaces_snapshot,
ks2:my_table:8eada2d0-d230-11ec-b12d-d9d82874084f:dropped-1652386867075-my_table,
system:size_estimates:618f817b-005f-3678-b8a4-53f3930b8e86:all_keyspaces_snapshot,
ks2:my_table:8eada2d0-d230-11ec-b12d-d9d82874084f:all_keyspaces_snapshot,
system:peer_events:59dfeaea-8db2-3341-91ef-109974d81484:all_keyspaces_snapshot,
system_auth:roles:5bc52802-de25-35ed-aeab-188eecebb090:all_keyspaces_snapshot,
system_schema:indexes:0feb57ac-311f-382f-ba6d-9024d305702f:all_keyspaces_snapshot,
system_schema:keyspaces:abac5682-dea6-31c5-b535-b3d6cffd0fb6:all_keyspaces_snapshot,
system:table_estimates:176c39cd-b93d-33a5-a218-8eb06a56f66e:all_keyspaces_snapshot,
system_auth:network_permissions:d46780c2-2f1c-3db9-b4c1-b8d9fbc0cc23:all_keyspaces_snapshot,
ks:my_table:704a1850-d230-11ec-b12d-d9d82874084f:dropped-1652388565491-my_table,
system:available_ranges:c539fcab-d65a-31d1-8133-d25605643ee3:all_keyspaces_snapshot,
ks:my_table:704a1850-d230-11ec-b12d-d9d82874084f:all_keyspaces_snapshot,
system_schema:aggregates:924c5587-2e3a-345b-b10c-12f37c1ba895:all_keyspaces_snapshot,
system:view_builds_in_progress:6c22df66-c3bd-3df6-b74d-21179c6a9fe9:all_keyspaces_snapshot,
system:batches:919a4bc5-7a33-3573-b03e-13fc3f68b465:all_keyspaces_snapshot,
system:available_ranges_v2:4224a088-2ac9-3d0c-889d-fbb5f0facda0:all_keyspaces_snapshot,
system_distributed:view_build_status:5582b59f-8e4e-35e1-b913-3acada51eb04:all_keyspaces_snapshot,
system:peer_events_v2:0e65065f-e401-38ed-9507-b9213fae8d11:all_keyspaces_snapshot,
system:paxos:b7b7f0c2-fd0a-3410-8c05-3ef614bb7c2d:all_keyspaces_snapshot,
system:paxos_repair_history:ecb86667-40b2-3316-bb91-e612c8047457:all_keyspaces_snapshot,
system:built_views:4b3c50a9-ea87-3d76-9101-6dbc9c38494a:all_keyspaces_snapshot,
system_traces:events:8826e8e9-e16a-3728-8753-3bc1fc713c25:all_keyspaces_snapshot,
system:sstable_activity:5a1ff267-ace0-3f12-8563-cfae6103c65e:all_keyspaces_snapshot,
ks2:my_table2:846d1fd0-d230-11ec-b12d-d9d82874084f:dropped-1652386867016-my_table2,
system_schema:triggers:4df70b66-6b05-3251-95a1-32b54005fd48:all_keyspaces_snapshot,
system_distributed:repair_history:759fffad-624b-3181-80ee-fa9a52d1f627:all_keyspaces_snapshot,
system_distributed:parent_repair_history:deabd734-b99d-3b9c-92e5-fd92eb5abf14:all_keyspaces_snapshot,
system_schema:tables:afddfb9d-bc1e-3068-8056-eed6c302ba09:all_keyspaces_snapshot,
system:compaction_history:b4dbb7b4-dc49-3fb5-b3bf-ce6e434832ca:all_keyspaces_snapshot,
system:sstable_activity_v2:62efe31f-3be8-310c-8d29-8963439c1288:all_keyspaces_snapshot,
system_auth:role_permissions:3afbe79f-2194-31a7-add7-f5ab90d8ec9c:all_keyspaces_snapshot,
ks:my_table2:74a02c00-d230-11ec-b12d-d9d82874084f:all_keyspaces_snapshot.
DEBUG [main] 2022-05-12 17:51:07,266 SnapshotManager.java:104 - Adding expiring
snapshot TableSnapshot{keyspaceName='ks', tableName='my_table2',
tableId=74a02c00-d230-11ec-b12d-d9d82874084f,
tag='dropped-1652388565604-my_table2', createdAt=2022-05-12T20:49:25.604Z,
expiresAt=2022-05-12T20:50:25.604Z,
snapshotDirs=[/Users/pmottagomes/.ccm/test/node1/data0/ks/my_table2-74a02c00d23011ecb12dd9d82874084f/snapshots/dropped-1652388565604-my_table2]}
DEBUG [main] 2022-05-12 17:51:07,266 SnapshotManager.java:104 - Adding expiring
snapshot TableSnapshot{keyspaceName='ks', tableName='my_table',
tableId=704a1850-d230-11ec-b12d-d9d82874084f,
tag='dropped-1652388565491-my_table', createdAt=2022-05-12T20:49:25.491Z,
expiresAt=2022-05-12T20:50:25.491Z,
snapshotDirs=[/Users/pmottagomes/.ccm/test/node1/data0/ks/my_table-704a1850d23011ecb12dd9d82874084f/snapshots/dropped-1652388565491-my_table]}
INFO [main] 2022-05-12 17:51:07,266 SnapshotManager.java:128 - Scheduling
expired snapshot cleanup with initialDelaySeconds={} and cleanupPeriodSeconds={}
DEBUG [SnapshotCleanup:1] 2022-05-12 17:51:12,269 SnapshotManager.java:141 -
Removing expired snapshot TableSnapshot{keyspaceName='ks',
tableName='my_table', tableId=704a1850-d230-11ec-b12d-d9d82874084f,
tag='dropped-1652388565491-my_table', createdAt=2022-05-12T20:49:25.491Z,
expiresAt=2022-05-12T20:50:25.491Z,
snapshotDirs=[/Users/pmottagomes/.ccm/test/node1/data0/ks/my_table-704a1850d23011ecb12dd9d82874084f/snapshots/dropped-1652388565491-my_table]}.
DEBUG [SnapshotCleanup:1] 2022-05-12 17:51:12,274 SnapshotManager.java:141 -
Removing expired snapshot TableSnapshot{keyspaceName='ks',
tableName='my_table2', tableId=74a02c00-d230-11ec-b12d-d9d82874084f,
tag='dropped-1652388565604-my_table2', createdAt=2022-05-12T20:49:25.604Z,
expiresAt=2022-05-12T20:50:25.604Z,
snapshotDirs=[/Users/pmottagomes/.ccm/test/node1/data0/ks/my_table2-74a02c00d23011ecb12dd9d82874084f/snapshots/dropped-1652388565604-my_table2]}.
{noformat}
I will create another ticket to fix the order items are displayed in nodetool
listsnapshots. Also we should probably expose nodetool listsnapshots output via
a virtual table.
> Expired snapshots of dropped tables are not removed after node restart
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-17619
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17619
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Snapshots
> Reporter: Paulo Motta
> Assignee: Paulo Motta
> Priority: Normal
> Fix For: 4.1-beta
>
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> Expired TTL snapshots from a dropped table are only removed if the node is
> not restarted after the table is dropped.
> The reason for this is because {{SnapshotManager}} is using the old
> implementation of {{listsnapshots}} to load snapshots in memory during node
> startup, before CASSANDRA-16843 is fixed so it doesn't keep track of
> snapshots of dropped tables.
> We should make {{SnapshotManager}} use the new implementation to load
> snapshots of dropped tables in memory, what ensures they will be removed when
> expired.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
