[
https://issues.apache.org/jira/browse/CASSANDRA-5195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13566128#comment-13566128
]
Ryan McGuire commented on CASSANDRA-5195:
-----------------------------------------
I have reproduced this issue. Omid's patch works as he described: it does not
fully fix the issue, but does allow the keyspace to be loaded on the 2nd
restart of cassandra. Below is my verification workflow:
* Checkout/build 1.0.12
** cd $CASSANDRA_DIR
** git checkout -b 5195-1.0.12
** git reset --hard cassandra-1.0.12
** git clean -f -d
** ant build
* Run 1.0.12 test:
** sudo rm -rf /var/lib/cassandra
** sudo cassandra
** cd tool/stress
** ant build
** ./bin/stress
** sudo pkill -f CassandraDaemon
* Verify the keyspace/cf was created by stress:
** 10:20 PM:~/git/datastax/cassandra/tools/stress[5195-1.0.12*]$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 2.0.0 | Cassandra unknown | CQL spec unknown | Thrift protocol
19.20.0]
Use HELP for help.
cqlsh> use Keyspace1 ;
cqlsh:Keyspace1> select count(*) from Standard1;
count
-------
10000
* Checkout/build 1.1.9
** cd $CASSANDRA_DIR
** git checkout -b 5195-1.1.9
** git reset --hard cassandra-1.1.9
** git clean -f -d
** ant build
* Run 1.1.9 test:
** sudo ./bin/sstablescrub Keyspace1 Standard1
*** stdout: Unknown keyspace/columnFamily Keyspace1.Standard1
** sudo cassandra
*** log: INFO [main] 2013-01-29 22:28:44,800 CommitLogReplayer.java (line 103)
Skipped 585748 mutations from unknown (probably removed) CF with id 1000
* Verify that Keyspace1 does or does not exist:
** 10:30 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol
19.33.0]
Use HELP for help.
cqlsh> use Keyspace1 ;
Bad Request: Keyspace 'Keyspace1' does not exist
* Run 1.1.9 test again without the sstablescrub (restoring /var/lib/cassandra
from before):
** sudo pkill -f CassandraDaemon
** sudo cassandra
*** log: INFO 22:33:01,240 Replaying
/var/lib/cassandra/commitlog/CommitLog-1359515707503.log,
/var/lib/cassandra/commitlog/CommitLog-1359515946450.log
INFO 22:33:01,244 Replaying
/var/lib/cassandra/commitlog/CommitLog-1359515707503.log
INFO 22:33:02,318 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1')
liveRatio is 4.55084790673026 (just-counted was 4.55084790673026). calculation
took 866ms for 4590 columns
INFO 22:33:02,930 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1')
liveRatio is 5.226616220760892 (just-counted was 5.226616220760892).
calculation took 357ms for 11635 columns
INFO 22:33:04,186 CFS(Keyspace='Keyspace1', ColumnFamily='Standard1')
liveRatio is 5.094053078093754 (just-counted was 4.9614899354266155).
calculation took 859ms for 26720 columns
* Verify that Keyspace1 does or does not exist:
** 10:36 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol
19.33.0]
Use HELP for help.
cqlsh> use Keyspace1;
cqlsh:Keyspace1> select count(*) from Standard1;
count
-------
10000
* Apply patch and retest:
** cd $CASSANDRA_DIR
** git apply ~/Downloads/5195.patch
** ant clean build
** sudo rm -rf /var/lib/cassandra
** (restore /var/lib/cassandra from 1.0.12)
** sudo pkill -f CassandraDaemon
** sudo ./bin/sstablescrub Keyspace1 Standard1
*** stdout:
Pre-scrub sstables snapshotted into snapshot pre-scrub-1359517364042
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-17-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-17-Data.db')
complete: 63608 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-10-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-10-Data.db')
complete: 258153 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-18-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-18-Data.db')
complete: 65207 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-15-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-15-Data.db')
complete: 254487 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-5-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-5-Data.db')
complete: 243561 rows in new sstable and 0 empty (tombstoned) rows dropped
Scrubbing
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-16-Data.db')
Scrub of
SSTableReader(path='/var/lib/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hd-16-Data.db')
complete: 64230 rows in new sstable and 0 empty (tombstoned) rows dropped
* Verify that Keyspace1 does or does not exist:
** sudo cassandra
** 10:44 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol
19.33.0]
Use HELP for help.
cqlsh> use Keyspace1;
Bad Request: Keyspace 'Keyspace1' does not exist
* Restart cassandra as suggested:
** sudo pkill -f CassandraDaemon
** sudo cassandra
* Verify that Keyspace1 does or does not exist:
** 10:47 PM:~/git/datastax/cassandra[5195-1.1.9*]$ cqlsh
Connected to Test Cluster at localhost:9160.
[cqlsh 2.2.0 | Cassandra 1.1.9-SNAPSHOT | CQL spec 2.0.0 | Thrift protocol
19.33.0]
Use HELP for help.
cqlsh> use Keyspace1;
cqlsh:Keyspace1> select count(*) from Standard1;
count
-------
10000
> Offline scrub does not migrate the directory structure on migration from
> 1.0.x to 1.1.x and causes the keyspace to disappear
> ----------------------------------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-5195
> URL: https://issues.apache.org/jira/browse/CASSANDRA-5195
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.9
> Reporter: Omid Aladini
> Assignee: Ryan McGuire
> Fix For: 1.1.9
>
> Attachments: 5195.patch
>
>
> Due to CASSANDRA-4411, upon migration from 1.0.x to 1.1.x containing
> LCS-compacted sstables, an offline scrub should be run before Cassandra 1.1.x
> is started. But Cassandra 1.1.x uses a new directory structure
> (CASSANDRA-2749) that offline scrubber doesn't detect or try to migrate.
> How to reproduce:
> 1- Run cassandra 1.0.12.
> 2- Run stress tool, let Cassandra flush Keyspace1 or flush manually.
> 3- Stop cassandra 1.0.12
> 4- Run ./bin/sstablescrub Keyspace1 Standard1
> which returns "Unknown keyspace/columnFamily Keyspace1.Standard1" and
> notice the data directory isn't migrated.
> 5- Run cassandra 1.1.9. Keyspace1 doesn't get loaded and Cassandra doesn't
> try to migrate the directory structure. Also commitlog entries get skipped:
> "Skipped XXXXX mutations from unknown (probably removed) CF with id 1000"
> Without the unsuccessful step 4, Cassandra 1.1.9 loads and migrates the
> Keyspace correctly.
>
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
