[
https://issues.apache.org/jira/browse/CASSANDRA-18153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700392#comment-17700392
]
Adriano Bonacin commented on CASSANDRA-18153:
---------------------------------------------
[~jlewandowski]
I have two nodes:
{{[root@ip-20-0-1-182 ~]# nodetool status}}
{{Datacenter: datacenter1}}
{{=======================}}
{{Status=Up/Down}}
{{|/ State=Normal/Leaving/Joining/Moving}}
{{-- Address Load Tokens Owns (effective) Host ID
Rack}}
{{UN 20.0.1.182 66.2 KiB 2 100.0%
1b0c03de-e885-4028-99c9-24122146f07d rack1}}
{{UN 20.0.1.239 66.2 KiB 2 100.0%
c54cf0ab-cf6e-46ce-9ebe-86b254fe5606 rack1}}
Creating a keyspace + table and writing something (wait to commitlog be flushed
to disk if commitlog_sync is periodic):
{{[root@ip-20-0-1-182 ~]# cqlsh $HOSTNAME}}
{{Connected to AnsibleLab at ip-20-0-1-182.ec2.internal:9042.}}
{{[cqlsh 5.0.1 | Cassandra 3.11.13 | CQL spec 3.4.4 | Native protocol v4]}}
{{Use HELP for help.}}
{{cqlsh> CREATE KEYSPACE abonacin WITH replication = \{'class':
'NetworkTopologyStrategy', 'datacenter1': 2};}}
{{cqlsh> CREATE TABLE abonacin.emp (}}
{{... emp_id int PRIMARY KEY,}}
{{... emp_city text,}}
{{... emp_name text,}}
{{... emp_phone varint,}}
{{... emp_sal varint,}}
{{... emp_hiredate date}}
{{... );}}
{{cqlsh> insert into abonacin.emp (emp_id, emp_city, emp_name, emp_phone,
emp_sal, emp_hiredate) values (1584, 'KVXKAAOHYH', 'KLEMTXEDBDIF', 60457532,
1000, '2022-08-25');}}
{{cqlsh> select * from abonacin.emp;}}
{{ emp_id | emp_city | emp_hiredate | emp_name | emp_phone | emp_sal}}
{{--------+------------+--------------+--------------+-----------+---------}}
{{ 1584 | KVXKAAOHYH | 2022-08-25 | KLEMTXEDBDIF | 60457532 | 1000}}
Now I'll kill -9 java process on the other node and restart cassandra service:
{{[root@ip-20-0-1-239 ~]# tail -f /var/log/cassandra/debug.log | grep "Ignored
commitLogIntervals from the following sstables" | grep abonacin &}}
{{[root@ip-20-0-1-239 ~]# ps -ef | grep -v grep | grep java | awk '\{print $2}'
| xargs kill -9}}{{[root@ip-20-0-1-239 ~]# systemctl restart cassandra}}
And I see this:
{{[root@ip-20-0-1-239 ~]# DEBUG [main] 2023-03-14 21:25:46,486
CommitLogReplayer.java:254 - Ignored commitLogIntervals from the following
sstables:
[/var/lib/cassandra/data/abonacin/emp-ee4bec70c2ac11ed85b715ea429a0971/me-1-big-Data.db]}}
What I found is that Storage Service is not yet available during
CommigLogReplay (not sure if it is always starts after commitlog replay or it's
just a coincidence). The following +StorageService.instance.getLocalHostUUID()+
returns {_}null{_}.
{{private static UUID getLocalHostId()}}
{{ {}}
{{ return StorageService.instance.getLocalHostUUID();}}
{{ }}}
I also find that{{
+Optional.ofNullable(StorageService.instance.getLocalHostUUID()).orElseGet(SystemKeyspace::getLocalHostId)+}}
returns the correct host uuid. This why I chose this approach instead of yours.
It's also worth to mention that it's the first time I tried to suggest a PR and
I don't have enough familiarity with all the context involved. :)
> Memtable being flushed without hostId in version "me" and newer during
> CommitLogReplay
> --------------------------------------------------------------------------------------
>
> Key: CASSANDRA-18153
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18153
> Project: Cassandra
> Issue Type: Bug
> Components: Local/SSTable
> Reporter: Adriano Bonacin
> Assignee: Adriano Bonacin
> Priority: Normal
> Time Spent: 50m
> Remaining Estimate: 0h
>
> On ticket CASSANDRA-16619 some files were changed to allow Cassandra to store
> HostID in the new "me" SSTable version.
> But SSTables flushed during CommitLogReplay miss this HostID info.
>
> In the next Cassandra startup, if these SSTables were still present,
> system.log will show:
> {{WARN Origin of 3 sstables is unknown or doesn't match the local node;
> commitLogIntervals for them were ignored}}
> {{WARN }}{{{}Origin of 3 sstables is unknown or doesn't match the local node;
> commitLogIntervals for them were ignored{}}}{{{}{}}}{{ }}
>
> And debug.log will show a list of SSTables, witch can include "md" and "me"
> version (before upgradesstables):
>
> {{Ignored commitLogIntervals from the following sstables:
> [/var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/me-3-big-Data.db,
>
> /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-1-big-Data.db,
>
> /var/lib/cassandra/data/system/compaction_history-b4dbb7b4dc493fb5b3bfce6e434832ca/md-2-big-Data.db]}}
>
> https://issues.apache.org/jira/browse/CASSANDRA-16619
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
