Re: Key count mismatch in cluster for a column family
Sylvain - We have similar problem but the discrepancy is not that big. Do we have to do major compaction to fix it? We did not do 'nodetool compact', just did repair regularly, which triggers minor compaction. Thanks, Daning On 10/26/2011 03:23 AM, Sylvain Lebresne wrote: The estimate for the number of keys is computed by summing the key estimate for each sstable of the CF. For each sstable, the estimate should be fairly good. However, that's when we sum all the sstable estimates that we can loose potentially a lot of precision if there is a lot of rows that have parts in different sstables. But that in turn would suggest a problem with compaction lacking badly behind, especially with leveled compaction. -- Sylvain On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatungecumar...@gmail.com wrote: I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node on one of the CFs for the number of keys is much larger than expected. The first node shows the key count estimate to be 9.2M whereas the rest report ~650K on each node. The 650K is in the correct neighborhood of the number of keys that have been inserted. The counts are comparable for all other CFs across the cluster. I'm using Level compaction, but no compression. The 'nodetool ring' shows that the load is equal across all nodes. What could cause this large disparity in the number of keys? Is this just a stats issue or does this suggest a functional problem? 1st node: Column Family: uid SSTable count: 395 Space used (live): 1375262 Space used (total): 5482088532 Number of Keys (estimate): 9215104 Memtable Columns Count: 514952 Memtable Data Size: 295213448 Memtable Switch Count: 290 Read Count: 193102511 Read Latency: 0.146 ms. Write Count: 176934874 Write Latency: 0.018 ms. Pending Tasks: 0 Key cache capacity: 8302131 Key cache size: 8302131 Key cache hit rate: 0.8644664668071792 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 8944 2nd node: Column Family: uid SSTable count: 402 Space used (live): 13723958304 Space used (total): 4044833220 Number of Keys (estimate): 652928 Memtable Columns Count: 170290 Memtable Data Size: 102378904 Memtable Switch Count: 272 Read Count: 192463595 Read Latency: 0.289 ms. Write Count: 176527238 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8783058 Key cache size: 8783058 Key cache hit rate: 0.7865727464740025 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 12151 3rd node: Column Family: uid SSTable count: 401 Space used (live): 13204714872 Space used (total): 4030024144 Number of Keys (estimate): 675968 Memtable Columns Count: 42881 Memtable Data Size: 30992298 Memtable Switch Count: 304 Read Count: 190769879 Read Latency: 0.224 ms. Write Count: 175381826 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8920108 Key cache size: 8920108 Key cache hit rate: 0.8053563128870577 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 4866323 Compacted row mean size: 12074
Re: Key count mismatch in cluster for a column family
The estimate for the number of keys is computed by summing the key estimate for each sstable of the CF. For each sstable, the estimate should be fairly good. However, that's when we sum all the sstable estimates that we can loose potentially a lot of precision if there is a lot of rows that have parts in different sstables. But that in turn would suggest a problem with compaction lacking badly behind, especially with leveled compaction. -- Sylvain On Wed, Oct 26, 2011 at 3:58 AM, Terry Cumaranatunge cumar...@gmail.com wrote: I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node on one of the CFs for the number of keys is much larger than expected. The first node shows the key count estimate to be 9.2M whereas the rest report ~650K on each node. The 650K is in the correct neighborhood of the number of keys that have been inserted. The counts are comparable for all other CFs across the cluster. I'm using Level compaction, but no compression. The 'nodetool ring' shows that the load is equal across all nodes. What could cause this large disparity in the number of keys? Is this just a stats issue or does this suggest a functional problem? 1st node: Column Family: uid SSTable count: 395 Space used (live): 1375262 Space used (total): 5482088532 Number of Keys (estimate): 9215104 Memtable Columns Count: 514952 Memtable Data Size: 295213448 Memtable Switch Count: 290 Read Count: 193102511 Read Latency: 0.146 ms. Write Count: 176934874 Write Latency: 0.018 ms. Pending Tasks: 0 Key cache capacity: 8302131 Key cache size: 8302131 Key cache hit rate: 0.8644664668071792 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 8944 2nd node: Column Family: uid SSTable count: 402 Space used (live): 13723958304 Space used (total): 4044833220 Number of Keys (estimate): 652928 Memtable Columns Count: 170290 Memtable Data Size: 102378904 Memtable Switch Count: 272 Read Count: 192463595 Read Latency: 0.289 ms. Write Count: 176527238 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8783058 Key cache size: 8783058 Key cache hit rate: 0.7865727464740025 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 12151 3rd node: Column Family: uid SSTable count: 401 Space used (live): 13204714872 Space used (total): 4030024144 Number of Keys (estimate): 675968 Memtable Columns Count: 42881 Memtable Data Size: 30992298 Memtable Switch Count: 304 Read Count: 190769879 Read Latency: 0.224 ms. Write Count: 175381826 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8920108 Key cache size: 8920108 Key cache hit rate: 0.8053563128870577 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 4866323 Compacted row mean size: 12074
Re: Key count mismatch in cluster for a column family
Looks like compaction for this column family stopped after some time.
The last message for this column family in the system.log is
INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
119) Applying migration 43f106c0-ff54-11e0--68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=null,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
bytes, 1 ops)
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
119) Applying migration 47b1b840-ff54-11e0--68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=null,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:06,694
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1978429475(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:06,694 Memtable.java (line
237) Writing Memtable-Migrations@1978429475(8806/11007 serialized/live
bytes, 1 ops)
The schema for this CF is
create column family uid
with column_type = 'Standard'
and comparator = 'BytesType'
and default_validation_class = 'BytesType'
and key_validation_class = 'BytesType'
and rows_cached = 0.0
and row_cache_save_period = 0
and row_cache_keys_to_save = 2147483647
and keys_cached = 1.5E7
and key_cache_save_period = 14400
and read_repair_chance = 1.0
and gc_grace = 3600
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and row_cache_provider = 'SerializingCacheProvider'
and
Re: Key count mismatch in cluster for a column family
I did some more analysis and I think the compaction for this CF stopped after
we did a update column family to increase the key cache. Other CF compactions
were going on without any issues.
I did another update column family to the same CF with same values as before
and the compaction started again.
It it interesting to note that this happened on only one of the node in a 8
node cluster. Also it is the same node where the command was issued using the
cli.
Is it normal to issue update column family to increase the key cache when
cassandra is actively serving traffic in a production environment?
Are there any known issues/interactions with compaction and updating column
family in 1.0?
Thanks
Ramesh
Ramesh Natarajan rames...@gmail.com wrote:
Looks like compaction for this column family stopped after some time.
The last message for this column family in the system.log is
INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
119) Applying migration 43f106c0-ff54-11e0--68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=null,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
bytes, 1 ops)
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
119) Applying migration 47b1b840-ff54-11e0--68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@23de1953[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=null,comment=,rowCacheSize=0.0,keyCacheSize=1.5E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@4a50aa8a,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:06,694
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1978429475(8806/11007
Re: Key count mismatch in cluster for a column family
On Wed, Oct 26, 2011 at 3:50 PM, Ramesh Natarajan rames...@gmail.com wrote:
I did some more analysis and I think the compaction for this CF stopped after
we did a update column family to increase the key cache. Other CF compactions
were going on without any issues.
I did another update column family to the same CF with same values as before
and the compaction started again.
It it interesting to note that this happened on only one of the node in a 8
node cluster. Also it is the same node where the command was issued using the
cli.
Is it normal to issue update column family to increase the key cache when
cassandra is actively serving traffic in a production environment?
Are there any known issues/interactions with compaction and updating column
family in 1.0?
If the update happens to change and set to 0 the
min_compaction_threshold or max_compaction_threshold, then that would
disable compaction. Now of course it should not do that unless you've
asked it to, and if it did, it should disable compaction cluster-wide,
otherwise that would be a bug. But that's the only way I see for an
update column family to stop compaction.
--
Sylvain
Thanks
Ramesh
Ramesh Natarajan rames...@gmail.com wrote:
Looks like compaction for this column family stopped after some time.
The last message for this column family in the system.log is
INFO [MigrationStage:1] 2011-10-25 16:57:00,385 Migration.java (line
119) Applying migration 43f106c0-ff54-11e0--68877f281daf Update
column family to
org.apache.cassandra.config.CFMetaData@86c50e4[cfId=1000,ksName=MSA,cfName=uid,cfType=Standard,comparator=org.apache.cassandra.db.marshal.BytesType,subcolumncomparator=null,comment=,rowCacheSize=0.0,keyCacheSize=1.3E7,readRepairChance=1.0,replicateOnWrite=true,gcGraceSeconds=3600,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,rowCacheSavePeriodInSeconds=0,keyCacheSavePeriodInSeconds=14400,rowCacheKeysToSave=2147483647,rowCacheProvider=org.apache.cassandra.cache.SerializingCacheProvider@7f32ad0d,mergeShardsChance=0.1,keyAlias=java.nio.HeapByteBuffer[pos=0
lim=3 cap=3],column_metadata={},compactionStrategyClass=class
org.apache.cassandra.db.compaction.LeveledCompactionStrategy,compactionStrategyOptions={sstable_size_in_mb=10},compressionOptions={}]
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Migrations@1386279942(8806/11007 serialized/live bytes, 1
ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,389 Memtable.java (line
237) Writing Memtable-Migrations@1386279942(8806/11007 serialized/live
bytes, 1 ops)
INFO [MigrationStage:1] 2011-10-25 16:57:00,389
ColumnFamilyStore.java (line 664) Enqueuing flush of
Memtable-Schema@1156898891(4336/5420 serialized/live bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Migrations-h-48-Data.db (8870 bytes)
INFO [FlushWriter:307] 2011-10-25 16:57:00,402 Memtable.java (line
237) Writing Memtable-Schema@1156898891(4336/5420 serialized/live
bytes, 3 ops)
INFO [FlushWriter:307] 2011-10-25 16:57:00,413 Memtable.java (line
273) Completed flushing
/var/lib/cassandra/data/system/Schema-h-48-Data.db (4486 bytes)
INFO [CompactionExecutor:23] 2011-10-25 16:57:00,929
CompactionTask.java (line 119) Compacting
[SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9016-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9046-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9042-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9039-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9043-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9045-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9044-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9015-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9040-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9054-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9047-Data.db'),
SSTableReader(path='/var/lib/cassandra/data/MSA/uid-h-9041-Data.db')]
INFO [MigrationStage:1] 2011-10-25 16:57:06,693 Migration.java (line
119) Applying migration 47b1b840-ff54-11e0--68877f281daf Update
column family to
Key count mismatch in cluster for a column family
I have a cluster of 8 nodes all running 1.0. The stats shown on the 1st node on one of the CFs for the number of keys is much larger than expected. The first node shows the key count estimate to be 9.2M whereas the rest report ~650K on each node. The 650K is in the correct neighborhood of the number of keys that have been inserted. The counts are comparable for all other CFs across the cluster. I'm using Level compaction, but no compression. The 'nodetool ring' shows that the load is equal across all nodes. What could cause this large disparity in the number of keys? Is this just a stats issue or does this suggest a functional problem? 1st node: Column Family: uid SSTable count: 395 Space used (live): 1375262 Space used (total): 5482088532 Number of Keys (estimate): 9215104 Memtable Columns Count: 514952 Memtable Data Size: 295213448 Memtable Switch Count: 290 Read Count: 193102511 Read Latency: 0.146 ms. Write Count: 176934874 Write Latency: 0.018 ms. Pending Tasks: 0 Key cache capacity: 8302131 Key cache size: 8302131 Key cache hit rate: 0.8644664668071792 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 8944 2nd node: Column Family: uid SSTable count: 402 Space used (live): 13723958304 Space used (total): 4044833220 Number of Keys (estimate): 652928 Memtable Columns Count: 170290 Memtable Data Size: 102378904 Memtable Switch Count: 272 Read Count: 192463595 Read Latency: 0.289 ms. Write Count: 176527238 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8783058 Key cache size: 8783058 Key cache hit rate: 0.7865727464740025 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 7007506 Compacted row mean size: 12151 3rd node: Column Family: uid SSTable count: 401 Space used (live): 13204714872 Space used (total): 4030024144 Number of Keys (estimate): 675968 Memtable Columns Count: 42881 Memtable Data Size: 30992298 Memtable Switch Count: 304 Read Count: 190769879 Read Latency: 0.224 ms. Write Count: 175381826 Write Latency: 0.014 ms. Pending Tasks: 0 Key cache capacity: 8920108 Key cache size: 8920108 Key cache hit rate: 0.8053563128870577 Row cache: disabled Compacted row minimum size: 87 Compacted row maximum size: 4866323 Compacted row mean size: 12074
