[jira] [Commented] (CASSANDRA-8267) Only stream from unrepaired sstables during incremental repair
[ https://issues.apache.org/jira/browse/CASSANDRA-8267?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14291694#comment-14291694 ] Flavien Charlon commented on CASSANDRA-8267: Won't this be part of 2.1.3? I hope it will. Only stream from unrepaired sstables during incremental repair -- Key: CASSANDRA-8267 URL: https://issues.apache.org/jira/browse/CASSANDRA-8267 Project: Cassandra Issue Type: Bug Reporter: Marcus Eriksson Assignee: Marcus Eriksson Fix For: 3.0 Attachments: 0001-Only-stream-from-unrepaired-sstables-during-incremen.patch, 8267-trunk.patch Seems we stream from all sstables even if we do incremental repair, we should limit this to only stream from the unrepaired sstables if we do incremental repair -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14284160#comment-14284160
]
Flavien Charlon commented on CASSANDRA-8641:
Memory pressure with 32 GB of RAM? If that's the case, then you should probably
look into making some optimizations. That's a bug either way.
In the meantime we have disabled vnodes as they don't seem to be production
ready.
Repair causes a large number of tiny SSTables
-
Key: CASSANDRA-8641
URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04
Reporter: Flavien Charlon
Fix For: 2.1.3
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am
running 2.1.2 with all the default settings. I'm seeing some strange
behaviors during incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
Here is an excerpt of the system logs showing the abnormal flushing:
{noformat}
INFO [AntiEntropyStage:1] 2015-01-17 15:28:43,807 ColumnFamilyStore.java:840
- Enqueuing flush of customers: 634484 (0%) on-heap, 2599489 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:06,823 ColumnFamilyStore.java:840
- Enqueuing flush of levels: 129504 (0%) on-heap, 222168 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:07,940 ColumnFamilyStore.java:840
- Enqueuing flush of chain: 4508 (0%) on-heap, 6880 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:08,124 ColumnFamilyStore.java:840
- Enqueuing flush of invoices: 1469772 (0%) on-heap, 2542675 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:09,471 ColumnFamilyStore.java:840
- Enqueuing flush of customers: 809844 (0%) on-heap, 3364728 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,368 ColumnFamilyStore.java:840
- Enqueuing flush of levels: 28212 (0%) on-heap, 44220 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,822 ColumnFamilyStore.java:840
- Enqueuing flush of chain: 860 (0%) on-heap, 1130 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,985 ColumnFamilyStore.java:840
- Enqueuing flush of invoices: 334480 (0%) on-heap, 568959 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:27,375 ColumnFamilyStore.java:840
- Enqueuing flush of customers: 221568 (0%) on-heap, 929962 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:35,755 ColumnFamilyStore.java:840
- Enqueuing flush of invoices: 7916 (0%)
[jira] [Updated] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavien Charlon updated CASSANDRA-8641:
---
Description:
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am running
2.1.2 with all the default settings. I'm seeing some strange behaviors during
incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
Here is an excerpt of the system logs showing the abnormal flushing:
{noformat}
INFO [AntiEntropyStage:1] 2015-01-17 15:28:43,807 ColumnFamilyStore.java:840 -
Enqueuing flush of customers: 634484 (0%) on-heap, 2599489 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:06,823 ColumnFamilyStore.java:840 -
Enqueuing flush of levels: 129504 (0%) on-heap, 222168 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:07,940 ColumnFamilyStore.java:840 -
Enqueuing flush of chain: 4508 (0%) on-heap, 6880 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:08,124 ColumnFamilyStore.java:840 -
Enqueuing flush of invoices: 1469772 (0%) on-heap, 2542675 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:09,471 ColumnFamilyStore.java:840 -
Enqueuing flush of customers: 809844 (0%) on-heap, 3364728 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,368 ColumnFamilyStore.java:840 -
Enqueuing flush of levels: 28212 (0%) on-heap, 44220 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,822 ColumnFamilyStore.java:840 -
Enqueuing flush of chain: 860 (0%) on-heap, 1130 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:24,985 ColumnFamilyStore.java:840 -
Enqueuing flush of invoices: 334480 (0%) on-heap, 568959 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:27,375 ColumnFamilyStore.java:840 -
Enqueuing flush of customers: 221568 (0%) on-heap, 929962 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:35,755 ColumnFamilyStore.java:840 -
Enqueuing flush of invoices: 7916 (0%) on-heap, 11080 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:36,239 ColumnFamilyStore.java:840 -
Enqueuing flush of customers: 9968 (0%) on-heap, 33041 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:37,935 ColumnFamilyStore.java:840 -
Enqueuing flush of invoices: 42108 (0%) on-heap, 69494 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:41,182 ColumnFamilyStore.java:840 -
Enqueuing flush of customers: 40936 (0%) on-heap, 159099 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:49,573 ColumnFamilyStore.java:840 -
Enqueuing flush of levels: 17236 (0%) on-heap, 27048 (0%) off-heap
INFO [AntiEntropyStage:1] 2015-01-17 15:29:50,440 ColumnFamilyStore.java:840 -
Enqueuing flush of chain: 548
[jira] [Commented] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14281407#comment-14281407
]
Flavien Charlon commented on CASSANDRA-8641:
I have rebuilt the cluster, and repeated the same setup (constant insert load),
while doing a non-incremental, non-parallel repair, and the same thing happens:
tons of flushes, creating hundreds of SSTables on disk.
Then I did it again, with the insert load only (no repair), and everything is
normal: rare flushes, and the number of SSTables remains stable.
So this is definitely triggered by repair in some way.
Repair causes a large number of tiny SSTables
-
Key: CASSANDRA-8641
URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04
Reporter: Flavien Charlon
Fix For: 2.1.3
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am
running 2.1.2 with all the default settings. I'm seeing some strange
behaviors during incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
At the end of the repair, the cluster has become unusable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
Flavien Charlon created CASSANDRA-8641:
--
Summary: Repair causes a large number of tiny SSTables
Key: CASSANDRA-8641
URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04
Reporter: Flavien Charlon
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am running
2.1.2 with all the default settings. I'm seeing some strange behaviors during
incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475 CompactionTask.java:251
- Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32 total
partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
At the end of the repair, the cluster has become unusable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavien Charlon updated CASSANDRA-8641:
---
Since Version: (was: 2.1.2)
Repair causes a large number of tiny SSTables
-
Key: CASSANDRA-8641
URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04
Reporter: Flavien Charlon
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am
running 2.1.2 with all the default settings. I'm seeing some strange
behaviors during incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
At the end of the repair, the cluster has become unusable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8641) Repair causes a large number of tiny SSTables
[
https://issues.apache.org/jira/browse/CASSANDRA-8641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flavien Charlon updated CASSANDRA-8641:
---
Fix Version/s: 2.1.3
Repair causes a large number of tiny SSTables
-
Key: CASSANDRA-8641
URL: https://issues.apache.org/jira/browse/CASSANDRA-8641
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Ubuntu 14.04
Reporter: Flavien Charlon
Fix For: 2.1.3
I have a 3 nodes cluster with RF = 3, quad core and 32 GB or RAM. I am
running 2.1.2 with all the default settings. I'm seeing some strange
behaviors during incremental repair (under write load).
Taking the example of one particular column family, before running an
incremental repair, I have about 13 SSTables. After finishing the incremental
repair, I have over 114000 SSTables.
{noformat}
Table: customers
SSTable count: 114688
Space used (live): 97203707290
Space used (total): 99175455072
Space used by snapshots (total): 0
SSTable Compression Ratio: 0.28281112416526505
Memtable cell count: 0
Memtable data size: 0
Memtable switch count: 1069
Local read count: 0
Local read latency: NaN ms
Local write count: 11548705
Local write latency: 0.030 ms
Pending flushes: 0
Bloom filter false positives: 0
Bloom filter false ratio: 0.0
Bloom filter space used: 144145152
Compacted partition minimum bytes: 311
Compacted partition maximum bytes: 1996099046
Compacted partition mean bytes: 3419
Average live cells per slice (last five minutes): 0.0
Maximum live cells per slice (last five minutes): 0.0
Average tombstones per slice (last five minutes): 0.0
Maximum tombstones per slice (last five minutes): 0.0
{noformat}
Looking at the logs during the repair, it seems Cassandra is struggling to
compact minuscule memtables (often just a few kilobytes):
{noformat}
INFO [CompactionExecutor:337] 2015-01-17 01:44:27,011
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-228341,].
8,332 bytes to 6,547 (~78% of original) in 80,476ms = 0.78MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:337] 2015-01-17 01:45:35,519
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229348,].
8,384 bytes to 6,563 (~78% of original) in 6,880ms = 0.000910MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
INFO [CompactionExecutor:339] 2015-01-17 01:47:46,475
CompactionTask.java:251 - Compacted 32 sstables to
[/mnt/data/cassandra/data/business/customers-d9d42d209ccc11e48ca54553c90a9d45/business-customers-ka-229351,].
8,423 bytes to 6,401 (~75% of original) in 10,416ms = 0.000586MB/s. 32
total partitions merged to 32. Partition merge counts were {1:32, }
{noformat}
At the end of the repair, the cluster has become unusable.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-8529) Cassandra suddenly stops responding to clients though process is still running
Flavien Charlon created CASSANDRA-8529: -- Summary: Cassandra suddenly stops responding to clients though process is still running Key: CASSANDRA-8529 URL: https://issues.apache.org/jira/browse/CASSANDRA-8529 Project: Cassandra Issue Type: Bug Components: Core Reporter: Flavien Charlon I am running a moderate write-only load onto a 3 nodes cluster. After some time, nodes become completely unresponsive to clients, even though the process is still running. tpstats on affected nodes indicate pending compaction, which never gets executued. This is tpstats on the affected node hours after the load has stopped: Pool NameActive Pending Completed Blocked All time blocked CounterMutationStage 0 0 0 0 0 ReadStage 0 0 243384 0 0 RequestResponseStage 0 03336833 0 0 MutationStage32 19024775909 0 0 ReadRepairStage 0 0 14445 0 0 GossipStage 0 0 128499 0 0 CacheCleanupExecutor 0 0 0 0 0 AntiEntropyStage 0 0 0 0 0 MigrationStage0 0 36 0 0 ValidationExecutor0 0 0 0 0 CommitLogArchiver 0 0 0 0 0 MiscStage 0 0 0 0 0 MemtableFlushWriter 2 7947 0 0 MemtableReclaimMemory 0 0947 0 0 PendingRangeCalculator0 0 5 0 0 MemtablePostFlush 1 8 1241 0 0 CompactionExecutor2 8 1035 0 0 InternalResponseStage 0 0 9 0 0 HintedHandoff 0 0 6 0 0 Message type Dropped RANGE_SLICE 0 READ_REPAIR 0 PAGED_RANGE 0 BINARY 0 READ 0 MUTATION 0 _TRACE 0 REQUEST_RESPONSE 0 COUNTER_MUTATION 0 Also, compactionstats shows that compaction is stalled: compaction type keyspace table completed totalunit progress Compactiontestnet transactions 117833347 117834891 bytes 100.00% Compactiontestnetscripts 206418064 206419414 bytes 100.00% Active compaction remaining time : 0h00m00s And again, this has been like this for hours. I have reproduced this on several clusters with various memory configurations. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-8502) Static columns returning null when paging is used
Flavien Charlon created CASSANDRA-8502: -- Summary: Static columns returning null when paging is used Key: CASSANDRA-8502 URL: https://issues.apache.org/jira/browse/CASSANDRA-8502 Project: Cassandra Issue Type: Bug Components: Core Reporter: Flavien Charlon Attachments: null-static-column.txt When paging is used for a query containing a static column, the first page contains the right value for the static column, but subsequent pages have null null for the static column instead of the expected value. Repro steps: - Create a table with a static column - Create a partition with 500 cells - Using cqlsh, query that partition Actual result: - You will see that first, the static column appears as expected, but if you press a key after ---MORE---, the static columns will appear as null. See the attached file for a repro of the output. I am using a single node cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-8502) Static columns returning null for pages after first
[ https://issues.apache.org/jira/browse/CASSANDRA-8502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavien Charlon updated CASSANDRA-8502: --- Summary: Static columns returning null for pages after first (was: Static columns returning null when paging is used) Static columns returning null for pages after first --- Key: CASSANDRA-8502 URL: https://issues.apache.org/jira/browse/CASSANDRA-8502 Project: Cassandra Issue Type: Bug Components: Core Reporter: Flavien Charlon Attachments: null-static-column.txt When paging is used for a query containing a static column, the first page contains the right value for the static column, but subsequent pages have null null for the static column instead of the expected value. Repro steps: - Create a table with a static column - Create a partition with 500 cells - Using cqlsh, query that partition Actual result: - You will see that first, the static column appears as expected, but if you press a key after ---MORE---, the static columns will appear as null. See the attached file for a repro of the output. I am using a single node cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
