[
https://issues.apache.org/jira/browse/CASSANDRA-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15802783#comment-15802783
]
Jon Haddad commented on CASSANDRA-12979:
----------------------------------------
Similar output for 3.0:
{code}DEBUG [SlabPoolCleaner] 2017-01-05 14:42:49,576
ColumnFamilyStore.java:854 - Enqueuing flush of standard1: 3393696 (32%)
on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,577 Memtable.java:364 -
Writing Memtable-standard1@1306908314(1.065MiB serialized bytes, 4002 ops,
32%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:3] 2017-01-05 14:42:49,628 Memtable.java:397 -
Completed flushing
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db
(878.008KiB) for commitlog position ReplayPosition(segmentId=1483655869016,
position=16570861)
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,205 CompactionTask.java:235 -
Compacted (48323b00-d398-11e6-a3e2-7b0b736bccfe) 32 sstables to
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big,]
to level=0. 28,817,720 bytes to 29,327,595 (~101% of original) in 4,091ms =
6.836708MB/s. 128,262 total partitions merged to 128,262. Partition merge
counts were {1:128262, }
DEBUG [CompactionExecutor:3] 2017-01-05 14:42:52,264 CompactionTask.java:235 -
Compacted (4347af80-d398-11e6-a3e2-7b0b736bccfe) 26 sstables to
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big,]
to level=0. 44,583,630 bytes to 45,115,065 (~101% of original) in 12,399ms =
3.470044MB/s. 196,716 total partitions merged to 196,716. Partition merge
counts were {1:196716, }
WARN [CompactionExecutor:4] 2017-01-05 14:42:52,264 CompactionTask.java:98 -
insufficient space to compact all requested files. 112.973495MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db')
WARN [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:319 -
Not enough space for compaction, 112.973495MB estimated. Reducing scope.
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:52,265 CompactionTask.java:158 -
Compacting (4aabc681-d398-11e6-a3e2-7b0b736bccfe)
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-318-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-319-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-320-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-323-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-293-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-325-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-324-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-322-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-300-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-321-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-258-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:235 -
Compacted (4aabc681-d398-11e6-a3e2-7b0b736bccfe) 11 sstables to
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big,]
to level=0. 73,346,235 bytes to 73,709,675 (~100% of original) in 4,129ms =
17.024709MB/s. 320,961 total partitions merged to 320,961. Partition merge
counts were {1:320961, }
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 -
insufficient space to compact all requested files. 227.1553MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 -
Not enough space for compaction, 227.1553MB estimated. Reducing scope.
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 -
insufficient space to compact all requested files. 156.86029MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:319 -
Not enough space for compaction, 156.86029MB estimated. Reducing scope.
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,395 CompactionTask.java:98 -
insufficient space to compact all requested files. 97.03554MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db')
WARN [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:319 -
Not enough space for compaction, 97.03554MB estimated. Reducing scope.
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:56,396 CompactionTask.java:158 -
Compacting (4d221db0-d398-11e6-a3e2-7b0b736bccfe)
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-284-big-Data.db:level=0,
]
DEBUG [CompactionExecutor:4] 2017-01-05 14:42:59,066 CompactionTask.java:235 -
Compacted (4d221db0-d398-11e6-a3e2-7b0b736bccfe) 1 sstables to
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big,]
to level=0. 45,115,065 bytes to 45,115,065 (~100% of original) in 2,670ms =
16.114262MB/s. 196,716 total partitions merged to 196,716. Partition merge
counts were {1:196716, }
WARN [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:98 -
insufficient space to compact all requested files. 227.1553MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-326-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN [CompactionExecutor:3] 2017-01-05 14:42:59,067 CompactionTask.java:319 -
Not enough space for compaction, 227.1553MB estimated. Reducing scope.
WARN [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:98 -
insufficient space to compact all requested files. 156.86029MB required,
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db'),
BigTableReader(path='/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-103-big-Data.db')
WARN [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:319 -
Not enough space for compaction, 156.86029MB estimated. Reducing scope.
DEBUG [CompactionExecutor:3] 2017-01-05 14:42:59,068 CompactionTask.java:158 -
Compacting (4eb9d4b0-d398-11e6-a3e2-7b0b736bccfe)
[/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-327-big-Data.db:level=0,
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-185-big-Data.db:level=0,
]
DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:1152 -
Flushing largest CFS(Keyspace='keyspace1', ColumnFamily='standard1') to free up
room. Used total: 0.33/0.00, live: 0.33/0.00, flushing: 0.00/0.00, this:
0.32/0.00
DEBUG [SlabPoolCleaner] 2017-01-05 14:43:04,830 ColumnFamilyStore.java:854 -
Enqueuing flush of standard1: 3378432 (32%) on-heap, 0 (0%) off-heap
DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,831 Memtable.java:364 -
Writing Memtable-standard1@279129348(1.060MiB serialized bytes, 3985 ops,
32%/0% of on/off-heap limit)
DEBUG [MemtableFlushWriter:4] 2017-01-05 14:43:04,897 Memtable.java:397 -
Completed flushing
/Volumes/tiny/keyspace1/standard1-07c5ebc0d39811e6a3e27b0b736bccfe/mc-329-big-Data.db
(877.544KiB) for commitlog position ReplayPosition(segmentId=1483655869016,
position=17624611){code}
> checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread
> scope
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-12979
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12979
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jon Haddad
> Assignee: Jon Haddad
> Fix For: 2.2.9, 3.0.11, 4.0, 3.x
>
> Attachments:
> 2.2-0001-Fixed-checkAvailableDiskSpace-to-properly-recalculat.patch
>
>
> If a compaction occurs that looks like it'll take up more space than
> remaining disk available, the compaction manager attempts to reduce the scope
> of the compaction by calling {{reduceScopeForLimitedSpace()}} repeatedly.
> Unfortunately, the while loop passes the {{estimatedWriteSize}} calculated
> from the original call to {{hasAvailableDiskSpace}}, so the comparisons that
> are done will always be against the size of the original compaction, rather
> than the reduced scope one.
> Full method below:
> {code}
> protected void checkAvailableDiskSpace(long estimatedSSTables, long
> expectedWriteSize)
> {
> if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType ==
> OperationType.COMPACTION)
> {
> logger.info("Compaction space check is disabled");
> return;
> }
> while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables,
> expectedWriteSize))
> {
> if (!reduceScopeForLimitedSpace())
> throw new RuntimeException(String.format("Not enough space
> for compaction, estimated sstables = %d, expected write size = %d",
> estimatedSSTables, expectedWriteSize));
>
> }
> }
> {code}
> I'm proposing to recalculate the {{estimatedSSTables}} and
> {{expectedWriteSize}} after each iteration of {{reduceScopeForLimitedSpace}}.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
