[
https://issues.apache.org/jira/browse/CASSANDRA-12979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15726381#comment-15726381
]
Jon Haddad commented on CASSANDRA-12979:
----------------------------------------
I've written a small patch against trunk to deal with this issue. A couple
notes.
* I moved the calculation for the space taken up by this compaction into
{{checkAvailableDiskSpace()}}
* I improved the logging in reduceScopeForLimitedSpace to print out the disk
space that was estimated / required. This is what I had initially done that
lead to finding this issue
* checkAvailableDiskSpace() no longer accepts the estimates as args since
they're recalculated internally
* as far as I can tell there's no way to test this (it wasn't tested before).
I'm creating follow up tickets for refactorings that will help test these sorts
of edge cases
* this bug has existed at least since 2.2, and probably before that. the patch
is trivial enough to be backported.
I consider this a critical fix for anyone who's running low on disk space, as
compactions completely freeze up without this patch, so I'd be happy to supply
patches for 2.1 -> 3.x.
For initial review:
https://github.com/apache/cassandra/compare/trunk...rustyrazorblade:12979_fix_check_disk_space?expand=1
> checkAvailableDiskSpace doesn't update expectedWriteSize when reducing thread
> scope
> -----------------------------------------------------------------------------------
>
> Key: CASSANDRA-12979
> URL: https://issues.apache.org/jira/browse/CASSANDRA-12979
> Project: Cassandra
> Issue Type: Bug
> Reporter: Jon Haddad
> Assignee: Jon Haddad
> Fix For: 2.2.9, 3.0.11, 3.x
>
>
> If a compaction occurs that looks like it'll take up more space than
> remaining disk available, the compaction manager attempts to reduce the scope
> of the compaction by calling {{reduceScopeForLimitedSpace()}} repeatedly.
> Unfortunately, the while loop passes the {{estimatedWriteSize}} calculated
> from the original call to {{hasAvailableDiskSpace}}, so the comparisons that
> are done will always be against the size of the original compaction, rather
> than the reduced scope one.
> Full method below:
> {code}
> protected void checkAvailableDiskSpace(long estimatedSSTables, long
> expectedWriteSize)
> {
> if(!cfs.isCompactionDiskSpaceCheckEnabled() && compactionType ==
> OperationType.COMPACTION)
> {
> logger.info("Compaction space check is disabled");
> return;
> }
> while (!getDirectories().hasAvailableDiskSpace(estimatedSSTables,
> expectedWriteSize))
> {
> if (!reduceScopeForLimitedSpace())
> throw new RuntimeException(String.format("Not enough space
> for compaction, estimated sstables = %d, expected write size = %d",
> estimatedSSTables, expectedWriteSize));
>
> }
> }
> {code}
> I'm proposing to recalculate the {{estimatedSSTables}} and
> {{expectedWriteSize}} after each iteration of {{reduceScopeForLimitedSpace}}.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
