[
https://issues.apache.org/jira/browse/CASSANDRA-18260?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brad Schoening updated CASSANDRA-18260:
---------------------------------------
Description:
When compaction fails, the log message should list a) the free space available
on disk at that point in time and b) perhaps the number and/or size of the
source sstables being compacted.
Free space can change from one moment to the next, so when the below compaction
failed because it needed 161GB, upon checking the server a few minutes later,
it had 184GB free. Similarly, the error message mentions it was writing out
one SSTable on this STCS table, but its not clear if it was combining X -> 1
tables, or something else.
[INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1
CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1
sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in
1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s,
Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375.
Partition merge counts were \{1:21375, }
[ERROR] [CompactionExecutor:77774] cluster_id=87 ip_address=127.1.1.1
CassandraDaemon.java:581 - Exception in thread
Thread[CompactionExecutor:77774,1,main]
java.lang.RuntimeException: Not enough space for compaction, estimated sstables
= 1, expected write size = 161228934093
at
org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
at
org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
was:
When compaction fails, the log message should list a) the free space available
on disk at that point in time and b) perhaps the number and/or size of the
source sstables being compacted.
Free space can change from one moment to the next, so when the below compaction
failed because it needed 161GB, upon checking the server a few minutes later,
it had 184GB free. Similarly, the error message mentions it was writing out
one SSTable on this STCS table, but its not clear if it was combining X -> 1
tables, or something else.
[INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1
CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1
sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in
1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s,
Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375.
Partition merge counts were \{1:21375, }
[ERROR] [CompactionExecutor:77774] cluster_id=87 ip_address=127.1.1.1
CassandraDaemon.java:581 - Exception in thread
Thread[CompactionExecutor:77774,1,main]
java.lang.RuntimeException: Not enough space for compaction, estimated sstables
= 1, expected write size = 161228934093
at
org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386)
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126)
at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
at
org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613)
at
org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
> Add details to Error message: Not enough space for compaction
> --------------------------------------------------------------
>
> Key: CASSANDRA-18260
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18260
> Project: Cassandra
> Issue Type: Improvement
> Components: Observability/Logging
> Reporter: Brad Schoening
> Priority: Normal
>
> When compaction fails, the log message should list a) the free space
> available on disk at that point in time and b) perhaps the number and/or size
> of the source sstables being compacted.
> Free space can change from one moment to the next, so when the below
> compaction failed because it needed 161GB, upon checking the server a few
> minutes later, it had 184GB free. Similarly, the error message mentions it
> was writing out one SSTable on this STCS table, but its not clear if it was
> combining X -> 1 tables, or something else.
> [INFO ] [CompactionExecutor:77758] cluster_id=87 ip_address=127.1.1.1
> CompactionTask.java:241 - Compacted (8a1cffe0-abb5-11ed-b3fc-8d2ac2c52f0d) 1
> sstables to [...] to level=0. 86.997GiB to 86.997GiB (~99% of original) in
> 1,508,155ms. Read Throughput = 59.069MiB/s, Write Throughput = 59.069MiB/s,
> Row Throughput = ~20,283/s. 21,375 total partitions merged to 21,375.
> Partition merge counts were \{1:21375, }
> [ERROR] [CompactionExecutor:77774] cluster_id=87 ip_address=127.1.1.1
> CassandraDaemon.java:581 - Exception in thread
> Thread[CompactionExecutor:77774,1,main]
> java.lang.RuntimeException: Not enough space for compaction, estimated
> sstables = 1, expected write size = 161228934093
> at
> org.apache.cassandra.db.compaction.CompactionTask.buildCompactionCandidatesForAvailableDiskSpace(CompactionTask.java:386)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:126)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:77)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:100)
> at
> org.apache.cassandra.db.compaction.CompactionManager$7.execute(CompactionManager.java:613)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:377)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
