[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16234951#comment-16234951
]
Jeff Jirsa commented on CASSANDRA-13973:
----------------------------------------
Branch [here|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.0-13973]
for a concept I think works, but I want some more opinions.
[~slebresne] / [~iamaleksey] in particular, do you two feel this is safe?
Basically while we build the ColumnIndex, we have the list of
{{List<IndexHelper.IndexInfo> columnsIndex}} in memory in a sorted list. At the
time we call {{close()}}, we see if either: there are too many of those
objects, OR, if they use too much memory (large keys). If so, we downsample the
list, in place, by a constant factor.
In ascii art:
{code}
[a...c][d...f][e...h][i...k][l...n][o...p][q...r][s...u][v...x][y...z]
{code}
If we set a target size of 3, we'd downsample this by a factor of 3 into:
{code}
[a...c][d...k][l...r][s...z]
{code}
We'll keep the latest / last lastName+deletion marker on merge, and we'll keep
earliest/first firstName + offset, and recalculate the width.
Not to be ignored: this has an added bonus of really making large partitions
much less painful if you crank down those column index targets - little bit
more IO, but much less GC. Running this in a test cluster, I was able to do
reads/writes to 80GB partitions with 5GB heap, which is a pretty cool side
effect.
> IllegalArgumentException in upgradesstables compaction
> ------------------------------------------------------
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Reporter: Dan Kinder
> Assignee: Jeff Jirsa
> Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
