[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359151#comment-16359151
]
Dinesh Joshi commented on CASSANDRA-13973:
--
Please disregard the earlier patch. [~jasobrown] and I looked over it. Looks
like 3.x is more likely to hit this overflow. 3.11 & trunk is a different
ballgame as it has the updated index file format (I think CASSANDRA-10314 is
the right ticket).
Here's a new patch for C* 3.0. This should be helpful to the user.
https://github.com/dineshjoshi/cassandra/tree/CASSANDRA-13973-3.0
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16358953#comment-16358953
]
Chris Lohfink commented on CASSANDRA-13973:
---
+1
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Dinesh Joshi
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16356358#comment-16356358
]
Dinesh Joshi commented on CASSANDRA-13973:
--
[~jjirsa] I have a patch that will log a meaningful error and warn the user
about this issue. Fixing this issue requires additional effort. Until then this
can be a helpful patch.
||13973||
|[branch|https://github.com/dineshjoshi/cassandra/tree/CASSANDRA-13973]|
|[utests
dtests|https://circleci.com/gh/dineshjoshi/workflows/cassandra/tree/CASSANDRA-13973]|
||
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16288381#comment-16288381
]
Dan Kinder commented on CASSANDRA-13973:
Yep [~jjirsa] that did the trick.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16275212#comment-16275212
]
Jeff Jirsa commented on CASSANDRA-13973:
[~dkinder] did changing {{column_index_size_in_kb}} work for you?
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236142#comment-16236142
]
Dan Kinder commented on CASSANDRA-13973:
Thanks [~jjirsa] I'll give it a shot.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16236109#comment-16236109
]
Jeff Jirsa commented on CASSANDRA-13973:
Thanks for the feedback [~slebresne]. I really appreciate you taking the time
to respond. The risky feeling here is why I'm moving slow on this myself - it
seems straightforward, but obviously the potential for unexpected surprises
here is pretty high.
Better messaging on that assert makes a lot of sense. Also good to see the
second confirmation that changing {{column_index_size_in_kb}} is a good
workaround, the drawback is that it's instance-wide, so if you have just a
handful of wide rows (like this user, the histogram shows their 99% size is
less than 1MB, but their max size is 394GB), you suffer a disk penalty on all
keyspaces/tables/rows in order to not crash on the one bad row.
[~dankinder] if you need to unblock yourself right now, changing
{{column_index_size_in_kb}} on your instance to 256k (7G of index data needs to
go under 2G in size, so multiplying factor of 4) PROBABLY works past this
issue, but expect a bit more disk IO (particularly reads) after the change
(+upgradesstables)
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16235374#comment-16235374
]
Sylvain Lebresne commented on CASSANDRA-13973:
--
bq. do you two feel this is safe?
I can't think of something this would hard break off the top of my head. That
said, and for what it's worth, my more complete initial reflexion is that:
* on 3.0/3.X, this feels a tad risky: we're adding new code to the file
indexing (granted, not excessively complex one) and it doesn't get a lot more
critical. It could also change the performance profile, and while it might be
on the good side in many cases, it may not always be (especially when the patch
rely on 2 new settings whose defaults may happen to not be the right ones for
someone's pratical workload). As few people will run into this problem in the
first place, asking those rare users to change {{column_index_size_in_kb}}
would probably be safer overall (tbc, I'm suggesting here to improve the error
message of the checked cast to point people to that work-around, not to leave
people on their own as done currently).
* on 4.0, we already have CASSANDRA-11206 (which is in fact in 3.11 as well) to
help work with large indexes, and things like CASSANDRA-9754 are supposed to
make that even better, so the benefit on memory of this aren't that clear.
CASSANDRA-11206 doesn't solve the {{AsssertionError}} of this ticket, but we
could move the index size from {{int}} to {{long}} (or varint) for that. Which
isn't necessarilly saying we shouldn't do this, but adding multiple ways to fix
the same problem, each with their own new config setting (CASSANDRA-11206 added
one, this patch adds 2) doesn't feel ideal, so that's to be taken into
consideration imo.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16234951#comment-16234951
]
Jeff Jirsa commented on CASSANDRA-13973:
Branch [here|https://github.com/jeffjirsa/cassandra/tree/cassandra-3.0-13973]
for a concept I think works, but I want some more opinions.
[~slebresne] / [~iamaleksey] in particular, do you two feel this is safe?
Basically while we build the ColumnIndex, we have the list of
{{List columnsIndex}} in memory in a sorted list. At the
time we call {{close()}}, we see if either: there are too many of those
objects, OR, if they use too much memory (large keys). If so, we downsample the
list, in place, by a constant factor.
In ascii art:
{code}
[a...c][d...f][e...h][i...k][l...n][o...p][q...r][s...u][v...x][y...z]
{code}
If we set a target size of 3, we'd downsample this by a factor of 3 into:
{code}
[a...c][d...k][l...r][s...z]
{code}
We'll keep the latest / last lastName+deletion marker on merge, and we'll keep
earliest/first firstName + offset, and recalculate the width.
Not to be ignored: this has an added bonus of really making large partitions
much less painful if you crank down those column index targets - little bit
more IO, but much less GC. Running this in a test cluster, I was able to do
reads/writes to 80GB partitions with 5GB heap, which is a pretty cool side
effect.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
>Priority: Major
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16223123#comment-16223123
]
Jeff Jirsa commented on CASSANDRA-13973:
The long term fix here is going to be 9754, but we can obviously do better than
we're doing. Linking 9754 and assigning myself so people know a fix is in the
works (waiting on formalities).
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>Assignee: Jeff Jirsa
> Fix For: 3.0.x, 3.11.x, 4.x
>
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216074#comment-16216074
]
Jeff Jirsa commented on CASSANDRA-13973:
Those knobs are actually for a slightly different index (the partition summary
itself). Though in yaml / {{Config.java}} there's {{column_index_size_in_kb}}
(defaults to 64), which I'm *pretty sure* does this, but not sure enough to
tell you to change it without testing in a lab (in particular, I don't know
exactly how it behaves if you change this with pre-existing data; I'd hope it'd
do the right thing, but I certainly haven't personally tested it).
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216043#comment-16216043
]
Dan Kinder commented on CASSANDRA-13973:
I see... messing with min/max_index_interval wouldn't help would it?
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216031#comment-16216031
]
Jeff Jirsa commented on CASSANDRA-13973:
OK, leaving some quick notes for whoever gets around to handling this (maybe
me, not self assigning because I don't have bandwidth right now, and to be
honest I haven't thought about the right fix yet).
The code here is trying to calculate the serialized size, so it can write the
index out as it rewrites that partition to a data file :
{code}
long size = TypeSizes.sizeofUnsignedVInt(headerLength)
+ DeletionTime.serializer.serializedSize(deletionTime)
+ TypeSizes.sizeofUnsignedVInt(columnsIndex.size()); //
number of entries
for (IndexHelper.IndexInfo info : columnsIndex)
size += idxSerializer.serializedSize(info);
size += columnsIndex.size() * TypeSizes.sizeof(0);
return Ints.checkedCast(size);
{code}
With 394GB and an index entry every 64k, you're going to write something like
{{6169617}} index markers, and the field there to handle it is a (signed)
integer (4 bytes), giving you a maximum size for all of the index markers of
{{2147483648}} , about 348 bytes per marker. The size of a marker is here:
{code}
long size = clusteringSerializer.serializedSize(info.firstName)
+ clusteringSerializer.serializedSize(info.lastName)
+ TypeSizes.sizeofUnsignedVInt(info.offset)
+ TypeSizes.sizeofVInt(info.width - WIDTH_BASE)
+ TypeSizes.sizeof(info.endOpenMarker != null);
if (info.endOpenMarker != null)
size +=
DeletionTime.serializer.serializedSize(info.endOpenMarker);
return size;
{code}
Note that it has both the first and last clustering within that marker - so for
you not to overflow, assuming no range tombstones which would take up even more
space, your clustering markers would have to average less than ~165 bytes each,
which clearly isn't happening, so we overflow that int and stop.
That's the short version of what's happening. I'm not sure why it's an {{int}}
instead of a {{long}} , and I'm not immediately sure why you're hitting it here
with {{upgradesstables}} when you didn't hit it previously.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
>
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16216010#comment-16216010
]
Dan Kinder commented on CASSANDRA-13973:
The partition key is a string, and there are 4 clustering columns (text, text,
text, timestamp). The partition key and two of the clustering columns are on
average small, on the order of 10 bytes. The third could be larger but have a
maximum of 2048 bytes.
Also, I don't recall running upgradesstables before, if that matters.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215992#comment-16215992
]
Jeff Jirsa commented on CASSANDRA-13973:
Ok, so a 394GB row on a 5TB table. Can you tell me just a little bit about your
clustering columns - how many and how long are they typically? I'm pretty sure
I see what the issue is, but I haven't yet checked the difference against 2.2
to see why it's only a problem now, and wasn't before.
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215984#comment-16215984
]
Dan Kinder commented on CASSANDRA-13973:
[~jjirsa]
This happens on about 12 out of 16 nodes, much greater than {{RF}}. (No real
need to anonymize the names.)
{noformat}
$ nodetool tablehistograms walker links
walker/links histograms
Percentile SSTables Write Latency Read LatencyPartition Size
Cell Count
(micros) (micros) (bytes)
50% 4.00 51.011155149.91 258
6
75% 5.00 73.463449259.15 1597
42
95% 5.00152.327152383.77 73457
2299
98% 5.00182.798582860.53315852
9887
99% 5.00219.34 53142810.15943127
24601
Min 0.00 8.24 88.1543
0
Max 5.001155149.91 53142810.15 394855527800
962624926
$ nodetool tablestats walker.links
Keyspace: walker
Read Count: 72932
Read Latency: 834.917616670323 ms.
Write Count: 15501219
Write Latency: 0.4392410533003888 ms.
Pending Flushes: 0
Table: links
SSTable count: 14269
SSTables in each level: [0, 9, 68, 349, 2843, 11000, 0, 0, 0]
Space used (live): 5310634842695
Space used (total): 5310634842695
Space used by snapshots (total): 12868175419329
Off heap memory used (total): 16339462
SSTable Compression Ratio: 0.0
Number of partitions (estimate): 20112302
Memtable cell count: 2216835
Memtable data size: 347672840
Memtable off heap memory used: 0
Memtable switch count: 4
Local read count: 80395
Local read latency: 747.603 ms
Local write count: 16891401
Local write latency: 0.118 ms
Pending flushes: 0
Bloom filter false positives: 178
Bloom filter false ratio: 0.00081
Bloom filter space used: 14912616
Bloom filter off heap memory used: 14798464
Index summary off heap memory used: 1540998
Compression metadata off heap memory used: 0
Compacted partition minimum bytes: 43
Compacted partition maximum bytes: 394855527800
Compacted partition mean bytes: 249068
Average live cells per slice (last five minutes):
11010.451219512195
Maximum live cells per slice (last five minutes): 11864
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
{noformat}
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
>
[jira] [Commented] (CASSANDRA-13973) IllegalArgumentException in upgradesstables compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-13973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16215703#comment-16215703
]
Jeff Jirsa commented on CASSANDRA-13973:
Is this happening on exactly 3 (or {{RF}}) nodes, or more than that?
Can you paste (anonymized) {{nodetool tablestats}} or {{nodetool
tablehistograms}} for that table?
> IllegalArgumentException in upgradesstables compaction
> --
>
> Key: CASSANDRA-13973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13973
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Dan Kinder
>
> After an upgrade from 2.2.6 to 3.0.15 (sstable version la to mc), when I try
> to run upgradesstables, most of them upgrade fine but I see the exception
> below on several nodes, and it doesn't complete.
> CASSANDRA-12717 looks similar but the stack trace is not the same, so I
> assumed it is not identical. The various nodes this happens on all give the
> same trace.
> Might be notable that this is an analytics cluster with some large
> partitions, in the GB size.
> {noformat}
> error: Out of range: 7316844981
> -- StackTrace --
> java.lang.IllegalArgumentException: Out of range: 7316844981
> at com.google.common.primitives.Ints.checkedCast(Ints.java:91)
> at
> org.apache.cassandra.db.RowIndexEntry$IndexedEntry.promotedSize(RowIndexEntry.java:329)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.serialize(RowIndexEntry.java:133)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter$IndexWriter.append(BigTableWriter.java:409)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.afterAppend(BigTableWriter.java:120)
> at
> org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:157)
> at
> org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:125)
> at
> org.apache.cassandra.db.compaction.writers.MaxSSTableSizeWriter.realAppend(MaxSSTableSizeWriter.java:88)
> at
> org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109)
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195)
> at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89)
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
> at
> org.apache.cassandra.db.compaction.CompactionManager$5.execute(CompactionManager.java:424)
> at
> org.apache.cassandra.db.compaction.CompactionManager$2.call(CompactionManager.java:311)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at
> org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
