[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16069678#comment-16069678 ] ZhaoYang commented on CASSANDRA-13565: -- I will mark this ticket as `not an issue` and 13622 is better place to fix all boundary cases. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065884#comment-16065884 ] Kurt Greaves commented on CASSANDRA-13565: -- Certainly covered under CASSANDRA-13622. If anyone would like to fix it in this case fixing the overflow should be enough. Under either ticket is reasonable. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065875#comment-16065875 ] Krishna Dattu Koneru commented on CASSANDRA-13565: -- This can be closed as duplicate of https://issues.apache.org/jira/browse/CASSANDRA-13622 , which is opened to fix similar possible overflows. May be [~KurtG] can comment. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[
https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16065841#comment-16065841
]
ZhaoYang commented on CASSANDRA-13565:
--
That is likely due to INT-32 overflow.. a clearer error message should be
thrown.
{code}
/**
* size of commitlog segments to allocate
*/
public static int getCommitLogSegmentSize()
{
return conf.commitlog_segment_size_in_mb * 1024 * 1024;
}
{code}
> Materialized view usage of commit logs requires large mutation but
> commitlog_segment_size_in_mb=2048 causes exception
> -
>
> Key: CASSANDRA-13565
> URL: https://issues.apache.org/jira/browse/CASSANDRA-13565
> Project: Cassandra
> Issue Type: Bug
> Components: Configuration, Materialized Views, Streaming and
> Messaging
> Environment: Cassandra 3.9.0, Windows
>Reporter: Tania S Engel
> Attachments: CQLforTable.png
>
>
> We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another
> scenario (not applyunsafe during JOIN) which leads to :
> java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large
> for the maximum size of 512.000MiB
> at
> org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.Mutation.apply(Mutation.java:227)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165)
> ~[apache-cassandra-3.9.0.jar:3.9.0]
> at
> org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591)
> [apache-cassandra-3.9.0.jar:3.9.0]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0_66]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> [na:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [na:1.8.0_66]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_66]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66]
> Due to the relationship of max_mutation_size_in_kb and
> commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and
> left Cassandra to calculate max_mutation_size_in_kb as half the size
> commitlog_segment_size_in_mb * 1024.
> However, we have found that if we set commitlog_segment_size_in_mb=2048 we
> get an exception upon starting Cassandra, when it is creating a new commit
> log.
> ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005
> JVMStabilityInspector.java:82 - Exiting due to error while processing commit
> log during initialization.
> org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was
> made to move the file pointer before the beginning of the file
> Perhaps the index you are using is not big enough and it goes negative.
> Is the relationship between max_mutation_size_in_kb and
> commitlog_segment_size_in_mb important to preserve? In our limited stress
> test we are finding mutation size already over 512mb and we expect more data
> in our sstables and associated materialized views.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064975#comment-16064975 ] Tania S Engel commented on CASSANDRA-13565: --- Fabulous. Thanks for the explanation, I understand enough to fix it. We actually switched away from using MVs from these particular tables due to their heavy inserts, and the fact we kept running into memory issues when joining. How about my other question ...setting commitlog_segment_size_in_mb=2048 we get : ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 JVMStabilityInspector.java:82 - Exiting due to error while processing commit log during initialization. org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was made to move the file pointer before the beginning of the file > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064346#comment-16064346 ] ZhaoYang commented on CASSANDRA-13565: -- Could you also share the view table schema? "wide" partition means "too many row * bytes_per_row". Changing it to `LogHour ` will reduce the # of rows in that partition. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064032#comment-16064032 ] Tania S Engel commented on CASSANDRA-13565: --- To make an unwieldy "wide" partition, is the problem that we have too many columns with potentially large varchar data? I am not sure how changing the key to LogDay or LogHour would change the "width" of the partition. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16064016#comment-16064016 ] ZhaoYang commented on CASSANDRA-13565: -- there is only 1 partition key: LogMonth...Maybe you can consider change it to LogDay or LogHour depending on your use case.. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063647#comment-16063647 ] Tania S Engel commented on CASSANDRA-13565: --- All of our partition keys have small values (not varchar). Some are composite, with 1 or 2 partition keys and four or 5 clustering keys. Would the composite of partition and clustering keys (albeit not big individual values) contribute to a "wide" partition? Here is an example of the CQL: !CQLforTable.png! > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > Attachments: CQLforTable.png > > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-13565) Materialized view usage of commit logs requires large mutation but commitlog_segment_size_in_mb=2048 causes exception
[ https://issues.apache.org/jira/browse/CASSANDRA-13565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16063196#comment-16063196 ] ZhaoYang commented on CASSANDRA-13565: -- Could you share your data modeling? it seems your partition is too wide. In my understanding, c* rebuilds MV from each base table's partition key. > Materialized view usage of commit logs requires large mutation but > commitlog_segment_size_in_mb=2048 causes exception > - > > Key: CASSANDRA-13565 > URL: https://issues.apache.org/jira/browse/CASSANDRA-13565 > Project: Cassandra > Issue Type: Bug > Components: Configuration, Materialized Views, Streaming and > Messaging > Environment: Cassandra 3.9.0, Windows >Reporter: Tania S Engel > > We will be upgrading to 3.10 for CASSANDRA-11670. However, there is another > scenario (not applyunsafe during JOIN) which leads to : > java.lang.IllegalArgumentException: Mutation of 525.847MiB is too large > for the maximum size of 512.000MiB > at > org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:262) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:493) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.applyFuture(Mutation.java:215) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.Mutation.apply(Mutation.java:227) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.batchlog.BatchlogManager.store(BatchlogManager.java:147) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.service.StorageProxy.mutateMV(StorageProxy.java:797) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.buildKey(ViewBuilder.java:96) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.view.ViewBuilder.run(ViewBuilder.java:165) > ~[apache-cassandra-3.9.0.jar:3.9.0] > at > org.apache.cassandra.db.compaction.CompactionManager$14.run(CompactionManager.java:1591) > [apache-cassandra-3.9.0.jar:3.9.0] > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > [na:1.8.0_66] > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) > [na:1.8.0_66] > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) > [na:1.8.0_66] > at java.lang.Thread.run(Thread.java:745) [na:1.8.0_66] > Due to the relationship of max_mutation_size_in_kb and > commitlog_segment_size_in_mb, we increased commitlog_segment_size_in_mb and > left Cassandra to calculate max_mutation_size_in_kb as half the size > commitlog_segment_size_in_mb * 1024. > However, we have found that if we set commitlog_segment_size_in_mb=2048 we > get an exception upon starting Cassandra, when it is creating a new commit > log. > ERROR [COMMIT-LOG-ALLOCATOR] 2017-05-31 17:01:48,005 > JVMStabilityInspector.java:82 - Exiting due to error while processing commit > log during initialization. > org.apache.cassandra.io.FSWriteError: java.io.IOException: An attempt was > made to move the file pointer before the beginning of the file > Perhaps the index you are using is not big enough and it goes negative. > Is the relationship between max_mutation_size_in_kb and > commitlog_segment_size_in_mb important to preserve? In our limited stress > test we are finding mutation size already over 512mb and we expect more data > in our sstables and associated materialized views. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
