[jira] [Comment Edited] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321703#comment-15321703 ] Chen Shen edited comment on CASSANDRA-10862 at 6/10/16 5:37 AM: [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. -And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification.- And sleep for a while to retry. Is this a right direction? -I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look.- I'm also planing to apply this patch to our production tier to see if this helps. was (Author: scv...@gmail.com): [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. -And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification.- And sleep for a while to retry. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland >Assignee: Chen Shen > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321703#comment-15321703 ] Chen Shen edited comment on CASSANDRA-10862 at 6/10/16 5:37 AM: [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. -And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification.- And sleep for a while to retry. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. was (Author: scv...@gmail.com): [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. -And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. - And sleep for a while to retry. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland >Assignee: Chen Shen > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321703#comment-15321703 ] Chen Shen edited comment on CASSANDRA-10862 at 6/10/16 5:36 AM: [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. -And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. - And sleep for a while to retry. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. was (Author: scv...@gmail.com): [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. ~~And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. ~~ Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland >Assignee: Chen Shen > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321703#comment-15321703 ] Chen Shen edited comment on CASSANDRA-10862 at 6/10/16 5:35 AM: [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. ~~And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. ~~ Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. was (Author: scv...@gmail.com): [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland >Assignee: Chen Shen > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11983) Migration task failed to complete
[ https://issues.apache.org/jira/browse/CASSANDRA-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323904#comment-15323904 ] Chris Love commented on CASSANDRA-11983: So how about `-Dcassandra.migration_task_wait_in_seconds=4` I am launch 1k nodes, and around 175 the boostrap process is slowing down. I am wondering if I need to let the ring settle down, or if there is another problem. I do not have any data in the keyspaces, but I would rather not turn off bootstrap. > Migration task failed to complete > - > > Key: CASSANDRA-11983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11983 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Docker / Kubernetes running > Linux cassandra-21 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) > x86_64 GNU/Linux > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14) > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > Cassnadra 3.5 installed from > deb-src http://www.apache.org/dist/cassandra/debian 35x main >Reporter: Chris Love > > When nodes are boostrapping I am getting mulitple errors: "Migration task > failed to complete", from MigrationManager.java > The errors increase as more nodes are added to the ring, as I am creating a > ring of 1k nodes. > Cassandra yaml i here > https://github.com/k8s-for-greeks/gpmr/blob/3d50ff91a139b9c4a7a26eda0fb4dcf9a008fbed/pet-race-devops/docker/cassandra-debian/files/cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sankalp kohli updated CASSANDRA-9754: - Status: Patch Available (was: Open) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (CASSANDRA-4663) Streaming sends one file at a time serially.
[ https://issues.apache.org/jira/browse/CASSANDRA-4663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] sankalp kohli reopened CASSANDRA-4663: -- The related JIRA is for stable loader whereas I opened it for streaming during bootstrap. This came up in NGCC today cc [~tjake] ...Close it if we already have a JIRA for this > Streaming sends one file at a time serially. > - > > Key: CASSANDRA-4663 > URL: https://issues.apache.org/jira/browse/CASSANDRA-4663 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Priority: Minor > > This is not fast enough when someone is using SSD and may be 10G link. We > should try to create multiple connections and send multiple files in > parallel. > Current approach under utilize the link(even 1G). > This change will improve the bootstrapping time of a node. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11988) NullPointerExpception when reading/compacting table
Nimi Wariboko Jr. created CASSANDRA-11988:
-
Summary: NullPointerExpception when reading/compacting table
Key: CASSANDRA-11988
URL: https://issues.apache.org/jira/browse/CASSANDRA-11988
Project: Cassandra
Issue Type: Bug
Reporter: Nimi Wariboko Jr.
Fix For: 3.6
I have a table that suddenly refuses to be read or compacted. Issuing a read on
the table causes a NPE.
On compaction, it returns the error
{code}
ERROR [CompactionExecutor:6] 2016-06-09 17:10:15,724 CassandraDaemon.java:213 -
Exception in thread Thread[CompactionExecutor:6,1,main]
java.lang.NullPointerException: null
at
org.apache.cassandra.db.transform.UnfilteredRows.isEmpty(UnfilteredRows.java:38)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:64)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.partitions.PurgeFunction.applyToPartition(PurgeFunction.java:24)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:76)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:182)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:82)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
~[apache-cassandra-3.6.jar:3.6]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:264)
~[apache-cassandra-3.6.jar:3.6]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_45]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_45]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_45]
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_45]
{code}
Schema:
{code}
CREATE TABLE cmpayments.report_payments (
reportid timeuuid,
userid timeuuid,
adjustedearnings decimal,
deleted set static,
earnings map,
gross map,
organizationid text,
payall timestamp static,
status text,
PRIMARY KEY (reportid, userid)
) WITH CLUSTERING ORDER BY (userid ASC)
AND bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '64', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.1
AND default_time_to_live = 0
AND gc_grace_seconds = 864000
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = '99PERCENTILE';
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323500#comment-15323500 ] Michael Kjellman commented on CASSANDRA-9754: - [~tjake] will do! > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Comment: was deleted (was: Integrate BirchIndexedEntry for CASSANDRA-9754) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Comment: was deleted (was: Initial refactor to abstract IndexedEntry implementations for CASSANDRA-9754) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Attachment: 9754_part2-v1.diff Integrate BirchIndexedEntry for CASSANDRA-9754 > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Attachment: (was: 9754_part1-v1.diff) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Attachment: 9754_part1-v1.diff Initial refactor to abstract IndexedEntry implementations for CASSANDRA-9754 > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Kjellman updated CASSANDRA-9754: Attachment: (was: 9754_part2-v1.diff) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11974) Failed assert causes OutboundTcpConnection to exit
[
https://issues.apache.org/jira/browse/CASSANDRA-11974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323462#comment-15323462
]
Sean Thornton commented on CASSANDRA-11974:
---
Agree. I was trying to focus on a single issue here but there are a number of
places in the code where critical threads can exit without proper handling and
it would be better, in my opinion, that these events are recognized and
handled, even if that is shutting the JVM down) vs. continuing to _appear_ to
be up and running normally (compaction thread, I'm looking at you). I think the
use of the Java assert keyword is possibly the root cause of this in a number
of places due to its raising of a true Error. Most people don't think or don't
know how to handle this appropriately (and really shouldn't). I would much
prefer to see something in the pattern of Spring's Assert or common-lang's
Validate be used.
I do think it's better for the community to provide concrete instances for the
developers to address one-by-one though. It's difficult to address more general
items without a larger effort and there are already a number of those.
> Failed assert causes OutboundTcpConnection to exit
> --
>
> Key: CASSANDRA-11974
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11974
> Project: Cassandra
> Issue Type: Bug
> Components: Streaming and Messaging
>Reporter: Sean Thornton
>
> I am seeing the following in a client's cluster:
> {noformat}
> ERROR [MessagingService-Outgoing-/10.0.0.1] 2016-06-06 03:38:19,305
> CassandraDaemon.java:229 - Exception in thread
> Thread[MessagingService-Outgoing-/10.0.0.1,5,main]
> java.lang.AssertionError: 635174
> at
> org.apache.cassandra.utils.ByteBufferUtil.writeWithShortLength(ByteBufferUtil.java:290)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:392)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.composites.AbstractCType$Serializer.serialize(AbstractCType.java:381)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:271)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.filter.ColumnSlice$Serializer.serialize(ColumnSlice.java:259)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:503)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.filter.SliceQueryFilter$Serializer.serialize(SliceQueryFilter.java:490)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.SliceFromReadCommandSerializer.serialize(SliceFromReadCommand.java:168)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:143)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.db.ReadCommandSerializer.serialize(ReadCommand.java:132)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:121)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:330)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:282)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> at
> org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:218)
> ~[cassandra-all-2.1.12.1046.jar:2.1.12.1046]
> {noformat}
> Obviously they somehow exceeded a 64K limit (quick and dirty suspects -
> https://docs.datastax.com/en/cql/3.1/cql/cql_reference/refLimits.html) but
> that is neither here nor there.
> The problem I see when this happens is
> {{ByteBufferUtil.writeWithShortLength}} can throw a
> {{java.lang.AssertionError}} which is a true {{Error}} that bubbles up and
> totally bypasses the {{catch (Exception e)}} clause in the message processing
> loop in {{OutboundTcpConnection.run()}} _which causes the thread to exit and
> that node to no longer communicate outgoing messages to other nodes_.
> At least from my perspective, there are two things I would like to see
> handled differently -
> * In the event of _any_ problem, I would like to see whatever details
> possible be logged about the problem Message - partition key, CF data,
> anything. Right now it can be very difficult to track this down
> * The {{java.lang.Error}} possibility needs to be handled somehow. If it's
> an assertion error, it
[jira] [Commented] (CASSANDRA-11873) Add duration type
[
https://issues.apache.org/jira/browse/CASSANDRA-11873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323457#comment-15323457
]
Benjamin Lerer commented on CASSANDRA-11873:
|[branch|https://github.com/apache/cassandra/compare/trunk...blerer:11873-trunk]|[utests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11873-trunk-testall/4/]|[dtests|http://cassci.datastax.com/view/Dev/view/blerer/job/blerer-11873-trunk-dtest/4/]|
The new patch serialize the duration as an unsigned long and accept {{us}} and
{{µs}} as microsecond symbol. The patch also add some testing for JSON. Adding
tests for MV was not possible due to the fact that the Java driver does not
recognize yet the {{Duration}} type.
> Add duration type
> -
>
> Key: CASSANDRA-11873
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11873
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: Benjamin Lerer
>Assignee: Benjamin Lerer
> Labels: client-impacting, doc-impacting
> Fix For: 3.x
>
>
> For CASSANDRA-11871 or to allow queries with {{WHERE}} clause like:
> {{... WHERE reading_time < now() - 2h}}, we need to support some duration
> type.
> In my opinion, it should be represented internally as a number of
> microseconds.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323406#comment-15323406 ] T Jake Luciani edited comment on CASSANDRA-9754 at 6/9/16 9:33 PM: --- [~mkjellman] for a > 2.1 patch, take a look at CASSANDRA-7443 which added an abstraction for IndexEntry and serializers which should hopefully be similar to what you did for this 2.1 version. was (Author: tjake): [~mkjellman] for a > 2.1 map take a look at CASSANDRA-7443 which added an abstraction for IndexEntry and serializers which should hopefully be similar to what you did for this 2.1 version. > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323406#comment-15323406 ] T Jake Luciani commented on CASSANDRA-9754: --- [~mkjellman] for a > 2.1 map take a look at CASSANDRA-7443 which added an abstraction for IndexEntry and serializers which should hopefully be similar to what you did for this 2.1 version. > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)
[
https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323402#comment-15323402
]
Branimir Lambov commented on CASSANDRA-8844:
bq. Thought I had that order fixed. Thanks for catching that.
My bad. You had fixed it in
[{{cab5c9de256348614cb1190875c44977e6289812}}|https://github.com/josh-mckenzie/cassandra/commit/cab5c9de256348614cb1190875c44977e6289812#diff-878dc31866184d5ef750ccd9befc8382R204]
and I looked at the individual commits, managed to miss it, and made you break
it again. Apologies.
Apart from that, the commit log changes look good to me.
> Change Data Capture (CDC)
> -
>
> Key: CASSANDRA-8844
> URL: https://issues.apache.org/jira/browse/CASSANDRA-8844
> Project: Cassandra
> Issue Type: New Feature
> Components: Coordination, Local Write-Read Paths
>Reporter: Tupshin Harper
>Assignee: Joshua McKenzie
>Priority: Critical
> Fix For: 3.x
>
>
> "In databases, change data capture (CDC) is a set of software design patterns
> used to determine (and track) the data that has changed so that action can be
> taken using the changed data. Also, Change data capture (CDC) is an approach
> to data integration that is based on the identification, capture and delivery
> of the changes made to enterprise data sources."
> -Wikipedia
> As Cassandra is increasingly being used as the Source of Record (SoR) for
> mission critical data in large enterprises, it is increasingly being called
> upon to act as the central hub of traffic and data flow to other systems. In
> order to try to address the general need, we (cc [~brianmhess]), propose
> implementing a simple data logging mechanism to enable per-table CDC patterns.
> h2. The goals:
> # Use CQL as the primary ingestion mechanism, in order to leverage its
> Consistency Level semantics, and in order to treat it as the single
> reliable/durable SoR for the data.
> # To provide a mechanism for implementing good and reliable
> (deliver-at-least-once with possible mechanisms for deliver-exactly-once )
> continuous semi-realtime feeds of mutations going into a Cassandra cluster.
> # To eliminate the developmental and operational burden of users so that they
> don't have to do dual writes to other systems.
> # For users that are currently doing batch export from a Cassandra system,
> give them the opportunity to make that realtime with a minimum of coding.
> h2. The mechanism:
> We propose a durable logging mechanism that functions similar to a commitlog,
> with the following nuances:
> - Takes place on every node, not just the coordinator, so RF number of copies
> are logged.
> - Separate log per table.
> - Per-table configuration. Only tables that are specified as CDC_LOG would do
> any logging.
> - Per DC. We are trying to keep the complexity to a minimum to make this an
> easy enhancement, but most likely use cases would prefer to only implement
> CDC logging in one (or a subset) of the DCs that are being replicated to
> - In the critical path of ConsistencyLevel acknowledgment. Just as with the
> commitlog, failure to write to the CDC log should fail that node's write. If
> that means the requested consistency level was not met, then clients *should*
> experience UnavailableExceptions.
> - Be written in a Row-centric manner such that it is easy for consumers to
> reconstitute rows atomically.
> - Written in a simple format designed to be consumed *directly* by daemons
> written in non JVM languages
> h2. Nice-to-haves
> I strongly suspect that the following features will be asked for, but I also
> believe that they can be deferred for a subsequent release, and to guage
> actual interest.
> - Multiple logs per table. This would make it easy to have multiple
> "subscribers" to a single table's changes. A workaround would be to create a
> forking daemon listener, but that's not a great answer.
> - Log filtering. Being able to apply filters, including UDF-based filters
> would make Casandra a much more versatile feeder into other systems, and
> again, reduce complexity that would otherwise need to be built into the
> daemons.
> h2. Format and Consumption
> - Cassandra would only write to the CDC log, and never delete from it.
> - Cleaning up consumed logfiles would be the client daemon's responibility
> - Logfile size should probably be configurable.
> - Logfiles should be named with a predictable naming schema, making it
> triivial to process them in order.
> - Daemons should be able to checkpoint their work, and resume from where they
> left off. This means they would have to leave some file artifact in the CDC
> log's directory.
> - A sophisticated daemon should be able to be written that could
> -- Catch up, in written-order, even when it is multiple logfiles behind in
>
[jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
[ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323330#comment-15323330 ] Michael Kjellman commented on CASSANDRA-9754: - Some additional thoughts while I'm thinking about them: * PageAlignedReader currently deserializes all the segments in the constructor. It might be more efficient to lazily deserialize the segments as we need the segment. I'm sure perf testing will quickly make it clear if the extra code complexity is worth the potential performance trade-off... * I picked 4kb for the page size based on an educated guess, but obviously other sizes need to be tested (less? more?) > Make index info heap friendly for large CQL partitions > -- > > Key: CASSANDRA-9754 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9754 > Project: Cassandra > Issue Type: Improvement >Reporter: sankalp kohli >Assignee: Michael Kjellman >Priority: Minor > Attachments: 9754_part1-v1.diff, 9754_part2-v1.diff > > > Looking at a heap dump of 2.0 cluster, I found that majority of the objects > are IndexInfo and its ByteBuffers. This is specially bad in endpoints with > large CQL partitions. If a CQL partition is say 6,4GB, it will have 100K > IndexInfo objects and 200K ByteBuffers. This will create a lot of churn for > GC. Can this be improved by not creating so many objects? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8844) Change Data Capture (CDC)
[ https://issues.apache.org/jira/browse/CASSANDRA-8844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323232#comment-15323232 ] Joshua McKenzie commented on CASSANDRA-8844: bq. I am worried about the cdcSizeCalculationExecutor task list size. I'm fairly certain this got lost in some refactoring somewhere - my original intent was to have a single-threaded DiscardPolicy executor but clearly that didn't happen. I believe only having a single recalc in flight is appropriate. A given write is going to be rejected regardless of the result of that async recalc call, so the in-flight recalc that causes discard of a new request should satisfy the intent of that submission. Nits: bq. I think it's safer to synchronize on the segment in processNewSegment/processDiscardedSegment and make setCDCState also synchronize if newState != cdcState. Originally I made cdcState volatile so as to work around the fact that we're essentially setting the state to CONTAINS on every successful CDC write and minimize the impact of repeatedly grabbing a lock for writes, and relying on the synchronization in CDCSizeTracker to keep allocations / discards from stomping over one another. I'm comfortable just using a synchronization on the setCDCState as well and synchronizing on the segment for the initial implementation and we can revisit this if it proves to be an issue. Added some comments in there to point that out; synchronizing on an object in two different classes makes me wary. bq. The setCDCstate comment is not what the code does. Old comment; removed. bq. We still have the same problem (as totalCDCSizeOnDisk does not sync). Since we use atomics that enforce happens-before, swapping the order in processDiscardedSegment will be sufficient to fix it. Thought I had that order fixed. Thanks for catching that. Fixed PascalCase that snuck in and rebased to curent trunk. [~carlyeks]: Going to have to chew on that one for a bit. May make sense in a v2 to either push pressure up to flushing a table if CDC is full and/or add some more intelligence into conflated non-CDC unflushed data that's causing CDC to hang around. > Change Data Capture (CDC) > - > > Key: CASSANDRA-8844 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8844 > Project: Cassandra > Issue Type: New Feature > Components: Coordination, Local Write-Read Paths >Reporter: Tupshin Harper >Assignee: Joshua McKenzie >Priority: Critical > Fix For: 3.x > > > "In databases, change data capture (CDC) is a set of software design patterns > used to determine (and track) the data that has changed so that action can be > taken using the changed data. Also, Change data capture (CDC) is an approach > to data integration that is based on the identification, capture and delivery > of the changes made to enterprise data sources." > -Wikipedia > As Cassandra is increasingly being used as the Source of Record (SoR) for > mission critical data in large enterprises, it is increasingly being called > upon to act as the central hub of traffic and data flow to other systems. In > order to try to address the general need, we (cc [~brianmhess]), propose > implementing a simple data logging mechanism to enable per-table CDC patterns. > h2. The goals: > # Use CQL as the primary ingestion mechanism, in order to leverage its > Consistency Level semantics, and in order to treat it as the single > reliable/durable SoR for the data. > # To provide a mechanism for implementing good and reliable > (deliver-at-least-once with possible mechanisms for deliver-exactly-once ) > continuous semi-realtime feeds of mutations going into a Cassandra cluster. > # To eliminate the developmental and operational burden of users so that they > don't have to do dual writes to other systems. > # For users that are currently doing batch export from a Cassandra system, > give them the opportunity to make that realtime with a minimum of coding. > h2. The mechanism: > We propose a durable logging mechanism that functions similar to a commitlog, > with the following nuances: > - Takes place on every node, not just the coordinator, so RF number of copies > are logged. > - Separate log per table. > - Per-table configuration. Only tables that are specified as CDC_LOG would do > any logging. > - Per DC. We are trying to keep the complexity to a minimum to make this an > easy enhancement, but most likely use cases would prefer to only implement > CDC logging in one (or a subset) of the DCs that are being replicated to > - In the critical path of ConsistencyLevel acknowledgment. Just as with the > commitlog, failure to write to the CDC log should fail that node's write. If > that means the requested consistency level was not met, then clients *should* > experience
[jira] [Updated] (CASSANDRA-11984) StorageService shutdown hook should use a volatile variable
[
https://issues.apache.org/jira/browse/CASSANDRA-11984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-11984:
Reviewer: Stefania
> StorageService shutdown hook should use a volatile variable
> ---
>
> Key: CASSANDRA-11984
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11984
> Project: Cassandra
> Issue Type: Bug
>Reporter: Edward Capriolo
>Assignee: Edward Capriolo
> Fix For: 3.8
>
>
> In StorageService.java there is a variable accessed from other threads that
> is not marked volatile.
> {noformat}
> private boolean inShutdownHook = false;
> public boolean isInShutdownHook()
>{
>return inShutdownHook;
>}
> drainOnShutdown = new Thread(new WrappedRunnable()
>{
>@Override
>public void runMayThrow() throws InterruptedException
>{
>inShutdownHook = true;
> {noformat}
> This is called from at least here:
> {noformat}
> ./src/java/org/apache/cassandra/concurrent/DebuggableScheduledThreadPoolExecutor.java:
> if (!StorageService.instance.isInShutdownHook())
> {noformat}
> This could cause issues in controlled shutdown like drain commands.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11966) When SEPWorker assigned work, set thread name to match pool
[ https://issues.apache.org/jira/browse/CASSANDRA-11966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323164#comment-15323164 ] Joshua McKenzie commented on CASSANDRA-11966: - [~snazy] to review. > When SEPWorker assigned work, set thread name to match pool > --- > > Key: CASSANDRA-11966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11966 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Chris Lohfink >Assignee: Chris Lohfink >Priority: Minor > Attachments: CASSANDRA-11966.patch, CASSANDRA-11966v2.patch > > > Currently in traces, logs, and stacktraces you cant really associate the > thread name with the pool since its just "SharedWorker-#". Calling setName > around the task could improve logging and tracing a little while being a > cheap operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11966) When SEPWorker assigned work, set thread name to match pool
[ https://issues.apache.org/jira/browse/CASSANDRA-11966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-11966: Reviewer: Robert Stupp > When SEPWorker assigned work, set thread name to match pool > --- > > Key: CASSANDRA-11966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11966 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Chris Lohfink >Assignee: Chris Lohfink >Priority: Minor > Attachments: CASSANDRA-11966.patch, CASSANDRA-11966v2.patch > > > Currently in traces, logs, and stacktraces you cant really associate the > thread name with the pool since its just "SharedWorker-#". Calling setName > around the task could improve logging and tracing a little while being a > cheap operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10869) paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest fails on 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-10869: Assignee: Alex Petrov (was: Sylvain Lebresne) > paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest > fails on 2.1 > -- > > Key: CASSANDRA-10869 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10869 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Alex Petrov > Labels: dtest > Fix For: 2.1.x > > > This test is failing hard on 2.1. Here is its history on the JDK8 job for > cassandra-2.1: > http://cassci.datastax.com/job/cassandra-2.1_dtest_jdk8/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > and on the JDK7 job: > http://cassci.datastax.com/job/cassandra-2.1_dtest/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > It fails because a read times out after ~1.5 minutes. If this is a test > error, it's specific to 2.1, because the test passes consistently on newer > versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10869) paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest fails on 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joshua McKenzie updated CASSANDRA-10869: Assignee: Sylvain Lebresne > paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest > fails on 2.1 > -- > > Key: CASSANDRA-10869 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10869 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey >Assignee: Sylvain Lebresne > Labels: dtest > Fix For: 2.1.x > > > This test is failing hard on 2.1. Here is its history on the JDK8 job for > cassandra-2.1: > http://cassci.datastax.com/job/cassandra-2.1_dtest_jdk8/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > and on the JDK7 job: > http://cassci.datastax.com/job/cassandra-2.1_dtest/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > It fails because a read times out after ~1.5 minutes. If this is a test > error, it's specific to 2.1, because the test passes consistently on newer > versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-11961:
Assignee: Achal Shah
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Achal Shah
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joshua McKenzie updated CASSANDRA-11961:
Reviewer: Robert Stupp
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Achal Shah
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323150#comment-15323150
]
Joshua McKenzie commented on CASSANDRA-11961:
-
[~snazy]: you ok taking review on this?
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Assignee: Achal Shah
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10664) Fix failing tests
[ https://issues.apache.org/jira/browse/CASSANDRA-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323143#comment-15323143 ] Russ Hatch commented on CASSANDRA-10664: +1 > Fix failing tests > - > > Key: CASSANDRA-10664 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10664 > Project: Cassandra > Issue Type: Test >Reporter: Sylvain Lebresne > Labels: dtest > Fix For: 3.0.x > > > This is the continuation of CASSANDRA-10166, just a meta-ticket to group all > tickets related to fixing any unit test or dtest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10664) Fix failing tests
[ https://issues.apache.org/jira/browse/CASSANDRA-10664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15323141#comment-15323141 ] Philip Thompson commented on CASSANDRA-10664: - Should we go ahead and close this, and break out the remaining failures into their own tickets? > Fix failing tests > - > > Key: CASSANDRA-10664 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10664 > Project: Cassandra > Issue Type: Test >Reporter: Sylvain Lebresne > Labels: dtest > Fix For: 3.0.x > > > This is the continuation of CASSANDRA-10166, just a meta-ticket to group all > tickets related to fixing any unit test or dtest. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10869) paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest fails on 2.1
[ https://issues.apache.org/jira/browse/CASSANDRA-10869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Philip Thompson updated CASSANDRA-10869: Issue Type: Bug (was: Sub-task) Parent: (was: CASSANDRA-10664) > paging_test.py:TestPagingWithDeletions.test_failure_threshold_deletions dtest > fails on 2.1 > -- > > Key: CASSANDRA-10869 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10869 > Project: Cassandra > Issue Type: Bug >Reporter: Jim Witschey > Labels: dtest > Fix For: 2.1.x > > > This test is failing hard on 2.1. Here is its history on the JDK8 job for > cassandra-2.1: > http://cassci.datastax.com/job/cassandra-2.1_dtest_jdk8/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > and on the JDK7 job: > http://cassci.datastax.com/job/cassandra-2.1_dtest/lastCompletedBuild/testReport/paging_test/TestPagingWithDeletions/test_failure_threshold_deletions/history/ > It fails because a read times out after ~1.5 minutes. If this is a test > error, it's specific to 2.1, because the test passes consistently on newer > versions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321703#comment-15321703 ] Chen Shen edited comment on CASSANDRA-10862 at 6/9/16 5:46 PM: --- [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/149d127c76f8f4e267524ed7f642d2ffdf6188e5 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. was (Author: scv...@gmail.com): [~pauloricardomg] I've done some investigation and I find it might not so easy to schedule a compaction on L0 table on reception as the only straightforward way to trigger a compaction is by submitting a task to CompactionManager.submitBackground, and 1) it's not guaranteed to be executed according to my knowledge 2) submitBackground need a `ColumnFamilyStore` as input, so we need either create a new CFS, or split the compaction strategy out of CompactionManager, each of which might need lots of work. So instead I am doing a different tricky approach: Don't add tables to CFS until the number of L0 sstables is smaller than a threshold. And subscribe to `SSTableListChangedNotification` so that the `OnCompletionRunnable` could sleep and wait on notification. Is this a right direction? I have a commit here https://github.com/scv119/cassandra/commit/3b48c092a7381d3074086476b12570db9b16dc16 if you want to take a look. I'm also planing to apply this patch to our production tier to see if this helps. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland >Assignee: Chen Shen > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11987) Cassandra support for JDK 9
shylaja kokoori created CASSANDRA-11987: --- Summary: Cassandra support for JDK 9 Key: CASSANDRA-11987 URL: https://issues.apache.org/jira/browse/CASSANDRA-11987 Project: Cassandra Issue Type: Bug Components: Core Environment: JDK9 Reporter: shylaja kokoori Hi, I tried to compile Cassandra with JDK 9 and ran into compilation issues because monitorEnter/monitorExit functions have been removed from sun.misc.Unsafe in JDK9 (http://mail.openjdk.java.net/pipermail/jdk9-hs-rt-changes/2015-January/000773.html). Is there a specific reason why Cassandra uses these Unsafe APIs instead of say java.util.concurrent.locks? Thanks, shylaja -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11904) Exception in thread Thread[CompactionExecutor:13358,1,main] java.lang.AssertionError: Memory was freed
[
https://issues.apache.org/jira/browse/CASSANDRA-11904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322869#comment-15322869
]
Dan Kinder commented on CASSANDRA-11904:
I am also seeing this exception, would be helpful to know if this causes any
issues. This happened on upgrade from 2.2.5 to Cassandra 3.5.0.
> Exception in thread Thread[CompactionExecutor:13358,1,main]
> java.lang.AssertionError: Memory was freed
> --
>
> Key: CASSANDRA-11904
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11904
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
>Reporter: Valentin Martinjuk
>
> We have Cassandra cluster 2.2.5 with two datacenters(3 nodes each).
> We observe ERRORs below on all nodes. The ERROR is repeated every minute.
> No any complains from customers. Do we have any chance to fix it without
> restart?
> {code}
> ERROR [CompactionExecutor:13996] 2016-05-26 21:20:46,700
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.findDroppableSSTable(LeveledCompactionStrategy.java:412)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.LeveledCompactionStrategy.getNextBackgroundTask(LeveledCompactionStrategy.java:101)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.WrappingCompactionStrategy.getNextBackgroundTask(WrappingCompactionStrategy.java:88)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:250)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[na:1.8.0_74]
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> ~[na:1.8.0_74]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [na:1.8.0_74]
> at java.lang.Thread.run(Thread.java:745) [na:1.8.0_74]
> ERROR [CompactionExecutor:13996] 2016-05-26 21:21:46,702
> CassandraDaemon.java:185 - Exception in thread
> Thread[CompactionExecutor:13996,1,main]
> java.lang.AssertionError: Memory was freed
> at
> org.apache.cassandra.io.util.SafeMemory.checkBounds(SafeMemory.java:103)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at org.apache.cassandra.io.util.Memory.getInt(Memory.java:292)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.getPositionInSummary(IndexSummary.java:148)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.fillTemporaryKey(IndexSummary.java:162)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.IndexSummary.binarySearch(IndexSummary.java:121)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getSampleIndexesForRanges(SSTableReader.java:1398)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.estimatedKeysForRanges(SSTableReader.java:1354)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionStrategy.worthDroppingTombstones(AbstractCompactionStrategy.java:403)
> ~[apache-cassandra-2.2.5.jar:2.2.5]
> at
>
[jira] [Commented] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322681#comment-15322681
]
Achal Shah commented on CASSANDRA-11961:
I actually don't seem to have permissions to assign tasks in Jira. Could you
assign it to me/grant me permission to do so?
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11920) bloom_filter_fp_chance needs to be validated up front
[
https://issues.apache.org/jira/browse/CASSANDRA-11920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322471#comment-15322471
]
Arindam Gupta commented on CASSANDRA-11920:
---
I am looking into this issue, I am very new to C* as this is the first issue I
am looking into, so my codebase knowledge is very limited. Following Tyler's
comment I feel validation should go into TableParams validate() method, but
BloomCalculations class is having default access modifier, so not accessible
from TableParams, is there any other place where bloom_filter_fp_chance minimum
supported value is kept?
> bloom_filter_fp_chance needs to be validated up front
> -
>
> Key: CASSANDRA-11920
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11920
> Project: Cassandra
> Issue Type: Bug
> Components: Lifecycle, Local Write-Read Paths
>Reporter: ADARSH KUMAR
> Labels: lhf
>
> Hi,
> I was doing some bench-marking on bloom_filter_fp_chance values. Everything
> worked fine for values .01(default for STCS), .001, .0001. But when I set
> bloom_filter_fp_chance = .1 i observed following behaviour:
> 1). Reads and writes looked normal from cqlsh.
> 2). SSttables are never created.
> 3). It just creates two files (*-Data.db and *-index.db) of size 0kb.
> 4). nodetool flush does not work and produce following exception:
> java.lang.UnsupportedOperationException: Unable to satisfy 1.0E-5 with 20
> buckets per element
> at
> org.apache.cassandra.utils.BloomCalculations.computeBloomSpec(BloomCalculations.java:150)
> .
> I checked BloomCalculations class and following lines are responsible for
> this exception:
> if (maxFalsePosProb < probs[maxBucketsPerElement][maxK]) {
> throw new UnsupportedOperationException(String.format("Unable to
> satisfy %s with %s buckets per element",
> maxFalsePosProb,
> maxBucketsPerElement));
> }
> From the code it looks like a hard coaded validation (unless we can change
> the nuber of buckets).
> So, if this validation is hard coaded then why it is even allowed to set such
> value of bloom_fileter_fp_chance, that can prevent ssTable generation?
> Please correct this issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11966) When SEPWorker assigned work, set thread name to match pool
[ https://issues.apache.org/jira/browse/CASSANDRA-11966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Lohfink updated CASSANDRA-11966: -- Attachment: CASSANDRA-11966v2.patch forgot to add property to disable, in v2 > When SEPWorker assigned work, set thread name to match pool > --- > > Key: CASSANDRA-11966 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11966 > Project: Cassandra > Issue Type: Improvement > Components: Observability >Reporter: Chris Lohfink >Assignee: Chris Lohfink >Priority: Minor > Attachments: CASSANDRA-11966.patch, CASSANDRA-11966v2.patch > > > Currently in traces, logs, and stacktraces you cant really associate the > thread name with the pool since its just "SharedWorker-#". Calling setName > around the task could improve logging and tracing a little while being a > cheap operation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11986) Repair using subranges (-st / -et) ignore Keyspace / Table name arguments
Alain RODRIGUEZ created CASSANDRA-11986:
---
Summary: Repair using subranges (-st / -et) ignore Keyspace /
Table name arguments
Key: CASSANDRA-11986
URL: https://issues.apache.org/jira/browse/CASSANDRA-11986
Project: Cassandra
Issue Type: Bug
Environment: Reproduced using ccm and Cassandra 2.1.12
Reporter: Alain RODRIGUEZ
When repairing, it is impossible to repair using subranges and a specific table
at the same time.
When running this:
{noformat}
date && echo "Repairing standard1 on 127.0.0.1" && time nodetool -h localhost
-p 7100 repair -dc datacenter1 -local -par -- keyspace1 standard1
{noformat}
*Without -st / -et* options, I have the following output:
{noformat}
MacBook-Pro:~ alain$ tail -100f ~/.ccm/test-2.1.12/node1/logs/system.log
INFO [Thread-33] 2016-06-09 14:18:52,193 StorageService.java:2939 - Starting
repair command #8, repairing 3 ranges for keyspace keyspace1
(parallelism=PARALLEL, full=true)
INFO [AntiEntropySessions:12] 2016-06-09 14:18:52,194 RepairSession.java:260 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync
/127.0.0.1, /127.0.0.2, /127.0.0.3 on range
(3074457345618258602,-9223372036854775808] for keyspace1.[standard1]
INFO [AntiEntropySessions:12] 2016-06-09 14:18:52,195 RepairJob.java:163 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for
standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1])
INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,433 RepairSession.java:171 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.2
INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,436 RepairSession.java:171 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.3
INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,439 RepairSession.java:171 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.1
INFO [AntiEntropySessions:13] 2016-06-09 14:18:57,439 RepairSession.java:260 -
[repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync
/127.0.0.1, /127.0.0.2, /127.0.0.3 on range
(-9223372036854775808,-3074457345618258603] for keyspace1.[standard1]
INFO [RepairJobTask:1] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair
#53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are
consistent for standard1
INFO [RepairJobTask:3] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair
#53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are
consistent for standard1
INFO [RepairJobTask:2] 2016-06-09 14:18:57,440 Differencer.java:67 - [repair
#53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are
consistent for standard1
INFO [AntiEntropySessions:13] 2016-06-09 14:18:57,440 RepairJob.java:163 -
[repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for
standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1])
INFO [AntiEntropyStage:1] 2016-06-09 14:18:57,440 RepairSession.java:237 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] standard1 is fully synced
INFO [AntiEntropySessions:12] 2016-06-09 14:18:57,440 RepairSession.java:299 -
[repair #53e6f820-2e3c-11e6-95ae-d1beb0ba4c9e] session completed successfully
INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,676 RepairSession.java:171 -
[repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.2
INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,684 RepairSession.java:171 -
[repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.3
INFO [AntiEntropyStage:1] 2016-06-09 14:19:03,758 RepairSession.java:171 -
[repair #57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Received merkle tree for
standard1 from /127.0.0.1
INFO [AntiEntropySessions:14] 2016-06-09 14:19:03,759 RepairSession.java:260 -
[repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] new session: will sync
/127.0.0.1, /127.0.0.2, /127.0.0.3 on range
(-3074457345618258603,3074457345618258602] for keyspace1.[standard1]
INFO [RepairJobTask:1] 2016-06-09 14:19:03,759 Differencer.java:67 - [repair
#57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.3 are
consistent for standard1
INFO [AntiEntropySessions:14] 2016-06-09 14:19:03,759 RepairJob.java:163 -
[repair #5acba5f0-2e3c-11e6-95ae-d1beb0ba4c9e] requesting merkle trees for
standard1 (to [/127.0.0.2, /127.0.0.3, /127.0.0.1])
INFO [RepairJobTask:3] 2016-06-09 14:19:03,760 Differencer.java:67 - [repair
#57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.3 and /127.0.0.1 are
consistent for standard1
INFO [RepairJobTask:2] 2016-06-09 14:19:03,760 Differencer.java:67 - [repair
#57074af0-2e3c-11e6-95ae-d1beb0ba4c9e] Endpoints /127.0.0.2 and /127.0.0.1 are
consistent for standard1
INFO [AntiEntropyStage:1] 2016-06-09
[jira] [Commented] (CASSANDRA-11944) sstablesInBounds might not actually give all sstables within the bounds due to having start positions moved in sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-11944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322365#comment-15322365 ] Benedict commented on CASSANDRA-11944: -- Probably - the earlier commits seemed to be for the other ticket. Thanks. I'll have a proper read of it later, but I would suggest renaming the methods to include the now implicit SSTableSet.LIVE, so it's still minimally front-and-centre when the functionality is used, by perhaps renaming select to selectLive, and sstablesInBounds to, perhaps, inBoundsLive(SSTables)? > sstablesInBounds might not actually give all sstables within the bounds due > to having start positions moved in sstables > --- > > Key: CASSANDRA-11944 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11944 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.0.x, 3.x > > > Same problem as with CASSANDRA-11886 - if we try to fetch sstablesInBounds > for CANONICAL_SSTABLES, we can miss some actually overlapping sstables. In > 3.0+ we state which SSTableSet we want when calling the method. > Looks like the only issue this could cause is that we include a few too many > sstables in compactions that we think contain only droppable tombstones -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11944) sstablesInBounds might not actually give all sstables within the bounds due to having start positions moved in sstables
[
https://issues.apache.org/jira/browse/CASSANDRA-11944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322357#comment-15322357
]
Marcus Eriksson commented on CASSANDRA-11944:
-
Maybe you were looking at the wrong commit? I have rebased and squashed
[here|https://github.com/krummas/cassandra/commits/marcuse/intervaltreesstableset]
{code}
-public Collection getOverlappingSSTables(SSTableSet
sstableSet, Iterable sstables)
+public Collection
getOverlappingSSTables(Iterable sstables)
{code}
> sstablesInBounds might not actually give all sstables within the bounds due
> to having start positions moved in sstables
> ---
>
> Key: CASSANDRA-11944
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11944
> Project: Cassandra
> Issue Type: Bug
>Reporter: Marcus Eriksson
>Assignee: Marcus Eriksson
> Fix For: 3.0.x, 3.x
>
>
> Same problem as with CASSANDRA-11886 - if we try to fetch sstablesInBounds
> for CANONICAL_SSTABLES, we can miss some actually overlapping sstables. In
> 3.0+ we state which SSTableSet we want when calling the method.
> Looks like the only issue this could cause is that we include a few too many
> sstables in compactions that we think contain only droppable tombstones
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11983) Migration task failed to complete
[ https://issues.apache.org/jira/browse/CASSANDRA-11983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322332#comment-15322332 ] Mike Adamson commented on CASSANDRA-11983: -- This is probably related to CASSANDRA-10731. This introduced a wait on the completion of in-flight migration tasks. This error is logged if the completion latch wait times out. The timeout is controlled by the cassandra.migration_task_wait_in_seconds system property and the default is 1. Could you try setting this system property to a higher value and see if that stops the error messages. > Migration task failed to complete > - > > Key: CASSANDRA-11983 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11983 > Project: Cassandra > Issue Type: Bug > Components: Lifecycle > Environment: Docker / Kubernetes running > Linux cassandra-21 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt25-1 (2016-03-06) > x86_64 GNU/Linux > openjdk version "1.8.0_91" > OpenJDK Runtime Environment (build 1.8.0_91-8u91-b14-1~bpo8+1-b14) > OpenJDK 64-Bit Server VM (build 25.91-b14, mixed mode) > Cassnadra 3.5 installed from > deb-src http://www.apache.org/dist/cassandra/debian 35x main >Reporter: Chris Love > > When nodes are boostrapping I am getting mulitple errors: "Migration task > failed to complete", from MigrationManager.java > The errors increase as more nodes are added to the ring, as I am creating a > ring of 1k nodes. > Cassandra yaml i here > https://github.com/k8s-for-greeks/gpmr/blob/3d50ff91a139b9c4a7a26eda0fb4dcf9a008fbed/pet-race-devops/docker/cassandra-debian/files/cassandra.yaml -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11944) sstablesInBounds might not actually give all sstables within the bounds due to having start positions moved in sstables
[ https://issues.apache.org/jira/browse/CASSANDRA-11944?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1533#comment-1533 ] Benedict commented on CASSANDRA-11944: -- I'm a little confused by the patch and the JIRA comment - I don't see (in this branch) any removal of an option to provide an SSTableSet... > sstablesInBounds might not actually give all sstables within the bounds due > to having start positions moved in sstables > --- > > Key: CASSANDRA-11944 > URL: https://issues.apache.org/jira/browse/CASSANDRA-11944 > Project: Cassandra > Issue Type: Bug >Reporter: Marcus Eriksson >Assignee: Marcus Eriksson > Fix For: 3.0.x, 3.x > > > Same problem as with CASSANDRA-11886 - if we try to fetch sstablesInBounds > for CANONICAL_SSTABLES, we can miss some actually overlapping sstables. In > 3.0+ we state which SSTableSet we want when calling the method. > Looks like the only issue this could cause is that we include a few too many > sstables in compactions that we think contain only droppable tombstones -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322096#comment-15322096
]
Mahdi Mohammadi commented on CASSANDRA-11961:
-
Seems good to me. Please assign yourself to the issue.
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8700) replace the wiki with docs in the git repo
[ https://issues.apache.org/jira/browse/CASSANDRA-8700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15322091#comment-15322091 ] Giampaolo commented on CASSANDRA-8700: -- I "vote" for Sphinx + reStructuredText too. Recently I did some documentation with it and was a smooth experience > replace the wiki with docs in the git repo > -- > > Key: CASSANDRA-8700 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8700 > Project: Cassandra > Issue Type: Improvement > Components: Documentation and Website >Reporter: Jon Haddad >Assignee: Sylvain Lebresne >Priority: Minor > Attachments: drivers_list.md, installation.md > > > The wiki as it stands is pretty terrible. It takes several minutes to apply > a single update, and as a result, it's almost never updated. The information > there has very little context as to what version it applies to. Most people > I've talked to that try to use the information they find there find it is > more confusing than helpful. > I'd like to propose that instead of using the wiki, the doc directory in the > cassandra repo be used for docs (already used for CQL3 spec) in a format that > can be built to a variety of output formats like HTML / epub / etc. I won't > start the bikeshedding on which markup format is preferable - but there are > several options that can work perfectly fine. I've personally use sphinx w/ > restructured text, and markdown. Both can build easily and as an added bonus > be pushed to readthedocs (or something similar) automatically. For an > example, see cqlengine's documentation, which I think is already > significantly better than the wiki: > http://cqlengine.readthedocs.org/en/latest/ > In addition to being overall easier to maintain, putting the documentation in > the git repo adds context, since it evolves with the versions of Cassandra. > If the wiki were kept even remotely up to date, I wouldn't bother with this, > but not having at least some basic documentation in the repo, or anywhere > associated with the project, is frustrating. > For reference, the last 3 updates were: > 1/15/15 - updating committers list > 1/08/15 - updating contributers and how to contribute > 12/16/14 - added a link to CQL docs from wiki frontpage (by me) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11985) 2.0.x leaks file handles (Again)
Amit Singh Chowdhery created CASSANDRA-11985: Summary: 2.0.x leaks file handles (Again) Key: CASSANDRA-11985 URL: https://issues.apache.org/jira/browse/CASSANDRA-11985 Project: Cassandra Issue Type: Bug Components: Compaction Environment: Unix kernel-2.6.32-431.56.1.el6.x86_64, Cassandra 2.0.14 Reporter: Amit Singh Chowdhery Priority: Critical We are running Cassandra 2.0.14 in production environment and disk usage is very high. On investigating it further we found that there are around 4-5 files(~ 150 GB) in stuck mode. Command Fired : lsof /var/lib/cassandra | grep -i deleted Output : java12158 cassandra 308r REG 8,16 34396638044 12727268 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-16481-Data.db (deleted) java12158 cassandra 327r REG 8,16 101982374806 12715102 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-126861-Data.db (deleted) java12158 cassandra 339r REG 8,16 12966304784 12714010 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-213548-Data.db (deleted) java12158 cassandra 379r REG 8,16 15323318036 12714957 /var/lib/cassandra/data/mykeyspace/mycolumnfamily/mykeyspace-mycolumnfamily-jb-182936-Data.db (deleted) we are not able to see these files in any directory. This is somewhat similar to CASSANDRA-6275 which is fixed but still issue is there on higher version. Also in logs no error related to compaction is reported. so could any one of you please provide any suggestion how to counter this. Restarting Cassandra is one solution but this issue keeps on occurring so we cannot restart production machine is not recommended so frequently. Also we know that this version is not supported but there is high probability that it can occur in higher version too. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-11961) Nonfatal NPE in CompactionMetrics
[
https://issues.apache.org/jira/browse/CASSANDRA-11961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Achal Shah updated CASSANDRA-11961:
---
Status: Patch Available (was: Open)
Hi, this is my first patch to Cassandra; apologies if I've messed up in any
way. Please let me know if there are any improvements I can make!
I've pushed a branch with my changes to github:
https://github.com/achals/cassandra/tree/11961-trunk
I looked for tests but there didn't seem to be any for this class.
> Nonfatal NPE in CompactionMetrics
> -
>
> Key: CASSANDRA-11961
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11961
> Project: Cassandra
> Issue Type: Bug
>Reporter: Robert Stupp
>Priority: Minor
> Labels: lhf
>
> Just saw the following NPE on trunk. Means, that {{metaData}} from
> {{CFMetaData metaData = compaction.getCompactionInfo().getCFMetaData();}} is
> {{null}}. A simple {{if (metaData == null) continue;}} should fix this.
> {code}
> Caused by: java.lang.NullPointerException: null
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:103)
> ~[main/:na]
> at
> org.apache.cassandra.metrics.CompactionMetrics$2.getValue(CompactionMetrics.java:78)
> ~[main/:na]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11182) Enable SASI index for collections
[
https://issues.apache.org/jira/browse/CASSANDRA-11182?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15321981#comment-15321981
]
Pavel Yaskevich commented on CASSANDRA-11182:
-
I think such a change deserves it's own ticket :)
> Enable SASI index for collections
> -
>
> Key: CASSANDRA-11182
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11182
> Project: Cassandra
> Issue Type: Improvement
> Components: CQL
>Reporter: DOAN DuyHai
>Assignee: Alex Petrov
>Priority: Minor
>
> This is a follow up ticket for post Cassandra 3.4 SASI integration.
> Right now it is possible with standard Cassandra 2nd index to:
> 1. index list and set elements ( {{WHERE list CONTAINS xxx}})
> 2. index map keys ( {{WHERE map CONTAINS KEYS 'abc'}} )
> 3. index map entries ( {{WHERE map\['key'\]=value}})
> It would be nice to enable these features in SASI too.
> With regard to tokenizing, we might want to allow wildcards ({{%}}) with the
> CONTAINS syntax as well as with index map entries. Ex:
> * {{WHERE list CONTAINS 'John%'}}
> * {{WHERE map CONTAINS KEY '%an%'}}
> * {{WHERE map\['key'\] LIKE '%val%'}}
> /cc [~xedin] [~rustyrazorblade] [~jkrupan]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
