[jira] [Commented] (CASSANDRA-8278) Build Issue
[
https://issues.apache.org/jira/browse/CASSANDRA-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542865#comment-14542865
]
Jeff Ferland commented on CASSANDRA-8278:
-
I've experienced the same issue. Experienced with clean install of Ubuntu
Trusty, clean pull of trunk, and having followed the ant realclean and rm -r
~/.m2 commands. Different
{quote}
Path to dependency:
1) org.apache.cassandra:cassandra-coverage-deps:jar:3.0.0-SNAPSHOT
2) net.sourceforge.cobertura:cobertura:jar:2.0.3
3) com.sun:tools:jar:0
--
1 required artifact is missing.
for artifact:
org.apache.cassandra:cassandra-coverage-deps:jar:3.0.0-SNAPSHOT
from the specified remote repositories:
central (http://repo1.maven.org/maven2)
Total time: 1 minute 18 seconds
/cassandra# git log | head -n 1
commit ea1d1614211b01d86d440dce8fe1a60a0bc3c7c1
{quote}
Build Issue
---
Key: CASSANDRA-8278
URL: https://issues.apache.org/jira/browse/CASSANDRA-8278
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: Mac OS X Yosemit
Reporter: David Laxer
BUILD FAILED
/Users/davidlaxer/cassandra/build.xml:545: Unable to resolve artifact:
Missing:
--
1) org.apache.hadoop:hadoop-minicluster:jar:1.0.3
Try downloading the file manually from the project website.
Then, install it using the command:
mvn install:install-file -DgroupId=org.apache.hadoop
-DartifactId=hadoop-minicluster -Dversion=1.0.3 -Dpackaging=jar
-Dfile=/path/to/file
Alternatively, if you host your own repository you can deploy the file
there:
mvn deploy:deploy-file -DgroupId=org.apache.hadoop
-DartifactId=hadoop-minicluster -Dversion=1.0.3 -Dpackaging=jar
-Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id]
Path to dependency:
1) org.apache.cassandra:cassandra-build-deps:jar:3.0.0-SNAPSHOT
2) org.apache.hadoop:hadoop-minicluster:jar:1.0.3
--
1 required artifact is missing.
for artifact:
org.apache.cassandra:cassandra-build-deps:jar:3.0.0-SNAPSHOT
from the specified remote repositories:
apache (https://repository.apache.org/content/repositories/releases),
central (http://repo1.maven.org/maven2),
java.net2 (http://download.java.net/maven/2)
Total time: 59 seconds
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-8278) Build Issue
[ https://issues.apache.org/jira/browse/CASSANDRA-8278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14542882#comment-14542882 ] Jeff Ferland commented on CASSANDRA-8278: - Occurs with openjdk-7-jre:amd64. Does not occur with Oracle Java 8 installation. Build Issue --- Key: CASSANDRA-8278 URL: https://issues.apache.org/jira/browse/CASSANDRA-8278 Project: Cassandra Issue Type: Bug Components: Core Environment: Mac OS X Yosemit Reporter: David Laxer BUILD FAILED /Users/davidlaxer/cassandra/build.xml:545: Unable to resolve artifact: Missing: -- 1) org.apache.hadoop:hadoop-minicluster:jar:1.0.3 Try downloading the file manually from the project website. Then, install it using the command: mvn install:install-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-minicluster -Dversion=1.0.3 -Dpackaging=jar -Dfile=/path/to/file Alternatively, if you host your own repository you can deploy the file there: mvn deploy:deploy-file -DgroupId=org.apache.hadoop -DartifactId=hadoop-minicluster -Dversion=1.0.3 -Dpackaging=jar -Dfile=/path/to/file -Durl=[url] -DrepositoryId=[id] Path to dependency: 1) org.apache.cassandra:cassandra-build-deps:jar:3.0.0-SNAPSHOT 2) org.apache.hadoop:hadoop-minicluster:jar:1.0.3 -- 1 required artifact is missing. for artifact: org.apache.cassandra:cassandra-build-deps:jar:3.0.0-SNAPSHOT from the specified remote repositories: apache (https://repository.apache.org/content/repositories/releases), central (http://repo1.maven.org/maven2), java.net2 (http://download.java.net/maven/2) Total time: 59 seconds -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9577) Cassandra not performing GC on stale SStables after compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14583407#comment-14583407 ] Jeff Ferland commented on CASSANDRA-9577: - Per timestamps above, this was the case more than 24 hours after completion. Restarting the host did clear the files. On the same host just after a restart I'm encountering the same pattern. Cassandra not performing GC on stale SStables after compaction -- Key: CASSANDRA-9577 URL: https://issues.apache.org/jira/browse/CASSANDRA-9577 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0.12.200 / DSE 4.6.1. Reporter: Jeff Ferland Assignee: Marcus Eriksson Space used (live), bytes: 878681716067 Space used (total), bytes: 2227857083852 jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java4473 cassandra 446r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 448r REG 0,26 62040962 37431 trends-trends-jb-144731-Data.db java4473 cassandra 449r REG 0,26 829935047545 21150 trends-trends-jb-143581-Data.db java4473 cassandra 452r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 454r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 462r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 463r REG 0,26 36158226 39629 trends-trends-jb-144889-Data.db java4473 cassandra 468r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 530r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 535r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 542r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 553u REG 0,26 6431729821 39556 trends-trends-tmp-jb-144884-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db trends-trends-jb-144883-Data.db trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db trends-trends-jb-144889-Data.db trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db trends-trends-tmp-jb-144884-Data.db trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd - /mnt/cassandra/data/trends/trends jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db trends-trends-jb-144890-Data.db trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db trends-trends-jb-41121-Data.db trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db trends-trends.trends_date_idx-jb-2100-Data.db trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db Compaction started INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515 CompactionTask.java (line 120) Compacting [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141806-Data.db'),
[jira] [Created] (CASSANDRA-9577) Cassandra not performing GC on stale SStables after compaction
Jeff Ferland created CASSANDRA-9577: --- Summary: Cassandra not performing GC on stale SStables after compaction Key: CASSANDRA-9577 URL: https://issues.apache.org/jira/browse/CASSANDRA-9577 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0.12.200 / DSE 4.6.1. Reporter: Jeff Ferland Space used (live), bytes: 878681716067 Space used (total), bytes: 2227857083852 jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java4473 cassandra 446r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 448r REG 0,26 62040962 37431 trends-trends-jb-144731-Data.db java4473 cassandra 449r REG 0,26 829935047545 21150 trends-trends-jb-143581-Data.db java4473 cassandra 452r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 454r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 462r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 463r REG 0,26 36158226 39629 trends-trends-jb-144889-Data.db java4473 cassandra 468r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 530r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 535r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 542r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 553u REG 0,26 6431729821 39556 trends-trends-tmp-jb-144884-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db trends-trends-jb-144883-Data.db trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db trends-trends-jb-144889-Data.db trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db trends-trends-tmp-jb-144884-Data.db trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd - /mnt/cassandra/data/trends/trends jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db trends-trends-jb-144890-Data.db trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db trends-trends-jb-41121-Data.db trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db trends-trends.trends_date_idx-jb-2100-Data.db trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db Compaction started INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515 CompactionTask.java (line 120) Compacting [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141806-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141102-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141123-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-41121-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141137-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141113-Data.db'),
[jira] [Commented] (CASSANDRA-9577) Cassandra not performing GC on stale SStables after compaction
[ https://issues.apache.org/jira/browse/CASSANDRA-9577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14615726#comment-14615726 ] Jeff Ferland commented on CASSANDRA-9577: - Sorry, I've missed the request for follow up. This has happened on more than one host. Yes, that major compaction took four days. No, I have no further logging available (or retained) at this point. Cassandra not performing GC on stale SStables after compaction -- Key: CASSANDRA-9577 URL: https://issues.apache.org/jira/browse/CASSANDRA-9577 Project: Cassandra Issue Type: Bug Components: Core Environment: 2.0.12.200 / DSE 4.6.1. Reporter: Jeff Ferland Assignee: Marcus Eriksson Space used (live), bytes: 878681716067 Space used (total), bytes: 2227857083852 jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ sudo lsof *-Data.db COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME java4473 cassandra 446r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 448r REG 0,26 62040962 37431 trends-trends-jb-144731-Data.db java4473 cassandra 449r REG 0,26 829935047545 21150 trends-trends-jb-143581-Data.db java4473 cassandra 452r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 454r REG 0,26 8980406 39503 trends-trends-jb-144882-Data.db java4473 cassandra 462r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 463r REG 0,26 36158226 39629 trends-trends-jb-144889-Data.db java4473 cassandra 468r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 530r REG 0,26 17582559172 39241 trends-trends-jb-144864-Data.db java4473 cassandra 535r REG 0,26105693505 39447 trends-trends-jb-144881-Data.db java4473 cassandra 542r REG 0,26 9487703 39542 trends-trends-jb-144883-Data.db java4473 cassandra 553u REG 0,26 6431729821 39556 trends-trends-tmp-jb-144884-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-142631-Data.db trends-trends-jb-143562-Data.db trends-trends-jb-143581-Data.db trends-trends-jb-144731-Data.db trends-trends-jb-144883-Data.db trends-trends-jb-142633-Data.db trends-trends-jb-143563-Data.db trends-trends-jb-144530-Data.db trends-trends-jb-144864-Data.db trends-trends-jb-144889-Data.db trends-trends-jb-143026-Data.db trends-trends-jb-143564-Data.db trends-trends-jb-144551-Data.db trends-trends-jb-144881-Data.db trends-trends-tmp-jb-144884-Data.db trends-trends-jb-143533-Data.db trends-trends-jb-143578-Data.db trends-trends-jb-144552-Data.db trends-trends-jb-144882-Data.db jbf@ip-10-0-2-98:/ebs/cassandra/data/trends/trends$ cd - /mnt/cassandra/data/trends/trends jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ sudo lsof * jbf@ip-10-0-2-98:/mnt/cassandra/data/trends/trends$ ls *-Data.db trends-trends-jb-124502-Data.db trends-trends-jb-141113-Data.db trends-trends-jb-141377-Data.db trends-trends-jb-141846-Data.db trends-trends-jb-144890-Data.db trends-trends-jb-125457-Data.db trends-trends-jb-141123-Data.db trends-trends-jb-141391-Data.db trends-trends-jb-141871-Data.db trends-trends-jb-41121-Data.db trends-trends-jb-130016-Data.db trends-trends-jb-141137-Data.db trends-trends-jb-141538-Data.db trends-trends-jb-141883-Data.db trends-trends.trends_date_idx-jb-2100-Data.db trends-trends-jb-139563-Data.db trends-trends-jb-141358-Data.db trends-trends-jb-141806-Data.db trends-trends-jb-142033-Data.db trends-trends-jb-141102-Data.db trends-trends-jb-141363-Data.db trends-trends-jb-141829-Data.db trends-trends-jb-144553-Data.db Compaction started INFO [CompactionExecutor:6661] 2015-06-05 14:02:36,515 CompactionTask.java (line 120) Compacting [SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-124502-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141358-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141883-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141846-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141871-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-141391-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-139563-Data.db'), SSTableReader(path='/mnt/cassandra/data/trends/trends/trends-trends-jb-125457-Data.db'),
[jira] [Created] (CASSANDRA-10864) Dropped mutations high until cluster restart
Jeff Ferland created CASSANDRA-10864: Summary: Dropped mutations high until cluster restart Key: CASSANDRA-10864 URL: https://issues.apache.org/jira/browse/CASSANDRA-10864 Project: Cassandra Issue Type: Bug Reporter: Jeff Ferland Originally raised and investigated in https://www.mail-archive.com/user@cassandra.apache.org/msg44586.html Cause is still unclear, but a rolling restart has on two occasions since been performed to cope with dropped mutations and timed-out reads. Pattern is indicative of some kind of code quality issue possibly involving locking operations. Stack flame graphs do not show a clear difference between restarts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
Jeff Ferland created CASSANDRA-10862: Summary: LCS repair: compact tables before making available in L0 Key: CASSANDRA-10862 URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 Project: Cassandra Issue Type: Improvement Components: Compaction, Streaming and Messaging Reporter: Jeff Ferland When doing repair on a system with lots of mismatched ranges, the number of tables in L0 goes up dramatically, as correspondingly goes the number of tables referenced for a query. Latency increases dramatically in tandem. Eventually all the copied tables are compacted down in L0, then copied into L1 (which may be a very large copy), finally reducing the number of SSTables per query into the manageable range. It seems to me that the cleanest answer is to compact after streaming, then mark tables available rather than marking available when the file itself is complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
Jeff Ferland created CASSANDRA-10979: Summary: LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress Key: CASSANDRA-10979 URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 Project: Cassandra Issue Type: Bug Components: Compaction Environment: 2.1.11 / 4.8.3 DSE. Reporter: Jeff Ferland Reading code from https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java and comparing with behavior shown in https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to incremental repair). Relevant code from here is if (Sets.intersection(l1overlapping, compacting).size() > 0) return Collections.emptyList(); Since there will be overlap between what is compacting and L1 (in my case, pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 smaller tables in L0 while awaiting the transition from L0 to L1 and destroy my performance. Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088139#comment-15088139 ] Jeff Ferland commented on CASSANDRA-10979: -- Debug logging using the current code: https://gist.github.com/autocracy/346afa253175af475770 > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland > Labels: compaction, leveled > Fix For: 2.1.x > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15088139#comment-15088139 ] Jeff Ferland edited comment on CASSANDRA-10979 at 1/7/16 9:24 PM: -- Debug logging using the current (unpatched) code: https://gist.github.com/autocracy/346afa253175af475770 was (Author: autocracy): Debug logging using the current code: https://gist.github.com/autocracy/346afa253175af475770 > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland > Labels: compaction, leveled > Fix For: 2.1.x > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15094350#comment-15094350 ] Jeff Ferland commented on CASSANDRA-10979: -- The problem I've got and why I considered it a bug is that the behavior as it is right now doesn't appear to be as it was intended, and particularly negative for me is that it destroys performance of my cluster by an order of magnitude with LCS tables during a repair as potentially thousands of tables build up in L0. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15098279#comment-15098279 ] Jeff Ferland commented on CASSANDRA-10979: -- [~sebastian.este...@datastax.com]: Can you cut a build of DSE 4.8.4 with this patch added? I'll try my best to test within a week. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11028) Streaming errors caused by corrupt tables need more logging
Jeff Ferland created CASSANDRA-11028: Summary: Streaming errors caused by corrupt tables need more logging Key: CASSANDRA-11028 URL: https://issues.apache.org/jira/browse/CASSANDRA-11028 Project: Cassandra Issue Type: Bug Reporter: Jeff Ferland Example output: ERROR [STREAM-IN-/10.0.10.218] 2016-01-17 16:01:38,431 StreamSession.java:505 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Streaming error occurred java.lang.IllegalArgumentException: Unknown type 0 In some cases logging shows a message more like: ERROR [STREAM-IN-/10.0.10.12] 2016-01-05 14:44:38,690 StreamSession.java:505 - [Stream #472d28e0-b347-11e5-8b40-bb4d80df86f4] Streaming error occurred java.io.IOException: Too many retries for Header (cfId: 6b262d58-8730-36ca-8e3e-f0a40beaf92f, #0, version: ka, estimated keys: 58880, transfer size: 2159040, compressed?: true, repairedAt: 0) In the majority of cases, however, no information identifying the column family is shown, and never identifying the source file that was being streamed. Errors do no stop the streaming process, but do mark the streaming as failed at the end. This usually results in a log message pattern like: INFO [StreamReceiveTask:252] 2016-01-18 04:45:01,190 StreamResultFuture.java:180 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Session with /10.0.10.219 is complete WARN [StreamReceiveTask:252] 2016-01-18 04:45:01,215 StreamResultFuture.java:207 - [Stream #e6ca4590-bc66-11e5-84be-571ffcecc993] Stream failed ERROR [main] 2016-01-18 04:45:01,217 CassandraDaemon.java:579 - Exception encountered during startup ... which is highly confusing given the error occurred hours before. Request: more detail in logging messages for stream failure indicating what column family was being used, and if possible a clarification between network issues and corrupt file issues. Actual cause of errors / solution is running nodetool scrub on the offending node. It's rather expensive scrubbing the whole space blindly versus targeting issue tables. In our particular case, out of order keys were caused by a bug in a previous version of Cassandra. WARN [CompactionExecutor:19552] 2016-01-18 16:02:10,155 OutputHandler.java:52 - 378490 out of order rows found while scrubbing SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-133-Data.db'); Those have been written (in order) to a new sstable (SSTableReader(path='/mnt/cassandra/data/keyspace/cf-888a52f96d1d389790ee586a6100916c/keyspace-cf-ka-179-Data.db')) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10979) LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress
[ https://issues.apache.org/jira/browse/CASSANDRA-10979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109177#comment-15109177 ] Jeff Ferland commented on CASSANDRA-10979: -- Applied the patch to a node that had just come up from streaming and was 700+ tables in L0. After restart, observed that as L0 was shifted into L1, L0 continued to compact new tables to prevent the extreme growth of tables. Thank you. Again still requesting a merge into 2.1 since streaming and repair with LCS are practically broken for us without this patch. > LCS doesn't do L0 STC on new tables while an L0->L1 compaction is in progress > - > > Key: CASSANDRA-10979 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10979 > Project: Cassandra > Issue Type: Bug > Components: Compaction > Environment: 2.1.11 / 4.8.3 DSE. >Reporter: Jeff Ferland >Assignee: Carl Yeksigian > Labels: compaction, leveled > Fix For: 3.x > > Attachments: 10979-2.1.txt > > > Reading code from > https://github.com/apache/cassandra/blob/cassandra-2.1/src/java/org/apache/cassandra/db/compaction/LeveledManifest.java > and comparing with behavior shown in > https://gist.github.com/autocracy/c95aca6b00e42215daaf, the following happens: > Score for L1,L2,and L3 is all < 1 (paste shows 20/10 and 200/100, due to > incremental repair). > Relevant code from here is > if (Sets.intersection(l1overlapping, compacting).size() > 0) > return Collections.emptyList(); > Since there will be overlap between what is compacting and L1 (in my case, > pushing over 1,000 tables in to L1 from L0 SCTS), I get a pile up of 1,000 > smaller tables in L0 while awaiting the transition from L0 to L1 and destroy > my performance. > Requested outcome is to continue to perform SCTS on non-compacting L0 tables. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
Jeff Ferland created CASSANDRA-11172:
Summary: Infinite loop bug adding high-level SSTableReader in
compaction
Key: CASSANDRA-11172
URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
Project: Cassandra
Issue Type: Bug
Components: Compaction
Environment: DSE 4.x / Cassandra 2.1.11.969
Reporter: Jeff Ferland
Observed that after a large repair on LCS that sometimes the system will enter
an infinite loop with vast amounts of logs lines recording, "Adding high-level
(L${LEVEL}) SSTableReader(path='${TABLE}') to candidates"
This results in an outage of the node and eventual crashing. The log spam
quickly rotates out possibly useful earlier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148988#comment-15148988
]
Jeff Ferland commented on CASSANDRA-11172:
--
It's this: `INFO [CompactionExecutor:12750] 2016-02-16 17:47:48,642
LeveledManifest.java:415 - Adding high-level (L0)
SSTableReader(path='/mnt/cassandra/data/youtube/youtube_videos-2d16275b7ff93269bea0aac894e1abaa/youtube-youtube_videos-ka-104968-Data.db')
to candidates` repeating endlessly. That one's extra special this time because
of the L0. I'll look in our aggregation server and try to get value from the
time when it starts.
> Infinite loop bug adding high-level SSTableReader in compaction
> ---
>
> Key: CASSANDRA-11172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: DSE 4.x / Cassandra 2.1.11.969
>Reporter: Jeff Ferland
>Assignee: Marcus Eriksson
>
> Observed that after a large repair on LCS that sometimes the system will
> enter an infinite loop with vast amounts of logs lines recording, "Adding
> high-level (L${LEVEL}) SSTableReader(path='${TABLE}') to candidates"
> This results in an outage of the node and eventual crashing. The log spam
> quickly rotates out possibly useful earlier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148994#comment-15148994
]
Jeff Ferland edited comment on CASSANDRA-11172 at 2/16/16 6:01 PM:
---
Possible duplicate of CASSANDRA-10831 ?
was (Author: autocracy):
Possible duplicate of https://issues.apache.org/jira/browse/CASSANDRA-10831 ?
> Infinite loop bug adding high-level SSTableReader in compaction
> ---
>
> Key: CASSANDRA-11172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: DSE 4.x / Cassandra 2.1.11.969
>Reporter: Jeff Ferland
>Assignee: Marcus Eriksson
>
> Observed that after a large repair on LCS that sometimes the system will
> enter an infinite loop with vast amounts of logs lines recording, "Adding
> high-level (L${LEVEL}) SSTableReader(path='${TABLE}') to candidates"
> This results in an outage of the node and eventual crashing. The log spam
> quickly rotates out possibly useful earlier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148994#comment-15148994
]
Jeff Ferland commented on CASSANDRA-11172:
--
Possible duplicate of https://issues.apache.org/jira/browse/CASSANDRA-10831 ?
> Infinite loop bug adding high-level SSTableReader in compaction
> ---
>
> Key: CASSANDRA-11172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: DSE 4.x / Cassandra 2.1.11.969
>Reporter: Jeff Ferland
>Assignee: Marcus Eriksson
>
> Observed that after a large repair on LCS that sometimes the system will
> enter an infinite loop with vast amounts of logs lines recording, "Adding
> high-level (L${LEVEL}) SSTableReader(path='${TABLE}') to candidates"
> This results in an outage of the node and eventual crashing. The log spam
> quickly rotates out possibly useful earlier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-11172) Infinite loop bug adding high-level SSTableReader in compaction
[
https://issues.apache.org/jira/browse/CASSANDRA-11172?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15149009#comment-15149009
]
Jeff Ferland commented on CASSANDRA-11172:
--
Yes. It's after incremental repair that I'm seeing this. Next time around I'll
check that the file listed doesn't exist before restart, but I think this is a
duplicate.
Alternately, though, I'm also seeing the gossip thread lockup at times after
incremental repair without mentioning higher level sstables, so that might be a
new ticket to file next time around.
> Infinite loop bug adding high-level SSTableReader in compaction
> ---
>
> Key: CASSANDRA-11172
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11172
> Project: Cassandra
> Issue Type: Bug
> Components: Compaction
> Environment: DSE 4.x / Cassandra 2.1.11.969
>Reporter: Jeff Ferland
>Assignee: Marcus Eriksson
>
> Observed that after a large repair on LCS that sometimes the system will
> enter an infinite loop with vast amounts of logs lines recording, "Adding
> high-level (L${LEVEL}) SSTableReader(path='${TABLE}') to candidates"
> This results in an outage of the node and eventual crashing. The log spam
> quickly rotates out possibly useful earlier debugging.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10862) LCS repair: compact tables before making available in L0
[ https://issues.apache.org/jira/browse/CASSANDRA-10862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15221843#comment-15221843 ] Jeff Ferland commented on CASSANDRA-10862: -- Note that CASSANDRA-10979 has helped immensely with the performance issues outlines here. > LCS repair: compact tables before making available in L0 > > > Key: CASSANDRA-10862 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10862 > Project: Cassandra > Issue Type: Improvement > Components: Compaction, Streaming and Messaging >Reporter: Jeff Ferland > > When doing repair on a system with lots of mismatched ranges, the number of > tables in L0 goes up dramatically, as correspondingly goes the number of > tables referenced for a query. Latency increases dramatically in tandem. > Eventually all the copied tables are compacted down in L0, then copied into > L1 (which may be a very large copy), finally reducing the number of SSTables > per query into the manageable range. > It seems to me that the cleanest answer is to compact after streaming, then > mark tables available rather than marking available when the file itself is > complete. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
