[jira] [Commented] (CASSANDRA-9935) Repair fails with RuntimeException
[
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030436#comment-15030436
]
mlowicki commented on CASSANDRA-9935:
-
Tried to run repair once again after online scrub and cleanup on all nodes.
Failed with the same error. This is what I've found in logs:
{code}
ERROR [ValidationExecutor:1089] 2015-11-28 04:33:15,865 Validator.java:245 -
Failed creating a merkle tree for [repair #0f9c5530-9589-11e5-b036-75bb514ae072
on sync/entity2, (-6842825601551036942,-6841068234348096268]], /10.210.3.221
(see log for details)
ERROR [ValidationExecutor:1089] 2015-11-28 04:33:15,866
CassandraDaemon.java:227 - Exception in thread
Thread[ValidationExecutor:1089,1,main]
java.lang.AssertionError: row DecoratedKey(-6842806631972123001,
000932383331343239333204c3c700) received out of order wrt
DecoratedKey(-6841074726771668561, 000932313637353230343404c3c700)
at org.apache.cassandra.repair.Validator.add(Validator.java:127)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionManager.doValidationCompaction(CompactionManager.java:1010)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionManager.access$600(CompactionManager.java:94)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionManager$9.call(CompactionManager.java:622)
~[apache-cassandra-2.1.11.jar:2.1.11]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
ERROR [AntiEntropySessions:1957] 2015-11-28 04:33:15,868 RepairSession.java:303
- [repair #0f9c5530-9589-11e5-b036-75bb514ae072] session completed with the
following error
org.apache.cassandra.exceptions.RepairException: [repair
#0f9c5530-9589-11e5-b036-75bb514ae072 on sync/entity2,
(-6842825601551036942,-6841068234348096268]] Validation failed in /10.210.3.221
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:406)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{code}
{code}
ERROR [AntiEntropySessions:1957] 2015-11-28 04:33:15,869
CassandraDaemon.java:227 - Exception in thread
Thread[AntiEntropySessions:1957,5,RMI Runtime]
java.lang.RuntimeException: org.apache.cassandra.exceptions.RepairException:
[repair #0f9c5530-9589-11e5-b036-75bb514ae072 on sync/entity2,
(-6842825601551036942,-6841068234348096268]] Validation failed in /10.210.3.221
at com.google.common.base.Throwables.propagate(Throwables.java:160)
~[guava-16.0.jar:na]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:32)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_80]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
Caused by: org.apache.cassandra.exceptions.RepairException: [repair
#0f9c5530-9589-11e5-b036-75bb514ae072 on sync/entity2,
(-6842825601551036942,-6841068234348096268]] Validation failed in /10.210.3.221
at
org.apache.cassandra.repair.RepairSession.validationComplete(RepairSession.java:166)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.service.ActiveRepairService.handleMessage(ActiveRepairService.java:406)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:134)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:64)
~[apache-cassandra-2.1.11.jar:2.1.11]
... 3 common frames omitted
{code}
{code}
ERROR
[jira] [Updated] (CASSANDRA-9935) Repair fails with RuntimeException
[
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mlowicki updated CASSANDRA-9935:
Attachment: system.log.10.210.3.117
> Repair fails with RuntimeException
> --
>
> Key: CASSANDRA-9935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9935
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.8, Debian Wheezy
>Reporter: mlowicki
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: db1.sync.lati.osa.cassandra.log,
> db5.sync.lati.osa.cassandra.log, system.log.10.210.3.117,
> system.log.10.210.3.221, system.log.10.210.3.230
>
>
> We had problems with slow repair in 2.1.7 (CASSANDRA-9702) but after upgrade
> to 2.1.8 it started to work faster but now it fails with:
> {code}
> ...
> [2015-07-29 20:44:03,956] Repair session 23a811b0-3632-11e5-a93e-4963524a8bde
> for range (-5474076923322749342,-5468600594078911162] finished
> [2015-07-29 20:44:03,957] Repair session 336f8740-3632-11e5-a93e-4963524a8bde
> for range (-8631877858109464676,-8624040066373718932] finished
> [2015-07-29 20:44:03,957] Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde
> for range (-5372806541854279315,-5369354119480076785] finished
> [2015-07-29 20:44:03,957] Repair session 59f129f0-3632-11e5-a93e-4963524a8bde
> for range (8166489034383821955,8168408930184216281] finished
> [2015-07-29 20:44:03,957] Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde
> for range (6084602890817326921,6088328703025510057] finished
> [2015-07-29 20:44:03,957] Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde
> for range (-781874602493000830,-781745173070807746] finished
> [2015-07-29 20:44:03,957] Repair command #4 finished
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
> {code}
> After running:
> {code}
> nodetool repair --partitioner-range --parallel --in-local-dc sync
> {code}
> Last records in logs regarding repair are:
> {code}
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 09ff9e40-3632-11e5-a93e-4963524a8bde for range
> (-7695808664784761779,-7693529816291585568] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 17d8d860-3632-11e5-a93e-4963524a8bde for range
> (806371695398849,8065203836608925992] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 23a811b0-3632-11e5-a93e-4963524a8bde for range
> (-5474076923322749342,-5468600594078911162] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 336f8740-3632-11e5-a93e-4963524a8bde for range
> (-8631877858109464676,-8624040066373718932] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde for range
> (-5372806541854279315,-5369354119480076785] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 59f129f0-3632-11e5-a93e-4963524a8bde for range
> (8166489034383821955,8168408930184216281] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde for range
> (6084602890817326921,6088328703025510057] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde for range
> (-781874602493000830,-781745173070807746] finished
> {code}
> but a bit above I see (at least two times in attached log):
> {code}
> ERROR [Thread-173887] 2015-07-29 20:44:03,853 StorageService.java:2959 -
> Repair session 1b07ea50-3608-11e5-a93e-4963524a8bde for range
> (5765414319217852786,5781018794516851576] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2,
> (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2,
> (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.7.0_80]
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> [na:1.7.0_80]
> at
> org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2950)
> ~[apache-cassandra-2.1.8.jar:2.1.8]
> at
[jira] [Updated] (CASSANDRA-9935) Repair fails with RuntimeException
[
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mlowicki updated CASSANDRA-9935:
Attachment: system.log.10.210.3.230
system.log.10.210.3.221
> Repair fails with RuntimeException
> --
>
> Key: CASSANDRA-9935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9935
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.8, Debian Wheezy
>Reporter: mlowicki
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: db1.sync.lati.osa.cassandra.log,
> db5.sync.lati.osa.cassandra.log, system.log.10.210.3.221,
> system.log.10.210.3.230
>
>
> We had problems with slow repair in 2.1.7 (CASSANDRA-9702) but after upgrade
> to 2.1.8 it started to work faster but now it fails with:
> {code}
> ...
> [2015-07-29 20:44:03,956] Repair session 23a811b0-3632-11e5-a93e-4963524a8bde
> for range (-5474076923322749342,-5468600594078911162] finished
> [2015-07-29 20:44:03,957] Repair session 336f8740-3632-11e5-a93e-4963524a8bde
> for range (-8631877858109464676,-8624040066373718932] finished
> [2015-07-29 20:44:03,957] Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde
> for range (-5372806541854279315,-5369354119480076785] finished
> [2015-07-29 20:44:03,957] Repair session 59f129f0-3632-11e5-a93e-4963524a8bde
> for range (8166489034383821955,8168408930184216281] finished
> [2015-07-29 20:44:03,957] Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde
> for range (6084602890817326921,6088328703025510057] finished
> [2015-07-29 20:44:03,957] Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde
> for range (-781874602493000830,-781745173070807746] finished
> [2015-07-29 20:44:03,957] Repair command #4 finished
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
> {code}
> After running:
> {code}
> nodetool repair --partitioner-range --parallel --in-local-dc sync
> {code}
> Last records in logs regarding repair are:
> {code}
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 09ff9e40-3632-11e5-a93e-4963524a8bde for range
> (-7695808664784761779,-7693529816291585568] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 17d8d860-3632-11e5-a93e-4963524a8bde for range
> (806371695398849,8065203836608925992] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 23a811b0-3632-11e5-a93e-4963524a8bde for range
> (-5474076923322749342,-5468600594078911162] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 336f8740-3632-11e5-a93e-4963524a8bde for range
> (-8631877858109464676,-8624040066373718932] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde for range
> (-5372806541854279315,-5369354119480076785] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 59f129f0-3632-11e5-a93e-4963524a8bde for range
> (8166489034383821955,8168408930184216281] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde for range
> (6084602890817326921,6088328703025510057] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde for range
> (-781874602493000830,-781745173070807746] finished
> {code}
> but a bit above I see (at least two times in attached log):
> {code}
> ERROR [Thread-173887] 2015-07-29 20:44:03,853 StorageService.java:2959 -
> Repair session 1b07ea50-3608-11e5-a93e-4963524a8bde for range
> (5765414319217852786,5781018794516851576] failed with error
> org.apache.cassandra.exceptions.RepairException: [repair
> #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2,
> (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.RepairException: [repair
> #1b07ea50-3608-11e5-a93e-4963524a8bde on sync/entity_by_id2,
> (5765414319217852786,5781018794516851576]] Validation failed in /10.195.15.162
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> [na:1.7.0_80]
> at java.util.concurrent.FutureTask.get(FutureTask.java:188)
> [na:1.7.0_80]
> at
> org.apache.cassandra.service.StorageService$4.runMayThrow(StorageService.java:2950)
>
[jira] [Commented] (CASSANDRA-9935) Repair fails with RuntimeException
[
https://issues.apache.org/jira/browse/CASSANDRA-9935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030439#comment-15030439
]
mlowicki commented on CASSANDRA-9935:
-
Also If I run repair for range where got this "Endpoint X died" it works fine:
{code}
root@db1:~# time nodetool repair --in-local-dc -st 8066543735336862962 -et
8074446636728465478
[2015-11-28 08:55:19,048] Nothing to repair for keyspace 'system'
[2015-11-28 08:55:19,069] Starting repair command #6, repairing 1 ranges for
keyspace OpsCenter (parallelism=SEQUENTIAL, full=true)
[2015-11-28 08:55:19,176] Repair command #6 finished
[2015-11-28 08:55:19,188] Starting repair command #7, repairing 1 ranges for
keyspace sync (parallelism=SEQUENTIAL, full=true)
[2015-11-28 09:03:49,529] Repair session c054ec60-95ad-11e5-b036-75bb514ae072
for range (8066543735336862962,8074446636728465478] finished
[2015-11-28 09:03:49,529] Repair command #7 finished
[2015-11-28 09:03:49,544] Starting repair command #8, repairing 1 ranges for
keyspace system_traces (parallelism=SEQUENTIAL, full=true)
[2015-11-28 09:03:49,562] Repair command #8 finished
real8m32.356s
user0m2.784s
sys 0m0.224s
{code}
> Repair fails with RuntimeException
> --
>
> Key: CASSANDRA-9935
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9935
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.8, Debian Wheezy
>Reporter: mlowicki
>Assignee: Yuki Morishita
> Fix For: 2.1.x
>
> Attachments: db1.sync.lati.osa.cassandra.log,
> db5.sync.lati.osa.cassandra.log, system.log.10.210.3.117,
> system.log.10.210.3.221, system.log.10.210.3.230
>
>
> We had problems with slow repair in 2.1.7 (CASSANDRA-9702) but after upgrade
> to 2.1.8 it started to work faster but now it fails with:
> {code}
> ...
> [2015-07-29 20:44:03,956] Repair session 23a811b0-3632-11e5-a93e-4963524a8bde
> for range (-5474076923322749342,-5468600594078911162] finished
> [2015-07-29 20:44:03,957] Repair session 336f8740-3632-11e5-a93e-4963524a8bde
> for range (-8631877858109464676,-8624040066373718932] finished
> [2015-07-29 20:44:03,957] Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde
> for range (-5372806541854279315,-5369354119480076785] finished
> [2015-07-29 20:44:03,957] Repair session 59f129f0-3632-11e5-a93e-4963524a8bde
> for range (8166489034383821955,8168408930184216281] finished
> [2015-07-29 20:44:03,957] Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde
> for range (6084602890817326921,6088328703025510057] finished
> [2015-07-29 20:44:03,957] Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde
> for range (-781874602493000830,-781745173070807746] finished
> [2015-07-29 20:44:03,957] Repair command #4 finished
> error: nodetool failed, check server logs
> -- StackTrace --
> java.lang.RuntimeException: nodetool failed, check server logs
> at
> org.apache.cassandra.tools.NodeTool$NodeToolCmd.run(NodeTool.java:290)
> at org.apache.cassandra.tools.NodeTool.main(NodeTool.java:202)
> {code}
> After running:
> {code}
> nodetool repair --partitioner-range --parallel --in-local-dc sync
> {code}
> Last records in logs regarding repair are:
> {code}
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 09ff9e40-3632-11e5-a93e-4963524a8bde for range
> (-7695808664784761779,-7693529816291585568] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 17d8d860-3632-11e5-a93e-4963524a8bde for range
> (806371695398849,8065203836608925992] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 23a811b0-3632-11e5-a93e-4963524a8bde for range
> (-5474076923322749342,-5468600594078911162] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,956 StorageService.java:2952 -
> Repair session 336f8740-3632-11e5-a93e-4963524a8bde for range
> (-8631877858109464676,-8624040066373718932] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 4ccd8430-3632-11e5-a93e-4963524a8bde for range
> (-5372806541854279315,-5369354119480076785] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 59f129f0-3632-11e5-a93e-4963524a8bde for range
> (8166489034383821955,8168408930184216281] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 6ae7a9a0-3632-11e5-a93e-4963524a8bde for range
> (6084602890817326921,6088328703025510057] finished
> INFO [Thread-173887] 2015-07-29 20:44:03,957 StorageService.java:2952 -
> Repair session 8938e4a0-3632-11e5-a93e-4963524a8bde for range
> (-781874602493000830,-781745173070807746] finished
> {code}
> but a bit above I see (at least two times in attached log):
> {code}
> ERROR
[jira] [Created] (CASSANDRA-10782) AssertionError at getApproximateKeyCount
mlowicki created CASSANDRA-10782:
Summary: AssertionError at getApproximateKeyCount
Key: CASSANDRA-10782
URL: https://issues.apache.org/jira/browse/CASSANDRA-10782
Project: Cassandra
Issue Type: Bug
Environment: C* 2.1.11, Debian Wheezy
Reporter: mlowicki
{code}
ERROR [CompactionExecutor:9797] 2015-11-28 09:20:10,361
CassandraDaemon.java:227 - Exception in thread
Thread[CompactionExecutor:9797,1,main]
java.lang.AssertionError:
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-6335-Data.db
at
org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:268)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:151)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
~[apache-cassandra-2.1.11.jar:2.1.11]at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.11.jar:2.1.11]at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236)
~[apache-cassandra-2.1.11.jar:2.1.11]at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_80]at
java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-10782) AssertionError at getApproximateKeyCount
[
https://issues.apache.org/jira/browse/CASSANDRA-10782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
mlowicki updated CASSANDRA-10782:
-
Description:
{code}
ERROR [CompactionExecutor:9845] 2015-11-28 09:26:10,525
CassandraDaemon.java:227 - Exception in thread
Thread[CompactionExecutor:9845,1,main]
java.lang.AssertionError:
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-6335-Data.db
at
org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:268)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:151)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_80]
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]
at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{code}
was:
{code}
ERROR [CompactionExecutor:9797] 2015-11-28 09:20:10,361
CassandraDaemon.java:227 - Exception in thread
Thread[CompactionExecutor:9797,1,main]
java.lang.AssertionError:
/var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-6335-Data.db
at
org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:268)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:151)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-2.1.11.jar:2.1.11]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
~[apache-cassandra-2.1.11.jar:2.1.11]at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
~[apache-cassandra-2.1.11.jar:2.1.11]at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236)
~[apache-cassandra-2.1.11.jar:2.1.11]at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
~[na:1.7.0_80]at
java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_80]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
~[na:1.7.0_80]at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[na:1.7.0_80]at java.lang.Thread.run(Thread.java:745) [na:1.7.0_80]
{code}
> AssertionError at getApproximateKeyCount
>
>
> Key: CASSANDRA-10782
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10782
> Project: Cassandra
> Issue Type: Bug
> Environment: C* 2.1.11, Debian Wheezy
>Reporter: mlowicki
>
> {code}
> ERROR [CompactionExecutor:9845] 2015-11-28 09:26:10,525
> CassandraDaemon.java:227 - Exception in thread
> Thread[CompactionExecutor:9845,1,main]
> java.lang.AssertionError:
> /var/lib/cassandra/data/system/sstable_activity-5a1ff267ace03f128563cfae6103c65e/system-sstable_activity-ka-6335-Data.db
> at
> org.apache.cassandra.io.sstable.SSTableReader.getApproximateKeyCount(SSTableReader.java:268)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:151)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:73)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:59)
> ~[apache-cassandra-2.1.11.jar:2.1.11]
> at
> org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:236)
>
[jira] [Created] (CASSANDRA-10783) Allow literal value as parameter of UDF & UDA
DOAN DuyHai created CASSANDRA-10783:
---
Summary: Allow literal value as parameter of UDF & UDA
Key: CASSANDRA-10783
URL: https://issues.apache.org/jira/browse/CASSANDRA-10783
Project: Cassandra
Issue Type: Improvement
Components: CQL
Environment: Cassandra 2.2.3
Reporter: DOAN DuyHai
Fix For: 2.2.3
I have defined the following UDF
{code:sql}
CREATE OR REPLACE FUNCTION maxOf(current int, testValue int) RETURNS NULL ON
NULL INPUT
RETURNS int
LANGUAGE java
AS 'return Math.max(current,testValue);'
CREATE TABLE maxValue(id int primary key, val int);
INSERT INTO maxValue(id, val) VALUES(1, 100);
SELECT maxOf(val, 101) FROM maxValue WHERE id=1;
{code}
I got the following error message:
{code}
SyntaxException:
{code}
It would be nice to allow literal value as parameter of UDF and UDA too.
I was thinking about an use-case for an UDA groupBy() function where the end
user can *inject* at runtime a literal value to select which aggregation he
want to display, something similar to GROUP BY ... HAVING
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9988) Introduce leaf-only iterator
[ https://issues.apache.org/jira/browse/CASSANDRA-9988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Jastrzebski updated CASSANDRA-9988: - Attachment: trunk-9988.txt > Introduce leaf-only iterator > > > Key: CASSANDRA-9988 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9988 > Project: Cassandra > Issue Type: Sub-task >Reporter: Benedict >Priority: Minor > Labels: patch > Fix For: 3.x > > Attachments: trunk-9988.txt > > > In many cases we have small btrees, small enough to fit in a single leaf > page. In this case it _may_ be more efficient to specialise our iterator. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9991) Implement efficient btree removal
[ https://issues.apache.org/jira/browse/CASSANDRA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030553#comment-15030553 ] Piotr Jastrzebski commented on CASSANDRA-9991: -- I made the code more efficient. Please have another look. > Implement efficient btree removal > - > > Key: CASSANDRA-9991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9991 > Project: Cassandra > Issue Type: Sub-task >Reporter: Benedict > Labels: patch > Fix For: 3.x > > Attachments: trunk-9991-v3.txt, trunk-9991-v4.txt, trunk-9991.txt, > trunk-9991.txt-v2 > > > Currently removal is implemented as a reconstruction by filtering and > iterator over the original btree. This could be much more efficient, editing > just the necessary nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (CASSANDRA-9991) Implement efficient btree removal
[ https://issues.apache.org/jira/browse/CASSANDRA-9991?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Jastrzebski updated CASSANDRA-9991: - Attachment: trunk-9991-v4.txt > Implement efficient btree removal > - > > Key: CASSANDRA-9991 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9991 > Project: Cassandra > Issue Type: Sub-task >Reporter: Benedict > Labels: patch > Fix For: 3.x > > Attachments: trunk-9991-v3.txt, trunk-9991-v4.txt, trunk-9991.txt, > trunk-9991.txt-v2 > > > Currently removal is implemented as a reconstruction by filtering and > iterator over the original btree. This could be much more efficient, editing > just the necessary nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-7904) Repair hangs
[ https://issues.apache.org/jira/browse/CASSANDRA-7904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030622#comment-15030622 ] Anuj Wadehra commented on CASSANDRA-7904: - If CASSANDRA-10113 is the issue,why repair message is mostly getting expired on only one of the nodes in one DC. And why most of the time only remote DC node fails to get the merkle tree message? Also, if you see attached logs, I am seeing hintedhandoff being timeout for the same node for which merkle tree response was missing. Are they related? > Repair hangs > > > Key: CASSANDRA-7904 > URL: https://issues.apache.org/jira/browse/CASSANDRA-7904 > Project: Cassandra > Issue Type: Bug > Environment: C* 2.0.10, ubuntu 14.04, Java HotSpot(TM) 64-Bit Server, > java version "1.7.0_45" >Reporter: Duncan Sands > Attachments: Repair_DEBUG_On_OutboundTcpConnection.txt, > ls-172.18.68.138, ls-192.168.21.13, ls-192.168.60.134, ls-192.168.60.136 > > > Cluster of 22 nodes spread over 4 data centres. Not used on the weekend, so > repair is run on all nodes (in a staggered fashion) on the weekend. Nodetool > options: -par -pr. There is usually some overlap in the repairs: repair on > one node may well still be running when repair is started on the next node. > Repair hangs for some of the nodes almost every weekend. It hung last > weekend, here are the details: > In the whole cluster, only one node had an exception since C* was last > restarted. This node is 192.168.60.136 and the exception is harmless: a > client disconnected abruptly. > tpstats > 4 nodes have a non-zero value for "active" or "pending" in > AntiEntropySessions. These nodes all have Active => 1 and Pending => 1. The > nodes are: > 192.168.21.13 (data centre R) > 192.168.60.134 (data centre A) > 192.168.60.136 (data centre A) > 172.18.68.138 (data centre Z) > compactionstats: > No compactions. All nodes have: > pending tasks: 0 > Active compaction remaining time :n/a > netstats: > All except one node have nothing. One node (192.168.60.131, not one of the > nodes listed in the tpstats section above) has (note the Responses Pending > value of 1): > Mode: NORMAL > Not sending any streams. > Read Repair Statistics: > Attempted: 4233 > Mismatch (Blocking): 0 > Mismatch (Background): 243 > Pool NameActive Pending Completed > Commandsn/a 0 34785445 > Responses n/a 1 38567167 > Repair sessions > I looked for repair sessions that failed to complete. On 3 of the 4 nodes > mentioned in tpstats above I found that they had sent merkle tree requests > and got responses from all but one node. In the log file for the node that > failed to respond there is no sign that it ever received the request. On 1 > node (172.18.68.138) it looks like responses were received from every node, > some streaming was done, and then... nothing. Details: > Node 192.168.21.13 (data centre R): > Sent merkle trees to /172.18.33.24, /192.168.60.140, /192.168.60.142, > /172.18.68.139, /172.18.68.138, /172.18.33.22, /192.168.21.13 for table > brokers, never got a response from /172.18.68.139. On /172.18.68.139, just > before this time it sent a response for the same repair session but a > different table, and there is no record of it receiving a request for table > brokers. > Node 192.168.60.134 (data centre A): > Sent merkle trees to /172.18.68.139, /172.18.68.138, /192.168.60.132, > /192.168.21.14, /192.168.60.134 for table swxess_outbound, never got a > response from /172.18.68.138. On /172.18.68.138, just before this time it > sent a response for the same repair session but a different table, and there > is no record of it receiving a request for table swxess_outbound. > Node 192.168.60.136 (data centre A): > Sent merkle trees to /192.168.60.142, /172.18.68.139, /192.168.60.136 for > table rollups7200, never got a response from /172.18.68.139. This repair > session is never mentioned in the /172.18.68.139 log. > Node 172.18.68.138 (data centre Z): > The issue here seems to be repair session > #a55c16e1-35eb-11e4-8e7e-51c077eaf311. It got responses for all its merkle > tree requests, did some streaming, but seems to have stopped after finishing > with one table (rollups60). I found it as follows: it is the only repair for > which there is no "session completed successfully" message in the log. > Some log file snippets are attached. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10753) Fix completion problems breaking clqshlib tests
[
https://issues.apache.org/jira/browse/CASSANDRA-10753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030805#comment-15030805
]
Stefania commented on CASSANDRA-10753:
--
bq. I guess we should be safe to update to the released 3.0.0 tag.
OK. There was a little bit of work to do because {{ColumnMetadata}} no longer
has a property called {{index}}. I still have some cqlshlib IO errors on trunk
that I haven't had time to investigate. I will resume mid next week.
bq. That's strange, it shouldn't take that long with such a small amount of
data even on HDDs IMO. Maybe it's a ccm problem? CASSANDRA-8816 reported
slowness on keyspace drop on Windows, so maybe you could run those tests and
check if they're failing in your machine? Is it possible to enable tracing on
drop keyspace statements? It would may be nice see what's making it so slow.
It's not a ccm problem because I am not using it; I'm using a simple
{{cassandra -f}}. I cannot reproduce the failures of CASSANDRA-8816. From what
I could see so far, the driver notifications were sent after several flush
operations. I will add more trace messages and investigate next week. For now
I've left the time-out to 20 seconds on all branches.
CI resubmitted on 2.2, 3.0 and trunk.
> Fix completion problems breaking clqshlib tests
> ---
>
> Key: CASSANDRA-10753
> URL: https://issues.apache.org/jira/browse/CASSANDRA-10753
> Project: Cassandra
> Issue Type: Bug
>Reporter: Stefania
>Assignee: Stefania
> Fix For: 3.0.x
>
>
> The recent changes in the python driver APIs have caused some completion
> problems as indicated by 2 of these failing tests:
> http://cassci.datastax.com/job/trunk_cqlshlib/579/testReport/
> The third failing test, {{test_timestamp_output}}, has been failing for some
> time due to uncertainty on what to do regarding timezone conversion but it
> too can be changed to reflect the fact that we convert the timestamp to UTC.
> Finally, {{max_window_size_seconds}} was recently added to the compaction
> properties and this caused 2 more tests to fail. It cannot be seen on Jenkins
> because of the relative import problem introduced by CASSANDRA-9304.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-10243) Warn or fail when changing cluster topology live
[ https://issues.apache.org/jira/browse/CASSANDRA-10243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15030772#comment-15030772 ] Stefania commented on CASSANDRA-10243: -- I've updated the test, thanks. I'll hold the P.R. until 9474 is committed since I want to test it locally again first. > Warn or fail when changing cluster topology live > > > Key: CASSANDRA-10243 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10243 > Project: Cassandra > Issue Type: Improvement > Components: Tools >Reporter: Jonathan Ellis >Assignee: Stefania >Priority: Critical > Fix For: 2.1.12, 2.2.4, 3.0.1, 3.1, 3.2 > > > Moving a node from one rack to another in the snitch, while it is alive, is > almost always the wrong thing to do. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
