[jira] [Updated] (CASSANDRA-18420) Connection without username not logged in auditlog
[
https://issues.apache.org/jira/browse/CASSANDRA-18420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-18420:
Description:
Hi,
If making connection *without username* to cassandra cluster with
PasswordAuthenticator enabled,
Connection will fail but not logged on auditlog.
How to reproduce:
# Enable "authenticator: PasswordAuthenticator" on cluster
# Enable audit : "nodetool enableauditlog"
# Open a new screen and run "auditlogviewer -f /audit/"
# Try to connect, and connection will fail:
{code:java}
[root@c1 ~]# cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
AuthenticationFailed('Remote end requires authentication',)}){code}
# *But nothing in auditlogviewer*.
Connection with incorrect usernames or password logged correct on auditlog ,
the problem only on connection without username.
How it's affecting:
# Security reason, hard to find unauthorized connections attempt .
# When migrating cluster into PasswordAuthenticator, hard to find applications
that didn't add username/password.
Thank you.
was:
Hi,
If making connection *without username* to cassandra cluster with
PasswordAuthenticator enabled,
Connection will fail but not logged on auditlog.
How to reproduce:
# Enable "authenticator: PasswordAuthenticator" on cluster
# Enable audit : "nodetool enableauditlog"
# Open a new screen and run "auditlogviewer -f /audit/"
# Try to connect, and connection will fail:
{code:java}
[root@c1 ~]# cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
AuthenticationFailed('Remote end requires authentication',)}){code}
# *But nothing in auditlogviewer*.
Connection with incorrect usernames or password logged correct on auditlog ,
the problem only on connection without username.
How it's affecting:
# Security reason, hard to find unauthorized connection .
# When migrating cluster into PasswordAuthenticator, hard to find applications
that didn't add username/password.
Thank you.
> Connection without username not logged in auditlog
> ---
>
> Key: CASSANDRA-18420
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18420
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/auditlogging
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi,
> If making connection *without username* to cassandra cluster with
> PasswordAuthenticator enabled,
> Connection will fail but not logged on auditlog.
> How to reproduce:
> # Enable "authenticator: PasswordAuthenticator" on cluster
> # Enable audit : "nodetool enableauditlog"
> # Open a new screen and run "auditlogviewer -f /audit/"
> # Try to connect, and connection will fail:
> {code:java}
> [root@c1 ~]# cqlsh
> Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
> AuthenticationFailed('Remote end requires authentication',)}){code}
> # *But nothing in auditlogviewer*.
> Connection with incorrect usernames or password logged correct on auditlog ,
> the problem only on connection without username.
> How it's affecting:
> # Security reason, hard to find unauthorized connections attempt .
> # When migrating cluster into PasswordAuthenticator, hard to find
> applications that didn't add username/password.
> Thank you.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-18420) Connection without username not logged in auditlog
[
https://issues.apache.org/jira/browse/CASSANDRA-18420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-18420:
Description:
Hi,
If making connection *without username* to cassandra cluster with
PasswordAuthenticator enabled,
Connection will fail but not logged on auditlog.
How to reproduce:
# Enable "authenticator: PasswordAuthenticator" on cluster
# Enable audit : "nodetool enableauditlog"
# Open a new screen and run "auditlogviewer -f /audit/"
# Try to connect, and connection will fail:
{code:java}
[root@c1 ~]# cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
AuthenticationFailed('Remote end requires authentication',)}){code}
# *But nothing in auditlogviewer*.
Connection with incorrect usernames or password logged correct on auditlog ,
the problem only on connection without username.
How it's affecting:
# Security reason, hard to find unauthorized connection .
# When migrating cluster into PasswordAuthenticator, hard to find applications
that didn't add username/password.
Thank you.
was:
Hi,
If making connection *without username* to cassandra cluster with
PasswordAuthenticator enabled,
Connection will fail but not logged on auditlog.
How to reproduce:
# Enable "authenticator: PasswordAuthenticator" on cluster
# Enable audit : "nodetool enableauditlog"
# Open a new screen and run "auditlogviewer -f /audit/"
# Try to connect, and connection will fail:
{code:java}
[root@c1 ~]# cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
AuthenticationFailed('Remote end requires authentication',)}){code}
# *But nothing in auditlogviewer*
How it's affecting:
# Security reason, hard to find unauthorized connection .
# When migrating cluster into PasswordAuthenticator, hard to find applications
that didn't add username/password.
Thank you.
> Connection without username not logged in auditlog
> ---
>
> Key: CASSANDRA-18420
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18420
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/auditlogging
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi,
> If making connection *without username* to cassandra cluster with
> PasswordAuthenticator enabled,
> Connection will fail but not logged on auditlog.
> How to reproduce:
> # Enable "authenticator: PasswordAuthenticator" on cluster
> # Enable audit : "nodetool enableauditlog"
> # Open a new screen and run "auditlogviewer -f /audit/"
> # Try to connect, and connection will fail:
> {code:java}
> [root@c1 ~]# cqlsh
> Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
> AuthenticationFailed('Remote end requires authentication',)}){code}
> # *But nothing in auditlogviewer*.
> Connection with incorrect usernames or password logged correct on auditlog ,
> the problem only on connection without username.
> How it's affecting:
> # Security reason, hard to find unauthorized connection .
> # When migrating cluster into PasswordAuthenticator, hard to find
> applications that didn't add username/password.
> Thank you.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-18420) Connection without username not logged in auditlog
Yakir Gibraltar created CASSANDRA-18420:
---
Summary: Connection without username not logged in auditlog
Key: CASSANDRA-18420
URL: https://issues.apache.org/jira/browse/CASSANDRA-18420
Project: Cassandra
Issue Type: Bug
Components: Tool/auditlogging
Reporter: Yakir Gibraltar
Hi,
If making connection *without username* to cassandra cluster with
PasswordAuthenticator enabled,
Connection will fail but not logged on auditlog.
How to reproduce:
# Enable "authenticator: PasswordAuthenticator" on cluster
# Enable audit : "nodetool enableauditlog"
# Open a new screen and run "auditlogviewer -f /audit/"
# Try to connect, and connection will fail:
{code:java}
[root@c1 ~]# cqlsh
Connection error: ('Unable to connect to any servers', {'127.0.0.1:9042':
AuthenticationFailed('Remote end requires authentication',)}){code}
# *But nothing in auditlogviewer*
How it's affecting:
# Security reason, hard to find unauthorized connection .
# When migrating cluster into PasswordAuthenticator, hard to find applications
that didn't add username/password.
Thank you.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-17426) metrics.html with UI bug
Yakir Gibraltar created CASSANDRA-17426: --- Summary: metrics.html with UI bug Key: CASSANDRA-17426 URL: https://issues.apache.org/jira/browse/CASSANDRA-17426 Project: Cassandra Issue Type: Bug Reporter: Yakir Gibraltar Attachments: image-2022-03-09-11-41-28-785.png Hi, Looks that https://cassandra.apache.org/doc/3.11/cassandra/operating/metrics.html with some UI bugs. !image-2022-03-09-11-41-28-785.png! -- This message was sent by Atlassian Jira (v8.20.1#820001) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16437) fault occurred in a recent unsafe memory access operation
[
https://issues.apache.org/jira/browse/CASSANDRA-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16437:
Resolution: Not A Problem
Status: Resolved (was: Triage Needed)
> fault occurred in a recent unsafe memory access operation
> -
>
> Key: CASSANDRA-16437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16437
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, testing Cassandra 4~beta4 and JDK 11, all nodes crashed with error of :
> {code}
> ERROR [MutationStage-55] 2021-02-10 01:07:15,015
> AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread
> Thread[MutationStage-55,5,main]
> java.lang.RuntimeException: java.lang.InternalError: a fault occurred in a
> recent unsafe memory access operation
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:104)
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:44)
> at
> org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.InternalError: a fault occurred in a recent unsafe
> memory access operation
> {code}
> Using Azul Zing JDK:
> zing-zst-8c.dkms.2.6.32-5.22.8.0.4.el.x86_64
> zing-jdk11.0.0-20.08.101.0-1.x86_64
> OS: CentOS 7, kernel 5.0.1
> The crash accrued after some stress tests on cluster.
> Thank you, Yakir Gibraltar
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16437) fault occurred in a recent unsafe memory access operation
[
https://issues.apache.org/jira/browse/CASSANDRA-16437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17282341#comment-17282341
]
Yakir Gibraltar commented on CASSANDRA-16437:
-
The disk was full
> fault occurred in a recent unsafe memory access operation
> -
>
> Key: CASSANDRA-16437
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16437
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, testing Cassandra 4~beta4 and JDK 11, all nodes crashed with error of :
> {code}
> ERROR [MutationStage-55] 2021-02-10 01:07:15,015
> AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread
> Thread[MutationStage-55,5,main]
> java.lang.RuntimeException: java.lang.InternalError: a fault occurred in a
> recent unsafe memory access operation
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:104)
> at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:44)
> at
> org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
> at
> org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
> at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Thread.java:834)
> Caused by: java.lang.InternalError: a fault occurred in a recent unsafe
> memory access operation
> {code}
> Using Azul Zing JDK:
> zing-zst-8c.dkms.2.6.32-5.22.8.0.4.el.x86_64
> zing-jdk11.0.0-20.08.101.0-1.x86_64
> OS: CentOS 7, kernel 5.0.1
> The crash accrued after some stress tests on cluster.
> Thank you, Yakir Gibraltar
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16437) fault occurred in a recent unsafe memory access operation
Yakir Gibraltar created CASSANDRA-16437:
---
Summary: fault occurred in a recent unsafe memory access operation
Key: CASSANDRA-16437
URL: https://issues.apache.org/jira/browse/CASSANDRA-16437
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Hi, testing Cassandra 4~beta4 and JDK 11, all nodes crashed with error of :
{code}
ERROR [MutationStage-55] 2021-02-10 01:07:15,015
AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread
Thread[MutationStage-55,5,main]
java.lang.RuntimeException: java.lang.InternalError: a fault occurred in a
recent unsafe memory access operation
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:104)
at org.apache.cassandra.net.InboundSink.accept(InboundSink.java:44)
at
org.apache.cassandra.net.InboundMessageHandler$ProcessMessage.run(InboundMessageHandler.java:432)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
at
org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:119)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
Caused by: java.lang.InternalError: a fault occurred in a recent unsafe memory
access operation
{code}
Using Azul Zing JDK:
zing-zst-8c.dkms.2.6.32-5.22.8.0.4.el.x86_64
zing-jdk11.0.0-20.08.101.0-1.x86_64
OS: CentOS 7, kernel 5.0.1
The crash accrued after some stress tests on cluster.
Thank you, Yakir Gibraltar
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17262198#comment-17262198
]
Yakir Gibraltar commented on CASSANDRA-16279:
-
Thank you [~aholmber] and [~mck]
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Assignee: Adam Holmberg
>Priority: Normal
> Fix For: 4.0-beta5, 4.0
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {code}
> echo "$JVM_OPTS" | grep -q "^-[X]log:gc"
> {code}
> Should be:
> {code}
> echo "$JVM_OPTS" | grep -qe "-[X]log:gc"
> {code}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
>
> Right now, jvm11-server.options with:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
> Will generate process of:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
>
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
> {code}
> With fix it will generate correct gc option of :
> {code}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17252524#comment-17252524
]
Yakir Gibraltar commented on CASSANDRA-16279:
-
Hi [~aholmber], please do it.
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {code}
> echo "$JVM_OPTS" | grep -q "^-[X]log:gc"
> {code}
> Should be:
> {code}
> echo "$JVM_OPTS" | grep -qe "-[X]log:gc"
> {code}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
>
> Right now, jvm11-server.options with:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
> Will generate process of:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
>
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
> {code}
> With fix it will generate correct gc option of :
> {code}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16283) Incorrect output in "nodetool status -r"
[
https://issues.apache.org/jira/browse/CASSANDRA-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17248448#comment-17248448
]
Yakir Gibraltar commented on CASSANDRA-16283:
-
Thank you [~wolfenhaut] :)
> Incorrect output in "nodetool status -r"
>
>
> Key: CASSANDRA-16283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16283
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Assignee: Scott Wolfenhaut
>Priority: Low
> Fix For: 4.0-beta
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> nodetool status -r not working well on C* 4,
> Version:
> {code:java}
> [root@foo001 ~]# nodetool version
> ReleaseVersion: 4.0-beta3
> {code}
> Without resolving:
> {code:java}
> [root@foo001 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns(effective) Host ID
> Rack
> UN 1.2.3.4 363.68 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 1.2.3.5 109.71 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> {code}
> With resolving:
> {code:java}
> [root@foo001 ~]# nodetool status -r
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> ?N foo001.tab.com ? 128 ? RAC1
> ?N foo002.tab.com ? 128 ? RAC1
> {code}
> I only changed here IPs and hostnames.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243587#comment-17243587
]
Yakir Gibraltar commented on CASSANDRA-16296:
-
I just change the following keyspace from RF 3 to 2 on the DC that i tried to
join and it's solved the issue:
{code}
CREATE KEYSPACE ks WITH replication = {'class': 'NetworkTopologyStrategy',
'V4C': '3', 'V4N': '3'} AND durable_writes = true;
{code}
> Join new node to cluster failing on C* version 4
>
>
> Key: CASSANDRA-16296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Yakir Gibraltar
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Hi, i'm using cassandra 4.0-beta3,
> Join new node failing,
> Current status:
> {code}
> [root@rc801 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 1.1.1.1 356.55 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 2.2.2.2 424.36 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> Datacenter: V4NJ
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 3.3.3.3 322.97 KiB 128 ?
> 0106ce0b-18be-4321-8247-843076ff42e6 RAC1
> UN 4.4.4.4 297.6 KiB 128 ?
> d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
> UN 5.5.5.5 324.75 KiB 128 ?
> 5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
> {code}
> Error on new host (7.7.7.7):
> {code}
> INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap:
> range Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists
> on Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for
> keyspace system_traces
> ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Multiple strict sources found for
> Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
> [Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
> Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849
> - No local state, state is in silent shutdown, or node hasn't joined, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
> MessagingService.java:441 - Waiting for messaging service to quiesce
> {code}
> Errors on seed node:
> {code}
> INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
> InboundConnectionInitiator.java:459 -
> /7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba
> messaging connection established, version = 12, framing = LZ4, encryption =
> disabled
> INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
> OutboundConnection.java:1151 -
> /1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
> successfully connected, version = 12, framing = LZ4, encryption = disabled
> INFO [GossipStage:1]
[jira] [Commented] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17243563#comment-17243563
]
Yakir Gibraltar commented on CASSANDRA-16296:
-
Hi [~dcapwell] ,
"have you repeated the process on 3.x line? How did you "join" with the new
node? "
I tried to join few times from different hosts, all join failed, with simple
"systemctl start cassandra"
" Was this upgrading the cluster or creating a new one?"
Fresh cluster of version 4 without any data.
> Join new node to cluster failing on C* version 4
>
>
> Key: CASSANDRA-16296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Yakir Gibraltar
>Assignee: Benjamin Lerer
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Hi, i'm using cassandra 4.0-beta3,
> Join new node failing,
> Current status:
> {code}
> [root@rc801 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 1.1.1.1 356.55 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 2.2.2.2 424.36 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> Datacenter: V4NJ
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 3.3.3.3 322.97 KiB 128 ?
> 0106ce0b-18be-4321-8247-843076ff42e6 RAC1
> UN 4.4.4.4 297.6 KiB 128 ?
> d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
> UN 5.5.5.5 324.75 KiB 128 ?
> 5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
> {code}
> Error on new host (7.7.7.7):
> {code}
> INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap:
> range Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists
> on Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for
> keyspace system_traces
> ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Multiple strict sources found for
> Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
> [Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
> Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849
> - No local state, state is in silent shutdown, or node hasn't joined, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
> MessagingService.java:441 - Waiting for messaging service to quiesce
> {code}
> Errors on seed node:
> {code}
> INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
> InboundConnectionInitiator.java:459 -
> /7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba
> messaging connection established, version = 12, framing = LZ4, encryption =
> disabled
> INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
> OutboundConnection.java:1151 -
> /1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
> successfully connected, version = 12,
[jira] [Comment Edited] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238243#comment-17238243
]
Yakir Gibraltar edited comment on CASSANDRA-16296 at 11/24/20, 4:29 PM:
Update, we have found the root cause, we added a keyspace with RF=3 but we had
only 2 nodes in DC and join the 3rd node failed.
I think that:
# Create keyspace should fail if RF>"number of nodes in DC".
# The exception thrown wasn't clear.
ALTER the keyspace to RF=2 solved the issue.
was (Author: yakir.g):
Update, we have found the root cause, we added a keyspace with RF=3 but we had
only 2 nodes in DC and join the 3th node failed.
I think that:
# Create keyspace should fail if RF>"number of nodes in DC".
# The exception thrown wasn't clear.
ALTER the keyspace to RF=2 solved the issue.
> Join new node to cluster failing on C* version 4
>
>
> Key: CASSANDRA-16296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm using cassandra 4.0-beta3,
> Join new node failing,
> Current status:
> {code}
> [root@rc801 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 1.1.1.1 356.55 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 2.2.2.2 424.36 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> Datacenter: V4NJ
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 3.3.3.3 322.97 KiB 128 ?
> 0106ce0b-18be-4321-8247-843076ff42e6 RAC1
> UN 4.4.4.4 297.6 KiB 128 ?
> d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
> UN 5.5.5.5 324.75 KiB 128 ?
> 5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
> {code}
> Error on new host (7.7.7.7):
> {code}
> INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap:
> range Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists
> on Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for
> keyspace system_traces
> ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Multiple strict sources found for
> Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
> [Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
> Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849
> - No local state, state is in silent shutdown, or node hasn't joined, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
> MessagingService.java:441 - Waiting for messaging service to quiesce
> {code}
> Errors on seed node:
> {code}
> INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
> InboundConnectionInitiator.java:459 -
> /7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba
> messaging connection established, version = 12, framing = LZ4, encryption =
>
[jira] [Commented] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238243#comment-17238243
]
Yakir Gibraltar commented on CASSANDRA-16296:
-
Update, we have found the root cause, we added a keyspace with RF=3 but we had
only 2 nodes in DC and join the 3th node failed.
I think that:
# Create keyspace should fail if RF>"number of nodes in DC".
# The exception thrown wasn't clear.
ALTER the keyspace to RF=2 solved the issue.
> Join new node to cluster failing on C* version 4
>
>
> Key: CASSANDRA-16296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm using cassandra 4.0-beta3,
> Join new node failing,
> Current status:
> {code}
> [root@rc801 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 1.1.1.1 356.55 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 2.2.2.2 424.36 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> Datacenter: V4NJ
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 3.3.3.3 322.97 KiB 128 ?
> 0106ce0b-18be-4321-8247-843076ff42e6 RAC1
> UN 4.4.4.4 297.6 KiB 128 ?
> d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
> UN 5.5.5.5 324.75 KiB 128 ?
> 5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
> {code}
> Error on new host (7.7.7.7):
> {code}
> INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap:
> range Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists
> on Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for
> keyspace system_traces
> ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Multiple strict sources found for
> Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
> [Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
> Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849
> - No local state, state is in silent shutdown, or node hasn't joined, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
> MessagingService.java:441 - Waiting for messaging service to quiesce
> {code}
> Errors on seed node:
> {code}
> INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
> InboundConnectionInitiator.java:459 -
> /7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba
> messaging connection established, version = 12, framing = LZ4, encryption =
> disabled
> INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
> OutboundConnection.java:1151 -
> /1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
> successfully connected, version = 12, framing = LZ4, encryption = disabled
> INFO [GossipStage:1] 2020-11-22 09:26:56,109 Gossiper.java:1248 - Node
> /7.7.7.7:7000 is now part of the
[jira] [Commented] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17238236#comment-17238236
]
Yakir Gibraltar commented on CASSANDRA-16296:
-
Hi [~aweisberg], any ideas?
> Join new node to cluster failing on C* version 4
>
>
> Key: CASSANDRA-16296
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm using cassandra 4.0-beta3,
> Join new node failing,
> Current status:
> {code}
> [root@rc801 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 1.1.1.1 356.55 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 2.2.2.2 424.36 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> Datacenter: V4NJ
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns (effective) Host ID
>Rack
> UN 3.3.3.3 322.97 KiB 128 ?
> 0106ce0b-18be-4321-8247-843076ff42e6 RAC1
> UN 4.4.4.4 297.6 KiB 128 ?
> d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
> UN 5.5.5.5 324.75 KiB 128 ?
> 5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
> {code}
> Error on new host (7.7.7.7):
> {code}
> INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap:
> range Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists
> on Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for
> keyspace system_traces
> ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
> encountered during startup
> java.lang.IllegalStateException: Multiple strict sources found for
> Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
> [Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
> Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
> at
> org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
> at
> org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
> at
> org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
> at
> org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
> at
> org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
> at
> org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
> at
> org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
> HintsService.java:220 - Paused hints dispatch
> WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849
> - No local state, state is in silent shutdown, or node hasn't joined, not
> announcing shutdown
> INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
> MessagingService.java:441 - Waiting for messaging service to quiesce
> {code}
> Errors on seed node:
> {code}
> INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
> InboundConnectionInitiator.java:459 -
> /7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba
> messaging connection established, version = 12, framing = LZ4, encryption =
> disabled
> INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
> OutboundConnection.java:1151 -
> /1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
> successfully connected, version = 12, framing = LZ4, encryption = disabled
> INFO [GossipStage:1] 2020-11-22 09:26:56,109 Gossiper.java:1248 - Node
> /7.7.7.7:7000 is now part of the cluster
> INFO [GossipStage:1] 2020-11-22 09:26:56,147 Gossiper.java:1206 -
> InetAddress /7.7.7.7:7000 is now UP
> INFO [Messaging-EventLoop-3-1] 2020-11-22 09:26:57,378
> InboundConnectionInitiator.java:459 -
>
[jira] [Updated] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
[
https://issues.apache.org/jira/browse/CASSANDRA-16296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16296:
Description:
Hi, i'm using cassandra 4.0-beta3,
Join new node failing,
Current status:
{code}
[root@rc801 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns (effective) Host ID
Rack
UN 1.1.1.1 356.55 KiB 128 ?
92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
UN 2.2.2.2 424.36 KiB 128 ?
d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
Datacenter: V4NJ
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns (effective) Host ID
Rack
UN 3.3.3.3 322.97 KiB 128 ?
0106ce0b-18be-4321-8247-843076ff42e6 RAC1
UN 4.4.4.4 297.6 KiB 128 ?
d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
UN 5.5.5.5 324.75 KiB 128 ?
5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
{code}
Error on new host (7.7.7.7):
{code}
INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap: range
Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists on
Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for keyspace
system_traces
ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
encountered during startup
java.lang.IllegalStateException: Multiple strict sources found for
Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
[Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
at
org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
at
org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849 -
No local state, state is in silent shutdown, or node hasn't joined, not
announcing shutdown
INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
MessagingService.java:441 - Waiting for messaging service to quiesce
{code}
Errors on seed node:
{code}
INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
InboundConnectionInitiator.java:459 -
/7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba messaging
connection established, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
OutboundConnection.java:1151 -
/1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
successfully connected, version = 12, framing = LZ4, encryption = disabled
INFO [GossipStage:1] 2020-11-22 09:26:56,109 Gossiper.java:1248 - Node
/7.7.7.7:7000 is now part of the cluster
INFO [GossipStage:1] 2020-11-22 09:26:56,147 Gossiper.java:1206 - InetAddress
/7.7.7.7:7000 is now UP
INFO [Messaging-EventLoop-3-1] 2020-11-22 09:26:57,378
InboundConnectionInitiator.java:459 -
/7.7.7.7:7000(/7.7.7.7:45504)->/1.1.1.1:7000-SMALL_MESSAGES-77c813d4 messaging
connection established, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-38] 2020-11-22 09:26:57,383
OutboundConnection.java:1151 -
/1.1.1.1:7000(/1.1.1.1:38680)->/7.7.7.7:7000-SMALL_MESSAGES-e75eaaa2
successfully connected, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-2] 2020-11-22 09:27:36,623
InboundConnectionInitiator.java:459 -
/7.7.7.7:7000(/7.7.7.7:45534)->/1.1.1.1:7000-LARGE_MESSAGES-defc02dd messaging
connection established, version = 12, framing = LZ4, encryption
[jira] [Created] (CASSANDRA-16296) Join new node to cluster failing on C* version 4
Yakir Gibraltar created CASSANDRA-16296:
---
Summary: Join new node to cluster failing on C* version 4
Key: CASSANDRA-16296
URL: https://issues.apache.org/jira/browse/CASSANDRA-16296
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Hi, i'm using cassandra 4.0-beta3,
Join new node failing,
Current status:
{code}
[root@rc801 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns (effective) Host ID
Rack
UN 1.1.1.1 356.55 KiB 128 ?
92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
UN 2.2.2.2 424.36 KiB 128 ?
d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
Datacenter: V4NJ
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns (effective) Host ID
Rack
UN 3.3.3.3 322.97 KiB 128 ?
0106ce0b-18be-4321-8247-843076ff42e6 RAC1
UN 4.4.4.4 297.6 KiB 128 ?
d81e2b7d-9221-4b1b-86c8-e8956e3eec07 RAC1
UN 5.5.5.5 324.75 KiB 128 ?
5dfbaf14-8310-4d64-b61a-9da0a7a8a620 RAC1
{code}
Error on new host (7.7.7.7):
{code}
INFO [main] 2020-11-22 09:27:36,592 RangeStreamer.java:323 - Bootstrap: range
Full(/7.7.7.7:7000,(-7734271291904743823,-7725025391901227341]) exists on
Full(/1.1.1.1:7000,(-7734271291904743823,-7725025391901227341]) for keyspace
system_traces
ERROR [main] 2020-11-22 09:27:36,609 CassandraDaemon.java:817 - Exception
encountered during startup
java.lang.IllegalStateException: Multiple strict sources found for
Full(/7.7.7.7:7000,(9208454697546654053,-9212763148842799409]), sources:
[Full(/2.2.2.2:7000,(9189373804905478622,-9212763148842799409]),
Full(/1.1.1.1:7000,(9189373804905478622,-9212763148842799409])]
at
org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:517)
at
org.apache.cassandra.dht.RangeStreamer.calculateRangesToFetchWithPreferredEndpoints(RangeStreamer.java:383)
at
org.apache.cassandra.dht.RangeStreamer.addRanges(RangeStreamer.java:320)
at org.apache.cassandra.dht.BootStrapper.bootstrap(BootStrapper.java:81)
at
org.apache.cassandra.service.StorageService.startBootstrap(StorageService.java:1635)
at
org.apache.cassandra.service.StorageService.bootstrap(StorageService.java:1612)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:931)
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:892)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:699)
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:635)
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:407)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:671)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:795)
INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,616
HintsService.java:220 - Paused hints dispatch
WARN [StorageServiceShutdownHook] 2020-11-22 09:27:36,617 Gossiper.java:1849 -
No local state, state is in silent shutdown, or node hasn't joined, not
announcing shutdown
INFO [StorageServiceShutdownHook] 2020-11-22 09:27:36,617
MessagingService.java:441 - Waiting for messaging service to quiesce
{code}
Errors on seed node:
{code}
INFO [Messaging-EventLoop-3-37] 2020-11-22 09:26:54,121
InboundConnectionInitiator.java:459 -
/7.7.7.7:7000(/7.7.7.7:45490)->/1.1.1.1:7000-URGENT_MESSAGES-f412b9ba messaging
connection established, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-40] 2020-11-22 09:26:54,157
OutboundConnection.java:1151 -
/1.1.1.1:7000(/1.1.1.1:38678)->/7.7.7.7:7000-URGENT_MESSAGES-3b23f639
successfully connected, version = 12, framing = LZ4, encryption = disabled
INFO [GossipStage:1] 2020-11-22 09:26:56,109 Gossiper.java:1248 - Node
/7.7.7.7:7000 is now part of the cluster
INFO [GossipStage:1] 2020-11-22 09:26:56,147 Gossiper.java:1206 - InetAddress
/7.7.7.7:7000 is now UP
INFO [Messaging-EventLoop-3-1] 2020-11-22 09:26:57,378
InboundConnectionInitiator.java:459 -
/7.7.7.7:7000(/7.7.7.7:45504)->/1.1.1.1:7000-SMALL_MESSAGES-77c813d4 messaging
connection established, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-38] 2020-11-22 09:26:57,383
OutboundConnection.java:1151 -
/1.1.1.1:7000(/1.1.1.1:38680)->/7.7.7.7:7000-SMALL_MESSAGES-e75eaaa2
successfully connected, version = 12, framing = LZ4, encryption = disabled
INFO [Messaging-EventLoop-3-2] 2020-11-22 09:27:36,623
InboundConnectionInitiator.java:459 -
[jira] [Created] (CASSANDRA-16289) nodetool removenode failing
Yakir Gibraltar created CASSANDRA-16289:
---
Summary: nodetool removenode failing
Key: CASSANDRA-16289
URL: https://issues.apache.org/jira/browse/CASSANDRA-16289
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Hi, i tied to remove node on Cassandra 4 and got a lot of errors and also
nodetool command failed.
{code}
ot working well with C* 4:
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges):
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing:
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help '.
{code}
In log:
{code}
INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..)
failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection
refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of
type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}
How to reproduce:
Create 2 DC and remove one node.
Maybe it possible to reproduce with one DC but in my case was 2 DCs.
I had to remove with "force"
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16289) nodetool removenode failing
[
https://issues.apache.org/jira/browse/CASSANDRA-16289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16289:
Description:
Hi, i tied to remove node on Cassandra 4.0-beta3 and got a lot of errors and
also nodetool command failed.
{code}
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges):
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing:
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help '.
{code}
In log:
{code}
INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..)
failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection
refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of
type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}
How to reproduce:
Create 2 DC and remove one node.
Maybe it possible to reproduce with one DC but in my case was 2 DCs.
I had to remove with "force"
was:
Hi, i tied to remove node on Cassandra 4 and got a lot of errors and also
nodetool command failed.
{code}
ot working well with C* 4:
[root@foo ~]# nodetool removenode 2dfa2fde-15ee-40ad-8afc-0cb6db9aaad9
nodetool: Conflicting replica added (expected unique ranges):
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585]); existing:
Full(/1.1.1.1:7000,(2012127519876180938,2107158795315043585])
See 'nodetool help' or 'nodetool help '.
{code}
In log:
{code}
INFO [Messaging-EventLoop-3-31] 2020-11-19 15:56:48,949 NoSpamLogger.java:92 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] failed to connect
io.netty.channel.AbstractChannel$AnnotatedConnectException: finishConnect(..)
failed: Connection refused: /2.2.2.2:7000
Caused by: java.net.ConnectException: finishConnect(..) failed: Connection
refused
at io.netty.channel.unix.Errors.throwConnectException(Errors.java:124)
at io.netty.channel.unix.Socket.finishConnect(Socket.java:243)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.doFinishConnect(AbstractEpollChannel.java:672)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.finishConnect(AbstractEpollChannel.java:649)
at
io.netty.channel.epoll.AbstractEpollChannel$AbstractEpollUnsafe.epollOutReady(AbstractEpollChannel.java:529)
at
io.netty.channel.epoll.EpollEventLoop.processReady(EpollEventLoop.java:465)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:378)
at
io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at
io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Thread.java:834)
WARN [Messaging-EventLoop-3-31] 2020-11-19 16:01:18,973 NoSpamLogger.java:95 -
/1.1.1.1:7000->/2.2.2.2:7000-URGENT_MESSAGES-[no-channel] dropping message of
type GOSSIP_DIGEST_SYN whose timeout expired before reaching the network
{code}
How to reproduce:
Create 2 DC and remove one node.
Maybe it possible to reproduce with one DC but in my case was 2 DCs.
I had to remove with "force"
> nodetool removenode failing
> ---
>
> Key: CASSANDRA-16289
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16289
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir
[jira] [Updated] (CASSANDRA-16283) Incorrect output in "nodetool status -r"
[
https://issues.apache.org/jira/browse/CASSANDRA-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16283:
Description:
nodetool status -r not working well on C* 4,
Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns(effective) Host ID
Rack
UN 1.2.3.4 363.68 KiB 128 ? 92ae4c39-edb3-4e67-8623-b49fd8301b66
RAC1
UN 1.2.3.5 109.71 KiB 128 ? d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2
RAC1
{code}
With resolving:
{code:java}
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N foo001.tab.com ? 128 ? RAC1
?N foo002.tab.com ? 128 ? RAC1
{code}
I changed IPs and hostnames.
was:
nodetool status -r not working well on C* 4,
Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns(effective) Host ID
Rack
UN 1.2.3.4 363.68 KiB 128 ? 92ae4c39-edb3-4e67-8623-b49fd8301b66
RAC1
UN 1.2.3.5 109.71 KiB 128 ? d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2
RAC1
{code}
With resolving:
{code:java}
I changed IPs and hostnames.
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N foo001.tab.com ? 128 ? RAC1
?N foo002.tab.com ? 128 ? RAC1
{code}
> Incorrect output in "nodetool status -r"
>
>
> Key: CASSANDRA-16283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16283
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> nodetool status -r not working well on C* 4,
> Version:
> {code:java}
> [root@foo001 ~]# nodetool version
> ReleaseVersion: 4.0-beta3
> {code}
> Without resolving:
> {code:java}
> [root@foo001 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns(effective) Host ID
> Rack
> UN 1.2.3.4 363.68 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 1.2.3.5 109.71 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> {code}
> With resolving:
> {code:java}
> [root@foo001 ~]# nodetool status -r
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> ?N foo001.tab.com ? 128 ? RAC1
> ?N foo002.tab.com ? 128 ? RAC1
> {code}
> I changed IPs and hostnames.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16283) Incorrect output in "nodetool status -r"
Yakir Gibraltar created CASSANDRA-16283:
---
Summary: Incorrect output in "nodetool status -r"
Key: CASSANDRA-16283
URL: https://issues.apache.org/jira/browse/CASSANDRA-16283
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
nodetool status -r not working well on C* 4,
Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns(effective) Host ID
Rack
UN 1.2.3.4 363.68 KiB 128 ? 92ae4c39-edb3-4e67-8623-b49fd8301b66
RAC1
UN 1.2.3.5 109.71 KiB 128 ? d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2
RAC1
{code}
With resolving:
{code:java}
I changed IPs and hostnames.
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N foo001.tab.com ? 128 ? RAC1
?N foo002.tab.com ? 128 ? RAC1
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16283) Incorrect output in "nodetool status -r"
[
https://issues.apache.org/jira/browse/CASSANDRA-16283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16283:
Description:
nodetool status -r not working well on C* 4,
Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns(effective) Host ID
Rack
UN 1.2.3.4 363.68 KiB 128 ? 92ae4c39-edb3-4e67-8623-b49fd8301b66
RAC1
UN 1.2.3.5 109.71 KiB 128 ? d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2
RAC1
{code}
With resolving:
{code:java}
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N foo001.tab.com ? 128 ? RAC1
?N foo002.tab.com ? 128 ? RAC1
{code}
I only changed here IPs and hostnames.
was:
nodetool status -r not working well on C* 4,
Version:
{code:java}
[root@foo001 ~]# nodetool version
ReleaseVersion: 4.0-beta3
{code}
Without resolving:
{code:java}
[root@foo001 ~]# nodetool status
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address LoadTokens Owns(effective) Host ID
Rack
UN 1.2.3.4 363.68 KiB 128 ? 92ae4c39-edb3-4e67-8623-b49fd8301b66
RAC1
UN 1.2.3.5 109.71 KiB 128 ? d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2
RAC1
{code}
With resolving:
{code:java}
[root@foo001 ~]# nodetool status -r
Datacenter: V4CH
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID Rack
?N foo001.tab.com ? 128 ? RAC1
?N foo002.tab.com ? 128 ? RAC1
{code}
I changed IPs and hostnames.
> Incorrect output in "nodetool status -r"
>
>
> Key: CASSANDRA-16283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16283
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> nodetool status -r not working well on C* 4,
> Version:
> {code:java}
> [root@foo001 ~]# nodetool version
> ReleaseVersion: 4.0-beta3
> {code}
> Without resolving:
> {code:java}
> [root@foo001 ~]# nodetool status
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address LoadTokens Owns(effective) Host ID
> Rack
> UN 1.2.3.4 363.68 KiB 128 ?
> 92ae4c39-edb3-4e67-8623-b49fd8301b66 RAC1
> UN 1.2.3.5 109.71 KiB 128 ?
> d80647a8-32b2-4a8f-8022-f5ae3ce8fbb2 RAC1
> {code}
> With resolving:
> {code:java}
> [root@foo001 ~]# nodetool status -r
> Datacenter: V4CH
>
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID Rack
> ?N foo001.tab.com ? 128 ? RAC1
> ?N foo002.tab.com ? 128 ? RAC1
> {code}
> I only changed here IPs and hostnames.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-16273) nodetool status owns (effective) question mark
[
https://issues.apache.org/jira/browse/CASSANDRA-16273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17234540#comment-17234540
]
Yakir Gibraltar commented on CASSANDRA-16273:
-
Same issue for me with Cassandra 4:
{code:java}
cqlsh> DESC KEYSPACE test;
CREATE KEYSPACE test WITH replication = {'class': 'NetworkTopologyStrategy',
'V4CH': '3'} AND durable_writes = true; {code}
> nodetool status owns (effective) question mark
> --
>
> Key: CASSANDRA-16273
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16273
> Project: Cassandra
> Issue Type: Bug
>Reporter: AaronTrazona
>Priority: Normal
> Fix For: 3.11.x
>
> Attachments: image-2020-11-13-13-55-12-609.png,
> image-2020-11-16-08-21-08-580.png, image-2020-11-16-08-23-18-474.png,
> image-2020-11-16-09-30-14-287.png, image-2020-11-16-09-34-23-103.png,
> image-2020-11-16-09-41-12-804.png, image-2020-11-16-12-42-56-725.png,
> image-2020-11-16-13-58-08-395.png
>
>
> !image-2020-11-13-13-55-12-609.png!
> I'm wondering why the owns (effective) become question mark..
> I already enabled the
> JVM_OPTS = "$JVM_OPTS -Djava.rmi.server.hostname=public name"
>
> Let me know if I missed something..
>
> Thanks
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16279:
Description:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{code}
echo "$JVM_OPTS" | grep -q "^-[X]log:gc"
{code}
Should be:
{code}
echo "$JVM_OPTS" | grep -qe "-[X]log:gc"
{code}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
Right now, jvm11-server.options with:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
Will generate process of:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
{code}
With fix it will generate correct gc option of :
{code}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
{code}
was:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
Right now, jvm11-server.options with:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
Will generate process of:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
{code}
With fix it will generate correct gc option of :
{code}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
{code}
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {code}
> echo "$JVM_OPTS" | grep -q "^-[X]log:gc"
> {code}
> Should be:
> {code}
> echo "$JVM_OPTS" | grep -qe "-[X]log:gc"
> {code}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
>
> Right now, jvm11-server.options with:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
> Will generate process of:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
>
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
> {code}
> With fix it will generate correct gc option of :
> {code}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail:
[jira] [Updated] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16279:
Description:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
Right now, jvm11-server.options with:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
Will generate process of:
{code:java}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
{code}
With fix it will generate correct gc option of :
{code}
-Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
{code}
was:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
Right now, let's say that i added
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
> Should be:
> {{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
>
> Right now, jvm11-server.options with:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M{code}
> Will generate process of:
> {code:java}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
>
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=10485760
> {code}
> With fix it will generate correct gc option of :
> {code}
> -Xlog:gc=info,heap*=trace,age*=debug,safepoint=info,promotion*=trace:file=/var/log/cassandra/gc.log:time,uptime,pid,tid,level:filecount=10,filesize=100M
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16279:
Description:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
Right now, let's say that i added
was:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^\-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "\-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{\-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
> Should be:
> {{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
>
> Right now, let's say that i added
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Updated] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
[
https://issues.apache.org/jira/browse/CASSANDRA-16279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-16279:
Description:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^\-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "\-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{\-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
was:
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
> Incorrect check for -Xlog in cassandra-env.sh
> --
>
> Key: CASSANDRA-16279
> URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
> Project: Cassandra
> Issue Type: Bug
> Components: Packaging
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 4.0-beta
>
>
> Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
> [https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
> The following command incorrect:
> {{echo "$JVM_OPTS" | grep -q "^\-[X]log:gc"}}
> Should be:
> {{echo "$JVM_OPTS" | grep -qe "\-[X]log:gc"}}
> The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always
> return 1, remove {{^}} and add {{\-qe}} will solve the problem.
> It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
> {{jvm11-server.options}} and {{jvm8-server.options}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Created] (CASSANDRA-16279) Incorrect check for -Xlog in cassandra-env.sh
Yakir Gibraltar created CASSANDRA-16279:
---
Summary: Incorrect check for -Xlog in cassandra-env.sh
Key: CASSANDRA-16279
URL: https://issues.apache.org/jira/browse/CASSANDRA-16279
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Possible small bug in {{cassandra-env.sh}} of Cassandra 4:
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-qe}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-9608) Support Java 11
[
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233556#comment-17233556
]
Yakir Gibraltar edited comment on CASSANDRA-9608 at 11/17/20, 12:41 PM:
Hi [~snazy] [~benedict] [~jasobrown] Possible small bug in {{cassandra-env.sh:}}
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-eq}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
was (Author: yakir.g):
Hi [~snazy] Possible small bug in {{cassandra-env.sh:}}
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^\-\[X\]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "\-\[X\]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-eq}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
> Support Java 11
> ---
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
> Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
> Labels: Java11
> Fix For: 4.0, 4.0-alpha1
>
> Attachments: jdk_9_10.patch
>
>
> This ticket is intended to group all issues found to support Java 9 in the
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved.
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods
> {{monitorEnter}} + {{monitorExit}}. These methods are used by
> {{o.a.c.utils.concurrent.Locks}} which is only used by
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early
> development phase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Comment Edited] (CASSANDRA-9608) Support Java 11
[
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233556#comment-17233556
]
Yakir Gibraltar edited comment on CASSANDRA-9608 at 11/17/20, 12:41 PM:
Hi [~snazy] Possible small bug in {{cassandra-env.sh:}}
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^\-\[X\]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "\-\[X\]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-eq}} will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
was (Author: yakir.g):
Hi [~snazy] Possible small bug in {{cassandra-env.sh:}}
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-e}}q will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
> Support Java 11
> ---
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
> Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
> Labels: Java11
> Fix For: 4.0, 4.0-alpha1
>
> Attachments: jdk_9_10.patch
>
>
> This ticket is intended to group all issues found to support Java 9 in the
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved.
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods
> {{monitorEnter}} + {{monitorExit}}. These methods are used by
> {{o.a.c.utils.concurrent.Locks}} which is only used by
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early
> development phase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-9608) Support Java 11
[
https://issues.apache.org/jira/browse/CASSANDRA-9608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17233556#comment-17233556
]
Yakir Gibraltar commented on CASSANDRA-9608:
Hi [~snazy] Possible small bug in {{cassandra-env.sh:}}
[https://github.com/apache/cassandra/blob/trunk/conf/cassandra-env.sh#L98]
The following command incorrect:
{{echo "$JVM_OPTS" | grep -q "^-[X]log:gc"}}
Should be:
{{echo "$JVM_OPTS" | grep -qe "-[X]log:gc"}}
The variable {{$JVM_OPTS}} not starting with {{-Xlog..}} , and always return
1, remove {{^}} and add {{-e}}q will solve the problem.
It's causing that {{cassandra-env.sh}} ignoring variable of {{-Xlog}} in
{{jvm11-server.options}} and {{jvm8-server.options}}
> Support Java 11
> ---
>
> Key: CASSANDRA-9608
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9608
> Project: Cassandra
> Issue Type: Task
>Reporter: Robert Stupp
>Assignee: Robert Stupp
>Priority: Low
> Labels: Java11
> Fix For: 4.0, 4.0-alpha1
>
> Attachments: jdk_9_10.patch
>
>
> This ticket is intended to group all issues found to support Java 9 in the
> future.
> From what I've found out so far:
> * Maven dependency {{com.sun:tools:jar:0}} via cobertura cannot be resolved.
> It can be easily solved using this patch:
> {code}
> - artifactId="cobertura"/>
> + artifactId="cobertura">
> +
> +
> {code}
> * Another issue is that {{sun.misc.Unsafe}} no longer contains the methods
> {{monitorEnter}} + {{monitorExit}}. These methods are used by
> {{o.a.c.utils.concurrent.Locks}} which is only used by
> {{o.a.c.db.AtomicBTreeColumns}}.
> I don't mind to start working on this yet since Java 9 is in a too early
> development phase.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17094204#comment-17094204
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Thank you [~jasonstack] and [~brandon.williams]
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Yakir Gibraltar
>Assignee: ZhaoYang
>Priority: Normal
> Fix For: 3.0.21, 3.11.7, 4.0-beta
>
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17045735#comment-17045735
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
I see the errors only in C* logs, example of logs with TRACE:
{code}
TRACE [Native-Transport-Requests-106] 2020-02-26 12:19:47,836
DigestResolver.java:96 - resolve: 0 ms.
TRACE [Native-Transport-Requests-207] 2020-02-26 12:19:47,836 Message.java:685
- Received: QUERY INSERT INTO ud.i_s (user_id,data) VALUES (?,?) USING TTL
7776000;[pageSize = 1000], v=4/v4
TRACE [Native-Transport-Requests-210] 2020-02-26 12:19:47,833
MessagingService.java:942 - /1.2.3.5 sending READ to -946659607@/1.2.3.15
TRACE [Native-Transport-Requests-44] 2020-02-26 12:19:47,834
StorageProxy.java:1352 - Adding FWD message to -946659552@/1.2.3.22
TRACE [Native-Transport-Requests-28] 2020-02-26 12:19:47,836
ReadCallback.java:101 - Blockfor is 2; setting up requests to /1.2.3.3,/1.2.3.14
TRACE [Native-Transport-Requests-118] 2020-02-26 12:19:47,833
ReadCallback.java:101 - Blockfor is 2; setting up requests to
/1.2.3.24,/1.2.3.23
TRACE [Native-Transport-Requests-18] 2020-02-26 12:19:47,833
StorageProxy.java:1357 - Sending message to -946659650@/1.2.3.25
TRACE [Native-Transport-Requests-127] 2020-02-26 12:19:47,824
StorageProxy.java:1352 - Adding FWD message to -946659949@/1.2.3.26
TRACE [Native-Transport-Requests-44] 2020-02-26 12:19:47,836
StorageProxy.java:1352 - Adding FWD message to -946659505@/1.2.3.27
TRACE [Native-Transport-Requests-28] 2020-02-26 12:19:47,836
AbstractReadExecutor.java:108 - reading data from /1.2.3.3
TRACE [Native-Transport-Requests-29] 2020-02-26 12:19:47,833 Message.java:704 -
Responding: EMPTY RESULT, v=4/v4
TRACE [Native-Transport-Requests-18] 2020-02-26 12:19:47,836
StorageProxy.java:1352 - Adding FWD message to -946659504@/1.2.3.37
TRACE [Native-Transport-Requests-22] 2020-02-26 12:19:47,834
ReadCallback.java:101 - Blockfor is 2; setting up requests to
/1.2.3.24,/1.2.3.12
TRACE [Native-Transport-Requests-44] 2020-02-26 12:19:47,836
MessagingService.java:942 - /1.2.3.5 sending MUTATION to -946659502@/1.2.3.28
TRACE [Native-Transport-Requests-62] 2020-02-26 12:19:47,833
StorageProxy.java:1357 - Sending message to -946659698@/1.2.3.37
TRACE [MutationStage-87] 2020-02-26 12:19:47,834 MessagingService.java:942 -
/1.2.3.5 sending REQUEST_RESPONSE to -946659658@/1.2.3.5
TRACE [Native-Transport-Requests-118] 2020-02-26 12:19:47,836
AbstractReadExecutor.java:108 - reading data from /1.2.3.24
TRACE [Native-Transport-Requests-5] 2020-02-26 12:19:47,834
MessagingService.java:942 - /1.2.3.5 sending MUTATION to -946659548@/1.2.3.29
TRACE [Native-Transport-Requests-64] 2020-02-26 12:19:47,833
MessagingService.java:942 - /1.2.3.5 sending MUTATION to -946659602@/1.2.3.30
TRACE [Native-Transport-Requests-3] 2020-02-26 12:19:47,833
MessagingService.java:942 - /1.2.3.5 sending READ to -946659600@/1.2.3.17
ERROR [ReadStage-110] 2020-02-26 12:19:47,836
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[ReadStage-110,5,main]
java.lang.AssertionError: Illegal bounds [-1777439288..-1777439280); size:
2761628520
at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
~[apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
~[apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
~[apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
~[caffeine-2.2.6.jar:na]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[na:1.8.0-zing_19.12.102.0]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
~[caffeine-2.2.6.jar:na]
at
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044824#comment-17044824
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Unable to see which specific query causing the error, don't see errors in
client side.
I see the errors only in C* log.
I tried to change slow_query_log_timeout_in_ms to 1 in order to catch all
queries and don't see any problematic query that trigger the error.
[~brandon.williams] do you have idea how to get more details on the query that
cause the error or which debug option can show that?
Thanks.
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 3.11.7
>
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
>
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044650#comment-17044650
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Don't see errors from application side, only from C* side.
The command the out application running:
{code:java}
SELECT data FROM ud.i_s WHERE user_id=?;
SELECT data FROM ud.i_g WHERE user_id=?;
INSERT INTO ud.i_s (user_id,data) VALUES (?,?) USING TTL 7776000;
INSERT INTO ud.i_g (user_id,data) VALUES (?,?) USING TTL 7776000;' {code}
Our metadata:
{code:java}
CREATE TABLE ud.i_g (
user_id blob PRIMARY KEY,
data blob
) WITH bloom_filter_fp_chance = 0.01
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '15', 'compaction_window_unit': 'DAYS',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '4', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 0
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = 'NONE';
CREATE TABLE ud.i_s (
user_id blob PRIMARY KEY,
data blob
) WITH bloom_filter_fp_chance = 0.05
AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
AND comment = ''
AND compaction = {'class':
'org.apache.cassandra.db.compaction.TimeWindowCompactionStrategy',
'compaction_window_size': '15', 'compaction_window_unit': 'DAYS',
'max_threshold': '32', 'min_threshold': '4'}
AND compression = {'chunk_length_in_kb': '4', 'class':
'org.apache.cassandra.io.compress.LZ4Compressor'}
AND crc_check_chance = 1.0
AND dclocal_read_repair_chance = 0.0
AND default_time_to_live = 0
AND gc_grace_seconds = 0
AND max_index_interval = 2048
AND memtable_flush_period_in_ms = 0
AND min_index_interval = 128
AND read_repair_chance = 0.0
AND speculative_retry = 'NONE'; {code}
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Caching
>Reporter: Yakir Gibraltar
>Priority: Normal
> Fix For: 3.11.7
>
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
>
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044613#comment-17044613
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
[~brandon.williams] do you have idea how to debug that error?
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504)
>
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17044412#comment-17044412
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Also change: "memtable_allocation_type: heap_buffers" didn't help.
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
>
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043653#comment-17043653
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Based on:
[https://github.com/ben-manes/caffeine/issues/397#issuecomment-590406804]
"It looks like it’s trying to read offheap but has an incorrect address offset."
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043632#comment-17043632
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Yes, same issue, getting same errors with jdk 1.8.0_131:
{code:java}
ERROR [ReadStage-42] 2020-02-24 15:58:51,767
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[ReadStage-42,5,main]
java.lang.AssertionError: Illegal bounds [-2049972896..-2049972888); size:
2754497496
at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
~[apache-cassandra-3.11.5.jar:3.11.5]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
~[caffeine-2.2.6.jar:na]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[na:1.8.0_131]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
~[caffeine-2.2.6.jar:na]
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:423)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043581#comment-17043581
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
Also update caffeine to 2.8.1 didn't help.
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504)
>
[jira] [Updated] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-15595:
Platform: Linux (was: All)
Severity: Normal
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
>
[jira] [Commented] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
[
https://issues.apache.org/jira/browse/CASSANDRA-15595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17043559#comment-17043559
]
Yakir Gibraltar commented on CASSANDRA-15595:
-
The errors was also on 3.11.5, i tried to upgrade and errors exist also on
3.11.6.
Right, I'm running Zing.
{code}
[root@udb001 ~]# rpm -qa | grep zing
zing-jdk1.8.0-19.12.102.0-3.x86_64
zing-zst-8c.3.10.0-5.22.6.0.1.2.el7.x86_64
{code}
> Many errors of "java.lang.AssertionError: Illegal bounds"
> -
>
> Key: CASSANDRA-15595
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
> Project: Cassandra
> Issue Type: Bug
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
> {code}
> ERROR [ReadStage-6] 2020-02-24 13:53:34,528
> AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
> Thread[ReadStage-6,5,main]
> java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
> 2761628520
> at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
> ~[apache-cassandra-3.11.6.ja
> r:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
> ~[caffeine-2.2.6.jar:na]
> at
> java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
> ~[na:1.8.0-zing_19.12.102.0]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
> ~[caffeine-2.2.6.jar:na]
> at
> com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
> ~[caffeine-2.2.6.jar:na]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
> org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
> ~[apache-cassandra-3.11.6.jar:3.11.6]
> at
>
[jira] [Created] (CASSANDRA-15595) Many errors of "java.lang.AssertionError: Illegal bounds"
Yakir Gibraltar created CASSANDRA-15595:
---
Summary: Many errors of "java.lang.AssertionError: Illegal bounds"
Key: CASSANDRA-15595
URL: https://issues.apache.org/jira/browse/CASSANDRA-15595
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Hi, i'm running cassandra 3.11.6 and getting on all hosts many errors of:
{code}
ERROR [ReadStage-6] 2020-02-24 13:53:34,528
AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread
Thread[ReadStage-6,5,main]
java.lang.AssertionError: Illegal bounds [-2102982480..-2102982472); size:
2761628520
at org.apache.cassandra.io.util.Memory.checkBounds(Memory.java:345)
~[apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.io.util.Memory.getLong(Memory.java:254)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.compress.CompressionMetadata.chunkFor(CompressionMetadata.java:234)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.util.CompressedChunkReader$Standard.readChunk(CompressedChunkReader.java:114)
~[apache-cassandra-3.11.6.ja
r:3.11.6]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:158)
~[apache-cassandra-3.11.6.jar:3.11.6]
at org.apache.cassandra.cache.ChunkCache.load(ChunkCache.java:39)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache$BoundedLocalLoadingCache.lambda$new$0(BoundedLocalCache.java:2949)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.lambda$doComputeIfAbsent$15(BoundedLocalCache.java:1807)
~[caffeine-2.2.6.jar:na]
at
java.util.concurrent.ConcurrentHashMap.compute(ConcurrentHashMap.java:1853)
~[na:1.8.0-zing_19.12.102.0]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.doComputeIfAbsent(BoundedLocalCache.java:1805)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.BoundedLocalCache.computeIfAbsent(BoundedLocalCache.java:1788)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalCache.computeIfAbsent(LocalCache.java:97)
~[caffeine-2.2.6.jar:na]
at
com.github.benmanes.caffeine.cache.LocalLoadingCache.get(LocalLoadingCache.java:66)
~[caffeine-2.2.6.jar:na]
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:236)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.cache.ChunkCache$CachingRebufferer.rebuffer(ChunkCache.java:214)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.util.RandomAccessReader.reBufferAt(RandomAccessReader.java:65)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.util.RandomAccessReader.seek(RandomAccessReader.java:207)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.util.FileHandle.createReader(FileHandle.java:150)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.sstable.format.SSTableReader.getFileDataInput(SSTableReader.java:1807)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.columniterator.AbstractSSTableIterator.(AbstractSSTableIterator.java:103)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.columniterator.SSTableIterator.(SSTableIterator.java:49)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:72)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:65)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.StorageHook$1.makeRowIterator(StorageHook.java:100)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:982)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:693)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:670)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:504)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:423)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:48)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:66)
~[apache-cassandra-3.11.6.jar:3.11.6]
at
[jira] [Updated] (CASSANDRA-15157) Missing commands in nodetool help
[
https://issues.apache.org/jira/browse/CASSANDRA-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-15157:
Resolution: Not A Problem
Status: Resolved (was: Triage Needed)
ok, thanks.
> Missing commands in nodetool help
> -
>
> Key: CASSANDRA-15157
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15157
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, how to gel *all* available commands of nodetool? like cfhistograms,
> cfstats, etc.
> "nodetool help" does not return all commands.
> for example:
> {code}
> [root@ctaz001 ~]# nodetool version
> ReleaseVersion: 3.11.4
> [root@ctaz001 ~]# nodetool help | grep cfh | wc -l
> 0
> [root@ctaz001 ~]# nodetool help
> usage: nodetool [(-p | --port )]
> [(-u | --username )]
> [(-pw | --password )]
> [(-pwf | --password-file )]
> [(-h | --host )] []
> The most commonly used nodetool commands are:
> assassinate Forcefully remove a dead node without
> re-replicating any data. Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
> cleanup Triggers the immediate cleanup of keys no
> longer belonging to a node. By default, clean all keyspaces
> clearsnapshotRemove the snapshot with the given name from
> the given keyspaces. If no snapshotName is specified we will remove all
> snapshots
> compact Force a (major) compaction on one or more
> tables or user-defined compaction on given SSTables
> compactionhistoryPrint history of compaction
> compactionstats Print statistics on compactions
> decommission Decommission the *node I am connecting to*
> describecluster Print the name, snitch, partitioner and
> schema version of a cluster
> describering Shows the token ranges info of a given
> keyspace
> disableautocompactionDisable autocompaction for the given
> keyspace and table
> disablebackupDisable incremental backup
> disablebinaryDisable native transport (binary protocol)
> disablegossipDisable gossip (effectively marking the node
> down)
> disablehandoff Disable storing hinted handoffs
> disablehintsfordcDisable hints for a data center
> disablethriftDisable thrift server
> drainDrain the node (stop accepting writes and
> flush all tables)
> enableautocompaction Enable autocompaction for the given keyspace
> and table
> enablebackup Enable incremental backup
> enablebinary Reenable native transport (binary protocol)
> enablegossip Reenable gossip
> enablehandoffReenable future hints storing on the current
> node
> enablehintsfordc Enable hints for a data center that was
> previsouly disabled
> enablethrift Reenable thrift server
> failuredetector Shows the failure detector information for
> the cluster
> flushFlush one or more tables
> garbagecollect Remove deleted data from one or more tables
> gcstats Print GC Statistics
> getcompactionthreshold Print min and max compaction thresholds for
> a given table
> getcompactionthroughput Print the MB/s throughput cap for compaction
> in the system
> getconcurrentcompactors Get the number of concurrent compactors in
> the system.
> getendpoints Print the end points that owns the key
> getinterdcstreamthroughput Print the Mb/s throughput cap for
> inter-datacenter streaming in the system
> getlogginglevels Get the runtime logging levels
> getsstables Print the sstable filenames that own the key
> getstreamthroughput Print the Mb/s throughput cap for streaming
> in the system
> gettimeout Print the timeout of the given type in ms
> gettraceprobability Print the current trace probability value
> gossipinfo Shows the gossip information for the cluster
> help Display help information
> info Print node information (uptime, load, ...)
> invalidatecountercache Invalidate the counter cache
> invalidatekeycache Invalidate the key cache
> invalidaterowcache Invalidate the row cache
> join
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.10.1.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED error:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.10.1.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.10.1.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.10.1.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED error:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.10.1.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.4
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.1.3.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Updated] (CASSANDRA-15157) Missing commands in nodetool help
[
https://issues.apache.org/jira/browse/CASSANDRA-15157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-15157:
Summary: Missing commands in nodetool help (was: missing command in
nodetool help)
> Missing commands in nodetool help
> -
>
> Key: CASSANDRA-15157
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15157
> Project: Cassandra
> Issue Type: Bug
> Components: Tool/nodetool
>Reporter: Yakir Gibraltar
>Priority: Normal
>
> Hi, how to gel *all* available commands of nodetool? like cfhistograms,
> cfstats, etc.
> "nodetool help" does not return all commands.
> for example:
> {code}
> [root@ctaz001 ~]# nodetool version
> ReleaseVersion: 3.11.4
> [root@ctaz001 ~]# nodetool help | grep cfh | wc -l
> 0
> [root@ctaz001 ~]# nodetool help
> usage: nodetool [(-p | --port )]
> [(-u | --username )]
> [(-pw | --password )]
> [(-pwf | --password-file )]
> [(-h | --host )] []
> The most commonly used nodetool commands are:
> assassinate Forcefully remove a dead node without
> re-replicating any data. Use as a last resort if you cannot removenode
> bootstrapMonitor/manage node's bootstrap process
> cleanup Triggers the immediate cleanup of keys no
> longer belonging to a node. By default, clean all keyspaces
> clearsnapshotRemove the snapshot with the given name from
> the given keyspaces. If no snapshotName is specified we will remove all
> snapshots
> compact Force a (major) compaction on one or more
> tables or user-defined compaction on given SSTables
> compactionhistoryPrint history of compaction
> compactionstats Print statistics on compactions
> decommission Decommission the *node I am connecting to*
> describecluster Print the name, snitch, partitioner and
> schema version of a cluster
> describering Shows the token ranges info of a given
> keyspace
> disableautocompactionDisable autocompaction for the given
> keyspace and table
> disablebackupDisable incremental backup
> disablebinaryDisable native transport (binary protocol)
> disablegossipDisable gossip (effectively marking the node
> down)
> disablehandoff Disable storing hinted handoffs
> disablehintsfordcDisable hints for a data center
> disablethriftDisable thrift server
> drainDrain the node (stop accepting writes and
> flush all tables)
> enableautocompaction Enable autocompaction for the given keyspace
> and table
> enablebackup Enable incremental backup
> enablebinary Reenable native transport (binary protocol)
> enablegossip Reenable gossip
> enablehandoffReenable future hints storing on the current
> node
> enablehintsfordc Enable hints for a data center that was
> previsouly disabled
> enablethrift Reenable thrift server
> failuredetector Shows the failure detector information for
> the cluster
> flushFlush one or more tables
> garbagecollect Remove deleted data from one or more tables
> gcstats Print GC Statistics
> getcompactionthreshold Print min and max compaction thresholds for
> a given table
> getcompactionthroughput Print the MB/s throughput cap for compaction
> in the system
> getconcurrentcompactors Get the number of concurrent compactors in
> the system.
> getendpoints Print the end points that owns the key
> getinterdcstreamthroughput Print the Mb/s throughput cap for
> inter-datacenter streaming in the system
> getlogginglevels Get the runtime logging levels
> getsstables Print the sstable filenames that own the key
> getstreamthroughput Print the Mb/s throughput cap for streaming
> in the system
> gettimeout Print the timeout of the given type in ms
> gettraceprobability Print the current trace probability value
> gossipinfo Shows the gossip information for the cluster
> help Display help information
> info Print node information (uptime, load, ...)
> invalidatecountercache Invalidate the counter cache
> invalidatekeycache Invalidate the key cache
> invalidaterowcache Invalidate the row cache
> join
[jira] [Created] (CASSANDRA-15157) missing command in nodetool help
Yakir Gibraltar created CASSANDRA-15157:
---
Summary: missing command in nodetool help
Key: CASSANDRA-15157
URL: https://issues.apache.org/jira/browse/CASSANDRA-15157
Project: Cassandra
Issue Type: Bug
Components: Tool/nodetool
Reporter: Yakir Gibraltar
Hi, how to gel *all* available commands of nodetool? like cfhistograms,
cfstats, etc.
"nodetool help" does not return all commands.
for example:
{code}
[root@ctaz001 ~]# nodetool version
ReleaseVersion: 3.11.4
[root@ctaz001 ~]# nodetool help | grep cfh | wc -l
0
[root@ctaz001 ~]# nodetool help
usage: nodetool [(-p | --port )]
[(-u | --username )]
[(-pw | --password )]
[(-pwf | --password-file )]
[(-h | --host )] []
The most commonly used nodetool commands are:
assassinate Forcefully remove a dead node without
re-replicating any data. Use as a last resort if you cannot removenode
bootstrapMonitor/manage node's bootstrap process
cleanup Triggers the immediate cleanup of keys no
longer belonging to a node. By default, clean all keyspaces
clearsnapshotRemove the snapshot with the given name from
the given keyspaces. If no snapshotName is specified we will remove all
snapshots
compact Force a (major) compaction on one or more
tables or user-defined compaction on given SSTables
compactionhistoryPrint history of compaction
compactionstats Print statistics on compactions
decommission Decommission the *node I am connecting to*
describecluster Print the name, snitch, partitioner and schema
version of a cluster
describering Shows the token ranges info of a given keyspace
disableautocompactionDisable autocompaction for the given keyspace
and table
disablebackupDisable incremental backup
disablebinaryDisable native transport (binary protocol)
disablegossipDisable gossip (effectively marking the node
down)
disablehandoff Disable storing hinted handoffs
disablehintsfordcDisable hints for a data center
disablethriftDisable thrift server
drainDrain the node (stop accepting writes and
flush all tables)
enableautocompaction Enable autocompaction for the given keyspace
and table
enablebackup Enable incremental backup
enablebinary Reenable native transport (binary protocol)
enablegossip Reenable gossip
enablehandoffReenable future hints storing on the current
node
enablehintsfordc Enable hints for a data center that was
previsouly disabled
enablethrift Reenable thrift server
failuredetector Shows the failure detector information for the
cluster
flushFlush one or more tables
garbagecollect Remove deleted data from one or more tables
gcstats Print GC Statistics
getcompactionthreshold Print min and max compaction thresholds for a
given table
getcompactionthroughput Print the MB/s throughput cap for compaction
in the system
getconcurrentcompactors Get the number of concurrent compactors in the
system.
getendpoints Print the end points that owns the key
getinterdcstreamthroughput Print the Mb/s throughput cap for
inter-datacenter streaming in the system
getlogginglevels Get the runtime logging levels
getsstables Print the sstable filenames that own the key
getstreamthroughput Print the Mb/s throughput cap for streaming in
the system
gettimeout Print the timeout of the given type in ms
gettraceprobability Print the current trace probability value
gossipinfo Shows the gossip information for the cluster
help Display help information
info Print node information (uptime, load, ...)
invalidatecountercache Invalidate the counter cache
invalidatekeycache Invalidate the key cache
invalidaterowcache Invalidate the row cache
join Join the ring
listsnapshotsLists all the snapshots along with the size on
disk and true size.
move Move node on the token ring to a new token
netstats Print network information on provided host
(connecting node by default)
pausehandoff Pause hints delivery process
proxyhistograms Print statistic histograms for network
[jira] [Updated] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
[
https://issues.apache.org/jira/browse/CASSANDRA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yakir Gibraltar updated CASSANDRA-14978:
Description:
Cassandra version: 3.11.3
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.1.3.el7.x86_64
JDK: jdk1.8.0_131
Heap: same errors with 16GB / 32GB / 64GB.
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
[jira] [Created] (CASSANDRA-14978) Cassandra going down with "java.lang.OutOfMemoryError: Map failed" and "LEAK DETECTED"
Yakir Gibraltar created CASSANDRA-14978:
---
Summary: Cassandra going down with "java.lang.OutOfMemoryError:
Map failed" and "LEAK DETECTED"
Key: CASSANDRA-14978
URL: https://issues.apache.org/jira/browse/CASSANDRA-14978
Project: Cassandra
Issue Type: Bug
Reporter: Yakir Gibraltar
Cassandra version: 3.11.3
OS: CentOS Linux release 7.4.1708 (Core)
Kernel: 3.10.0-957.1.3.el7.x86_64
JDK: jdk1.8.0_131
*We are seeing this errors in production:*
*java.io.IOException: Map failed:*
{code:java}
ERROR [CompactionExecutor:5017] 2019-01-14 00:02:04,763
CassandraDaemon.java:228 - Exception in thread
Thread[CompactionExecutor:5017,1,main]
org.apache.cassandra.io.FSReadError: java.io.IOException: Map failed
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:157)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.add(MmappedRegions.java:310)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions$State.access$400(MmappedRegions.java:246)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.updateState(MmappedRegions.java:181)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:73)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.(MmappedRegions.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.MmappedRegions.map(MmappedRegions.java:104)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.util.FileHandle$Builder.complete(FileHandle.java:362)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.format.big.BigTableWriter.openEarly(BigTableWriter.java:290)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.maybeReopenEarly(SSTableRewriter.java:179)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:134)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:65)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:142)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:201)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:85)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:274)
~[apache-cassandra-3.11.3.jar:3.11.3]
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
~[na:1.8.0_131]
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
~[na:1.8.0_131]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
[na:1.8.0_131]
at
org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:81)
[apache-cassandra-3.11.3.jar:3.11.3]
at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_131]
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:940)
~[na:1.8.0_131]
at org.apache.cassandra.io.util.ChannelProxy.map(ChannelProxy.java:153)
~[apache-cassandra-3.11.3.jar:3.11.3]
... 23 common frames omitted
Caused by: java.lang.OutOfMemoryError: Map failed
at sun.nio.ch.FileChannelImpl.map0(Native Method) ~[na:1.8.0_131]
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:937)
~[na:1.8.0_131]
... 24 common frames omitted
{code}
*LEAK DETECTED:*
{code:java}
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,469 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6a4ef142) to class
org.apache.cassandra.io.util.SafeMemory$MemoryTidy@1651696741:Memory@[6b91a27c5290..6b91a27de290)
was not released before the reference was garbage collected
ERROR [Reference-Reaper:1] 2019-01-14 00:03:46,520 Ref.java:224 - LEAK
DETECTED: a reference
(org.apache.cassandra.utils.concurrent.Ref$State@6c458f8a) to class
