Re: Cannot achieve consistency level LOCAL_ONE

2017-07-09 Thread Justin Cameron 
It's best-practice to disable the default user ("cassandra" user) after
enabling password authentication on your cluster. The default user reads
with a CL.QUORUM when authenticating, while other users use CL.LOCAL_ONE.
This means it's more likely you could experience authentication issues,
even if you increase the replication factor of your system_auth keyspace.
See the docs for more info:
http://cassandra.apache.org/doc/latest/operating/security.html#enabling-password-authentication

Also, accessing Cassandra via a load-balancer is considered an
anti-pattern. The Cassandra drivers load-balance requests to the cluster
transparently, so the only thing you get by adding a load balancer to the
mix is potentially increased query latency.

Cheers,
Justin


On Fri, 7 Jul 2017 at 21:42 Oleksandr Shulgin 
wrote:

> On Thu, Jul 6, 2017 at 6:58 PM, Charulata Sharma (charshar) <
> chars...@cisco.com> wrote:
>
>> Hi,
>>
>> I am facing similar issues with SYSTEM_AUTH keyspace and wanted to know
>> the implication of disabling the "*cassandra*" superuser.
>>
>
> Unless you have scheduled any tasks that require the user with that name
> to be there, there are no implications.  This user is not used by Cassandra
> tools or the server process internally, so nothing really depends on it.
>
> Of course, in order to drop a superuser account, you need to create
> another superuser, so in the end you still have superuser access to your
> cluster.
>
> Cheers,
> --
> Alex
>
> --


*Justin Cameron*Senior Software Engineer





This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
and Instaclustr Inc (USA).

This email and any attachments may contain confidential and legally
privileged information.  If you are not the intended recipient, do not copy
or disclose its content, but please reply to this email immediately and
highlight the error to the sender and then immediately delete the message.

Re: Cannot achieve consistency level LOCAL_ONE

2017-07-07 Thread Oleksandr Shulgin 
On Thu, Jul 6, 2017 at 6:58 PM, Charulata Sharma (charshar) <
chars...@cisco.com> wrote:

> Hi,
>
> I am facing similar issues with SYSTEM_AUTH keyspace and wanted to know
> the implication of disabling the "*cassandra*" superuser.
>

Unless you have scheduled any tasks that require the user with that name to
be there, there are no implications.  This user is not used by Cassandra
tools or the server process internally, so nothing really depends on it.

Of course, in order to drop a superuser account, you need to create another
superuser, so in the end you still have superuser access to your cluster.

Cheers,
--
Alex

Re: Cannot achieve consistency level LOCAL_ONE

2017-07-06 Thread Charulata Sharma (charshar) 
Hi,

I am facing similar issues with SYSTEM_AUTH keyspace and wanted to know the 
implication of disabling the "cassandra" superuser.

Thanks,
Charu

From: "wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com>" 
<wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com>>
Date: Wednesday, June 14, 2017 at 2:16 AM
To: Oleksandr Shulgin 
<oleksandr.shul...@zalando.de<mailto:oleksandr.shul...@zalando.de>>
Cc: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
<user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
Subject: Re: Cannot achieve consistency level LOCAL_ONE

Thanks for the detail explanation. You did solve my problem.

Cheers,
-Simon

From: Oleksandr Shulgin<mailto:oleksandr.shul...@zalando.de>
Date: 2017-06-14 17:09
To: wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com>
CC: user<mailto:user@cassandra.apache.org>
Subject: Re: Cannot achieve consistency level LOCAL_ONE
On Wed, Jun 14, 2017 at 10:46 AM, 
wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com> 
<wxn...@zjqunshuo.com<mailto:wxn...@zjqunshuo.com>> wrote:
Thanks for the reply.
My system_auth settings is as below and what should I do with it? And I'm 
interested why the newly added node is responsible for the user authentication?

CREATE KEYSPACE system_auth WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;

You should change the replication options to use NetworkTopologyStrategy and a 
replication factor greater than 1 in each DC.  It is not uncommon to set it to 
the number of nodes in the DC, and is actually recommended by the following 
piece of documentation:

http://docs.datastax.com/en/cassandra/2.1/cassandra/security/security_config_native_authenticate_t.html

For version 3, the official doc says to set it to 3-5 nodes per DC: 
http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigNativeAuth.html

In general there is no drawback when setting the RF for system_auth to the 
number of nodes in DC, unless you're relying on the default superuser named 
"cassandra" being able to login at all times.  This user is special and it 
requires LOCAL_QUORUM in order to log in, while any other user (including 
non-default superusers) require only LOCAL_ONE.

As to the reason why the new node is responsible for authenticating your 
application user.  There is no particular reason for that.  The new node is 
assigned a random set of tokens and it happened to be responsible for that 
user, while some of the old nodes is no longer responsible (remember, you have 
RF=1).

Hope this helps,
--
Alex

Re: Cannot achieve consistency level LOCAL_ONE

2017-06-14 Thread wxn...@zjqunshuo.com 
Thanks for the detail explanation. You did solve my problem.

Cheers,
-Simon
 
From: Oleksandr Shulgin
Date: 2017-06-14 17:09
To: wxn...@zjqunshuo.com
CC: user
Subject: Re: Cannot achieve consistency level LOCAL_ONE
On Wed, Jun 14, 2017 at 10:46 AM, wxn...@zjqunshuo.com <wxn...@zjqunshuo.com> 
wrote:
Thanks for the reply.
My system_auth settings is as below and what should I do with it? And I'm 
interested why the newly added node is responsible for the user authentication?

CREATE KEYSPACE system_auth WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;

You should change the replication options to use NetworkTopologyStrategy and a 
replication factor greater than 1 in each DC.  It is not uncommon to set it to 
the number of nodes in the DC, and is actually recommended by the following 
piece of documentation: 

http://docs.datastax.com/en/cassandra/2.1/cassandra/security/security_config_native_authenticate_t.html

For version 3, the official doc says to set it to 3-5 nodes per DC: 
http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigNativeAuth.html

In general there is no drawback when setting the RF for system_auth to the 
number of nodes in DC, unless you're relying on the default superuser named 
"cassandra" being able to login at all times.  This user is special and it 
requires LOCAL_QUORUM in order to log in, while any other user (including 
non-default superusers) require only LOCAL_ONE.

As to the reason why the new node is responsible for authenticating your 
application user.  There is no particular reason for that.  The new node is 
assigned a random set of tokens and it happened to be responsible for that 
user, while some of the old nodes is no longer responsible (remember, you have 
RF=1).

Hope this helps,
--
Alex

Re: Cannot achieve consistency level LOCAL_ONE

2017-06-14 Thread Oleksandr Shulgin 
On Wed, Jun 14, 2017 at 10:46 AM, wxn...@zjqunshuo.com  wrote:

> Thanks for the reply.
> My system_auth settings is as below and what should I do with it? And I'm
> interested why the newly added node is responsible for the user
> authentication?
>
> CREATE KEYSPACE system_auth WITH replication = {'class': '
> SimpleStrategy', 'replication_factor': '1'}  AND durable_writes = true;
>

You should change the replication options to use NetworkTopologyStrategy
and a replication factor greater than 1 in each DC.  It is not uncommon to
set it to the number of nodes in the DC, and is actually recommended by the
following piece of documentation:

http://docs.datastax.com/en/cassandra/2.1/cassandra/security/security_config_native_authenticate_t.html

For version 3, the official doc says to set it to 3-5 nodes per DC:
http://docs.datastax.com/en/cassandra/3.0/cassandra/configuration/secureConfigNativeAuth.html

In general there is no drawback when setting the RF for system_auth to the
number of nodes in DC, unless you're relying on the default superuser named
"cassandra" being able to login at all times.  This user is special and it
requires LOCAL_QUORUM in order to log in, while any other user (including
non-default superusers) require only LOCAL_ONE.

As to the reason why the new node is responsible for authenticating your
application user.  There is no particular reason for that.  The new node is
assigned a random set of tokens and it happened to be responsible for that
user, while some of the old nodes is no longer responsible (remember, you
have RF=1).

Hope this helps,
--
Alex

Re: Cannot achieve consistency level LOCAL_ONE

2017-06-14 Thread wxn...@zjqunshuo.com 
Thanks for the reply.
My system_auth settings is as below and what should I do with it? And I'm 
interested why the newly added node is responsible for the user authentication?

CREATE KEYSPACE system_auth WITH replication = {'class': 'SimpleStrategy', 
'replication_factor': '1'}  AND durable_writes = true;

-Simon

 
From: Oleksandr Shulgin
Date: 2017-06-14 16:36
To: wxn...@zjqunshuo.com
CC: user
Subject: Re: Cannot achieve consistency level LOCAL_ONE
On Wed, Jun 14, 2017 at 9:11 AM, wxn...@zjqunshuo.com <wxn...@zjqunshuo.com> 
wrote:
Hi,
Cluster set up:
1 DC with 5 nodes (each node having 700GB data)
1 kespace with RF of 2
write CL is LOCAL_ONE
read CL is LOCAL_QUORUM

One node was down for about 1 hour because of OOM issue. During the down 
period, all 4 other nodes report "Cannot achieve consistency level LOCAL_ONE" 
constantly until I brought up the dead node. My data seems lost during that 
down time. To me this could not happen because the write CL is LOCAL_ONE and 
only one node was dead. I encountered node down before because of OOM issue and 
I believe I didn't lose data because of the hinted handoff feature.  

Hi,

The problem here is at a different level: not a single replica of the data 
could be written because no coordinator was available to serve the 
(authentication, see below) request.

One more thing, the dead node was added recently and the only difference is the 
other 4 nodes are behind an internal SLB(Service Load Balance) with VIP, and 
the new one not.
Our application access Casssandra cluster by the SLB VIP.

Any thoughts are appreciated.

Best regards,
-Simon
  
System log:
57659 Caused by: com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.RuntimeException: org.apache.cassandra.exceptions.Unavai
lableException: Cannot achieve consistency level LOCAL_ONE
  57660 at 
com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2201) 
~[guava-16.0.jar:na]
  57661 at com.google.common.cache.LocalCache.get(LocalCache.java:3934) 
~[guava-16.0.jar:na]
  57662 at 
com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3938) 
~[guava-16.0.jar:na]
  57663 at 
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4821) 
~[guava-16.0.jar:na]
  57664 at 
org.apache.cassandra.auth.RolesCache.getRoles(RolesCache.java:70) 
~[apache-cassandra-2.2.8.jar:2.2.8]
  57665 at 
org.apache.cassandra.auth.Roles.hasSuperuserStatus(Roles.java:51) 
~[apache-cassandra-2.2.8.jar:2.2.8]
  57666 at 
org.apache.cassandra.auth.AuthenticatedUser.isSuper(AuthenticatedUser.java:71) 
~[apache-cassandra-2.2.8.jar:2.2.8]
  57667 at 
org.apache.cassandra.auth.CassandraAuthorizer.authorize(CassandraAuthorizer.java:76)
 ~[apache-cassandra-2.2.8.jar:2.2.8]

What are the replication settings of your system_auth keyspace?  It looks like 
the node being down was responsible for the only replica of the user info 
needed to check its credentials/permissions.

Cheers,
--
Alex

Re: Cannot achieve consistency level LOCAL_ONE

2017-06-14 Thread Oleksandr Shulgin 
On Wed, Jun 14, 2017 at 9:11 AM, wxn...@zjqunshuo.com 
wrote:

> Hi,
> Cluster set up:
> 1 DC with 5 nodes (each node having 700GB data)
> 1 kespace with RF of 2
> write CL is LOCAL_ONE
> read CL is LOCAL_QUORUM
>
> One node was down for about 1 hour because of OOM issue. During the down
> period, all 4 other nodes report "Cannot achieve consistency
> level LOCAL_ONE" constantly until I brought up the dead node. My data
> seems lost during that down time. To me this could not happen because the
> write CL is LOCAL_ONE and only one node was dead. I encountered node down
> before because of OOM issue and I believe I didn't lose data because of the
> hinted handoff feature.
>

Hi,

The problem here is at a different level: not a single replica of the data
could be written because no coordinator was available to serve the
(authentication, see below) request.

One more thing, the dead node was added recently and the only difference is
> the other 4 nodes are behind an internal SLB(Service Load Balance) with
> VIP, and the new one not.
> Our application access Casssandra cluster by the SLB VIP.
>
> Any thoughts are appreciated.
>
> Best regards,
> -Simon
>
> System log:
> 57659 Caused by: com.google.common.util.concurrent.
> UncheckedExecutionException: java.lang.RuntimeException:
> org.apache.cassandra.exceptions.UnavailableException: Cannot
> achieve consistency level LOCAL_ONE
>   57660 at com.google.common.cache.LocalCache$
> Segment.get(LocalCache.java:2201) ~[guava-16.0.jar:na]
>   57661 at com.google.common.cache.LocalCache.get(
> LocalCache.java:3934) ~[guava-16.0.jar:na]
>   57662 at com.google.common.cache.LocalCache.
> getOrLoad(LocalCache.java:3938) ~[guava-16.0.jar:na]
>   57663 at com.google.common.cache.LocalCache$
> LocalLoadingCache.get(LocalCache.java:4821) ~[guava-16.0.jar:na]
>   57664 at org.apache.cassandra.auth.RolesCache.
> getRoles(RolesCache.java:70) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57665 at org.apache.cassandra.auth.Roles.
> hasSuperuserStatus(Roles.java:51) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57666 at org.apache.cassandra.auth.AuthenticatedUser.isSuper(
> AuthenticatedUser.java:71) ~[apache-cassandra-2.2.8.jar:2.2.8]
>   57667 at org.apache.cassandra.auth.
> CassandraAuthorizer.authorize(CassandraAuthorizer.java:76) ~
> [apache-cassandra-2.2.8.jar:2.2.8]
>

What are the replication settings of your system_auth keyspace?  It looks
like the node being down was responsible for the only replica of the user
info needed to check its credentials/permissions.

Cheers,
--
Alex