Re: Cassandra Authentication

[Jai Bheemsen Rao Dhanwada]

Thank you

On Wed, Jan 18, 2017 at 10:10 PM, Ben Bromhead  wrote:

> the volume of data is pretty low + you still want to be able to
> authenticate even if you have more nodes down than the RF for other
> keyspaces. Essentially you don't want auth to be the thing that stops you
> serving requests.
>
> On Wed, 18 Jan 2017 at 14:57 Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Thanks Ben,
>>
>> RF 3 isn't sufficient for system_auth? as we are using 3 RF for other
>> production KS, do you see any challenges?
>>
>> On Wed, Jan 18, 2017 at 2:39 PM, Ben Bromhead 
>> wrote:
>>
>> We have a process that syncs and manages RF==N and we also control and
>> manage users, however that entails it's own set of challenges and
>> maintenance.
>>
>> For most users I would suggest 3 < RF <=5 is sufficient. Also make sure
>> you don't use the user "Cassandra" in production as authentication queries
>> are done at QUORUM.
>>
>> On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
>> jaibheem...@gmail.com> wrote:
>>
>> Hello,
>>
>> When enabling Authentication on cassandra, is it required to set the RF
>> same as the no.of nodes(https://docs.datastax.
>> com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)? or can I live with RF
>> of 3 in each DC (other KS are using 3)
>>
>> If it has to be equal to the number of nodes then, every time adding or
>> removing a node requires update of RF.
>>
>> Thanks in advance.
>>
>> --
>> Ben Bromhead
>> CTO | Instaclustr 
>> +1 650 284 9692 <+1%20650-284-9692>
>> Managed Cassandra / Spark on AWS, Azure and Softlayer
>>
>>
>> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692 <+1%20650-284-9692>
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>

Re: Cassandra Authentication

[Ben Bromhead]

the volume of data is pretty low + you still want to be able to
authenticate even if you have more nodes down than the RF for other
keyspaces. Essentially you don't want auth to be the thing that stops you
serving requests.

On Wed, 18 Jan 2017 at 14:57 Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Thanks Ben,
>
> RF 3 isn't sufficient for system_auth? as we are using 3 RF for other
> production KS, do you see any challenges?
>
> On Wed, Jan 18, 2017 at 2:39 PM, Ben Bromhead  wrote:
>
> We have a process that syncs and manages RF==N and we also control and
> manage users, however that entails it's own set of challenges and
> maintenance.
>
> For most users I would suggest 3 < RF <=5 is sufficient. Also make sure
> you don't use the user "Cassandra" in production as authentication queries
> are done at QUORUM.
>
> On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
> Hello,
>
> When enabling Authentication on cassandra, is it required to set the RF
> same as the no.of nodes(
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)?
> or can I live with RF of 3 in each DC (other KS are using 3)
>
> If it has to be equal to the number of nodes then, every time adding or
> removing a node requires update of RF.
>
> Thanks in advance.
>
> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692 <+1%20650-284-9692>
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>
>
> --
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Re: Cassandra Authentication

[Jai Bheemsen Rao Dhanwada]

Thanks Ben,

RF 3 isn't sufficient for system_auth? as we are using 3 RF for other
production KS, do you see any challenges?

On Wed, Jan 18, 2017 at 2:39 PM, Ben Bromhead  wrote:

> We have a process that syncs and manages RF==N and we also control and
> manage users, however that entails it's own set of challenges and
> maintenance.
>
> For most users I would suggest 3 < RF <=5 is sufficient. Also make sure
> you don't use the user "Cassandra" in production as authentication queries
> are done at QUORUM.
>
> On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
> jaibheem...@gmail.com> wrote:
>
>> Hello,
>>
>> When enabling Authentication on cassandra, is it required to set the RF
>> same as the no.of nodes(https://docs.datastax.
>> com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)? or can I live with RF
>> of 3 in each DC (other KS are using 3)
>>
>> If it has to be equal to the number of nodes then, every time adding or
>> removing a node requires update of RF.
>>
>> Thanks in advance.
>>
> --
> Ben Bromhead
> CTO | Instaclustr 
> +1 650 284 9692 <+1%20650-284-9692>
> Managed Cassandra / Spark on AWS, Azure and Softlayer
>

Re: Cassandra Authentication

[Ben Bromhead]

We have a process that syncs and manages RF==N and we also control and
manage users, however that entails it's own set of challenges and
maintenance.

For most users I would suggest 3 < RF <=5 is sufficient. Also make sure you
don't use the user "Cassandra" in production as authentication queries are
done at QUORUM.

On Wed, 18 Jan 2017 at 13:41 Jai Bheemsen Rao Dhanwada <
jaibheem...@gmail.com> wrote:

> Hello,
>
> When enabling Authentication on cassandra, is it required to set the RF
> same as the no.of nodes(
> https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)?
> or can I live with RF of 3 in each DC (other KS are using 3)
>
> If it has to be equal to the number of nodes then, every time adding or
> removing a node requires update of RF.
>
> Thanks in advance.
>
-- 
Ben Bromhead
CTO | Instaclustr 
+1 650 284 9692
Managed Cassandra / Spark on AWS, Azure and Softlayer

Cassandra Authentication

[Jai Bheemsen Rao Dhanwada]

Hello,

When enabling Authentication on cassandra, is it required to set the RF
same as the no.of nodes(
https://docs.datastax.com/en/cql/3.1/cql/cql_using/update_ks_rf_t.html)? or
can I live with RF of 3 in each DC (other KS are using 3)

If it has to be equal to the number of nodes then, every time adding or
removing a node requires update of RF.

Thanks in advance.

ApacheCon CFP closing soon (11 February)

[Rich Bowen]

Hello, fellow Apache enthusiast. Thanks for your participation, and
interest in, the projects of the Apache Software Foundation.

I wanted to remind you that the Call For Papers (CFP) for ApacheCon
North America, and Apache: Big Data North America, closes in less than a
month. If you've been putting it off because there was lots of time
left, it's time to dig for that inspiration and get those talk proposals in.

It's also time to discuss with your developer and user community whether
there's a track of talks that you might want to propose, so that you
have more complete coverage of your project than a talk or two.

We're looking for talks directly, and indirectly, related to projects at
the Apache Software Foundation. These can be anything from in-depth
technical discussions of the projects you work with, to talks about
community, documentation, legal issues, marketing, and so on. We're also
very interested in talks about projects and services built on top of
Apache projects, and case studies of how you use Apache projects to
solve real-world problems.

We are particularly interested in presentations from Apache projects
either in the Incubator, or recently graduated. ApacheCon is where
people come to find out what technology they'll be using this time next
year.

Important URLs are:

To submit a talk for Apache: Big Data -
http://events.linuxfoundation.org/events/apache-big-data-north-america/program/cfp
To submit a talk for ApacheCon -
http://events.linuxfoundation.org/events/apachecon-north-america/program/cfp

To register for Apache: Big Data -
http://events.linuxfoundation.org/events/apache-big-data-north-america/attend/register-
To register for ApacheCon -
http://events.linuxfoundation.org/events/apachecon-north-america/attend/register-

Early Bird registration rates end March 12th, but if you're a committer
on an Apache project, you get the low committer rate, which is less than
half of the early bird rate!

For further updated about ApacheCon, follow us on Twitter, @ApacheCon,
or drop by our IRC channel, #apachecon on the Freenode IRC network. Or
contact me - rbo...@apache.org - with any questions or concerns.

Thanks!

Rich Bowen, VP Conferences, Apache Software Foundation

-- 
(You've received this email because you're on a dev@ or users@ mailing
list of an Apache Software Foundation project. For subscription and
unsubscription information, consult the headers of this email message,
as this varies from one list to another.)

Nodes becoming unresponsive intermediately (Gossip stage pending)

[Sermandurai Konar]

Hi,

We have 11/11 node cluster running Cassandra 2.1.15 version.
We are observing that 3 nodes from each data center are becoming
unresponsive for short period of time.
This behavior is happening only in 6 nodes (i.e. 3 from each data center)
and we are seeing a lot of Gossip stage has pending task and periodic
commit log syncer issue. This is resulting in huge mutation drop and the
node also has high cpu usage compared to other nodes.

Log entry from system.log:
WARN  [GossipTasks:1] 2017-01-18 10:15:51,328  Gossiper.java:748 - Gossip
stage has 2 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:52,428  Gossiper.java:748 - Gossip
stage has 3 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:53,529  Gossiper.java:748 - Gossip
stage has 3 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:54,629  Gossiper.java:748 - Gossip
stage has 5 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:55,730  Gossiper.java:748 - Gossip
stage has 7 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:56,830  Gossiper.java:748 - Gossip
stage has 9 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:57,930  Gossiper.java:748 - Gossip
stage has 12 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:15:59,031  Gossiper.java:748 - Gossip
stage has 15 pending tasks; skipping status check (no nodes will be marked
down)
WARN  [GossipTasks:1] 2017-01-18 10:16:00,131  Gossiper.java:748 - Gossip
stage has 16 pending tasks; skipping status check (no nodes will be marked
down)



WARN  [PERIODIC-COMMIT-LOG-SYNCER] 2017-01-18
06:35:44,186  AbstractCommitLogService.java:105 - Out of 33 commit log
syncs over the past 309s with average duration of 9506.03ms, 13 have
exceeded the configured commit interval by an average of 8676.77ms


*Observation:*
These 6 nodes having issues are running in Red hat version 6.8 and all
other nodes in clusters are in 6.7 version. The node in 6.7 version is not
having any gossip issue/ mutation drop/ CPU issues/ IO.
Also, the IOStats in these nodes is bad compared to other nodes. We also
see that threads are blocked at HintedHandoff.

*-bash-4.1$ sar 2 7*

10:29:51 AM CPU %user %nice   %system   %iowait%steal
%idle
10:29:53 AM all 36.83  4.86  5.51  4.65  0.00
48.15
10:29:55 AM all 41.09  4.98  6.68  2.28  0.00
44.97
10:29:57 AM all 37.53  4.71  5.52  2.04  0.00
50.21
10:29:59 AM all 35.82  4.76  4.97  2.14  0.00
52.31
10:30:01 AM all 33.90  3.53  3.82  2.62  0.00
56.13
10:30:03 AM all 31.78  2.64  3.53  4.65  0.00
57.39
10:30:05 AM all 33.27  3.90  3.95  2.76  0.00
56.12
Average:all 35.74  4.20  4.85  3.02  0.00
52.19

*Thread are also blocked HintedHandoff:*

"HintedHandoff:1" daemon prio=1 os_prio=4 tid=0x440 nid=0x44  [ JVM locked
by VM (w/poll advisory bit) acquiring VM lock
'org.apache.cassandra.db.HintedHandOffManager', polling bits: safep rstak
gpgc_clean_new ]

  java.lang.Thread.State: BLOCKED (on object monitor)

at
org.apache.cassandra.db.HintedHandOffManager.compact(HintedHandOffManager.java:269)

at
org.apache.cassandra.db.HintedHandOffManager$5.run(HintedHandOffManager.java:563)

at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)

at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)

We are working with system team to see if there is any IO issue and is this
issue associated with any futex_wait.

Any other pointer to fix this issue will be great.

Thanks,
Sermandurai.