Re: Host not available

[Kyrylo Lebediev]

First of all you need to identify what's the bottleneck in your case.

First things to check:

1) jvm - heap is too small for such workloads

Enable GC logging in /etc/cassandra/cassandra-env.sh, then analyze its output 
during workload. In case you observe messages about long pauses  or Full GC's, 
most probably it's the root-cause.

2) hardware - monitor cpu, disks when writes are active.


there is a really good command 'nodetool tpstats' which helps identify 
bottleneck / understand what's going on.


Two articles that may help (credits to Chris and Jon):

https://blog.pythian.com/guide-to-cassandra-thread-pools/

http://thelastpickle.com/blog/2018/04/11/gc-tuning.html


Regards,

Kyrill


From: Soheil Pourbafrani 
Sent: Monday, April 30, 2018 10:40:10 AM
To: user@cassandra.apache.org
Subject: Host not available

I have 3 node Cassandra 3.11 cluster, one node 4GB memory, and others 2GB. All 
of them have 4 CPU core. I want to insert data into the table and read it to 
visualize at the san\me time. When I insert data using Flink Cassandra 
Connector, in rate 200 inserts per sec, the reader application can't connect to 
the hosts, but when the insertion process is not running, the reader 
application can connect and fetch data.

All Cassandra,yaml properties are defaults.
Two nodes are seed nodes and keyspace replication factor is 2.

How can I optimize the cluster and solve the problem?

In addition, I have the following log:

 INFO  [epollEventLoopGroup-2-3] 2018-04-30 12:05:11,225 Message.java:623 - 
Unexpected exception during request; channel = [id: 0xb84fd5cc, 
L:/192.168.1.218:9042 - R:/1$
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed: 
Connection reset by peer

Re: Does node add/remove requires all cluster nodes be present?

[Jeff Jirsa]

nodetool decommission streams data from the losing replica, so only that 
instance has to be online (and decom should be preferred to removenode)

If that instance is offline, you can use removenode, but you risk violating 
consistency guarantees

Adding nodes is similar - bootstrap streams from the losing range

-- 
Jeff Jirsa


> On Apr 30, 2018, at 8:57 PM, Jinhua Luo  wrote:
> 
> Hi All,
> 
> When a new node added, due to the even distribution of the new tokens,
> the current nodes of the ring should migrate data to this new node.
> 
> So, does it requires all nodes be present? If not, then if some nodes
> are down, then it will miss the data migration of those parts, how and
> when to fix it? When those nodes come back?
> 
> Similarly, the node removal would migrate its data to other nodes, so
> it seems that all other nodes must be present, otherwise it would lost
> data?
> 
> -
> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: user-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Does node add/remove requires all cluster nodes be present?

[Jinhua Luo]

Hi All,

When a new node added, due to the even distribution of the new tokens,
the current nodes of the ring should migrate data to this new node.

So, does it requires all nodes be present? If not, then if some nodes
are down, then it will miss the data migration of those parts, how and
when to fix it? When those nodes come back?

Similarly, the node removal would migrate its data to other nodes, so
it seems that all other nodes must be present, otherwise it would lost
data?

-
To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org
For additional commands, e-mail: user-h...@cassandra.apache.org

Re: Schema disagreement

[Gábor Auth]

Hi,

On Mon, Apr 30, 2018 at 11:03 PM Ali Hubail 
wrote:

> What steps have you performed to add the new DC? Have you tried to follow
> certain procedures like this?
>
> https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html
>

Yes, exactly. :/

Bye,
Gábor Auth

Re: Schema disagreement

[Ali Hubail]

Hi,

What steps have you performed to add the new DC? Have you tried to follow 
certain procedures like this?
https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html

Node can appear offline to other nodes for various reasons. It would help 
greatly to know what steps you have taken in order to know why you're 
facing this

Ali Hubail

Confidentiality warning: This message and any attachments are intended 
only for the persons to whom this message is addressed, are confidential, 
and may be privileged. If you are not the intended recipient, you are 
hereby notified that any review, retransmission, conversion to hard copy, 
copying, modification, circulation or other use of this message and any 
attachments is strictly prohibited. If you receive this message in error, 
please notify the sender immediately by return email, and delete this 
message and any attachments from your system. Petrolink International 
Limited its subsidiaries, holding companies and affiliates disclaims all 
responsibility from and accepts no liability whatsoever for the 
consequences of any unauthorized person acting, or refraining from acting, 
on any information contained in this message. For security purposes, staff 
training, to assist in resolving complaints and to improve our customer 
service, email communications may be monitored and telephone calls may be 
recorded.



Gábor Auth  
04/30/2018 03:40 PM
Please respond to
user@cassandra.apache.org


To
"user@cassandra.apache.org" , 
cc

Subject
Re: Schema disagreement






Hi,

On Mon, Apr 30, 2018 at 11:39 AM Gábor Auth  wrote:
've just tried to add a new DC and new node to my cluster (3 DCs and 10 
nodes) and the new node has a different schema version:

Is it normal? Node is marked down but doing a repair successfully?

WARN  [MigrationStage:1] 2018-04-30 20:36:56,579 MigrationTask.java:67 - 
Can't send schema pull request: node /x.x.216.121 is down.
INFO  [AntiEntropyStage:1] 2018-04-30 20:36:56,611 Validator.java:281 - 
[repair #323bf873-4cb6-11e8-bdd5-5feb84046dc9] Sending completed merkle 
tree to /x.x.216.121 for keyspace.table

The `nodetool status` is looking good:
UN  x.x.216.121  959.29 MiB  32   ? 
  322e4e9b-4d9e-43e3-94a3-bbe012058516  RACK01

Bye,
Gábor Auth

Re: Schema disagreement

[Gábor Auth]

Hi,

On Mon, Apr 30, 2018 at 11:39 AM Gábor Auth  wrote:

> 've just tried to add a new DC and new node to my cluster (3 DCs and 10
> nodes) and the new node has a different schema version:
>

Is it normal? Node is marked down but doing a repair successfully?

WARN  [MigrationStage:1] 2018-04-30 20:36:56,579 MigrationTask.java:67 -
Can't send schema pull request: node /x.x.216.121 is down.
INFO  [AntiEntropyStage:1] 2018-04-30 20:36:56,611 Validator.java:281 -
[repair #323bf873-4cb6-11e8-bdd5-5feb84046dc9] Sending completed merkle
tree to /x.x.216.121 for keyspace.table

The `nodetool status` is looking good:
UN  x.x.216.121  959.29 MiB  32   ?
  322e4e9b-4d9e-43e3-94a3-bbe012058516  RACK01

Bye,
Gábor Auth

Schema disagreement

[Gábor Auth]

Hi,

I've just tried to add a new DC and new node to my cluster (3 DCs and 10
nodes) and the new node has a different schema version:

Cluster Information:
Name: cluster
Snitch: org.apache.cassandra.locator.DynamicEndpointSnitch
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Schema versions:
7e12a13e-dcca-301b-a5ce-b1ad29fbbacb: [x.x.x.x, ..., ...]
bb186922-82b5-3a61-9c12-bf4eb87b9155: [new.new.new.new]

I've tried:
- node decommission and node re-addition
- resetlocalschema
- rebuild
- replace node
- repair
- cluster restart (node-by-node)

The MigrationManager constantly running on the new node and try to migrate
schema:
DEBUG [NonPeriodicTasks:1] 2018-04-30 09:33:22,405
MigrationManager.java:125 - submitting migration task for /x.x.x.x

What also can I do? :(

Bye,
Gábor Auth

Does repair handles indexes

[Akshit Jain]

Hi,
Does repair in cassandra handles secondary indexes repair/rebuild?
Has someone observed this/ is there any documentation related to it.

Regards

Re: Frequency of rebuild_index

[Akshit Jain]

Hi,
This looks good but when can the discrepancy in the index arise. Any
specific example?
Is there any documentation which says the index automatically
rebuilds/keeps itself up to date after updations and deletions. Also if
there anything specific to stratio-lucene-index.

Regards
Akshit Jain
9891724697

On Fri, Apr 27, 2018 at 9:59 AM, Anup Shirolkar <
anup.shirol...@instaclustr.com> wrote:

> Hi,
>
> The secondary indices in Cassandra are maintained continuously as data is
> written. Also index rebuilding is kicked off automatically when you create
> a new index. So, there is no good reason to schedule nodetool rebuild_index
> regularly.
>
> However, if you find any discrepancy in the index and data you should run
> it. Ideally, this should not happen but if it is required as a result of
> any major activity/failure you can use it.
>
> Talking about the load it puts on system, it depends upon the size of
> index itself. Although it will consume resources, it should not give a
> major performance hit to the system.
>
> Regards,
> Anup
>
> On 27 April 2018 at 13:46, Akshit Jain  wrote:
>
>> Hi,
>> How frequently one should run nodetool rebuild_index and what's its
>> impact on performance in terms of iops,cpu utilisation etc.
>>
>> Regards
>>
>>
>

Host not available

[Soheil Pourbafrani]

I have 3 node Cassandra 3.11 cluster, one node 4GB memory, and others 2GB.
All of them have 4 CPU core. I want to insert data into the table and read
it to visualize at the san\me time. When I insert data using Flink
Cassandra Connector, in rate 200 inserts per sec, the reader application
can't connect to the hosts, but when the insertion process is not running,
the reader application can connect and fetch data.

All Cassandra,yaml properties are defaults.
Two nodes are seed nodes and keyspace replication factor is 2.

How can I optimize the cluster and solve the problem?

In addition, I have the following log:

 INFO  [epollEventLoopGroup-2-3] 2018-04-30 12:05:11,225 Message.java:623 -
Unexpected exception during request; channel = [id: 0xb84fd5cc, L:/
192.168.1.218:9042 - R:/1$
io.netty.channel.unix.Errors$NativeIoException: syscall:read(...)() failed:
Connection reset by peer