Hi Jeff

Thank a lot for your answer.
The reference to "fat client" is very interesting… On debug log on classical 
node, we have sometimes message like :

INFO  [GossipTasks:1] 2021-03-17 16:21:01,135 Gossiper.java:894 - FatClient 
/10.120.1.183 has been silent for 30000ms, removing from gossip

Does it means that the fat client (our proxies) is removed from gossip, only 
after 30 seconds ? In such case, the delay I ask for is 30 sec :-)
Does someone know if this parameter can be changed ?

PS : yes proxies work really well… we indeed use PHP with FPM. That the reason 
why we have a lot of connections and so need proxies. Basically if counting all 
FPM on all our PHP servers, I’d said 8000 to 10000 clients… maybe more.
Advantages are multiple but basically we had a lot of pressure on Cassandra 
node when restarting all our PHP servers during a new code rollout requiring 
PHP reload for instance (many times per day). Proxies saved us. We can continue 
to talk in private if you want.

De : Jeff Jirsa <jji...@gmail.com>
Envoyé : mercredi 17 mars 2021 14:52
À : cassandra <user@cassandra.apache.org>
Objet : Re: Delay between stop/start cassandra

-Dcassandra.join_ring=false is basically a pre-bootstrap phase that says "this 
machine is about to join the cluster, but hasn't yet, so don't give it a token"

It's taking advantage of a stable but non-terminal state to let you do things 
like serve queries without owning data - it's a side effect that works, but 
it's rough because it wasn't exactly built for this purpose. In this state, 
you're considered a "fat client" - your presence exists in the ring as a "I'm 
here, about to join the ring with IP a.b.c.d", and you just conveniently decide 
not to join the ring. If you go away at any time, the cluster says "cool, no 
big deal, they didn't join the ring anyway".

Your hypothesis is probably mostly right here - it's not so much UP or DOWN, 
it's "still here" or "gone". Because once the instance is DOWN, it gets removed 
because it hadn't finished joining. Once it's removed, it can come back and say 
"Hi, me again, about to join this cluster". But, until it's removed as a fat 
client, when it comes back and says "Hi, me again, about to join this cluster", 
cassandra says "not so fast friend, you're already here and we haven't yet 
given up on you joining".

Random aside: There are relatively few people on earth who run like this, so 
I'm super interested in knowing how it's working for you. Does the PHP client 
still reconnect on every page load, or does it finally support long lived 
connections / pooling if you're using something like php-fpm or a fastcgi pool? 
Are the coordinators/proxies here just to handle a ridiculous number of 
clients, or is it the cost of connecting that's hurting as you blow up the 
native thread pool on connect for expensive auth?





On Wed, Mar 17, 2021 at 5:44 AM Regis Le Bretonnic 
<r.lebreton...@meetic-corp.com<mailto:r.lebreton...@meetic-corp.com>> wrote:
Hi all,

Following a discussion with our adminsys, I have a very practical question.
We use cassandra proxies (-Dcassandra.join_ring=false) as coordinators for PHP 
clients (a loooooot of PHP clients).

Our problem is that restarting Cassandra on proxies sometimes fails with the 
following error :

ERROR [main] 2021-03-16 14:18:46,236 CassandraDaemon.java:803 - Exception 
encountered during startup
java.lang.RuntimeException: A node with address XXXXXXXXXXXXXXXX/10.120.1.XXX 
already exists, cancelling join. Use cassandra.replace_address if you want to 
replace this node.

The node mentioned in the ERROR is the one we are restarting… and the start 
fails. Of course doing a manual start after works fine.
This message doesn’t make sense… hostId didn’t changed for this proxy (I am 
sure of me : system.local, IP, hostname, … nothing changed… just the restart).

What I suppose (we don’t all agree about this) is that, as proxies don’t have 
data, they start very quickly. Too quickly for gossip protocol knows that the 
node was down.
Could this ERROR log be explained if the node is still known as UP by seeds 
servers if the state of the proxy in gossip protocol is not updated because 
stop/start is made too quickly ?
If this hypothesis seems possible, what reasonable delay (with technical 
arguments) should be implemented between stop and start ?
We have ~ 100 proxies and 12 classical Cassandra (4 of them are seeds)…

Thx in advance

Reply via email to