You can increase the max number of open files on the new node. We find that 65K is too low for most production clusters and you can bump it up to 100 or 200K. We generally recommend 1 million but YMMV:
<cassandra_user> - nofile 1048576 On Tue, Jan 28, 2020 at 11:55 AM Eunsu Kim <eunsu.bil...@gmail.com> wrote: > Hi experts > > I had a problem adding a new node. > > Joining node in datacenterA stops streaming while joining. So it keeps the > UJ. > (datacenterB is fine.) > > I try 'nodetool netstats' on a stopped node and it looks like this: > > Mode: JOINING > Not sending any streams. > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > > When I try 'nodetool rebuild' it changes to the following but no steaming > occurs. > > Mode: JOINING > Rebuild 1df64590-4166-11ea-86a0-4b3cc5e92e4a > Read Repair Statistics: > Attempted: 0 > Mismatch (Blocking): 0 > Mismatch (Background): 0 > > I think this is related to the number of open file descriptors. > > > Incoming Streaming Bytes went to zero after the number of open file > descriptors reached the host's MAX (65536). > Since then, the number of open file descriptors has decreased, but > steaming has not resumed. > > And when I drop that joining process, it automatically was removed from > the cluster. > > What should I do to add nodes to this data center in this case? > > Please advice. > > Thank you. >
