Re: Streaming failure Issue

2021-10-07 Thread Jim Shaw
The error message indicates source side stream session failed. if source side load is heavy, consider reduce some, like stop repair, etc. If every trying failed at same file, may check that file too. Hope it helps. On Wed, Oct 6, 2021 at 11:20 PM MyWorld wrote: > Hi Jim, > It's 600 Megabits

Re: Streaming failure Issue

2021-10-06 Thread MyWorld
Hi Jim, It's 600 Megabits not MegaBytes. So near around 600/8=75MBps. Also, streaming is happening from any 3 nodes at a time. However we have tried with default Streaming throughput which is 200 Megabits per sec (25MBps), but still the same issue. Heap we have setup to 8gb on GCP and seems pretty

Re: Streaming failure Issue

2021-10-06 Thread Jim Shaw
I met similar issue before. What I did was: reduce Heap size for rebuild, reduce streamthroughput. But it depends on version, and your env., may not your case, just hope it helpful. ps -ef | grep , you will see a new java process for rebuild, see what memory size used, if use default, it

Re: Streaming failure Issue

2021-10-05 Thread MyWorld
Logged "nodetool failuredetector" every 5sec. Doesn't seems to be an issue for phi_convict_threshold value On Tue, Oct 5, 2021 at 4:35 PM Surbhi Gupta wrote: > Hi , > > Try to adjust phi_convict_threshold and see if that helps. > When we did migration from on prim to AWS, this was one of the

Re: Streaming failure Issue

2021-10-05 Thread Surbhi Gupta
Hi , Try to adjust phi_convict_threshold and see if that helps. When we did migration from on prim to AWS, this was one of the factor to consider. Thanks On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: > Hi all, > > Need urgent help. > We have one Physical Data Center of 5 nodes with 1 TB data

Streaming failure Issue

2021-10-05 Thread MyWorld
Hi all, Need urgent help. We have one Physical Data Center of 5 nodes with 1 TB data on each (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it to the existing one. While running nodetool rebuild