Re: Vulnerability in libthrift library (CVE-2019-0205)
Thank you. On Tue, Oct 5, 2021 at 9:19 AM Aaron Ploetz wrote: > In reading the Jira ticket, I see this line: > > "a server or client may run into an endless loop *when fed with specific > input data.*" > > That seems to suggest that if the cassandra.yaml contains start_rpc: false, > you should be ok. > > On Mon, Oct 4, 2021 at 8:12 PM rammohan ganapavarapu < > rammohanga...@gmail.com> wrote: > >> We are not using thrift but the lib is there with Cassandra binary right? >> Does it cause any risk? >> >> On Mon, Oct 4, 2021, 5:53 PM Erick Ramirez >> wrote: >> >>> See https://issues.apache.org/jira/browse/CASSANDRA-15420. It only >>> applies if you're still using Thrift in 2021. Cheers! >>> >>
Re: Vulnerability in libthrift library (CVE-2019-0205)
In reading the Jira ticket, I see this line: "a server or client may run into an endless loop *when fed with specific input data.*" That seems to suggest that if the cassandra.yaml contains start_rpc: false, you should be ok. On Mon, Oct 4, 2021 at 8:12 PM rammohan ganapavarapu < rammohanga...@gmail.com> wrote: > We are not using thrift but the lib is there with Cassandra binary right? > Does it cause any risk? > > On Mon, Oct 4, 2021, 5:53 PM Erick Ramirez > wrote: > >> See https://issues.apache.org/jira/browse/CASSANDRA-15420. It only >> applies if you're still using Thrift in 2021. Cheers! >> >
Re: Streaming failure Issue
Logged "nodetool failuredetector" every 5sec. Doesn't seems to be an issue for phi_convict_threshold value On Tue, Oct 5, 2021 at 4:35 PM Surbhi Gupta wrote: > Hi , > > Try to adjust phi_convict_threshold and see if that helps. > When we did migration from on prim to AWS, this was one of the factor to > consider. > > Thanks > > > On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: > >> Hi all, >> >> Need urgent help. >> We have one Physical Data Center of 5 nodes with 1 TB data on each >> (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are >> Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it >> to the existing one. >> >> While running nodetool rebuild command, we are getting following error : >> On GCP node (where we ran rebuild command) : >> >>> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 >>> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>> Remote peer 192.x.x.x failed stream session. >>> INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 >>> StreamResultFuture.java:183 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete >> >> >> On DL source node : >> >>> INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 >>> StreamResultFuture.java:183 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete >>> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 >>> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >>> Streaming error occurred >>> java.lang.RuntimeException: Transfer of file >>> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db >>> already completed or aborted (perhaps session failed?). >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at >>> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) >>> ~[apache-cassandra-3.0.9.jar:3.0.9] >>> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] >>> WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 >>> StreamResultFuture.java:210 - [Stream >>> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed >> >> >> Before starting this rebuild, we have made the following changes: >> 1. Set setstreamthroughput to 600 Mb/sec >> 2. Set setinterdcstreamthroughput to 600 Mb/sec >> 3. streaming_socket_timeout_in_ms is 24 hrs >> 4. Disabled autocompaction on GCP node as this was heavily utilising CPU >> resource >> >> FYI, GCP rebuild process starts with data streaming from 3 nodes, and all >> fails one by one after streaming for a few hours. >> Please help out how to correct this issue. >> Is there any other way to rebuild such big data. >> We have a few tables with 200 - 400GB of data and some smaller tables. >> Also, we have Mviews in our environment >> >> Regards, >> Ashish Gupta >> >> >
Re: Streaming failure Issue
Hi , Try to adjust phi_convict_threshold and see if that helps. When we did migration from on prim to AWS, this was one of the factor to consider. Thanks On Tue, Oct 5, 2021 at 4:00 AM MyWorld wrote: > Hi all, > > Need urgent help. > We have one Physical Data Center of 5 nodes with 1 TB data on each > (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are > Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it > to the existing one. > > While running nodetool rebuild command, we are getting following error : > On GCP node (where we ran rebuild command) : > >> ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 >> StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >> Remote peer 192.x.x.x failed stream session. >> INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 >> StreamResultFuture.java:183 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete > > > On DL source node : > >> INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 >> StreamResultFuture.java:183 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete >> ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 >> StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] >> Streaming error occurred >> java.lang.RuntimeException: Transfer of file >> /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db >> already completed or aborted (perhaps session failed?). >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at >> org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) >> ~[apache-cassandra-3.0.9.jar:3.0.9] >> at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] >> WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 >> StreamResultFuture.java:210 - [Stream >> #66646d30-25a2-11ec-903b-774f88efe725] Stream failed > > > Before starting this rebuild, we have made the following changes: > 1. Set setstreamthroughput to 600 Mb/sec > 2. Set setinterdcstreamthroughput to 600 Mb/sec > 3. streaming_socket_timeout_in_ms is 24 hrs > 4. Disabled autocompaction on GCP node as this was heavily utilising CPU > resource > > FYI, GCP rebuild process starts with data streaming from 3 nodes, and all > fails one by one after streaming for a few hours. > Please help out how to correct this issue. > Is there any other way to rebuild such big data. > We have a few tables with 200 - 400GB of data and some smaller tables. > Also, we have Mviews in our environment > > Regards, > Ashish Gupta > >
Streaming failure Issue
Hi all, Need urgent help. We have one Physical Data Center of 5 nodes with 1 TB data on each (Location: Dallas). Currently we are using Cassandra ver 3.0.9. Now we are Adding one more Data Center of 5 nodes(Location GCP-US) and have joined it to the existing one. While running nodetool rebuild command, we are getting following error : On GCP node (where we ran rebuild command) : > ERROR [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,246 > StreamSession.java:639 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Remote peer 192.x.x.x failed stream session. > INFO [STREAM-IN-/192.x.x.x] 2021-10-05 15:56:52,266 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /192.x.x.x is complete On DL source node : > INFO [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamResultFuture.java:183 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Session with /34.x.x.x is complete > ERROR [STREAM-OUT-/34.x.x.x] 2021-10-05 15:55:53,785 > StreamSession.java:534 - [Stream #66646d30-25a2-11ec-903b-774f88efe725] > Streaming error occurred > java.lang.RuntimeException: Transfer of file > /var/lib/cassandra/data/clickstream/glusr_usr_paid_url_mv-3c49c392b35511e9bd0a8f42dfb09617/mc-45676-big-Data.db > already completed or aborted (perhaps session failed?). > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage.startTransfer(OutgoingFileMessage.java:120) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:50) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.OutgoingFileMessage$1.serialize(OutgoingFileMessage.java:42) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.messages.StreamMessage.serialize(StreamMessage.java:48) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.sendMessage(ConnectionHandler.java:387) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at > org.apache.cassandra.streaming.ConnectionHandler$OutgoingMessageHandler.run(ConnectionHandler.java:367) > ~[apache-cassandra-3.0.9.jar:3.0.9] > at java.lang.Thread.run(Thread.java:748) [na:1.8.0_192] > WARN [STREAM-IN-/34.x.x.x] 2021-10-05 15:55:53,786 > StreamResultFuture.java:210 - [Stream > #66646d30-25a2-11ec-903b-774f88efe725] Stream failed Before starting this rebuild, we have made the following changes: 1. Set setstreamthroughput to 600 Mb/sec 2. Set setinterdcstreamthroughput to 600 Mb/sec 3. streaming_socket_timeout_in_ms is 24 hrs 4. Disabled autocompaction on GCP node as this was heavily utilising CPU resource FYI, GCP rebuild process starts with data streaming from 3 nodes, and all fails one by one after streaming for a few hours. Please help out how to correct this issue. Is there any other way to rebuild such big data. We have a few tables with 200 - 400GB of data and some smaller tables. Also, we have Mviews in our environment Regards, Ashish Gupta
