Re: multiple node bootstrapping
You can use auto_bootstrap set to false to add a new node to the ring, it will calculate the token range for the new node, but will not start streaming the data. In this case you can add several nodes into the ring quickly. After that you can start nodetool rebuild -dc <> to start streaming data. In your case 50Tb of data per node is quite a large amount of data i would recommend, based on own experience keeping 1Tb per node, since when streaming can be interrupted for some reason and it cannot be resumed so you'll have to restart streaming. Also there will be compaction problems. Vitali. On Wed, Nov 28, 2018 at 12:03 PM Osman YOZGATLIOĞLU < osman.yozgatlio...@krontech.com> wrote: > Hello, > > I have 2 dc cassandra 3.0.14 setup. I need to add 2 new nodes to each dc. > > I started one node in dc1 and its already joining. 3TB of 50TB finished in > 2 weeks. One year ttl time series data with twcs. > > I know, its not best practise.. > > I want to start one node in dc2 and cassandra refused to start with > mentioning already one node in joining state. > > I find some workaround with jmx directives, but i'm not sure if I broke > something on the way. > > Is it wise to bootstrap in both dc at the same time? > > > Regards, > > Osman >
Re: system_auth keyspace replication factor
Attaching the runner log snippet, where we can see that "Rebuilding token map" took most of the time. getAllroles is using quorum, don't if it is used during login https://github.com/apache/cassandra/blob/cc12665bb7645d17ba70edcf952ee6a1ea63127b/src/java/org/apache/cassandra/auth/CassandraRoleManager.java#L260 Vitali Djatsuk, On Fri, Nov 23, 2018 at 8:32 PM Jeff Jirsa wrote: > I suspect some of the intermediate queries (determining role, etc) happen > at quorum in 2.2+, but I don’t have time to go read the code and prove it. > > In any case, RF > 10 per DC is probably excessive > > Also want to crank up the validity times so it uses cached info longer > > > -- > Jeff Jirsa > > > On Nov 23, 2018, at 10:18 AM, Vitali Dyachuk wrote: > > no its not a cassandra user and as i understood all other users login > local_one. > > On Fri, 23 Nov 2018, 19:30 Jonathan Haddad >> Any chance you’re logging in with the Cassandra user? It uses quorum >> reads. >> >> >> On Fri, Nov 23, 2018 at 11:38 AM Vitali Dyachuk >> wrote: >> >>> Hi, >>> We have recently met a problem when we added 60 nodes in 1 region to the >>> cluster >>> and set an RF=60 for the system_auth ks, following this documentation >>> https://docs.datastax.com/en/cql/3.3/cql/cql_using/useUpdateKeyspaceRF.html >>> However we've started to see increased login latencies in the cluster 5x >>> bigger than before changing RF of system_auth ks. >>> We have casandra runner written is csharp, running against the cluster, >>> when analyzing the logs we notices that Rebuilding token map is >>> taking most of the time ~20s. >>> When we changed RF to 3 the issue has resolved. >>> We are using C* 3.0.17 , 4 DC, system_auth RF=3, "CassandraCSharpDriver" >>> version="3.2.1" >>> I've found somehow related to my problem ticket >>> https://datastax-oss.atlassian.net/browse/CSHARP-436 but it says in the >>> related tickets, that the issue with the token map rebuild time has been >>> fixed in the previous versions of the driver. >>> So my question is what is the best recommendation of the setting >>> system_auth ks RF? >>> >>> Regards, >>> Vitali Djatsuk. >>> >>> >>> -- >> Jon Haddad >> http://www.rustyrazorblade.com >> twitter: rustyrazorblade >> > ControlConnection: 11/22/2018 10:30:32.170 +00:00 : Trying to connect the ControlConnection TcpSocket: 11/22/2018 10:30:32.170 +00:00 Socket connected, starting SSL client authentication TcpSocket: 11/22/2018 10:30:32.170 +00:00 Starting SSL authentication TcpSocket: 11/22/2018 10:30:32.217 +00:00 SSL authentication successful Connection: 11/22/2018 10:30:32.217 +00:00 Sending #0 for StartupRequest to node1:9042 Connection: 11/22/2018 10:30:32.217 +00:00 Received #0 from node1:9042 Connection: 11/22/2018 10:30:32.217 +00:00 Sending #0 for AuthResponseRequest to node1:9042 Connection: 11/22/2018 10:30:32.329 +00:00 Received #0 from node1:9042 ControlConnection: 11/22/2018 10:30:32.329 +00:00 : Connection established to node1:9042 Connection: 11/22/2018 10:30:32.329 +00:00 Sending #0 for RegisterForEventRequest to node1:9042 Connection: 11/22/2018 10:30:32.329 +00:00 Received #0 from node1:9042 ControlConnection: 11/22/2018 10:30:32.329 +00:00 : Refreshing node list Connection: 11/22/2018 10:30:32.329 +00:00 Sending #0 for QueryRequest to node1:9042 Connection: 11/22/2018 10:30:32.342 +00:00 Received #0 from node1:9042 Connection: 11/22/2018 10:30:32.342 +00:00 Sending #0 for QueryRequest to node1:9042 Connection: 11/22/2018 10:30:32.373 +00:00 Received #0 from node1:9042 ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Node list retrieved successfully ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Retrieving keyspaces metadata Connection: 11/22/2018 10:30:32.389 +00:00 Sending #0 for QueryRequest to node1:9042 Connection: 11/22/2018 10:30:32.389 +00:00 Received #0 from node1:9042 ControlConnection: 11/22/2018 10:30:32.389 +00:00 : Rebuilding token map Cluster: 11/22/2018 10:30:55.233 +00:00 : Cluster Connected using binary protocol version: [4] - To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org For additional commands, e-mail: user-h...@cassandra.apache.org
Re: system_auth keyspace replication factor
no its not a cassandra user and as i understood all other users login local_one. On Fri, 23 Nov 2018, 19:30 Jonathan Haddad Any chance you’re logging in with the Cassandra user? It uses quorum > reads. > > > On Fri, Nov 23, 2018 at 11:38 AM Vitali Dyachuk > wrote: > >> Hi, >> We have recently met a problem when we added 60 nodes in 1 region to the >> cluster >> and set an RF=60 for the system_auth ks, following this documentation >> https://docs.datastax.com/en/cql/3.3/cql/cql_using/useUpdateKeyspaceRF.html >> However we've started to see increased login latencies in the cluster 5x >> bigger than before changing RF of system_auth ks. >> We have casandra runner written is csharp, running against the cluster, >> when analyzing the logs we notices that Rebuilding token map is >> taking most of the time ~20s. >> When we changed RF to 3 the issue has resolved. >> We are using C* 3.0.17 , 4 DC, system_auth RF=3, "CassandraCSharpDriver" >> version="3.2.1" >> I've found somehow related to my problem ticket >> https://datastax-oss.atlassian.net/browse/CSHARP-436 but it says in the >> related tickets, that the issue with the token map rebuild time has been >> fixed in the previous versions of the driver. >> So my question is what is the best recommendation of the setting >> system_auth ks RF? >> >> Regards, >> Vitali Djatsuk. >> >> >> -- > Jon Haddad > http://www.rustyrazorblade.com > twitter: rustyrazorblade >
system_auth keyspace replication factor
Hi, We have recently met a problem when we added 60 nodes in 1 region to the cluster and set an RF=60 for the system_auth ks, following this documentation https://docs.datastax.com/en/cql/3.3/cql/cql_using/useUpdateKeyspaceRF.html However we've started to see increased login latencies in the cluster 5x bigger than before changing RF of system_auth ks. We have casandra runner written is csharp, running against the cluster, when analyzing the logs we notices that Rebuilding token map is taking most of the time ~20s. When we changed RF to 3 the issue has resolved. We are using C* 3.0.17 , 4 DC, system_auth RF=3, "CassandraCSharpDriver" version="3.2.1" I've found somehow related to my problem ticket https://datastax-oss.atlassian.net/browse/CSHARP-436 but it says in the related tickets, that the issue with the token map rebuild time has been fixed in the previous versions of the driver. So my question is what is the best recommendation of the setting system_auth ks RF? Regards, Vitali Djatsuk.
Re: nodetool rebuild
Dinesh this is my understanding of streamng options in C* 3.0 1) nodetool rebuild - is a default option to stream data to a new node, when adding new regions pros: simply run rebuild to stream data to a new node: nodetool rebuild -dc & cons: If internode compression enabled or per table compression enabled it takes too much time to stream data because of compression, If compression disabled then we need to have much more disk space and increase in streaming speed is only 15%. Also it is single threaded per 1 remote node, which affects the overall streaming time. If the streaming process failed you have to rebuild again from scratch and before delete all streamed data. 2) Manually get a new node tokens with SELECT tokens FROM system.local; then find all replicas for these token ranges, then create sstable snapshots for these token ranges on the remote nodes, then find manually in which sstables are these ranges, only after that copy to the new node. pros: Potentially if we already have snapshots on all nodes, then finding these ranges in the snapshotted sstables is more or less possible cons: very difficult manual procedure, which in the end need repairing to validate the data consistency 3) running repair on the new node with cassandra-reaper pros: if the rebuild process has failed at 80% then its makes sense to run repair which will eventually stream missing data to the new node cons: takes too much time to finish since it calculates merkle tree for each token range finds the difference and then streams. 6 hours for 10Gb of data If we are scaling out the existing data centers then the bootstrap process will take care of streaming data to a new node, we just need to add a new node to the region. Vitali On Sun, Sep 16, 2018 at 11:02 PM Vitali Dyachuk wrote: > Yes, we are using 256 vnodes.Keyspace is configured with > NetworkTopologyStrategy in 4 regions, with RF3. > Copying sstabes and running cleanup is a good idea. > > On Sun, Sep 16, 2018 at 9:26 PM Dinesh Joshi > wrote: > >> It would be helpful to describe your setup - specifically are you using >> vnodes? How is the keyspace setup? One option would be to copy SSTables >> from the replicas and running clean up. That might actually be faster. >> Since the SSTables are compressed you should use a tool that copies without >> compressing the data stream in transit. >> >> Dinesh >> >> On Sep 16, 2018, at 2:07 AM, Vitali Dyachuk wrote: >> >> Both stream throughput settings are set to 0, meaning that there is no >> stream throttling on the C* side. Yes, i see high cpu used by STREAM-IN >> thread, sstables are compressed up to 80% >> What about copying sstables with rsync and then running repair? Probably >> its not that simple, vut If the data is RF3 so one node should have all the >> key ranges and repair will not recalculate all the hashes? >> >> Vitali >> >> On Sun, Sep 16, 2018, 02:33 dinesh.jo...@yahoo.com.INVALID < >> dinesh.jo...@yahoo.com.invalid> wrote: >> >>> Its a long shot but do you have >>> stream_throughput_outbound_megabits_per_sec or >>> inter_dc_stream_throughput_outbound_megabits_per_sec set to a low value? >>> >>> You're right in that 3.0 streaming uses 1 thread for incoming and >>> outgoing connection each per peer. It not only reads the bytes off of the >>> channel but also deserializes the partitions on that same thread. If you >>> see high CPU use by STREAM-IN thread then your streaming is CPU bound. In >>> this situation a powerful CPU will definitely help. Dropping internode >>> compression and encryption will also help. Are your SSTables compressed? >>> >>> Dinesh >>> >>> >>> On Friday, September 14, 2018, 4:15:28 AM PDT, Vitali Dyachuk < >>> vdjat...@gmail.com> wrote: >>> >>> >>> None of these throttling are helpful for streaming if you have even a >>> 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network >>> tcp memory, window size etc does not help, the bottleneck is not the >>> network. >>> These are my findings on how streaming is limited in C* 3.0.* >>> >>> 1) Streaming of the particular range which needs to be steamed to the >>> new node is limited with one 1 thread and no tweaking of cpu affinity etc >>> helps, probably the powerfull computing VM will help >>> 2) Disabling compression internode_compression and disabling compression >>> per table in our case helps a bit >>> 3) When streaming has been dropped there is no resume available for the >>> streaming range so it will start from the beginning >>> >>> One of the options
Re: nodetool rebuild
Yes, we are using 256 vnodes.Keyspace is configured with NetworkTopologyStrategy in 4 regions, with RF3. Copying sstabes and running cleanup is a good idea. On Sun, Sep 16, 2018 at 9:26 PM Dinesh Joshi wrote: > It would be helpful to describe your setup - specifically are you using > vnodes? How is the keyspace setup? One option would be to copy SSTables > from the replicas and running clean up. That might actually be faster. > Since the SSTables are compressed you should use a tool that copies without > compressing the data stream in transit. > > Dinesh > > On Sep 16, 2018, at 2:07 AM, Vitali Dyachuk wrote: > > Both stream throughput settings are set to 0, meaning that there is no > stream throttling on the C* side. Yes, i see high cpu used by STREAM-IN > thread, sstables are compressed up to 80% > What about copying sstables with rsync and then running repair? Probably > its not that simple, vut If the data is RF3 so one node should have all the > key ranges and repair will not recalculate all the hashes? > > Vitali > > On Sun, Sep 16, 2018, 02:33 dinesh.jo...@yahoo.com.INVALID < > dinesh.jo...@yahoo.com.invalid> wrote: > >> Its a long shot but do you have >> stream_throughput_outbound_megabits_per_sec or >> inter_dc_stream_throughput_outbound_megabits_per_sec set to a low value? >> >> You're right in that 3.0 streaming uses 1 thread for incoming and >> outgoing connection each per peer. It not only reads the bytes off of the >> channel but also deserializes the partitions on that same thread. If you >> see high CPU use by STREAM-IN thread then your streaming is CPU bound. In >> this situation a powerful CPU will definitely help. Dropping internode >> compression and encryption will also help. Are your SSTables compressed? >> >> Dinesh >> >> >> On Friday, September 14, 2018, 4:15:28 AM PDT, Vitali Dyachuk < >> vdjat...@gmail.com> wrote: >> >> >> None of these throttling are helpful for streaming if you have even a >> 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network >> tcp memory, window size etc does not help, the bottleneck is not the >> network. >> These are my findings on how streaming is limited in C* 3.0.* >> >> 1) Streaming of the particular range which needs to be steamed to the >> new node is limited with one 1 thread and no tweaking of cpu affinity etc >> helps, probably the powerfull computing VM will help >> 2) Disabling compression internode_compression and disabling compression >> per table in our case helps a bit >> 3) When streaming has been dropped there is no resume available for the >> streaming range so it will start from the beginning >> >> One of the options could be to create snapshots of sstables on the source >> node and just copy all sstable snapshots to new node and then run repair, >> data is ~5TB, RF3 ? >> How is it possible at all to stream data fast to a new node/nodes ? >> >> Vitali. >> >> On Wed, Sep 12, 2018 at 5:02 PM Surbhi Gupta >> wrote: >> >> Increase 3 throughput >> Compaction throughput >> Stream throughput >> Interdcstream throughput (if rebuilding from another DC) >> >> Make all of the above to 0 and see if there is any improvement and later >> set the value if u can’t leave these values to 0. >> >> On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk >> wrote: >> >> Hi, >> I'm currently streaming data with nodetool rebuild on 2 nodes, each node >> is streaming from different location. The problem is that it takes ~7 days >> to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so >> it should take around >> ~2,5 days . Although there are resources on the destnodes and in the >> source regions. >> I've increased stream throughput, but its only affects outbound >> connections. >> Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week >> i've changed the CS from ST to LC because of huge sstables and compaction >> of them is still ongoing. >> How does rebuild command works ? Does it calculate the range then request >> the needed sstables from that node and start streaming ? How is it possible >> to speed up the streaming ? >> >> Vitali. >> >>
Re: nodetool rebuild
Both stream throughput settings are set to 0, meaning that there is no stream throttling on the C* side. Yes, i see high cpu used by STREAM-IN thread, sstables are compressed up to 80% What about copying sstables with rsync and then running repair? Probably its not that simple, vut If the data is RF3 so one node should have all the key ranges and repair will not recalculate all the hashes? Vitali On Sun, Sep 16, 2018, 02:33 dinesh.jo...@yahoo.com.INVALID wrote: > Its a long shot but do you have > stream_throughput_outbound_megabits_per_sec or > inter_dc_stream_throughput_outbound_megabits_per_sec set to a low value? > > You're right in that 3.0 streaming uses 1 thread for incoming and outgoing > connection each per peer. It not only reads the bytes off of the channel > but also deserializes the partitions on that same thread. If you see high > CPU use by STREAM-IN thread then your streaming is CPU bound. In this > situation a powerful CPU will definitely help. Dropping internode > compression and encryption will also help. Are your SSTables compressed? > > Dinesh > > > On Friday, September 14, 2018, 4:15:28 AM PDT, Vitali Dyachuk < > vdjat...@gmail.com> wrote: > > > None of these throttling are helpful for streaming if you have even a > 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network > tcp memory, window size etc does not help, the bottleneck is not the > network. > These are my findings on how streaming is limited in C* 3.0.* > > 1) Streaming of the particular range which needs to be steamed to the new > node is limited with one 1 thread and no tweaking of cpu affinity etc > helps, probably the powerfull computing VM will help > 2) Disabling compression internode_compression and disabling compression > per table in our case helps a bit > 3) When streaming has been dropped there is no resume available for the > streaming range so it will start from the beginning > > One of the options could be to create snapshots of sstables on the source > node and just copy all sstable snapshots to new node and then run repair, > data is ~5TB, RF3 ? > How is it possible at all to stream data fast to a new node/nodes ? > > Vitali. > > On Wed, Sep 12, 2018 at 5:02 PM Surbhi Gupta > wrote: > > Increase 3 throughput > Compaction throughput > Stream throughput > Interdcstream throughput (if rebuilding from another DC) > > Make all of the above to 0 and see if there is any improvement and later > set the value if u can’t leave these values to 0. > > On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk wrote: > > Hi, > I'm currently streaming data with nodetool rebuild on 2 nodes, each node > is streaming from different location. The problem is that it takes ~7 days > to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so > it should take around > ~2,5 days . Although there are resources on the destnodes and in the > source regions. > I've increased stream throughput, but its only affects outbound > connections. > Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week > i've changed the CS from ST to LC because of huge sstables and compaction > of them is still ongoing. > How does rebuild command works ? Does it calculate the range then request > the needed sstables from that node and start streaming ? How is it possible > to speed up the streaming ? > > Vitali. > >
Re: nodetool rebuild
None of these throttling are helpful for streaming if you have even a 150-200 Mbit/s bandwidth which is affordable in any cloud. Tweaking network tcp memory, window size etc does not help, the bottleneck is not the network. These are my findings on how streaming is limited in C* 3.0.* 1) Streaming of the particular range which needs to be steamed to the new node is limited with one 1 thread and no tweaking of cpu affinity etc helps, probably the powerfull computing VM will help 2) Disabling compression internode_compression and disabling compression per table in our case helps a bit 3) When streaming has been dropped there is no resume available for the streaming range so it will start from the beginning One of the options could be to create snapshots of sstables on the source node and just copy all sstable snapshots to new node and then run repair, data is ~5TB, RF3 ? How is it possible at all to stream data fast to a new node/nodes ? Vitali. On Wed, Sep 12, 2018 at 5:02 PM Surbhi Gupta wrote: > Increase 3 throughput > Compaction throughput > Stream throughput > Interdcstream throughput (if rebuilding from another DC) > > Make all of the above to 0 and see if there is any improvement and later > set the value if u can’t leave these values to 0. > > On Wed, Sep 12, 2018 at 5:42 AM Vitali Dyachuk wrote: > >> Hi, >> I'm currently streaming data with nodetool rebuild on 2 nodes, each node >> is streaming from different location. The problem is that it takes ~7 days >> to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so >> it should take around >> ~2,5 days . Although there are resources on the destnodes and in the >> source regions. >> I've increased stream throughput, but its only affects outbound >> connections. >> Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week >> i've changed the CS from ST to LC because of huge sstables and compaction >> of them is still ongoing. >> How does rebuild command works ? Does it calculate the range then request >> the needed sstables from that node and start streaming ? How is it possible >> to speed up the streaming ? >> >> Vitali. >> >
nodetool rebuild
Hi, I'm currently streaming data with nodetool rebuild on 2 nodes, each node is streaming from different location. The problem is that it takes ~7 days to stream 4Tb of data to 1 node, the speed on each side is ~150Mbit/s so it should take around ~2,5 days . Although there are resources on the destnodes and in the source regions. I've increased stream throughput, but its only affects outbound connections. Tested with iperf the bandwidth is 600Mibt/s from both sides. Last week i've changed the CS from ST to LC because of huge sstables and compaction of them is still ongoing. How does rebuild command works ? Does it calculate the range then request the needed sstables from that node and start streaming ? How is it possible to speed up the streaming ? Vitali.
Re: Large sstables
What i have done is: 1) added more disks, so the compaction will carry on 2) when i've switched to LCS from STCS the STCS queues for the processing big sstables have remained, so i've stopped these queues with nodetool stop -id queue_id and LCS compaction has started to process sstables , i'm using 3.0.17 C* with RF3 However the question remains if i use sstablesplit on 200Gb sstables to split it to 200Mb files, will it help the LCS compaction? Will LCS just take some data from that big sstable and try to merge with other sstable on L0 adn other levels so i just have to wait until the LCS compaction will finish? On Sun, Sep 2, 2018 at 9:55 AM shalom sagges wrote: > If there are a lot of droppable tombstones, you could also run User > Defined Compaction on that (and on other) SSTable(s). > > This blog post explains it well: > http://thelastpickle.com/blog/2016/10/18/user-defined-compaction.html > > On Fri, Aug 31, 2018 at 12:04 AM Mohamadreza Rostami < > mohamadrezarosta...@gmail.com> wrote: > >> Hi,Dear Vitali >> The best option for you is migrating data to the new table and change >> portion key patterns to a better distribution of data and you sstables >> become smaller but if your data already have good distribution and your >> data is really big you must add new server to your datacenter, if you >> change compassion strategy it has some risk. >> >> > On Shahrivar 8, 1397 AP, at 19:54, Jeff Jirsa wrote: >> > >> > Either of those are options, but there’s also sstablesplit to break it >> up a bit >> > >> > Switching to LCS can be a problem depending on how many sstables >> /overlaps you have >> > >> > -- >> > Jeff Jirsa >> > >> > >> >> On Aug 30, 2018, at 8:05 AM, Vitali Dyachuk >> wrote: >> >> >> >> Hi, >> >> Some of the sstables got too big 100gb and more so they are not >> compactiong any more so some of the disks are running out of space. I'm >> running C* 3.0.17, RF3 with 10 disks/jbod with STCS. >> >> What are my options? Completely delete all data on this node and >> rejoin it to the cluster, change CS to LCS then run repair? >> >> Vitali. >> >> >> > >> > - >> > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> > For additional commands, e-mail: user-h...@cassandra.apache.org >> > >> >> >> - >> To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: user-h...@cassandra.apache.org >> >>
Large sstables
Hi, Some of the sstables got too big 100gb and more so they are not compactiong any more so some of the disks are running out of space. I'm running C* 3.0.17, RF3 with 10 disks/jbod with STCS. What are my options? Completely delete all data on this node and rejoin it to the cluster, change CS to LCS then run repair? Vitali.
Re: Reading cardinality from Statistics.db failed
Dinesh, here is the ticket https://issues.apache.org/jira/browse/CASSANDRA-14647 On Thu, Aug 16, 2018 at 10:13 AM dinesh.jo...@yahoo.com.INVALID wrote: > Vitali, > > It doesn't look like there is an existing Jira. It would be helpful if you > could create one with as much information as possible. Can you reduce this > issue to a short, repeatable set of steps that we can reproduce? That'll be > helpful to debug this problem. > > Dinesh > On Wednesday, August 15, 2018, 1:07:21 AM PDT, Vitali Dyachuk < > vdjat...@gmail.com> wrote: > > > I've upgraded to 3.0.17 and the issue is still there, Is there a jira > ticket for that bug or should i create one? > > On Wed, Jul 25, 2018 at 2:57 PM Vitali Dyachuk wrote: > > I'm using 3.0.15. I see that there is some fix for sstable metadata in > 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a > fix for "reading cardinalyti from statistics.db" ? > > > On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger wrote: > > What version of Cassandra are you running? There is a bug in 3.10.0 and > certain 3.0.x that occurs in certain conditions and corrupts that file. > > Hannu > > Vitali Dyachuk kirjoitti 25.7.2018 kello 10.48: > > Hi, > I have noticed in the cassandra system.log that there is some issue with > sstable metadata, the messages says: > WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading > cardinality from Statistics.db failed for > /opt/data/disk5/data/keyspace/table/mc-big-Data.db > Although there is no such file. The message has appeared after i've > changed the compaction strategy from SizeTiered to Leveled. > Currently i'm running nodetool scrub to rebuilt the sstable, and it takes > a lot of time to scrub all sstables. > Reading the code it is said that if this metada is broken, then estimating > the keys will be done using index summary. How expensive it is ? > > https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 > > The main question is why has this happened? > > Thanks, > Vitali Djatsuk. > >
Re: Reading cardinality from Statistics.db failed
I've upgraded to 3.0.17 and the issue is still there, Is there a jira ticket for that bug or should i create one? On Wed, Jul 25, 2018 at 2:57 PM Vitali Dyachuk wrote: > I'm using 3.0.15. I see that there is some fix for sstable metadata in > 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a > fix for "reading cardinalyti from statistics.db" ? > > > On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger wrote: > >> What version of Cassandra are you running? There is a bug in 3.10.0 and >> certain 3.0.x that occurs in certain conditions and corrupts that file. >> >> Hannu >> >> Vitali Dyachuk kirjoitti 25.7.2018 kello 10.48: >> >> Hi, >> I have noticed in the cassandra system.log that there is some issue with >> sstable metadata, the messages says: >> WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading >> cardinality from Statistics.db failed for >> /opt/data/disk5/data/keyspace/table/mc-big-Data.db >> Although there is no such file. The message has appeared after i've >> changed the compaction strategy from SizeTiered to Leveled. >> Currently i'm running nodetool scrub to rebuilt the sstable, and it takes >> a lot of time to scrub all sstables. >> Reading the code it is said that if this metada is broken, then >> estimating the keys will be done using index summary. How expensive it is ? >> >> https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 >> >> The main question is why has this happened? >> >> Thanks, >> Vitali Djatsuk. >> >>
Adding new datacenter to the cluster
Hello, I'm going to follow this documentation to add a new datacenter to the C* cluster https://docs.datastax.com/en/cassandra/3.0/cassandra/operations/opsAddDCToCluster.html The main step is to run nodetool rebuild which will sync data to the new datacenter, this will load cluster badly since the main keyspace size is 2TB. 1) What are the best practicies to add a new datacenter with a lot of data? 2) How is it possible to stop rebuild? 3) What are the throttling possibilities I'm using C* 3.0.17 with RF3 Vitali.
Re: Reading cardinality from Statistics.db failed
I'm using 3.0.15. I see that there is some fix for sstable metadata in 3.0.16 https://issues.apache.org/jira/browse/CASSANDRA-14217 - is that a fix for "reading cardinalyti from statistics.db" ? On Wed, Jul 25, 2018 at 1:02 PM Hannu Kröger wrote: > What version of Cassandra are you running? There is a bug in 3.10.0 and > certain 3.0.x that occurs in certain conditions and corrupts that file. > > Hannu > > Vitali Dyachuk kirjoitti 25.7.2018 kello 10.48: > > Hi, > I have noticed in the cassandra system.log that there is some issue with > sstable metadata, the messages says: > WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading > cardinality from Statistics.db failed for > /opt/data/disk5/data/keyspace/table/mc-big-Data.db > Although there is no such file. The message has appeared after i've > changed the compaction strategy from SizeTiered to Leveled. > Currently i'm running nodetool scrub to rebuilt the sstable, and it takes > a lot of time to scrub all sstables. > Reading the code it is said that if this metada is broken, then estimating > the keys will be done using index summary. How expensive it is ? > > https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 > > The main question is why has this happened? > > Thanks, > Vitali Djatsuk. > >
Reading cardinality from Statistics.db failed
Hi, I have noticed in the cassandra system.log that there is some issue with sstable metadata, the messages says: WARN [Thread-6] 2018-07-25 07:12:47,928 SSTableReader.java:249 - Reading cardinality from Statistics.db failed for /opt/data/disk5/data/keyspace/table/mc-big-Data.db Although there is no such file. The message has appeared after i've changed the compaction strategy from SizeTiered to Leveled. Currently i'm running nodetool scrub to rebuilt the sstable, and it takes a lot of time to scrub all sstables. Reading the code it is said that if this metada is broken, then estimating the keys will be done using index summary. How expensive it is ? https://github.com/apache/cassandra/blob/cassandra-3.0.15/src/java/org/apache/cassandra/io/sstable/format/SSTableReader.java#L245 The main question is why has this happened? Thanks, Vitali Djatsuk.
Re: Cassandra 2FA
Thanks, checked the ticket which is about a client hostname verification, but this is not an optimal solution for us; maintaining the allowed hosts list is not convenient way, once new hosts added you have reissue a new cert.and deploy it. What we are looking for is for example certificate validation based on CN, which adds additional small level of security. I'm also thinking to try OID "challengePassword" as a pre-shared key, but thats not related to C*. On Tue, Jul 10, 2018 at 10:43 AM Stefan Podkowinski wrote: > You may want to keep an eye on the following ticket: > https://issues.apache.org/jira/browse/CASSANDRA-13404 > > > On 09.07.2018 17:12, Vitali Dyachuk wrote: > > Hi, > > There is a certificate validation based on the mutual CA this is a 1st > > factor, the 2nd factor could be checking the common name of the client > > certificate, probably this requires writing a patch, but probably some > > has already done that ? > > > > Vitali Djatsuk. > > - > To unsubscribe, e-mail: user-unsubscr...@cassandra.apache.org > For additional commands, e-mail: user-h...@cassandra.apache.org > >
Cassandra 2FA
Hi, There is a certificate validation based on the mutual CA this is a 1st factor, the 2nd factor could be checking the common name of the client certificate, probably this requires writing a patch, but probably some has already done that ? Vitali Djatsuk.
