A lot of things depend on actual cluster config - compaction settings (LCS vs STCS vs TWCS) and token allocation (single token, vnodes, etc) matter a ton.
With 4.0 and LCS, streaming for replacement is MUCH faster, so much so that most people should be fine with 4-8TB/node, because the rebuild time is decreased by an order of magnitude. If you happen to have large physical machines, running multiple instances on a machine (each with a single token, and making sure you match rack awareness) sorta approximates vnodes without some of the unpleasant side effects. If you happen to run on more-reliable-storage (like EBS, or a SAN, and you understand what that means from a business continuity perspective), then you can assume that your rebuild frequency is probably an order of magnitude less often, so you can adjust your risk calculation based on measured reliability there (again, EBS and other disaggregated disks still fail, just less often than single physical flash devices). Seed nodes never really need to change significantly. You should be fine with 2-3 per DC no matter the instance count. On Wed, Aug 16, 2023 at 8:34 AM Joe Obernberger < joseph.obernber...@gmail.com> wrote: > General question on how to configure Cassandra. Say I have 1PByte of > data to store. The general rule of thumb is that each node (or at least > instance of Cassandra) shouldn't handle more than 2TBytes of disk. That > means 500 instances of Cassandra. > > Assuming you have very fast persistent storage (such as a NetApp, > PorterWorx etc.), would using Kubernetes or some orchestration layer to > handle those nodes be a viable approach? Perhaps the worker nodes would > have enough RAM to run 4 instances (pods) of Cassandra, you would need > 125 servers. > Another approach is to build your servers with 5 (or more) SSD devices - > one for OS, four for each instance of Cassandra running on that server. > Then build some scripts/ansible/puppet that would manage Cassandra > start/stops, and other maintenance items. > > Where I think this runs into problems is with repairs, or sstablescrubs > that can take days to run on a single instance. How is that handled 'in > the real world'? With seed nodes, how many would you have in such a > configuration? > Thanks for any thoughts! > > -Joe > > > -- > This email has been checked for viruses by AVG antivirus software. > www.avg.com >
