CQLSH definitely won't work for that amount of data, so you need to use other tools.
But before selecting them, you need to define requirements. For example: 1. Are you copying the data into tables with exactly the same structure? 2. Do you need to preserve metadata, like, writetime & TTL? Depending on that, you may have following choices: - use sstableloader - it will preserve all metadata, like, ttl and writetime. You just need to copy SSTable files, or stream directly from the source cluster. But this will require copying of data into tables with exactly same structure (and in case of UDTs, the keyspace names should be the same) - use DSBulk - it's a very effective tool for unloading & loading data from/to Cassandra/DSE. Use zstd compression for offloaded data to save disk space (see blog links below for more details). But the preserving metadata could be a problem. - use Spark + Spark Cassandra Connector. But also, preserving the metadata is not an easy task, and requires programming to handle all edge cases (see https://datastax-oss.atlassian.net/browse/SPARKC-596 for details) blog series on DSBulk: - https://www.datastax.com/blog/2019/03/datastax-bulk-loader-introduction-and-loading - https://www.datastax.com/blog/2019/04/datastax-bulk-loader-more-loading - https://www.datastax.com/blog/2019/04/datastax-bulk-loader-common-settings - https://www.datastax.com/blog/2019/06/datastax-bulk-loader-unloading - https://www.datastax.com/blog/2019/07/datastax-bulk-loader-counting - https://www.datastax.com/blog/2019/12/datastax-bulk-loader-examples-loading-other-locations On Tue, Jul 14, 2020 at 1:47 AM Jai Bheemsen Rao Dhanwada < jaibheem...@gmail.com> wrote: > Hello, > > I would like to copy some data from one cassandra cluster to another > cassandra cluster using the CQLSH copy command. Is this the good approach > if the dataset size on the source cluster is very high(500G - 1TB)? If not > what is the safe approach? and are there any limitations/known issues to > keep in mind before attempting this? > -- With best wishes, Alex Ott http://alexott.net/ Twitter: alexott_en (English), alexott (Russian)
