Fabulous tip. Thanks, Sean. I will definitely check out dsbulk. Great to see it's a Cassandra-general tool and not just limited to DataStax Enterprise.
On Fri, Jul 24, 2020 at 12:58 PM Durity, Sean R <sean_r_dur...@homedepot.com> wrote: > I would use dsbulk to unload and load. Then the schemas don’t really > matter. You define which fields in the resulting file are loaded into which > columns. You also won’t have the limitations and slowness of COPY TO/FROM. > > > > > > Sean Durity > > > > *From:* Mitch Gitman <mgit...@gmail.com> > *Sent:* Friday, July 24, 2020 2:22 PM > *To:* user@cassandra.apache.org > *Subject:* [EXTERNAL] Re: Restore a table with dropped columns to a new > cluster fails > > > > I'm reviving this thread because I'm looking for a non-hacky way to > migrate data from one cluster to another using nodetool snapshot and > sstableloader without having to preserve dropped columns in the new schema. > In my view, that's just cruft and confusion that keeps building. > > The best idea I can come up with is to do the following in the source > cluster: > > 1. Use the cqlsh COPY FROM command to export the data in the table. > 2. Drop the table. > 3. Re-create the table. > 4. Use the cqlsh COPY TO command to import the data into the new > incarnation of the table. > > > This approach is predicated on two assumptions: > > - The re-created table has no knowledge of the history of the old > table by the same name. > - The amount of data in the table doesn't exceed what the COPY command > can handle. > > > If the dropped columns exist in the table in an environment where there's > a lot of data, then we'd have to use some other mechanism to capture and > reload the data. > > If you see something wrong about this approach or you have a better way to > do it, I'd be glad to hear from you. > > > > On Tue, Feb 19, 2019 at 11:31 AM Jeff Jirsa <jji...@gmail.com> wrote: > > You can also manually add the dropped column to the appropriate table to > eliminate the issue. Has to be done by a human, a new cluster would have no > way of learning about a dropped column, and the missing metadata cannot be > inferred. > > > > > > On Tue, Feb 19, 2019 at 10:58 AM Elliott Sims <elli...@backblaze.com> > wrote: > > When a snapshot is taken, it includes a "schema.cql" file. That should be > sufficient to restore whatever you need to restore. I'd argue that neither > automatically resurrecting a dropped table nor silently failing to restore > it is a good behavior, so it's not unreasonable to have the user re-create > the table then choose if they want to re-drop it. > > > > On Tue, Feb 19, 2019 at 7:28 AM Hannu Kröger <hkro...@gmail.com> wrote: > > Hi, > > > > I would like to bring this issue to your attention. > > > > Link to the ticket: > > https://issues.apache.org/jira/browse/CASSANDRA-14336 [issues.apache.org] > <https://urldefense.com/v3/__https:/issues.apache.org/jira/browse/CASSANDRA-14336__;!!M-nmYVHPHQ!eJ1PiiThRyq9y1v7PnYgHnaxFUJ6Lloy4Zs_wSgCcg_DSsLcbHgZxGqhKQ0vCapZPSmg3JY$> > > > > Basically if a table contains dropped columns and you try to restore a > snapshot to a new cluster, that will fail because of an error like > "java.lang.RuntimeException: Unknown column XXX during deserialization”. > > > > I feel this is quite serious problem for backup and restore functionality > of Cassandra. You cannot restore a backup to a new cluster if columns have > been dropped. > > > > There have been other similar tickets that have been apparently closed but > based on my test with 3.11.4, the issue still persists. > > > > Best Regards, > > Hannu Kröger > > > ------------------------------ > > The information in this Internet Email is confidential and may be legally > privileged. It is intended solely for the addressee. Access to this Email > by anyone else is unauthorized. If you are not the intended recipient, any > disclosure, copying, distribution or any action taken or omitted to be > taken in reliance on it, is prohibited and may be unlawful. When addressed > to our clients any opinions or advice contained in this Email are subject > to the terms and conditions expressed in any applicable governing The Home > Depot terms of business or client engagement letter. The Home Depot > disclaims all responsibility and liability for the accuracy and content of > this attachment and for any damages or losses arising from any > inaccuracies, errors, viruses, e.g., worms, trojan horses, etc., or other > items of a destructive nature, which may be contained in this attachment > and shall not be liable for direct, indirect, consequential or special > damages in connection with this e-mail message or its attachment. >
