hey, Did you get a chance to review my queries please. On Thu, Feb 3, 2022 at 4:40 PM Surinder Mehra <redni...@gmail.com> wrote:
> Hi, > So the way I am thinking to use it is if we lose the EBS volume and we > need to restore the cluster state back. I would have a secondary EBS as my > snapshot directory so I can restore from it. > It means Application would need to be restarted after EBS data is copied > back to the work directory. I see two options here > 1. Manual as described in previous reply. manually copy data from snapshot > directory to work/db and restart cluster > 2. Use control script : I am not clear on how will this work because If I > restart cluster, it is going to create directory structure again and then > when we run restore command, it does not copy data > > Could you please suggest how it would work. directory structure is > attached. > Also, can you suggest a better way to copy snapshot directory data to S3 . > I am thinking of using a kubernetes CSI driver to do it. Any objections to > it > > > > On Thu, Feb 3, 2022 at 4:23 PM Maxim Muzafarov <mmu...@apache.org> wrote: > >> Hello, >> >> You don't need to stop the cluster or delete/move any snapshot files >> in case you are using the restore procedure from the control.sh, so >> the following should work: >> - create snapshot >> - stop the caches you are intended to restore >> - run ./control.sh --snapshot restore snapshot_1 --start >> >> Can you provide the directory structure of the Ignite working >> directory? (use `tree` command) >> >> On Wed, 2 Feb 2022 at 22:15, Surinder Mehra <redni...@gmail.com> wrote: >> > >> > Hi, >> > Could you please point out if i missed something? >> > >> > On Wed, Feb 2, 2022, 13:39 Surinder Mehra <redni...@gmail.com> wrote: >> >> >> >> Hey thanks for your suggestions. >> >> >> >> I tried restoring using control.sh but it doesn't seem to work. Below >> are steps >> >> 1. Started 3 nodes and added data using a thick client >> >> 2. created a snapshot using with ./control.sh --snapshot create >> snapshot_1 >> >> 3. I verified, the snapshot directory has data >> >> 4. Stopped the cluster and cleared binary_data, marshaler and nodes >> directory /db >> >> 5. Started the cluster again, all 3 nodes >> >> 6. Activate the cluster using ./control.sh --set-state ACTIVE >> >> 7. Run restore command : ./control.sh --snapshot restore snapshot_1 >> --start >> >> 8. Command was successful but data is not copied to cluster nodes. >> >> >> >> Please note that when I restarted the cluster, it created binary_data, >> marshaler and nodes directories by default. >> >> >> >> Did I miss anything ? >> >> >> >> >> >> On Tue, Feb 1, 2022 at 8:21 PM Maxim Muzafarov <mmu...@apache.org> >> wrote: >> >>> >> >>> Hello, >> >>> >> >>> Your case looks correct to me, however, I'd like to mention some >> >>> important points that may help you: >> >>> - the directories structure of the snapshot has the same structure as >> >>> the Ignite native persistence, so you may backup the original cluster >> >>> node directory (for binary_data, marshaller and db) and move all the >> >>> files right from the snapshot. >> >>> - do not forget to backup and clear the original wal directory in case >> >>> of restoration. >> >>> - you may use control.sh --snapshot restore command to restore from a >> >>> snapshot (this was added in 2.11) >> >>> >> >>> [1] https://issues.apache.org/jira/browse/IGNITE-13805 >> >>> >> >>> On Tue, 1 Feb 2022 at 16:28, Surinder Mehra <redni...@gmail.com> >> wrote: >> >>> > >> >>> > Hi, >> >>> > After a few hiccups, I managed to restore the cluster state from >> the snapshot. Please confirm if they look correct. If so documentation page >> needs to be updated >> >>> > >> >>> > Create N nodes >> >>> > Add some data to them >> >>> > Create snapshot >> >>> > Stop all nodes(cluster) >> >>> > Delete binary_data, marsheller and sub directories of /work/db >> >>> > Copy snapshots/snapshotname/db/binary_data to /work/db/, >> >>> > Copy snapshots/snapshotname/db/marshaller to /work/db/ >> >>> > Copy snapshots/snapshotname/db/{nodeid} dir to /work/db/ >> >>> > Start cluster >> >>> > Cluster should auto activate after all nodes join it >> >>> > Cluster is ready >> >>> > >> >>> > >> >>> > On Mon, Jan 31, 2022 at 7:14 PM Surinder Mehra <redni...@gmail.com> >> wrote: >> >>> >> >> >>> >> Hi, >> >>> >> We are using ignite 2.11.1 to experiment with ignite snapshots. We >> tried steps mentioned on below page to restore ignite data from snapshot >> >>> >> https://ignite.apache.org/docs/latest/snapshots/snapshots >> >>> >> >> >>> >> But we get the below error when we start a cluster after copying >> data manually as mentioned on the page. >> >>> >> >> >>> >> Steps: >> >>> >> 1.Created 3 nodes and added 3 records >> >>> >> >> >>> >> 2.Created snapshot. >> >>> >> 3. Stopped the cluster and removed files from binary_data and >> marshellar, not the directories. they are present but empty >> >>> >> 4. removed nodeId directories and files under them from /work/db/ >> >>> >> >> >>> >> 5. Copied node id directories from snapshot directory to >> /work/db/. I guess the below step meant to say $IGNITE_HOME/work/db/ right ? >> >>> >> >> >>> >> Copy the files belonging to a node with the {node_id} from the >> snapshot into the $IGNITE_HOME/work/ directory. If the db/{node_id} >> directory is not located under the Ignite work dir then you need to copy >> data files there. >> >>> >> >> >>> >> Error : do we need to copy binary_data and marshaler files as well >> or something else missing ? >> >>> >> >> >>> >> Caused by: class org.apache.ignite.IgniteCheckedException: Cannot >> find metadata for object with compact footer (Ignite work directory might >> have been cleared after restart. Make sure that IGNITE_HOME does not point >> to a temp folder or any other folder that is destroyed/cleared on restarts) >> [typeId=-88020438, IGNITE_HOME='null'] >> >>> >> >> >>> >> Please note that ignite HOEM/work/db directory has all nodes data >> copied from snapshot, it is not cleared as indicated by error above >> >>> >> >> >>> >> >> >
