[Zope-dev] Help! Need to extract files out of Data.fs!
Hiya, My company has been using a Zope server to store some reference material, but it has grown beyond belief, and I need to extract all the collected files (about 1.3 terabytes!) from the Data.fs into distinct files onto a different server. Unfortunately, I've run into some problems. Firstly, the FTP transfer process seems to be extremely slow, and if I try and transfer too large or deep a folder structure in one go then things grind to a halt. To beat this I've been carefully recreating the folder structure manually, and copying things in easier-to-digest chunks. Unfortunately, because I was a bit rushed over the weekend I have accidentally deleted about 130,000 files! I've tried going through the Undo procedure, but the idea of manually ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread. So ... Please please /please/ does anyone happen to have the file format for the Data.fs file so that I could write a C program to extract and recreate the files and folders to another server? Or has anyone written such a program already? If so, I'll happily have your children if you can help me out! As it stands, I'm going to have to break it to the librarian (the poor man who has been collating all the reference material for the last six months) that I've lost half of his work, and blaming it on the poor FTP client in Windows seems a bit lame! If it helps: I'm running Zope 2.9.2 on a SuSE Linux box. I have full access to the Linux box ... and a large hammer, if I need it! :( Help! Many thanks in advance, Kris (considering giving up this computing job and going to live in a cave somewhere). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help! Need to extract files out of Data.fs!
Kris, I assume you have a backup of the Data.fs. If you do, nothing is really lost. Have you compressed the Data.fs? There is an option in the control panel to do this. I can be run on a live machine, but you may not want to do so given the size of stuff. I'd do it on a quiet machine. This will remove any cruft you have in the Data.fs and will improve your speed of access somewhat, Things are probably slow because of swapping. Reducing the size of Data.fs may speed things up. Take a look at the documentation for the ZODB (Google is your friend). You could write a program in Python to simply open the ZODB and extract the files you need. It might save a bit of overhead. Remember 1.3TB is a large file and is going to take some time to process no matter what. On Mon, 28 Aug 2006, Kris Adcock wrote: Hiya, My company has been using a Zope server to store some reference material, but it has grown beyond belief, and I need to extract all the collected files (about 1.3 terabytes!) from the Data.fs into distinct files onto a different server. Unfortunately, I've run into some problems. Firstly, the FTP transfer process seems to be extremely slow, and if I try and transfer too large or deep a folder structure in one go then things grind to a halt. To beat this I've been carefully recreating the folder structure manually, and copying things in easier-to-digest chunks. Unfortunately, because I was a bit rushed over the weekend I have accidentally deleted about 130,000 files! I've tried going through the Undo procedure, but the idea of manually ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread. So ... Please please /please/ does anyone happen to have the file format for the Data.fs file so that I could write a C program to extract and recreate the files and folders to another server? Or has anyone written such a program already? If so, I'll happily have your children if you can help me out! As it stands, I'm going to have to break it to the librarian (the poor man who has been collating all the reference material for the last six months) that I've lost half of his work, and blaming it on the poor FTP client in Windows seems a bit lame! If it helps: I'm running Zope 2.9.2 on a SuSE Linux box. I have full access to the Linux box ... and a large hammer, if I need it! :( Help! Many thanks in advance, Kris (considering giving up this computing job and going to live in a cave somewhere). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help! Need to extract files out of Data.fs!
One more thing, what does the disk where the Data.fs resides look like? On Mon, 28 Aug 2006, Dennis Allison wrote: Kris, I assume you have a backup of the Data.fs. If you do, nothing is really lost. Have you compressed the Data.fs? There is an option in the control panel to do this. I can be run on a live machine, but you may not want to do so given the size of stuff. I'd do it on a quiet machine. This will remove any cruft you have in the Data.fs and will improve your speed of access somewhat, Things are probably slow because of swapping. Reducing the size of Data.fs may speed things up. Take a look at the documentation for the ZODB (Google is your friend). You could write a program in Python to simply open the ZODB and extract the files you need. It might save a bit of overhead. Remember 1.3TB is a large file and is going to take some time to process no matter what. On Mon, 28 Aug 2006, Kris Adcock wrote: Hiya, My company has been using a Zope server to store some reference material, but it has grown beyond belief, and I need to extract all the collected files (about 1.3 terabytes!) from the Data.fs into distinct files onto a different server. Unfortunately, I've run into some problems. Firstly, the FTP transfer process seems to be extremely slow, and if I try and transfer too large or deep a folder structure in one go then things grind to a halt. To beat this I've been carefully recreating the folder structure manually, and copying things in easier-to-digest chunks. Unfortunately, because I was a bit rushed over the weekend I have accidentally deleted about 130,000 files! I've tried going through the Undo procedure, but the idea of manually ticking 130,000 boxes (in pages of 20) and undoing them fills me with dread. So ... Please please /please/ does anyone happen to have the file format for the Data.fs file so that I could write a C program to extract and recreate the files and folders to another server? Or has anyone written such a program already? If so, I'll happily have your children if you can help me out! As it stands, I'm going to have to break it to the librarian (the poor man who has been collating all the reference material for the last six months) that I've lost half of his work, and blaming it on the poor FTP client in Windows seems a bit lame! If it helps: I'm running Zope 2.9.2 on a SuSE Linux box. I have full access to the Linux box ... and a large hammer, if I need it! :( Help! Many thanks in advance, Kris (considering giving up this computing job and going to live in a cave somewhere). ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope ) -- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help! Need to extract files out of Data.fs!
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 28 Aug 2006, at 22:33, Dennis Allison wrote: Have you compressed the Data.fs? There is an option in the control panel to do this. I can be run on a live machine, but you may not want to do so given the size of stuff. I'd do it on a quiet machine. This will remove any cruft you have in the Data.fs and will improve your speed of access somewhat, Things are probably slow because of swapping. Reducing the size of Data.fs may speed things up. Umh, packing the database is something you expressly *do not want to do* in this situation, because that will immediately destroy the Undo history which currently allows him access to the deleted items. jens -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.1 (Darwin) iD8DBQFE81pMRAx5nvEhZLIRAt3KAJ4lzO25kjckhX3+WKeUqqGOXuyDWQCdF1Ya tWJpfMlJCIdP8EfdlXr8Nio= =mJU8 -END PGP SIGNATURE- ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Help! Need to extract files out of Data.fs!
--- Dennis Allison [EMAIL PROTECTED] wrote: Kris, I assume you have a backup of the Data.fs. If you do, nothing is really lost. Have you compressed the Data.fs? There is an option in the control panel Wouldn't this remove all the deleted files? Or is compression different to packing? Cheers, Kris. ___ Zope-Dev maillist - Zope-Dev@zope.org http://mail.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://mail.zope.org/mailman/listinfo/zope-announce http://mail.zope.org/mailman/listinfo/zope )
