Re: [Zope-dev] Fw: 2gb file size
Itamar Shtull-Trauring wrote: Shane Hathaway wrote: Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. Yes please! Perhaps it could be integrated into the ZODB codebase while you're at it? I'm working on cleaning it up so that it integrates cleanly with FileStorage. Then I'll release it. BTW your roadmap document is fantastic. I hope you're getting lots of good feedback. It really brings everything together. Shane ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote: Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. Yes please! Perhaps it could be integrated into the ZODB codebase while you're at it? ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
RE: [Zope-dev] Fw: 2gb file size
Shane This would be very helpfull to share. One of the reasons I tried the Berkeley Database was because I kept blowing this 2Gb limit. Prehaps if this was released I could finally finish the zope mail archives. :-) Cheers -Andy -Original Message- From: [EMAIL PROTECTED] [mailto:[EMAIL PROTECTED]]On Behalf Of Shane Hathaway Sent: 06 November 2000 21:51 To: Andy McKay; [EMAIL PROTECTED] Subject: Re: [Zope-dev] Fw: 2gb file size Andy McKay wrote: I did do a How-to as well http://www.zope.org/Members/andym/2gig Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. I tried it with FileStorage and even integrated it in. That's how we were able to find and fix some arithmetic errors that occurred in FileStorage when reaching the 2GB barrier. If there's interest in the community I can release the code (in fact, there is no dependency on Zope and it's pretty fast.) OTOH, I can verify there are production sites that have succesfully reached around 10 GB in Data.fs size. Shane -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue. Thanks. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Itamar Shtull-Trauring" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size Shane Hathaway wrote: Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. So on W2K, Zope can have 2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. [EMAIL PROTECTED] Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue. Thanks. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Itamar Shtull-Trauring" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size Shane Hathaway wrote: Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. So on W2K, Zope can have 2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. [EMAIL PROTECTED] Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
I did do a How-to as well http://www.zope.org/Members/andym/2gig -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue. Thanks. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Itamar Shtull-Trauring" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size Shane Hathaway wrote: Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. So on W2K, Zope can have 2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. [EMAIL PROTECTED] Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Andy McKay wrote: I did do a How-to as well http://www.zope.org/Members/andym/2gig Not long ago I created a wrapper around File objects which automatically splits the file before it reached a predefined limit. It could handle any number of partitioned file segements. I tried it with FileStorage and even integrated it in. That's how we were able to find and fix some arithmetic errors that occurred in FileStorage when reaching the 2GB barrier. If there's interest in the community I can release the code (in fact, there is no dependency on Zope and it's pretty fast.) OTOH, I can verify there are production sites that have succesfully reached around 10 GB in Data.fs size. Shane -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Monday, November 06, 2000 9:26 AM Subject: Re: [Zope-dev] Fw: 2gb file size Thank you everyone for your help. I truncated the file and restarted Zope successfully. I had a brief period of problems when I forgot use binary mode for truncating the file - duh. Anyway, I am running 2.2.1 and it doesnt work with a 2gb file. I didnt analyse where the problem lies, however I do believe its a python issue. Thanks. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Itamar Shtull-Trauring" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Sunday, November 05, 2000 2:37 AM Subject: Re: [Zope-dev] Fw: 2gb file size Shane Hathaway wrote: Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. So on W2K, Zope can have 2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. [EMAIL PROTECTED] Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Shane Hathaway wrote: Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. So on W2K, Zope can have 2GB databases? I'd really like to hear if you get it to work Andy (or if anyone else did.) -- Itamar S.T. [EMAIL PROTECTED] Fingerprint = D365 7BE8 B81E 2B18 6534 025E D0E7 92DB E441 411C ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Andy, Are you running Zope 2.2.1 or later? There were pointer arithmetic errors that occurred on earlier versions, even with operating systems that support 2 GB files. Shane Chris McDonough wrote: I didn't think this was a problem on Windows? I wonder if Python needs to be compiled in a special way to support large files on Windows. Have you tried Tranalyzer to see if will tell you where the transaction breaks are in the file? There's a howto by Itamar on Zope.org about how to do this. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: "Andy McKay" [EMAIL PROTECTED]; [EMAIL PROTECTED] Sent: Friday, November 03, 2000 6:39 PM Subject: Re: [Zope-dev] Fw: 2gb file size Ive tried truncating the file and then running fsrecover.py, but get: File "ZODB\FileStorage.py", line 1530, in read_index if read(4) != packed_version: raise FileStorageFormatError, name FileStorage.FileStorageFormatError: q:\var\Data.fs As a desperation I commented out line 1530 and reran. It runs through the entire file and reduces the data.fs to 1k. Am I totally hooped? I thought ZODB could handle being truncated, ive never had an problems reading from a data.fs even when my box has blown up. Anyway looks like Im talking to myself here... -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 03, 2000 1:28 PM Subject: [Zope-dev] Fw: 2gb file size Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so? This is kinda urgent... :( Thanks. -- Andy McKay, Developer. ActiveState. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
Re: [Zope-dev] Fw: 2gb file size
Ive tried truncating the file and then running fsrecover.py, but get: File "ZODB\FileStorage.py", line 1530, in read_index if read(4) != packed_version: raise FileStorageFormatError, name FileStorage.FileStorageFormatError: q:\var\Data.fs As a desperation I commented out line 1530 and reran. It runs through the entire file and reduces the data.fs to 1k. Am I totally hooped? I thought ZODB could handle being truncated, ive never had an problems reading from a data.fs even when my box has blown up. Anyway looks like Im talking to myself here... -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 03, 2000 1:28 PM Subject: [Zope-dev] Fw: 2gb file size Hmmm ive looked at fsrecover.py and its not happy either. Anyone else solved this problem? Im on Win2k BTW. -- Andy McKay, Developer. ActiveState. - Original Message - From: "Andy McKay" [EMAIL PROTECTED] To: [EMAIL PROTECTED] Sent: Friday, November 03, 2000 1:04 PM Subject: 2gb file size My database has suddenly ballooned to 2 gig's stopping my Zope from running. Are there any utils to delete the last transaction or so? This is kinda urgent... :( Thanks. -- Andy McKay, Developer. ActiveState. ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope ) ___ Zope-Dev maillist - [EMAIL PROTECTED] http://lists.zope.org/mailman/listinfo/zope-dev ** No cross posts or HTML encoding! ** (Related lists - http://lists.zope.org/mailman/listinfo/zope-announce http://lists.zope.org/mailman/listinfo/zope )
