Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Tres Seaver
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

Christopher N. Deckard wrote:
> On 1/7/10 3:10 PM, Andrew Milton wrote:
> 
>> Set the TMPDIR env var to point to where you want tmp files created in
>> the shell script that starts your ZEO server
> 
> 
> We got our tmpdir issues worked out, but then got this error:
> 
> 2010-01-07 15:40:34 CRITICAL txn.47988082355376 A storage error occurred
> during the second phase of the two-phase commit.  Resources may be in an
> inconsistent state.
> Traceback (most recent call last):
>   File "./migrate_site.py", line 45, in ?
> transaction.get().commit()
>   File
> "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
> line 325, in commit
> self._commitResources()
>   File
> "/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
> line 432, in _commitResources
> rm.tpc_finish(self)
>   File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
> line 758, in tpc_finish
> self._storage.tpc_finish(transaction, callback)
>   File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py",
> line 1079, in tpc_finish
> raise ClientDisconnected(
> ZEO.Exceptions.ClientDisconnected: Calling tpc_finish() on a
> disconnected transaction

Try restarting the appservers after the restart of the ZEO server.


Tres.
- --
===
Tres Seaver  +1 540-429-0999  tsea...@palladion.com
Palladion Software   "Excellence by Design"http://palladion.com
-BEGIN PGP SIGNATURE-
Version: GnuPG v1.4.9 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org

iEYEARECAAYFAktGS3IACgkQ+gerLs4ltQ5vyQCdHUGdM13eHGh0YFibqovvxuPu
rIsAmgL02kC/XczD59oGhyY8pDhicByN
=+owh
-END PGP SIGNATURE-

___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Christopher N. Deckard
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 1/7/10 3:10 PM, Andrew Milton wrote:

> Set the TMPDIR env var to point to where you want tmp files created in
> the shell script that starts your ZEO server
> 

We got our tmpdir issues worked out, but then got this error:

2010-01-07 15:40:34 CRITICAL txn.47988082355376 A storage error occurred
during the second phase of the two-phase commit.  Resources may be in an
inconsistent state.
Traceback (most recent call last):
  File "./migrate_site.py", line 45, in ?
transaction.get().commit()
  File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 325, in commit
self._commitResources()
  File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 432, in _commitResources
rm.tpc_finish(self)
  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 758, in tpc_finish
self._storage.tpc_finish(transaction, callback)
  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py",
line 1079, in tpc_finish
raise ClientDisconnected(
ZEO.Exceptions.ClientDisconnected: Calling tpc_finish() on a
disconnected transaction

- -Chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktGSBwACgkQ+zGkEDwn8UY/TwCggnFf2KLjId+J/wI4LIESuuM+
b1wAoKz8p/JpbA/WOQHZQ3fgMyglmpaE
=zKI2
-END PGP SIGNATURE-
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Andrew Milton
+---[ Christopher N. Deckard ]--
| -BEGIN PGP SIGNED MESSAGE-
| Hash: SHA1
| 
| On 1/7/10 2:39 PM, Andrew Milton wrote:
| > +---[ Christopher N. Deckard ]--
| > | -BEGIN PGP SIGNED MESSAGE-
| > | Hash: SHA1
| > | 
| > | On 1/7/10 2:30 PM, Andrew Milton wrote:
| > | > +---[ Brian Brinegar ]--
| > | > | I'm fairly well versed on the inner workings of Zope, but yesterday 
I've
| > | > | run into a problem that I can grasp.
| > | > 
| > | > [snip]
| > | > 
| > | > | Anyway, here's what I see. The copy starts, a tempfile is created (on
| > | > | the client) and grows to the appropriate size (3.1GB). Then, a
| > | > | Data.fs.temp appears on the new server and grows to the appropriate
| > | > | size. However, when my script tries to commit the transaction I get an
| > | > | EOF error:
| > | > 
| > | > File size limits on the new server restricting to say 2GB?
| > | 
| > | No.  It's an x86-64 RedHat 5 install.  We've had 250GB files on this 
system.
| > 
| > Got any errors in your ZEO log ?
| 
| Aha!  Our zeo log shows that we are running out of space in /tmp.  What
| is the best way to redefine the location for temp files for the ZEO server?

Set the TMPDIR env var to point to where you want tmp files created in
the shell script that starts your ZEO server

-- 
Andrew Milton
a...@theinternet.com.au
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Christopher N. Deckard
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 1/7/10 2:39 PM, Andrew Milton wrote:
> +---[ Christopher N. Deckard ]--
> | -BEGIN PGP SIGNED MESSAGE-
> | Hash: SHA1
> | 
> | On 1/7/10 2:30 PM, Andrew Milton wrote:
> | > +---[ Brian Brinegar ]--
> | > | I'm fairly well versed on the inner workings of Zope, but yesterday I've
> | > | run into a problem that I can grasp.
> | > 
> | > [snip]
> | > 
> | > | Anyway, here's what I see. The copy starts, a tempfile is created (on
> | > | the client) and grows to the appropriate size (3.1GB). Then, a
> | > | Data.fs.temp appears on the new server and grows to the appropriate
> | > | size. However, when my script tries to commit the transaction I get an
> | > | EOF error:
> | > 
> | > File size limits on the new server restricting to say 2GB?
> | 
> | No.  It's an x86-64 RedHat 5 install.  We've had 250GB files on this system.
> 
> Got any errors in your ZEO log ?

Aha!  Our zeo log shows that we are running out of space in /tmp.  What
is the best way to redefine the location for temp files for the ZEO server?

- -Chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktGPuYACgkQ+zGkEDwn8UbojQCglylx+tFbmuALBmEK7dFXACRK
6BcAoMooVxYsbfSCwRjUbV3VDAyqfYMn
=XHFb
-END PGP SIGNATURE-
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Andrew Milton
+---[ Christopher N. Deckard ]--
| -BEGIN PGP SIGNED MESSAGE-
| Hash: SHA1
| 
| On 1/7/10 2:30 PM, Andrew Milton wrote:
| > +---[ Brian Brinegar ]--
| > | I'm fairly well versed on the inner workings of Zope, but yesterday I've
| > | run into a problem that I can grasp.
| > 
| > [snip]
| > 
| > | Anyway, here's what I see. The copy starts, a tempfile is created (on
| > | the client) and grows to the appropriate size (3.1GB). Then, a
| > | Data.fs.temp appears on the new server and grows to the appropriate
| > | size. However, when my script tries to commit the transaction I get an
| > | EOF error:
| > 
| > File size limits on the new server restricting to say 2GB?
| 
| No.  It's an x86-64 RedHat 5 install.  We've had 250GB files on this system.

Also have you done a cmp(1) of the chunk and the data.fs.tmp file to
make sure you're not just seeing a sparse file of the right size that's
internally truncated?

-- 
Andrew Milton
a...@theinternet.com.au
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Andrew Milton
+---[ Christopher N. Deckard ]--
| -BEGIN PGP SIGNED MESSAGE-
| Hash: SHA1
| 
| On 1/7/10 2:30 PM, Andrew Milton wrote:
| > +---[ Brian Brinegar ]--
| > | I'm fairly well versed on the inner workings of Zope, but yesterday I've
| > | run into a problem that I can grasp.
| > 
| > [snip]
| > 
| > | Anyway, here's what I see. The copy starts, a tempfile is created (on
| > | the client) and grows to the appropriate size (3.1GB). Then, a
| > | Data.fs.temp appears on the new server and grows to the appropriate
| > | size. However, when my script tries to commit the transaction I get an
| > | EOF error:
| > 
| > File size limits on the new server restricting to say 2GB?
| 
| No.  It's an x86-64 RedHat 5 install.  We've had 250GB files on this system.

Got any errors in your ZEO log ?


-- 
Andrew Milton
a...@theinternet.com.au
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Christopher N. Deckard
-BEGIN PGP SIGNED MESSAGE-
Hash: SHA1

On 1/7/10 2:30 PM, Andrew Milton wrote:
> +---[ Brian Brinegar ]--
> | I'm fairly well versed on the inner workings of Zope, but yesterday I've
> | run into a problem that I can grasp.
> 
> [snip]
> 
> | Anyway, here's what I see. The copy starts, a tempfile is created (on
> | the client) and grows to the appropriate size (3.1GB). Then, a
> | Data.fs.temp appears on the new server and grows to the appropriate
> | size. However, when my script tries to commit the transaction I get an
> | EOF error:
> 
> File size limits on the new server restricting to say 2GB?

No.  It's an x86-64 RedHat 5 install.  We've had 250GB files on this system.

- -Chris
-BEGIN PGP SIGNATURE-
Version: GnuPG/MacGPG2 v2.0.12 (Darwin)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iEYEARECAAYFAktGNxoACgkQ+zGkEDwn8Uam5QCcDOe62Znr9CrfuwpvCPXXCW0T
oKEAoMjde1r8SKzdv38UTbf6ozaAeNWE
=O65C
-END PGP SIGNATURE-
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


Re: [Zope] Error in Copy/Import during Migration

2010-01-07 Thread Andrew Milton
+---[ Brian Brinegar ]--
| I'm fairly well versed on the inner workings of Zope, but yesterday I've
| run into a problem that I can grasp.

[snip]

| Anyway, here's what I see. The copy starts, a tempfile is created (on
| the client) and grows to the appropriate size (3.1GB). Then, a
| Data.fs.temp appears on the new server and grows to the appropriate
| size. However, when my script tries to commit the transaction I get an
| EOF error:

File size limits on the new server restricting to say 2GB?


-- 
Andrew Milton
a...@theinternet.com.au
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )


[Zope] Error in Copy/Import during Migration

2010-01-07 Thread Brian Brinegar
I'm fairly well versed on the inner workings of Zope, but yesterday I've
run into a problem that I can grasp.

First, some background on what I'm doing.

I've got a 200GB Data.fs file which I'm splitting up into multiple
storages. The way I'm doing this is to create a zope instance which
mounts a new storage and the old storage and then a python script
connects to the application and uses a modified version of manage_clone
(which maintains ownership) to copy content into the new storage.

This works correctly for most things, but yesterday I tried to move a
folder with about 3.1GB of stuff in it.

On a side note, I ran into an issue with running out of /tmp space
during the clone. The _getCopy method of CopySourse exports an object to
a tempfile and then imports it to create the copy. When copying
something larger than available tmp space zope crashes. I worked around
this by explicitly setting tempfile.tempdir to somewhere with plenty of
free space. There may be an environment variable to do this as well.
Just an FYI.

Anyway, here's what I see. The copy starts, a tempfile is created (on
the client) and grows to the appropriate size (3.1GB). Then, a
Data.fs.temp appears on the new server and grows to the appropriate
size. However, when my script tries to commit the transaction I get an
EOF error:

Traceback (most recent call last):
  File "./migrate_site.py", line 45, in ?
transaction.get().commit()
  File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 325, in commit
self._commitResources()
  File
"/var/local/zope/Zope-2.11.4-1/lib/python/transaction/_transaction.py",
line 427, in _commitResources
rm.tpc_vote(self)
  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZODB/Connection.py",
line 744, in tpc_vote
s = vote(transaction)
  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ClientStorage.py",
line 1004, in tpc_vote
self._server.vote(id(txn))
  File "/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/ServerStub.py",
line 262, in vote
return self.rpc.call('vote', trans_id)
  File
"/var/local/zope/Zope-2.11.4-1/lib/python/ZEO/zrpc/connection.py", line
699, in call
raise inst # error raised by server
EOFError

I don't seem to have any space issues on any devices, and the process
works for smaller site. Any input would be greatly appreciated.

Thanks,

-- 
Brian Brinegar
Web Services Coordinator
Engineering Computer Network
___
Zope maillist  -  Zope@zope.org
https://mail.zope.org/mailman/listinfo/zope
**   No cross posts or HTML encoding!  **
(Related lists - 
 https://mail.zope.org/mailman/listinfo/zope-announce
 https://mail.zope.org/mailman/listinfo/zope-dev )