[sage-devel] Re: sage -upgrade and file corruption
md5 sums (or sha1 for extra security) could be useful if there's ever any interest in signing spkgs in the future (official or 3rd party ones). - Robert On Oct 21, 2007, at 3:28 PM, Pablo De Napoli wrote: My idea was actually the second one, so nothing has to be changed in current sage packages.I don't see this as so painfull (as the Debian is currently doing something similar for debian packages (actually for each Debian package there are 3 sources files: a .dsc file, with description and checksum, .diff.gz (the differencies as a patch to pristine sources) and .orig.tar.gz (the pristine sources) I think that this good be a good model to follow. But yes, perhaps is just having tar to report if the opeation of unpacking was sucessfull or not. Pablo On 10/21/07, William Stein [EMAIL PROTECTED] wrote: On 10/21/07, Pablo De Napoli [EMAIL PROTECTED] wrote: I'm currently working on ticket #329 My idea is adding to each .spkg file a .spkg.md5 file with the md5checksum This should prevent file corruption. Are you literally adding to each .spkg file. If so, make sure this is completely automatic. I.e., whenever anybody does sage -pkg directory-version the md5 file is created inside the resulting spkg. What are you going to create the md5 hash of, by the way, given that the spkg doesn't exist when you create the md5 hash to add to the spkg? The alternative is that we have to have separate files directory-version.spkg and directory-version.spkg.md5 and then whenever anybody ever wants to trade spkgs, they have to copy around, get, etc. 2 separate files. That would be painful in practice. Just out of curiosity, shouldn't tar report if the file it is unpacking is somehow corrupt? Why do we need md5 hashes at all if the whole point is to determine whether or not a download of a .tar.bz2 file (an spkg) was corrupted or not? Should we be able to get that information from tar during the extract process, or at least change how we make the tarball so that information is available. I really don't want to have to keep track of twice as many files if it isn't absolutely necessary. I've already reimplemented the md5sum standard utility (from the coreutils package) in python (using the md5 module), so that we don't need to add an extra dependency to sage. I still have to modify the logic of the scripts (sage-download- package, etc.) so that they do the right thing. Pablo On 10/20/07, Timothy Clemans [EMAIL PROTECTED] wrote: Hi, I think I have done sage -upgrade a few times when William was in the process of uploading a new release. I think it would be helpful if Sage would check a file on sagemath that gave the latest release that had been completed uploaded. Another possibility might be that William would upload the files to directories that Sage doesn't look in and then move them over to the release directories after they have been completely uploaded. Timothy -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: sage -upgrade and file corruption
I think you can easily make tar-archives that contain a checksum, if you agree on some extremely mild file naming convention for such a checksum (i.e., the archive is not allowed to contain a filename that clashes with the file that stores the checksum). Of course, the key is that when you add something to the archive, the file changes, so the plain md5sum of the total archive changes. You have to md5sum something that is easily extracted and independent of the later added md5sum. The options -O (dump to stdout), -r (append file) and -- exclude provide the necessary features for tar. Procedure for storing a checksum in a tar archive: -- (tar xf file.tar --exclude md5sum.check -O; \ tar tvf file.tar --exclude md5sum.check ) | md5sum md5sum.check tar -rf file.tar md5sum.check -- Procedure for checking that the stored sum agrees with the computed one: -- tar xf file.tar md5sum.check -O storedcheck (tar xf file.tar --exclude md5sum.check -O; \ tar tvf file.tar --exclude md5sum.check ) | md5sum computedcheck cmp storedcheck computedcheck -- Note that we need to include the directory listing information as well, because the output of -O does not include file names (i.e., one could move files around and still have the same checksum) If it is ever decided that .spkgs should be signed, then you could include a .gpg-file via the same procedure. --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: sage -upgrade and file corruption
On 10/22/07, Nils Bruin [EMAIL PROTECTED] wrote: I think you can easily make tar-archives that contain a checksum, if you agree on some extremely mild file naming convention for such a checksum (i.e., the archive is not allowed to contain a filename that clashes with the file that stores the checksum). Of course, the key is that when you add something to the archive, the file changes, so the plain md5sum of the total archive changes. You have to md5sum something that is easily extracted and independent of the later added md5sum. The options -O (dump to stdout), -r (append file) and -- exclude provide the necessary features for tar. Procedure for storing a checksum in a tar archive: -- (tar xf file.tar --exclude md5sum.check -O; \ tar tvf file.tar --exclude md5sum.check ) | md5sum md5sum.check tar -rf file.tar md5sum.check -- Procedure for checking that the stored sum agrees with the computed one: -- tar xf file.tar md5sum.check -O storedcheck (tar xf file.tar --exclude md5sum.check -O; \ tar tvf file.tar --exclude md5sum.check ) | md5sum computedcheck cmp storedcheck computedcheck -- Note that we need to include the directory listing information as well, because the output of -O does not include file names (i.e., one could move files around and still have the same checksum) If it is ever decided that .spkgs should be signed, then you could include a .gpg-file via the same procedure. I really like this idea a lot! It's vastly better -- I think -- from a usability point of view than having to constantly pass around .spkg's and .md5 files together. It will just work 100% automatically and transparently to users, once we modify some scripts in local/bin/sage-*. While we're at it, we should make the following work: 1) sage -unpkg packagename-version.spkg which just does tar jxvf and does the above consistency checks. I suggest sage -unpkg, since making a package is sage -pkg. Another option would be sage -extract blah.spkg, or even sage -x blah.spkg.Please note, sage spkg's can be either bzip2'd or not, so that has to be taken account of. 2) sage -i packagename-version where packagename-version is the name of a *directory*, does sage -pkg on the directory, then installs it. -- William --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: sage -upgrade and file corruption
I'm currently working on ticket #329 My idea is adding to each .spkg file a .spkg.md5 file with the md5checksum This should prevent file corruption. I've already reimplemented the md5sum standard utility (from the coreutils package) in python (using the md5 module), so that we don't need to add an extra dependency to sage. I still have to modify the logic of the scripts (sage-download-package, etc.) so that they do the right thing. Pablo On 10/20/07, Timothy Clemans [EMAIL PROTECTED] wrote: Hi, I think I have done sage -upgrade a few times when William was in the process of uploading a new release. I think it would be helpful if Sage would check a file on sagemath that gave the latest release that had been completed uploaded. Another possibility might be that William would upload the files to directories that Sage doesn't look in and then move them over to the release directories after they have been completely uploaded. Timothy --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
[sage-devel] Re: sage -upgrade and file corruption
My idea was actually the second one, so nothing has to be changed in current sage packages.I don't see this as so painfull (as the Debian is currently doing something similar for debian packages (actually for each Debian package there are 3 sources files: a .dsc file, with description and checksum, .diff.gz (the differencies as a patch to pristine sources) and .orig.tar.gz (the pristine sources) I think that this good be a good model to follow. But yes, perhaps is just having tar to report if the opeation of unpacking was sucessfull or not. Pablo On 10/21/07, William Stein [EMAIL PROTECTED] wrote: On 10/21/07, Pablo De Napoli [EMAIL PROTECTED] wrote: I'm currently working on ticket #329 My idea is adding to each .spkg file a .spkg.md5 file with the md5checksum This should prevent file corruption. Are you literally adding to each .spkg file. If so, make sure this is completely automatic. I.e., whenever anybody does sage -pkg directory-version the md5 file is created inside the resulting spkg. What are you going to create the md5 hash of, by the way, given that the spkg doesn't exist when you create the md5 hash to add to the spkg? The alternative is that we have to have separate files directory-version.spkg and directory-version.spkg.md5 and then whenever anybody ever wants to trade spkgs, they have to copy around, get, etc. 2 separate files. That would be painful in practice. Just out of curiosity, shouldn't tar report if the file it is unpacking is somehow corrupt? Why do we need md5 hashes at all if the whole point is to determine whether or not a download of a .tar.bz2 file (an spkg) was corrupted or not? Should we be able to get that information from tar during the extract process, or at least change how we make the tarball so that information is available. I really don't want to have to keep track of twice as many files if it isn't absolutely necessary. I've already reimplemented the md5sum standard utility (from the coreutils package) in python (using the md5 module), so that we don't need to add an extra dependency to sage. I still have to modify the logic of the scripts (sage-download-package, etc.) so that they do the right thing. Pablo On 10/20/07, Timothy Clemans [EMAIL PROTECTED] wrote: Hi, I think I have done sage -upgrade a few times when William was in the process of uploading a new release. I think it would be helpful if Sage would check a file on sagemath that gave the latest release that had been completed uploaded. Another possibility might be that William would upload the files to directories that Sage doesn't look in and then move them over to the release directories after they have been completely uploaded. Timothy -- William Stein Associate Professor of Mathematics University of Washington http://wstein.org --~--~-~--~~~---~--~~ To post to this group, send email to sage-devel@googlegroups.com To unsubscribe from this group, send email to [EMAIL PROTECTED] For more options, visit this group at http://groups.google.com/group/sage-devel URLs: http://sage.scipy.org/sage/ and http://modular.math.washington.edu/sage/ -~--~~~~--~~--~--~---
