Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On 15/06/2015 18:05, Ritesh Raj Sarraf wrote: On Monday 15 June 2015 07:08 PM, Peter Laird wrote: Changing to AptType="apt" works OK. I tried doing an update, an upgrade and an install. Compared with the original apt-offline v1.5, the .sig files and corresponding .zip files for the upgrade and install were the same. The .sig files and .zip files for the update were different, but the original version 1.5 does not work for update. Thanks Peter, So I devise out of it that it should suffice ? In that case, these changes are minimal and should be acceptable by the release team. I'll roll something up in coming week and prep for stable. I'll keep you posted through this bug report. Attached is a patch for the changes that I tested. I am happy that the change around line 1155 fixes bug#787730. I am happy that the change around line 1798 fixes bug#787736. I would also be happy if you made changes around line 1798 to match version 1.6 because that would be equivalent to what I tested. diff -ru orig/apt-offline-1.5/apt_offline_core/AptOfflineCoreLib.py patch_method/apt-offline-1.5.patch_method/apt_offline_core/AptOfflineCoreLib.py --- orig/apt-offline-1.5/apt_offline_core/AptOfflineCoreLib.py 2015-06-11 22:44:24.0 +0100 +++ patch_method/apt-offline-1.5.patch_method/apt_offline_core/AptOfflineCoreLib.py 2015-06-11 22:44:28.0 +0100 @@ -1155,7 +1155,7 @@ os.rename(temp_filename, filename) else: os.unlink(temp_filename) -elif magicMIME.file( archive_file ) == "application/x-gnupg-keyring": +elif magicMIME.file( archive_file ) == "application/x-gnupg-keyring" or magicMIME.file( archive_file ) == "application/pgp-signature": filename = os.path.join(apt_update_target_path, filename) shutil.copy2(archive_file, filename) # PGP armored data should be bypassed @@ -1798,7 +1798,7 @@ #Instantiate Apt based on what we have. For now, fall to apt only -AptInst = AptManip(Str_SetArg, Simulate=Bool_TestWindows, AptType="python-apt") +AptInst = AptManip(Str_SetArg, Simulate=Bool_TestWindows, AptType="apt") if Bool_SetUpdate: if platform.system() in supported_platforms:
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On Monday 15 June 2015 07:08 PM, Peter Laird wrote: > Changing to AptType="apt" works OK. > I tried doing an update, an upgrade and an install. > > Compared with the original apt-offline v1.5, the .sig files and > corresponding .zip files for the upgrade and install were the same. > The .sig files and .zip files for the update were different, but the > original version 1.5 does not work for update. Thanks Peter, So I devise out of it that it should suffice ? In that case, these changes are minimal and should be acceptable by the release team. I'll roll something up in coming week and prep for stable. I'll keep you posted through this bug report. -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com "Necessity is the mother of invention." signature.asc Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On 12/06/2015 11:30, Ritesh Raj Sarraf wrote: I have added the bug report back so that your root cause doesn't get lost in our mail box. On Thursday 11 June 2015 06:27 PM, Peter Laird wrote: Another option for fixing bug#787736 is to change the line: AptInst = AptManip(Str_SetArg, Simulate=Bool_TestWindows, AptType="python-apt") to match version 1.6 ie. change to ...AptType="apt" It will take me a few days to test this. Thank you for the root cause Peter. I guess this will be the rightest change and should also allow for migration into stable, easily. Upon receiving your test results, I will prepare the upload accordingly. Changing to AptType="apt" works OK. I tried doing an update, an upgrade and an install. Compared with the original apt-offline v1.5, the .sig files and corresponding .zip files for the upgrade and install were the same. The .sig files and .zip files for the update were different, but the original version 1.5 does not work for update. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
I have added the bug report back so that your root cause doesn't get lost in our mail box. On Thursday 11 June 2015 06:27 PM, Peter Laird wrote: > > > Another option for fixing bug#787736 is to change the line: > AptInst = AptManip(Str_SetArg, Simulate=Bool_TestWindows, > AptType="python-apt") > to match version 1.6 ie. change to ...AptType="apt" > It will take me a few days to test this. Thank you for the root cause Peter. I guess this will be the rightest change and should also allow for migration into stable, easily. Upon receiving your test results, I will prepare the upload accordingly. -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com "Necessity is the mother of invention." signature.asc Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On Thursday 11 June 2015 04:26 PM, Peter Laird wrote:
>
> Attached is a patch for apt-offline version 1.5 to fix bug#787730 and
> bug#787736. I've done an update using this patched version and it
> works OK.
> The changes for bug#787730 (around line 1155) means that the code is
> the same as in version 1.6.
> The changes for bug#787736 (around line 1643) apply to code which is
> specific to an update, therefore other uses (eg. upgrade, installing
> packages) should not be affected.
>
> Is it possible for this change to be included in the stable release?
>
Oh!! Thank you. I hadn't realized that the bug fixes were that small.
Some comments below.
>
> apt_offline_v1.5.patch
>
>
> diff -ur apt-offline-1.5.orig/apt_offline_core/AptOfflineCoreLib.py
> apt-offline-1.5.patch/apt_offline_core/AptOfflineCoreLib.py
> --- apt-offline-1.5.orig/apt_offline_core/AptOfflineCoreLib.py
> 2014-09-15 19:28:06.0 +0100
> +++ apt-offline-1.5.patch/apt_offline_core/AptOfflineCoreLib.py
> 2015-06-10 21:25:06.0 +0100
> @@ -1155,7 +1155,7 @@
> os.rename(temp_filename, filename)
> else:
> os.unlink(temp_filename)
> -elif magicMIME.file( archive_file ) ==
> "application/x-gnupg-keyring":
> +elif magicMIME.file( archive_file ) ==
> "application/x-gnupg-keyring" or magicMIME.file( archive_file ) ==
> "application/pgp-signature":
> filename = os.path.join(apt_update_target_path,
> filename)
> shutil.copy2(archive_file, filename)
> # PGP armored data should be bypassed
This one looks good. Stable team shouldn't have any issue allowing its
inclusion.
> @@ -1643,6 +1643,10 @@
> # We strip item.destfile because that's how
> apt-get had historically presented it to us
> destFile = item.destfile.split("/")[-1]
>
> +# strip any compression file extension
> (bug#787736)
> +if os.path.splitext(destFile)[1] in [".bz2",
> ".gz", ".lzma"]:
> +destFile =
> os.path.splitext(destFile)[0]
> +
> writeFH.write("'" + item.desc_uri + "'" + "
> " + destFile + " " + str(item.filesize) + " " + ":" + "\n")
> log.verbose("Writing string %s %s %d %s to
> file %s\n" % (item.desc_uri, destFile, item.filesize, ":", self.WriteTo) )
> writeFH.flush()
Ah!! Now I get the context here.
I think this one is wrong. I doubt if that code path is even executed.
In 1.5 (and 1.6, and my repo), we have following code.
#INFO: Check if python-apt is installed
PythonApt = True
try:
import apt
import apt_pkg
except ImportError:
PythonApt = False
PythonApt = False #Remove it after porting to python-apt
That means python-apt is never used. I just haven't had enough time to
test and enable it.
That said Your actual issue was about .
After applying the fix to apt-offline described in bug #787730, I did:
apt-offline install update.zip
The package list files etc. are correctly synced to /var/lib/apt/lists
but with filename extension .bz2 eg.
-rw-r--r-- 1 root root 33775581 Jun 1 21:12
/var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages
-rw-r--r-- 1 root root 33775581 Jun 3 21:45
/var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2
Although they have a .bz2 extension, they are actually plain text files
- the output from the "file" command is:
/var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2:
ASCII text, with very long lines
Where as the code referenced above is in the setter(), which only is
invoked in the `set` operation.
Your bug report has to do with some issue in the installer() function.
The way it works is:
* We copy the update data in a temp location.
* then we verify its integrity (using the gpg keys)
* If it is pristine, we push it to the right location
We may have a problem in that flow, somewhere, when upgrading from
Wheezy to Jessie.
--
Ritesh Raj Sarraf
RESEARCHUT - http://www.researchut.com
"Necessity is the mother of invention."
signature.asc
Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On 09/06/2015 11:57, Ritesh Raj Sarraf wrote:
On Tuesday 09 June 2015 04:05 PM, Peter Laird wrote:
When apt-offline in jessie is upgraded to version 1.6 I suggest
including the steps above in the
release notes (but change the URL to refer to jessie instead of
stretch) because apt-offline
version 1.5 cannot be used as-is to do the upgrade.
I am just reading the stable upload policy,
https://www.debian.org/doc/manuals/developers-reference/pkgs.html#upload-stable
I doubt if 1.6 can be pushed to stable, because the changes are big.
rs@learner:~/devel/apt-offline/apt-offline (master)$ git diff
v1.5..v1.6 | diffstat
apt-offline-tests.sh |2
apt-offline.8 | 16 ++--
apt-offline.html | 44 +--
apt_offline_core/AptOfflineCoreLib.py | 128
++
apt_offline_core/AptOfflineLib.py | 116 ++
5 files changed, 157 insertions(+), 149 deletions(-)
16:25 ♒♒♒ ☺
I guess a wiser option would be backports.
Never done a backport before, so I guess this is the right time. :-)
Attached is a patch for apt-offline version 1.5 to fix bug#787730 and
bug#787736. I've done an update using this patched version and it works OK.
The changes for bug#787730 (around line 1155) means that the code is the
same as in version 1.6.
The changes for bug#787736 (around line 1643) apply to code which is
specific to an update, therefore other uses (eg. upgrade, installing
packages) should not be affected.
Is it possible for this change to be included in the stable release?
diff -ur apt-offline-1.5.orig/apt_offline_core/AptOfflineCoreLib.py
apt-offline-1.5.patch/apt_offline_core/AptOfflineCoreLib.py
--- apt-offline-1.5.orig/apt_offline_core/AptOfflineCoreLib.py 2014-09-15
19:28:06.0 +0100
+++ apt-offline-1.5.patch/apt_offline_core/AptOfflineCoreLib.py 2015-06-10
21:25:06.0 +0100
@@ -1155,7 +1155,7 @@
os.rename(temp_filename, filename)
else:
os.unlink(temp_filename)
-elif magicMIME.file( archive_file ) ==
"application/x-gnupg-keyring":
+elif magicMIME.file( archive_file ) ==
"application/x-gnupg-keyring" or magicMIME.file( archive_file ) ==
"application/pgp-signature":
filename = os.path.join(apt_update_target_path,
filename)
shutil.copy2(archive_file, filename)
# PGP armored data should be bypassed
@@ -1643,6 +1643,10 @@
# We strip item.destfile because that's how
apt-get had historically presented it to us
destFile = item.destfile.split("/")[-1]
+# strip any compression file extension
(bug#787736)
+if os.path.splitext(destFile)[1] in [".bz2",
".gz", ".lzma"]:
+destFile =
os.path.splitext(destFile)[0]
+
writeFH.write("'" + item.desc_uri + "'" + " "
+ destFile + " " + str(item.filesize) + " " + ":" + "\n")
log.verbose("Writing string %s %s %d %s to
file %s\n" % (item.desc_uri, destFile, item.filesize, ":", self.WriteTo) )
writeFH.flush()
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On Tuesday 09 June 2015 04:27 PM, Ritesh Raj Sarraf wrote: > I guess a wiser option would be backports. > > Never done a backport before, so I guess this is the right time. :-) This may take some time before I can push, based on how much free time I can get. :-( http://backports.debian.org/Contribute/ If you are versed with Debian, and are interested to help with the backport, that'll be welcome. -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com "Necessity is the mother of invention." signature.asc Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On Tuesday 09 June 2015 04:05 PM, Peter Laird wrote: > When apt-offline in jessie is upgraded to version 1.6 I suggest > including the steps above in the > release notes (but change the URL to refer to jessie instead of > stretch) because apt-offline > version 1.5 cannot be used as-is to do the upgrade. I am just reading the stable upload policy, https://www.debian.org/doc/manuals/developers-reference/pkgs.html#upload-stable I doubt if 1.6 can be pushed to stable, because the changes are big. rs@learner:~/devel/apt-offline/apt-offline (master)$ git diff v1.5..v1.6 | diffstat apt-offline-tests.sh |2 apt-offline.8 | 16 ++-- apt-offline.html | 44 +-- apt_offline_core/AptOfflineCoreLib.py | 128 ++ apt_offline_core/AptOfflineLib.py | 116 ++ 5 files changed, 157 insertions(+), 149 deletions(-) 16:25 ♒♒♒ ☺ I guess a wiser option would be backports. Never done a backport before, so I guess this is the right time. :-) -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com "Necessity is the mother of invention." signature.asc Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On 05/06/2015 10:22, Ritesh Raj Sarraf wrote: On Thursday 04 June 2015 08:39 PM, Peter Laird wrote: After applying the fix to apt-offline described in bug #787730, I did: apt-offline install update.zip The package list files etc. are correctly synced to /var/lib/apt/lists but with filename extension .bz2 eg. -rw-r--r-- 1 root root 33775581 Jun 1 21:12 /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages -rw-r--r-- 1 root root 33775581 Jun 3 21:45 /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 Although they have a .bz2 extension, they are actually plain text files - the output from the "file" command is: /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2: ASCII text, with very long lines When I run aptitude, it reports: BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC) Problem opening /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 The package lists or status file could not be parsed or opened. and it does not show any packages. As a workaround, I've renamed the files to remove the .bz2 extension ie. (as root): cd /var/lib/apt/lists for i in *.bz2; do mv $i `basename $i .bz2`; done aptitude then works as expected. I did not have this issue when I upgraded from wheezy to jessie -I assume that when I first got the jessie package list files etc. usingapt-offline it was still the wheezy version ie. version 1.2. This too should be fixed with version 1.6. Have you verified it with 1.6 ? If it works, then we'll target this too for "1.6 for Jessie". Yes, bug#787730 and bug#787736 are fixed in apt-offline version 1.6. Using version 1.6 for all 3 steps (set, get, install) works OK. Using version 1.6 for some steps and version 1.5 for others does not work (in most cases). In case other people are having this issue, the steps I took to install apt-offline version 1.6 : - On the box with a good internet connection, download apt-offline_1.6.1_all.deb e.g. via https://packages.debian.org/stretch/all/apt-offline/download - copy apt-offline_1.6.1_all.deb to the Debian box - Optional: on the Debian box, verify apt-offline_1.6.1_all.deb: # Test file size, expect output "57416 apt-offline_1.6.1_all.deb" wc -c apt-offline_1.6.1_all.deb # Test MD5 hash, expect output "apt-offline_1.6.1_all.deb: OK" echo "5c699069347782f676ac17fd049155e8 apt-offline_1.6.1_all.deb" | md5sum -c - On the Debian box install the new version; do the command as root dpkg --install apt-offline_1.6.1_all.deb When apt-offline in jessie is upgraded to version 1.6 I suggest including the steps above in the release notes (but change the URL to refer to jessie instead of stretch) because apt-offline version 1.5 cannot be used as-is to do the upgrade. -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
On Thursday 04 June 2015 08:39 PM, Peter Laird wrote: > > After applying the fix to apt-offline described in bug #787730, I did: > apt-offline install update.zip > > The package list files etc. are correctly synced to /var/lib/apt/lists > but with filename extension .bz2 eg. > -rw-r--r-- 1 root root 33775581 Jun 1 21:12 > /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages > -rw-r--r-- 1 root root 33775581 Jun 3 21:45 > /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 > > Although they have a .bz2 extension, they are actually plain text > files - the output from the "file" command is: > /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2: > ASCII text, with very long lines > > When I run aptitude, it reports: > BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC) > Problem opening > /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 > The package lists or status file could not be parsed or opened. > and it does not show any packages. > > As a workaround, I've renamed the files to remove the .bz2 extension > ie. (as root): > cd /var/lib/apt/lists > for i in *.bz2; do mv $i `basename $i .bz2`; done > > aptitude then works as expected. > > > I did not have this issue when I upgraded from wheezy to jessie -I > assume that when I first got the jessie package list files etc. > usingapt-offline it was still the wheezy version ie. version 1.2. This too should be fixed with version 1.6. Have you verified it with 1.6 ? If it works, then we'll target this too for "1.6 for Jessie". -- Ritesh Raj Sarraf RESEARCHUT - http://www.researchut.com "Necessity is the mother of invention." signature.asc Description: OpenPGP digital signature
Bug#787736: apt-offline: wrong file extension for Packages/Sources files
Package: apt-offline Version: 1.5 Severity: important Tags: jessie Dear Maintainer, After applying the fix to apt-offline described in bug #787730, I did: apt-offline install update.zip The package list files etc. are correctly synced to /var/lib/apt/lists but with filename extension .bz2 eg. -rw-r--r-- 1 root root 33775581 Jun 1 21:12 /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages -rw-r--r-- 1 root root 33775581 Jun 3 21:45 /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 Although they have a .bz2 extension, they are actually plain text files - the output from the "file" command is: /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2: ASCII text, with very long lines When I run aptitude, it reports: BZ2_bzread: Read error (-5: DATA_ERROR_MAGIC) Problem opening /var/lib/apt/lists/ftp.uk.debian.org_debian_dists_jessie_main_binary-amd64_Packages.bz2 The package lists or status file could not be parsed or opened. and it does not show any packages. As a workaround, I've renamed the files to remove the .bz2 extension ie. (as root): cd /var/lib/apt/lists for i in *.bz2; do mv $i `basename $i .bz2`; done aptitude then works as expected. I did not have this issue when I upgraded from wheezy to jessie -I assume that when I first got the jessie package list files etc. usingapt-offline it was still the wheezy version ie. version 1.2. -- System Information: Debian Release: 8.0 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Foreign Architectures: i386 Kernel: Linux 3.16.0-4-amd64 (SMP w/2 CPU cores) Locale: LANG=en_GB.utf8, LC_CTYPE=en_GB.utf8 (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: systemd (via /run/systemd/system) Versions of packages apt-offline depends on: ii apt1.0.9.8 ii less 458-3 ii libpython2.7-stdlib [python-argparse] 2.7.9-2 ii python 2.7.9-1 Versions of packages apt-offline recommends: ii python-magic 1:5.22+15-2 ii python-soappy 0.12.22-1 apt-offline suggests no packages. -- no debconf information -- To UNSUBSCRIBE, email to debian-bugs-dist-requ...@lists.debian.org with a subject of "unsubscribe". Trouble? Contact listmas...@lists.debian.org
