[Touch-packages] [Bug 1375637] Re: Premature end of data in tag file line 247651 with libxml2
Hi Sebastien, The problem is I've found no consistent trigger for the bug. I have 225 packages in my repository, built out for two different branches and two oses (so 225x4). createrepo just randomly corrupts when executing 'createrepo --update .' when adding a new source/binary rpm pair. It is trivially solvable (rm -rf repodata .repodata; createrepo .), but having to briefly have a 100% repo outage is the problem. I've not seen anything specific about the packages that would cause this to occur. It seems to be some type of race condition during the update process, because of it complaining about " I/O error : No such file or directory". -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to libxml2 in Ubuntu. https://bugs.launchpad.net/bugs/1375637 Title: Premature end of data in tag file line 247651 with libxml2 Status in libxml2 package in Ubuntu: Confirmed Bug description: Hello, I'm getting the following behaviour of createrepo on Ubuntu 14.04.1 LTS with libxml2=2.9.1+dfsg1-3ubuntu4.3(trusty-updates/main) and libxml2=2.9.1+dfsg1-3ubuntu4(trusty/main). createrepo -g /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/comps.xml \ -o /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/ /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/ I/O error : No such file or directory (process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory. This indicates a bug in someone's code. You must ensure an error is NULL before it's set. The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag package line 246828 (process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory. This indicates a bug in someone's code. You must ensure an error is NULL before it's set. The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag filelists line 2 Spawning worker 0 with 54 pkgs Spawning worker 1 with 54 pkgs Spawning worker 2 with 54 pkgs Spawning worker 3 with 54 pkgs Spawning worker 4 with 54 pkgs Spawning worker 5 with 54 pkgs Spawning worker 6 with 54 pkgs Spawning worker 7 with 54 pkgs Spawning worker 8 with 54 pkgs Spawning worker 9 with 54 pkgs Spawning worker 10 with 53 pkgs Spawning worker 11 with 53 pkgs Spawning worker 12 with 53 pkgs Spawning worker 13 with 53 pkgs Spawning worker 14 with 53 pkgs Spawning worker 15 with 53 pkgs Spawning worker 16 with 53 pkgs Spawning worker 17 with 53 pkgs Spawning worker 18 with 53 pkgs Spawning worker 19 with 53 pkgs Spawning worker 20 with 53 pkgs Spawning worker 21 with 53 pkgs Spawning worker 22 with 53 pkgs Spawning worker 23 with 53 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Traceback (most recent call last): File "/usr/share/createrepo/genpkgmetadata.py", line 308, in main(sys.argv[1:]) File "/usr/share/createrepo/genpkgmetadata.py", line 280, in main mdgen.doRepoMetadata() File "/usr/lib/python2.7/dist-packages/createrepo/__init__.py", line 999, in doRepoMetadata rp.getFilelists(complete_path, csum) File "/usr/lib/python2.7/dist-packages/sqlitecachec.py", line 55, in getFilelists self.repoid)) TypeError: Parsing filelists.xml error: Premature end of data in tag file line 247651 Problem fixes by installing libxml2=2.9.1+dfsg1 from debian sid. Comparing patchsets in debian package sources between 2.9.1+dfsg1 and 2.9.1+dfsg1-3ubuntu4.3 I see in debian package much more patches, I suppose one of them helps in my problem: 2.9.1+dfsg1-3ubuntu4.3: 0001-modify-xml2-config-and-pkgconfig-behaviour.patch 0002-Fix-an-error-in-xmlCleanupParser.patch 0003-Fix-missing-break-on-last-function-for-attributes.patch 0004-xmllint-memory-should-fail-on-empty-files.patch 0005-properly-quote-the-namespace-uris-written-out-during.patch 0006-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch 0006-fix-python-multiarch-includes.patch 0007-Fix-XPath-optimization-with-predicates.patch CVE-2014-0191.patch lp1321869.patch xmllint_pretty.patch 2.9.1+dfsg1: 0001-modify-xml2-config-and-pkgconfig-behaviour.patch 0002-fix-python-multiarch-includes.patch 0003-Fix-an-error-in-xmlCleanupParser.patch 0004-Fix-missing-break-on-last-function-for-attributes.patch 0005-xmllint-memory-should-fail-on-empty-files.patch 0006-properly-quote-the-namespace-uris-written-out-during.patch 0007-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch 0008-missing-else-in-xlink.c.patch 0009-Catch-malloc-error-and-exit-accordingly.patch 0010-Fix-handling-of-mmap-errors.patch 0011-Avoid-crash-if-allocation-fails.patch 0012-Fix-a
[Touch-packages] [Bug 1375637] Re: Premature end of data in tag file line 247651 with libxml2
When will this be fixed? I'm currently using a single server to host both debs and rpms, and this bug triggers fairly frequently, causing me to have to take down my repo and regenerate it. -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to libxml2 in Ubuntu. https://bugs.launchpad.net/bugs/1375637 Title: Premature end of data in tag file line 247651 with libxml2 Status in libxml2 package in Ubuntu: Confirmed Bug description: Hello, I'm getting the following behaviour of createrepo on Ubuntu 14.04.1 LTS with libxml2=2.9.1+dfsg1-3ubuntu4.3(trusty-updates/main) and libxml2=2.9.1+dfsg1-3ubuntu4(trusty/main). createrepo -g /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/comps.xml \ -o /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/ /home/jenkins/workspace/tmp/fuel_5_1_mirror/local_mirror/centos/os/x86_64/ I/O error : No such file or directory (process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory. This indicates a bug in someone's code. You must ensure an error is NULL before it's set. The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag package line 246828 (process:29830): GLib-WARNING **: GError set over the top of a previous GError or uninitialized memory. This indicates a bug in someone's code. You must ensure an error is NULL before it's set. The overwriting error message was: Parsing filelists.xml error: Premature end of data in tag filelists line 2 Spawning worker 0 with 54 pkgs Spawning worker 1 with 54 pkgs Spawning worker 2 with 54 pkgs Spawning worker 3 with 54 pkgs Spawning worker 4 with 54 pkgs Spawning worker 5 with 54 pkgs Spawning worker 6 with 54 pkgs Spawning worker 7 with 54 pkgs Spawning worker 8 with 54 pkgs Spawning worker 9 with 54 pkgs Spawning worker 10 with 53 pkgs Spawning worker 11 with 53 pkgs Spawning worker 12 with 53 pkgs Spawning worker 13 with 53 pkgs Spawning worker 14 with 53 pkgs Spawning worker 15 with 53 pkgs Spawning worker 16 with 53 pkgs Spawning worker 17 with 53 pkgs Spawning worker 18 with 53 pkgs Spawning worker 19 with 53 pkgs Spawning worker 20 with 53 pkgs Spawning worker 21 with 53 pkgs Spawning worker 22 with 53 pkgs Spawning worker 23 with 53 pkgs Workers Finished Saving Primary metadata Saving file lists metadata Saving other metadata Generating sqlite DBs Traceback (most recent call last): File "/usr/share/createrepo/genpkgmetadata.py", line 308, in main(sys.argv[1:]) File "/usr/share/createrepo/genpkgmetadata.py", line 280, in main mdgen.doRepoMetadata() File "/usr/lib/python2.7/dist-packages/createrepo/__init__.py", line 999, in doRepoMetadata rp.getFilelists(complete_path, csum) File "/usr/lib/python2.7/dist-packages/sqlitecachec.py", line 55, in getFilelists self.repoid)) TypeError: Parsing filelists.xml error: Premature end of data in tag file line 247651 Problem fixes by installing libxml2=2.9.1+dfsg1 from debian sid. Comparing patchsets in debian package sources between 2.9.1+dfsg1 and 2.9.1+dfsg1-3ubuntu4.3 I see in debian package much more patches, I suppose one of them helps in my problem: 2.9.1+dfsg1-3ubuntu4.3: 0001-modify-xml2-config-and-pkgconfig-behaviour.patch 0002-Fix-an-error-in-xmlCleanupParser.patch 0003-Fix-missing-break-on-last-function-for-attributes.patch 0004-xmllint-memory-should-fail-on-empty-files.patch 0005-properly-quote-the-namespace-uris-written-out-during.patch 0006-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch 0006-fix-python-multiarch-includes.patch 0007-Fix-XPath-optimization-with-predicates.patch CVE-2014-0191.patch lp1321869.patch xmllint_pretty.patch 2.9.1+dfsg1: 0001-modify-xml2-config-and-pkgconfig-behaviour.patch 0002-fix-python-multiarch-includes.patch 0003-Fix-an-error-in-xmlCleanupParser.patch 0004-Fix-missing-break-on-last-function-for-attributes.patch 0005-xmllint-memory-should-fail-on-empty-files.patch 0006-properly-quote-the-namespace-uris-written-out-during.patch 0007-Fix-a-parsing-bug-on-non-ascii-element-and-CR-LF-usa.patch 0008-missing-else-in-xlink.c.patch 0009-Catch-malloc-error-and-exit-accordingly.patch 0010-Fix-handling-of-mmap-errors.patch 0011-Avoid-crash-if-allocation-fails.patch 0012-Fix-a-possible-NULL-dereference.patch 0013-Clear-up-a-potential-NULL-dereference.patch 0014-Fix-XPath-optimization-with-predicates.patch 0015-xmllint-pretty-crashed-without-following-numeric-arg.patch 0016-Fix-potential-NULL-pointer-dereferences-in-regexp-co.patch 0017-Fix-a-potential-NULL-dereference-in-tree-code.patch 0018-Fix-pointer-dereferenced-before-null-check.patch 0019-Fix-a-bug-loading-some-compressed-files.patch 0020-Avoid-a-possibility-
[Touch-packages] [Bug 1003633] Re: Escape plus character in apt HTTP requests to work around Amazon S3 bug
I filed https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1512845 for
the https issue
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1003633
Title:
Escape plus character in apt HTTP requests to work around Amazon S3
bug
Status in apt package in Ubuntu:
Fix Released
Bug description:
Amazon S3 performs incorrect query-string un-escaping on URLs and improperly
converts the plus ('+') character to a space (' ') when processing incoming GET
requests for files. This causes apt-get to get HTTP 404 errors when
downloading packages from S3-hosted mirrors which include a + character in the
name of the package.
For more background info, see:
https://forums.aws.amazon.com/thread.jspa?threadID=55746
One possibly solution is to change the call to QuoteString in http.cc
to include '+' in the character set, which would escape the plus
character to the %2B form, which will be properly processed by Amazon
S3.
Thanks,
Doug
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1003633/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1512845] [NEW] Escape plus character in apt HTTPS requests to work around Amazon S3 bug
Public bug reported: As noted in bug#1003633, S3 is broken when a + sign exists unless it is escaped. This was fixed in 2013 for http based repos. However, it is still broken for https based repositories. apt-get install zimbra-openssl Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: zimbra-openssl-lib The following NEW packages will be installed: zimbra-openssl zimbra-openssl-lib 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 1,324 kB of archives. After this operation, 6,313 kB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl-lib amd64 1.0.1p-1zimbra8.7b1+14.04 [828 kB] Err https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl-lib amd64 1.0.1p-1zimbra8.7b1+14.04 HttpError403 Get:2 https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl amd64 1.0.1p-1zimbra8.7b1+14.04 [497 kB] Err https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl amd64 1.0.1p-1zimbra8.7b1+14.04 HttpError403 E: Failed to fetch https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl-lib_1.0.1p-1zimbra8.7b1+14.04_amd64.deb HttpError403 E: Failed to fetch https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra- openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1+14.04_amd64.deb HttpError403 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Escaping the + sign fixes the issue when testing with wget: root@zre-ldap001:/home/quanah# wget https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1%2B14.04_amd64.deb --2015-11-03 13:11:03-- https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1%2B14.04_amd64.deb Resolving repo.zimbra.com (repo.zimbra.com)... 54.230.4.69, 54.230.6.152, 54.192.4.217, ... Connecting to repo.zimbra.com (repo.zimbra.com)|54.230.4.69|:443... connected. HTTP request sent, awaiting response... 200 OK Length: 496784 (485K) [application/x-debian-package] Saving to: ‘zimbra-openssl_1.0.1p-1zimbra8.7b1+14.04_amd64.deb’ 100%[===>] 496,784 1.62MB/s in 0.3s 2015-11-03 13:11:04 (1.62 MB/s) - ‘zimbra-openssl_1.0.1p- 1zimbra8.7b1+14.04_amd64.deb’ saved [496784/496784] ** Affects: apt (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1512845 Title: Escape plus character in apt HTTPS requests to work around Amazon S3 bug Status in apt package in Ubuntu: New Bug description: As noted in bug#1003633, S3 is broken when a + sign exists unless it is escaped. This was fixed in 2013 for http based repos. However, it is still broken for https based repositories. apt-get install zimbra-openssl Reading package lists... Done Building dependency tree Reading state information... Done The following extra packages will be installed: zimbra-openssl-lib The following NEW packages will be installed: zimbra-openssl zimbra-openssl-lib 0 upgraded, 2 newly installed, 0 to remove and 0 not upgraded. Need to get 1,324 kB of archives. After this operation, 6,313 kB of additional disk space will be used. Do you want to continue? [Y/n] y Get:1 https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl-lib amd64 1.0.1p-1zimbra8.7b1+14.04 [828 kB] Err https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl-lib amd64 1.0.1p-1zimbra8.7b1+14.04 HttpError403 Get:2 https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl amd64 1.0.1p-1zimbra8.7b1+14.04 [497 kB] Err https://repo.zimbra.com/apt/87/ trusty/zimbra zimbra-openssl amd64 1.0.1p-1zimbra8.7b1+14.04 HttpError403 E: Failed to fetch https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl-lib_1.0.1p-1zimbra8.7b1+14.04_amd64.deb HttpError403 E: Failed to fetch https://repo.zimbra.com/apt/87/pool/zimbra/z /zimbra-openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1+14.04_amd64.deb HttpError403 E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing? Escaping the + sign fixes the issue when testing with wget: root@zre-ldap001:/home/quanah# wget https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1%2B14.04_amd64.deb --2015-11-03 13:11:03-- https://repo.zimbra.com/apt/87/pool/zimbra/z/zimbra-openssl/zimbra-openssl_1.0.1p-1zimbra8.7b1%2B14.04_amd64.deb Resolving repo.zimbra.com (repo.zimbra.com)... 54.230.4.69, 54.230.6.152, 54.192.4.217, ... Connecting to repo.zimbra.com (repo.zimbra.com)|54.230.4.69|:443... connected. HTTP request sent, aw
[Touch-packages] [Bug 1003633] Re: Escape plus character in apt HTTP requests to work around Amazon S3 bug
I would note this is broken for https based repositories still.
--
You received this bug notification because you are a member of Ubuntu
Touch seeded packages, which is subscribed to apt in Ubuntu.
https://bugs.launchpad.net/bugs/1003633
Title:
Escape plus character in apt HTTP requests to work around Amazon S3
bug
Status in apt package in Ubuntu:
Fix Released
Bug description:
Amazon S3 performs incorrect query-string un-escaping on URLs and improperly
converts the plus ('+') character to a space (' ') when processing incoming GET
requests for files. This causes apt-get to get HTTP 404 errors when
downloading packages from S3-hosted mirrors which include a + character in the
name of the package.
For more background info, see:
https://forums.aws.amazon.com/thread.jspa?threadID=55746
One possibly solution is to change the call to QuoteString in http.cc
to include '+' in the character set, which would escape the plus
character to the %2B form, which will be properly processed by Amazon
S3.
Thanks,
Doug
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1003633/+subscriptions
--
Mailing list: https://launchpad.net/~touch-packages
Post to : touch-packages@lists.launchpad.net
Unsubscribe : https://launchpad.net/~touch-packages
More help : https://help.launchpad.net/ListHelp
[Touch-packages] [Bug 1442775] [NEW] apt-get --ignore-missing doesn't ignore missing packages
Public bug reported: The man page for apt-get states: -m, --ignore-missing, --fix-missing Ignore missing packages; if packages cannot be retrieved or fail the integrity check after retrieval (corrupted package files), hold back those packages and handle the result. Use of this option together with -f may produce an error in some situations. If a package is selected for installation (particularly if it is mentioned on the command line) and it could not be downloaded then it will be silently held back. Configuration Item: APT::Get::Fix-Missing. However, when removing or purging packages, it does /not/ actually ignore missing packages: build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ sudo apt-get purge -y --ignore-missing zimbra-openssl-lib Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package zimbra-openssl-lib build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ echo $? 100 This causes some pain when using apt-get in a Makefile and wanting to ensure custom packages have been removed. dpkg handles this correctly: build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ sudo dpkg --purge zimbra-openssl-lib dpkg: warning: ignoring request to remove zimbra-openssl-lib which isn't installed build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ echo $? 0 however you then lose dependency tracking. ** Affects: apt (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Touch seeded packages, which is subscribed to apt in Ubuntu. https://bugs.launchpad.net/bugs/1442775 Title: apt-get --ignore-missing doesn't ignore missing packages Status in apt package in Ubuntu: New Bug description: The man page for apt-get states: -m, --ignore-missing, --fix-missing Ignore missing packages; if packages cannot be retrieved or fail the integrity check after retrieval (corrupted package files), hold back those packages and handle the result. Use of this option together with -f may produce an error in some situations. If a package is selected for installation (particularly if it is mentioned on the command line) and it could not be downloaded then it will be silently held back. Configuration Item: APT::Get::Fix-Missing. However, when removing or purging packages, it does /not/ actually ignore missing packages: build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ sudo apt-get purge -y --ignore-missing zimbra-openssl-lib Reading package lists... Done Building dependency tree Reading state information... Done E: Unable to locate package zimbra-openssl-lib build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ echo $? 100 This causes some pain when using apt-get in a Makefile and wanting to ensure custom packages have been removed. dpkg handles this correctly: build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ sudo dpkg --purge zimbra-openssl-lib dpkg: warning: ignoring request to remove zimbra-openssl-lib which isn't installed build@ub14test:~/p4/zimbra/main/ThirdParty/openldap$ echo $? 0 however you then lose dependency tracking. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/apt/+bug/1442775/+subscriptions -- Mailing list: https://launchpad.net/~touch-packages Post to : touch-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~touch-packages More help : https://help.launchpad.net/ListHelp
