Bug#944696: python-apt: relies on MD5 internally to download packages
I am no expert but on my opinion a temporary fix would be to bring back MD5 for the time being and then fixing/extending python-apt functions to check for more secure sums like SHA* first and then if not found fallback to MD5. But just for the backwards-compatibility since MD5 are know to be insecure and colissions are possible to find with decent hardware. Security issue as well? On November 14, 2019 12:36:14 AM UTC, Cyril Brulebois wrote: >Package: python-apt >Version: 1.8.4 >Severity: serious >Justification: some people want to get rid of MD5Sum in indices > >Hi, > >While debugging a live-wrapper (lwr) failure that started occurring >(literally) overnight, I ended up discovering it was triggered by the >intel-microcode package's getting a security upgrade. > >live-wrapper 0.10 isn't affected, but live-wrapper's master branch has >an extra commit that automatically enables security sources for stable >releases. > >Here's the traceback for a simple build (with a local mirror but anyone >would do) with that master branch: > >$ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode >[…] >DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian' >DEBUG environment: LWR_EXTRA_PACKAGES = '' >DEBUG environment: LWR_BASE_DEBS = '' >DEBUG environment: LWR_DISTRIBUTION = 'buster' >DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode' >DEBUG environment: LWR_TASK_PACKAGES = '' >[…] >Downloading udebs for Debian Installer... >INFO Downloading udebs for Debian Installer... >Updating a local cache for amd64 buster ... >DEBUG Updating local cache... >CRITICAL Traceback (most recent call last): >File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in >_run >self.process_args(args) >File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in >process_args >self.start_ops() >File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in >start_ops >apt_udeb.download_udebs(exclude_list) >File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in >download_udebs >self.download_apt_file(pkg_name, pool_dir, False) >File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in >download_apt_file >version.fetch_binary(destdir=pkg_dir) >File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in >fetch_binary >if _file_is_same(destfile, self.size, self._records.md5_hash): >SystemError: error return without exception set > > >After some debugging, it turned out that merely accessing the >self._records.md5_hash item is sufficient to reproduce this issue. > >Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for >buster/updates on security.debian.org, one can only see SHA256 entries >in Release and Packages files, which is likely the reason for >python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries >back at least for buster/updates, and will file another bug report for >that in a moment to make sure it isn't lost. > >Looking at even the most recent python-apt code in experimental >(1.9.0), >MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary(): > > >def fetch_binary(self, destdir='', progress=None): ># type: (str, AcquireProgress) -> str >"""Fetch the binary version of the package. > > The parameter *destdir* specifies the directory where the package will >be fetched to. > > The parameter *progress* may refer to an apt_pkg.AcquireProgress() > object. If not specified or None, apt.progress.text.AcquireProgress() >is used. > >.. versionadded:: 0.7.10 >""" >base = os.path.basename(self._records.filename) >destfile = os.path.join(destdir, base) >if _file_is_same(destfile, self.size, self._records.md5_hash): > logging.debug('Ignoring already existing file: %s' % destfile) >return os.path.abspath(destfile) > acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress()) >acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # >type: ignore # TODO: Do not use MD5 # nopep8 >self.size, base, destfile=destfile) >acq.run() > >if acqfile.status != acqfile.STAT_DONE: >raise FetchError("The item %r could not be fetched: %s" % > (acqfile.destfile, acqfile.error_text)) > >return os.path.abspath(destfile) > > >Notice the TODO on the apt_pkg.AcquireFile(), but it would probably >break in the same way as in the live-wrapper case a few lines before, >on >the self._records.md5_hash item. > >The same goes for fetch_source(). > > >Since getting rid of MD5Sum entirely is a topic that comes up on a >regular fashion (with fingers being pointed at jigdo in particular), it >looks to me python-apt should get some attention as well; hence filing >at serious severity. Feel free to adjust as required. > > >Cheers, >-- >Cyril
Bug#944696: python-apt: relies on MD5 internally to download packages
Cyril Brulebois (2019-11-14): > Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for > buster/updates on security.debian.org, one can only see SHA256 entries > in Release and Packages files, which is likely the reason for > python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries > back at least for buster/updates, and will file another bug report for > that in a moment to make sure it isn't lost. For completeness, this is tracked in #944697. Cheers, -- Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/ signature.asc Description: PGP signature
Bug#944696: python-apt: relies on MD5 internally to download packages
Package: python-apt Version: 1.8.4 Severity: serious Justification: some people want to get rid of MD5Sum in indices Hi, While debugging a live-wrapper (lwr) failure that started occurring (literally) overnight, I ended up discovering it was triggered by the intel-microcode package's getting a security upgrade. live-wrapper 0.10 isn't affected, but live-wrapper's master branch has an extra commit that automatically enables security sources for stable releases. Here's the traceback for a simple build (with a local mirror but anyone would do) with that master branch: $ sudo lwr -d buster -m http://wodi.home/debian -f intel-microcode […] DEBUG environment: LWR_MIRROR = 'http://wodi.home/debian' DEBUG environment: LWR_EXTRA_PACKAGES = '' DEBUG environment: LWR_BASE_DEBS = '' DEBUG environment: LWR_DISTRIBUTION = 'buster' DEBUG environment: LWR_FIRMWARE_PACKAGES = 'intel-microcode' DEBUG environment: LWR_TASK_PACKAGES = '' […] Downloading udebs for Debian Installer... INFO Downloading udebs for Debian Installer... Updating a local cache for amd64 buster ... DEBUG Updating local cache... CRITICAL Traceback (most recent call last): File "/usr/lib/python2.7/dist-packages/cliapp/app.py", line 193, in _run self.process_args(args) File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 143, in process_args self.start_ops() File "/usr/lib/python2.7/dist-packages/lwr/run.py", line 286, in start_ops apt_udeb.download_udebs(exclude_list) File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 157, in download_udebs self.download_apt_file(pkg_name, pool_dir, False) File "/usr/lib/python2.7/dist-packages/lwr/apt_udeb.py", line 141, in download_apt_file version.fetch_binary(destdir=pkg_dir) File "/usr/lib/python2.7/dist-packages/apt/package.py", line 867, in fetch_binary if _file_is_same(destfile, self.size, self._records.md5_hash): SystemError: error return without exception set After some debugging, it turned out that merely accessing the self._records.md5_hash item is sufficient to reproduce this issue. Looking at the current (as of 2019-11-14 00:27:00 UTC) indices for buster/updates on security.debian.org, one can only see SHA256 entries in Release and Packages files, which is likely the reason for python-apt's explosion. I've asked #debian-ftp to add MD5Sum entries back at least for buster/updates, and will file another bug report for that in a moment to make sure it isn't lost. Looking at even the most recent python-apt code in experimental (1.9.0), MD5 still seems hardwired, e.g. in apt/packages.py's fetch_binary(): def fetch_binary(self, destdir='', progress=None): # type: (str, AcquireProgress) -> str """Fetch the binary version of the package. The parameter *destdir* specifies the directory where the package will be fetched to. The parameter *progress* may refer to an apt_pkg.AcquireProgress() object. If not specified or None, apt.progress.text.AcquireProgress() is used. .. versionadded:: 0.7.10 """ base = os.path.basename(self._records.filename) destfile = os.path.join(destdir, base) if _file_is_same(destfile, self.size, self._records.md5_hash): logging.debug('Ignoring already existing file: %s' % destfile) return os.path.abspath(destfile) acq = apt_pkg.Acquire(progress or apt.progress.text.AcquireProgress()) acqfile = apt_pkg.AcquireFile(acq, self.uri, self._records.md5_hash, # type: ignore # TODO: Do not use MD5 # nopep8 self.size, base, destfile=destfile) acq.run() if acqfile.status != acqfile.STAT_DONE: raise FetchError("The item %r could not be fetched: %s" % (acqfile.destfile, acqfile.error_text)) return os.path.abspath(destfile) Notice the TODO on the apt_pkg.AcquireFile(), but it would probably break in the same way as in the live-wrapper case a few lines before, on the self._records.md5_hash item. The same goes for fetch_source(). Since getting rid of MD5Sum entirely is a topic that comes up on a regular fashion (with fingers being pointed at jigdo in particular), it looks to me python-apt should get some attention as well; hence filing at serious severity. Feel free to adjust as required. Cheers, -- Cyril Brulebois -- Debian Consultant @ DEBAMAX -- https://debamax.com/