Re: [gentoo-portage-dev] [PATCH 3/3] Add an emaint module that can scan for failed merges and that can fix failed merges.
On Fri, Mar 14, 2014 at 10:35 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:
On 03/13/14 15:09, Alec Warner wrote:
On Wed, Mar 12, 2014 at 11:10 PM, Pavel Kazakov
nullishz...@gentoo.org mailto:nullishz...@gentoo.org wrote:
---
pym/portage/emaint/main.py| 6 +-
...
Perhaps add an __iter__ function to TrackingFile, that implicitly
calls load?
Then this can be:
for pkg, mtime in self._tracking_file:
?
Just a thought.
That's actually a neat idea. Going to do it.
+ for pkg, mtime in self._tracking_file.load().items():
+ if pkg not in failed_pkgs:
+ failed_pkgs[pkg] = mtime
+ return failed_pkgs
+
+
+ def _remove_failed_dirs(self, failed_pkgs):
+
+ Remove the directories of packages that failed to
merge.
+
+ @param failed_pkgs: failed packages whose
directories to remove
+ @type failed_pkg: dict
+
+ for failed_pkg in failed_pkgs:
+ pkg_path = os.path.join(self._vardb_path,
failed_pkg)
+ # delete failed merge directory if it
exists (it might not exist
+ # if loaded from tracking file)
+ if os.path.exists(pkg_path):
+ shutil.rmtree(pkg_path)
+ # TODO: try removing package contents to
prevent orphaned
+ # files
I don't get this TODO, can you elaborate?
Right now the current code deletes only the failed merge directory (e.g.
/var/db/pkg/app-misc/-MERGING-lsx-0.1/).
In /var/db/pkg/app-misc/-MERGING-lsx-0.1/, there should be a file called
CONTENTS that records the package contents (e.g. obj
/usr/bin/lsx-suckless 478333aa1cb7761bc8e3a400cbd976ab 1394688298).
However, during a failed merge, the CONTENTS file may not be there or
may be corrupt, so my code doesn't handle removing the contents right
now, only the failed merge directory.
Any suggestions for better wording?
an upcase CONTENTS, or just talk about vdb or metadata specifically.
+
+
+ def _get_pkg_atoms(self, failed_pkgs, pkg_atoms, pkg_dne):
+
+ Get the package atoms for the specified failed
packages.
+
+ @param failed_pkgs: failed packages to iterate
+ @type failed_pkgs: dict
+ @param pkg_atoms: append package atoms to this set
+ @type pkg_atoms: set
+ @param pkg_dne: append any packages atoms that are
invalid to this set
+ @type pkg_dne: set
Not following what dne means.
How about pkg_invalid_entries? And I can change the comment to '@param
pkg_invalid_entries: append any packages that are invalid to this set'
After looking at the function more, I think it might make sense to
instead return pkg_atoms and pkg_invalid_entries as a tuple, instead of
having the calling code pass them in.
sgtm
+
+
+ emerge_config = load_emerge_config()
+ portdb =
emerge_config.target_config.trees['porttree'].dbapi
+ for failed_pkg in failed_pkgs:
+ # validate pkg name
+ pkg_name = '%s' %
failed_pkg.replace(MERGING_IDENTIFIER, '')
+ pkg_atom = '=%s' % pkg_name
+
+ if not isvalidatom(pkg_atom):
+ pkg_dne.append( '%s' is an
invalid package atom. % pkg_atom)
+ if not portdb.cpv_exists(pkg_name):
+ pkg_dne.append( '%s' does not
exist in the portage tree.
+ % pkg_name)
+ pkg_atoms.add(pkg_atom)
+
+
+ def _emerge_pkg_atoms(self, module_output, pkg_atoms):
+
+ Emerge the specified packages atoms.
+
+ @param module_output: output will be written to
+ @type module_output: Class
+ @param pkg_atoms: packages atoms to emerge
+ @type pkg_atoms: set
+ @rtype: list
+ @return: List of results
+
+ # TODO: rewrite code to use portage's APIs instead
of a subprocess
+ env = {
+ FEATURES : -collision-detect
-protect-owned,
+ PATH : os.environ[PATH]
+ }
+
Re: [gentoo-portage-dev] plugin-sync progress report
Am 15.03.2014 21:32, schrieb Brian Dolbec: I've started working on the repository/config.py changes needed for the plugin-sync system. First: The following class performed checks on the repos.conf entries for the sync variables regardless if they were being used or not. [...] Second: - Should all the repos.conf entires to be synced be validated prior to syncing and fail if there are any errors? - Or, just call each sync module's sync() and let each fail individually and continue syncing remaining repos? Maybe that's just me, but I don't like things that sometimes work and sometimes not. Having some emerge invocations fail with broken config and some not, seems rather ugly to me. (This also implies that syncing a repo with working config is not possible as long as there are repos with broken config.) Third: - I would like to add a new config item for all repos. It is a variable to define which sources of sync operations which the repo will be allowed to be synced from. It could be a space separated list which could be used to prevent it from being synced via certain commands. What exactly does this variable mean? What are emaint or layman doing here?
Re: [gentoo-portage-dev] plugin-sync progress report
On Sat, 15 Mar 2014 21:48:59 +0100 Sebastian Luther sebastianlut...@gmx.de wrote: Am 15.03.2014 21:32, schrieb Brian Dolbec: I've started working on the repository/config.py changes needed for the plugin-sync system. First: The following class performed checks on the repos.conf entries for the sync variables regardless if they were being used or not. [...] Second: - Should all the repos.conf entires to be synced be validated prior to syncing and fail if there are any errors? - Or, just call each sync module's sync() and let each fail individually and continue syncing remaining repos? Maybe that's just me, but I don't like things that sometimes work and sometimes not. Having some emerge invocations fail with broken config and some not, seems rather ugly to me. (This also implies that syncing a repo with working config is not possible as long as there are repos with broken config.) So, what your saying is EVERY time emerge is run or portage is imported for api use. It _MUST_ load the sync modules and verify the sync parameters regardless if those setting will be used or NOT. That is a waste of resources and time to me. Portage is already far too slow. I admit, that this will not be much of a time/resource consumer for the majority of gentoo systems. But what about low power systems, where resources including memory are at a premium? After all, how many times do you edit your repos.conf files? Why test them for every possible error every time portage code is started up? I was intending on adding a check-config option to the emaint sync module I planned on adding. I think that is a good compromise between over checking and no checking at all. Third: - I would like to add a new config item for all repos. It is a variable to define which sources of sync operations which the repo will be allowed to be synced from. It could be a space separated list which could be used to prevent it from being synced via certain commands. What exactly does this variable mean? What are emaint or layman doing here? emaint is here because I intend on making an emaint sync module for doing more detailed sync operations. Like syncing individual repos. If a user had an overlay that they did not want to be synced via emerge then not putting emerge there, when the emerge --sync command was started. It would skip syncing that repo, even though it had a sync-type and sync-url. Layman was an entry there since most overlays are installed managed by it. So, would have been a possible entry. Alex's solution seems like a good fit in place of this. See his email reply. -- Brian Dolbec dolsen
