Re: [gentoo-dev] dev-lang/go
On Sat, 15 Feb 2014 16:44:09 -0600 William Hubbs willi...@gentoo.org wrote: On Sat, Feb 15, 2014 at 12:48:44AM +0100, yac wrote: On Fri, 14 Feb 2014 11:02:49 -0500 Emery Hemingway em...@vfemail.net wrote: The default GOROOT that go looks at for base libraries seems to be compiled in so this should be pretty easy, like python but simplier. I'm not sure what you are trying to solve here. Afaik GOROOT is used to determine where to install and it can be overriden from env. Not overridden, but extended. See go help gopath. An eclass could look at a GO_MINIMUM variable and install for each version go that is present and matches. It might be good idea to learn from others who'd been through this and I think the new python eclasses are good ones, going with something like PYTHON_TARGETS array (GOLANG_TARGETS ?) I would prefer go_targets if this becomes an issue, golang is more search friendly but it isn't at this point because there is only one target, go1, and we do not know if there will be a go2 or not. There still are different compilers at least, even if changing minor version would be a non-issue. But I'm not familiar with those, I think those are used for compiling for other than the supported archs (iirc only x86 and x86_64) Dropping old versions of go will be easy because linking wont break, and new releases should be forwards compatible. So far yes I think but I guess that may be quite different with in the future with 1.x, and should be so there may be corner cases where the user does need to use earlier version. Highly unlikely in the context of go1, and again, we don't know if there is going to even be a go2 or not. The only reason there will be a go2 is if there needs to be a change at the source level which can only be done in a backward incompatible way. The question really should be, do we want a system-wide workspace to store third-party libraries [1]? and if so, where do we put it -- maybe /usr/lib/go-gentoo should exist along side /usr/lib/go? I assume you are talking about thirdparty packages installed by portage, not by localy/manually by user. Well, without the system-wide workspace to store the libraries, this whole go eclass would be kinda pointless, no? Currently /usr/lib/go/gentoo is used and I see no reason to change it. The catch would be that every time you upgrade dev-lang/go, everything stored in /usr/lib/go-gentoo has to be recompiled because there is no guarantee that the libraries we have there are compatible with each minor release of go1, only the source. Then, the executables we have in /usr/bin will still run, but it would be good to rebuild them as well to get the new libraries linked into them. If we had a work space in, say, /usr/lib/go-gentoo, we could leave the executables in there and symlink them to /usr/bin. If we did that, it would be easy for a user to rebuild everything in the workspace for the new go by doing emerge /usr/lib/go-gentoo/bin Good idea. Thoughts? William [1] http://golang.org/doc/code.html --- Jan Matějka| Gentoo Developer https://gentoo.org | Gentoo Linux GPG: A33E F5BC A9F6 DAFD 2021 6FB6 3EBF D45B EEB6 CA8B signature.asc Description: PGP signature
[gentoo-dev] first steps in gentoo devolment
Hello, I have updated a ebuild to a new version. Am I right I have to make a local repo to test if it's build. And does ebuild x.ebuild manifest also builds the package or do I have to use emerge x for that. And for testing Is repoman scan enough after looking if the package can installed of course. I have looked into the manuals but I cannot find a clear answer. Roelof --- Dit e-mailbericht bevat geen virussen en malware omdat avast! Antivirus-bescherming actief is. http://www.avast.com
Re: [gentoo-dev] RFC: GTK USE flag situation (gtk, gtk2, gtk3; relevant to bug #420493)
18.02.2014 13:39, Ben de Groot пишет: On 12 February 2014 07:04, Samuli Suominen ssuomi...@gentoo.org wrote: [...] It's sad that people don't follow common sense (which happens to be the GNOME highlights) and that everything must be turned into a policy of somesort so people get it. [...] Just make the gnome gtk3 policy the guideline if you must. It's just documenting common sense. It seems that only the Gnome team regards this as common sense. Others, such as the Qt team and QA regard the versioned useflag solution as more sensible. I think we can conclude that there is no perfect solution, and we simply have different preferences. That said, I'm not sure we absolutely need a policy. Though I would agree it would be more elegant if we can solve this matter, to make it easier to grasp by our users. In that case the QA proposal makes sense to me. We definitely NEED the policy, cause if we do not - current situation will continue to confuse users and even developers about how to do things properly. But we should come to the policy together and then - force it together. We really do not want to force anyone in a way he does not like, but if we will do nothing about this - it will remain a problem as it is now. -- Best regards, Sergey Popov Gentoo developer Gentoo Desktop-effects project lead Gentoo Qt project lead Gentoo Proxy maintainers project lead signature.asc Description: OpenPGP digital signature
Re: [gentoo-dev] first steps in gentoo devolment
On Wed, 19 Feb 2014 17:05:09 +0100, Roelof Wobben r.wob...@home.nl wrote: Hello, I have updated a ebuild to a new version. Am I right I have to make a local repo to test if it's build. And does ebuild x.ebuild manifest also builds the package or do I have to use emerge x for that. And for testing Is repoman scan enough after looking if the package can installed of course. I have looked into the manuals but I cannot find a clear answer. Roelof --- Dit e-mailbericht bevat geen virussen en malware omdat avast! Antivirus-bescherming actief is. http://www.avast.com `ebuild ... manifest` only downloads the files listed in SRC_URI of the ebuild and creates/updates the file 'Manifest' in the same directory as the ebuild file. The same can also be accomplished by running `repoman manifest` from the directory containing the ebuild file. To build the package, you can use emerge like you would for a main-tree package. This requires you to set PORTDIR_OVERLAY in make.conf (or your environment) to point to your overlay. It also requires superuser access for most actions. Alternatively, you can run individual phases with `ebuild ... $command` (see ebuild(1) [1] for the right commands for the different phases). Prerequisite phases are run for most commands, notably excepting 'qmerge'. Dependencies are not automatically installed when using `ebuild` directly, so you must first make sure they are installed, for example using `emerge --oneshot --noreplace list of dependencies`. repoman does not run the code in the /functions/ of an ebuild (src_configure and so on), but only performs static analysis on them. Thus, problems arising from actually running the package's build system can not be caught this way. You should probably also verify that the stuff that gets installed by your ebuild actually works before you share it with others, which would require you to actually build and install it to your system as described above. 1: https://dev.gentoo.org/~zmedico/portage/doc/man/ebuild.1.html -- eroen signature.asc Description: PGP signature
[gentoo-dev] February 2014 QA policy updates
Hello all, The following are the policy changes from this month's QA team meeting: -USE=multislot (and other USE-dependent SLOT values) need to be removed from the tree. Toolchain can keep it in an overlay if they want. We would consider it acceptable to remove USE=multislot from the tree but keep the eclasses as-is, so that toolchain doesn't need to maintain multiple eclasses. This does not affect sys-boot/grub's USE=multislot, as that does not mangle the SLOT value like the others (as I understand it). -Regarding the gtk/gtk2/gtk3 USE flag situation: we mandate that gtk move to versioned USE flags. For simplicity of migration, we will allow USE=gtk to mean depend on gtk2, since there are only a few USE=gtk2 remaining in tree. USE=gtk3 will mean depend on gtk3, and in the future, USE=gtk4 will mean depend on gtk4, and so on. USE=gtk may not be used to mean depend on some version of gtk. -More generally: we recommend that in future situations like this (a package can optionally depend on different versions of a library), we recommend the use of versioned USE flags. It should be discussed with the QA team either way. Also, on a non-policy note, we recommend that the Council deprecate EAPIs 0 and 3 (0 pending discussion with toolchain) and ban EAPI 1. As always, if you have questions, feel free to ping us in #gentoo-qa. The meeting summary and these policies will be available on the Quality Assurance page on the Gentoo Wiki tonight or tomorrow. Chris Reffett Gentoo QA Lead
[gentoo-dev] RFD: new global USE flag gtk3
Following up to today's QA meeting: The gtk3 USE flag is used by 27 packages, so I suggest making it a global flag: gtk3 - Add support for x11-libs/gtk+ (The GIMP Toolkit) version 3 Ulrich pgpo0uXzcQQml.pgp Description: PGP signature
[gentoo-dev] Re: Thank you
Canek Peláez Valdés posted on Thu, 06 Feb 2014 00:30:10 -0600 as excerpted: TL;DR, this is basically just a THANK YOU to the Gentoo devs, so you can go on your daily business if you don't want to read the rest of it. No biggie. Along these same lines... Who helps your Linux distribution run smoothly? Thank a packager today http://opensource.com/business/14/2/thank-a-linux-packager-today Quoting... In many cases, the process of packaging uncovers issues with the package that require the upstream developers to make changes and adjustments. A packager also works in close coordination with other packagers in the same Linux distribution because many packages have dependencies on other packages or provide services for other packages, making it vital that the community of packagers coordinate their updates to ensure the consistency of the final Linux distribution. As Linux users, it is often easy to forget (disregard?) how much work goes into the creation and maintenance of a Linux distribution. [...] After having learned the ropes of Linux packaging, and having seen first hand the dedication of this community, I developed a great deal of respect and appreciation for their work. Now, every time I install a package [...] I pause and think: Thank you to the person who spent many hours configuring and building this application so that I didn’t have to. End quote. So here I am. Thanks! =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] dev-lang/go
On Wed, Feb 19, 2014 at 12:34:22PM +0100, yac wrote: On Sat, 15 Feb 2014 16:44:09 -0600 William Hubbs willi...@gentoo.org wrote: On Sat, Feb 15, 2014 at 12:48:44AM +0100, yac wrote: On Fri, 14 Feb 2014 11:02:49 -0500 Emery Hemingway em...@vfemail.net wrote: The default GOROOT that go looks at for base libraries seems to be compiled in so this should be pretty easy, like python but simplier. I'm not sure what you are trying to solve here. Afaik GOROOT is used to determine where to install and it can be overriden from env. Not overridden, but extended. See go help gopath. An eclass could look at a GO_MINIMUM variable and install for each version go that is present and matches. It might be good idea to learn from others who'd been through this and I think the new python eclasses are good ones, going with something like PYTHON_TARGETS array (GOLANG_TARGETS ?) I would prefer go_targets if this becomes an issue, golang is more search friendly but it isn't at this point because there is only one target, go1, and we do not know if there will be a go2 or not. There still are different compilers at least, even if changing minor version would be a non-issue. But I'm not familiar with those, I think those are used for compiling for other than the supported archs (iirc only x86 and x86_64) Also add arm to the supported list along with amd64/x86-fbsd. The only other compiler is gccgo [1], but it is part of the gcc toolchain, so I have no idea what they are doing there, or if our toolchain guys are even supporting it. Also, if you look at the comments on that page, it is a bit behind dev-lang/go. It is not clear to me either whether gccgo works on other architectures, and whether it is a compiler or a front end like the old cfront used to be for c++. Also, I don't know whether it will even share the same libraries as dev-lang/go; it may just use standard libraries stored in /usr/lib. In short, I have no clue how gccgo works. ;-) Dropping old versions of go will be easy because linking wont break, and new releases should be forwards compatible. So far yes I think but I guess that may be quite different with in the future with 1.x, and should be so there may be corner cases where the user does need to use earlier version. Highly unlikely in the context of go1, and again, we don't know if there is going to even be a go2 or not. The only reason there will be a go2 is if there needs to be a change at the source level which can only be done in a backward incompatible way. The question really should be, do we want a system-wide workspace to store third-party libraries [1]? and if so, where do we put it -- maybe /usr/lib/go-gentoo should exist along side /usr/lib/go? I assume you are talking about thirdparty packages installed by portage, not by localy/manually by user. Well, without the system-wide workspace to store the libraries, this whole go eclass would be kinda pointless, no? Currently /usr/lib/go/gentoo is used and I see no reason to change it. Well, I was suggesting go-gentoo to keep third party libraries out of the standard go tree and based on the definition of a workspace from the go documentation. The catch would be that every time you upgrade dev-lang/go, everything stored in /usr/lib/go-gentoo has to be recompiled because there is no guarantee that the libraries we have there are compatible with each minor release of go1, only the source. Then, the executables we have in /usr/bin will still run, but it would be good to rebuild them as well to get the new libraries linked into them. If we had a work space in, say, /usr/lib/go-gentoo, we could leave the executables in there and symlink them to /usr/bin. If we did that, it would be easy for a user to rebuild everything in the workspace for the new go by doing emerge /usr/lib/go-gentoo/bin Good idea. Thoughts? William [1] http://golang.org/doc/code.html --- Jan Matějka| Gentoo Developer https://gentoo.org | Gentoo Linux GPG: A33E F5BC A9F6 DAFD 2021 6FB6 3EBF D45B EEB6 CA8B [1] http://golang.org/doc/install/gccgo signature.asc Description: Digital signature
Re: [gentoo-dev] RFD: new global USE flag gtk3
On 20/02/14 00:23, Ulrich Mueller wrote: Following up to today's QA meeting: The gtk3 USE flag is used by 27 packages, so I suggest making it a global flag: gtk3 - Add support for x11-libs/gtk+ (The GIMP Toolkit) version 3 Ulrich that would suggest it's fine to use, and is anything but temporary -1 from here
Re: [gentoo-dev] RFD: new global USE flag gtk3
On Thu, 2014-02-20 at 07:55 +0200, Samuli Suominen wrote: On 20/02/14 00:23, Ulrich Mueller wrote: Following up to today's QA meeting: The gtk3 USE flag is used by 27 packages, so I suggest making it a global flag: gtk3 - Add support for x11-libs/gtk+ (The GIMP Toolkit) version 3 Ulrich that would suggest it's fine to use, and is anything but temporary -1 from here MATE desktop (which I hope to bring in to Portage soon) can be built against gtk+ 2 or gtk+ 3, and upstream supports doing both, so +1 from me. Just because gtk+ 3 is the latest, does not mean it's the greatest, and I really wish people would realize that newest != bestest.
Re: [gentoo-portage-dev] [PATCH v2] Add --output-style option to repoman
On 02/13/2014 10:42 AM, Brian Dolbec wrote:
On Thu, 13 Feb 2014 03:19:35 -0500
Mike Frysinger vap...@gentoo.org wrote:
On Monday, February 10, 2014 20:22:36 Chris Reffett wrote:
This patch adds a --output-style option to repoman, which gives the
user a choice of output formats for the repoman checks. Choices are
default (current style) and column (a greppable format), but it
should be easy to add more. Fixes bug 481584.
i'd expect a proper structured output would make sense to include in
the default set. like JSON. just create a dict and send it to
json.dump().
He is working on more changes to repoman and the output. So, if you
can, Chris, then do it, add a json option.
Will do that after my next set of changes to repoman (to be emailed shortly)
v2: Fix docstring to be complete and in the standard format, make
use of default choices in --output-style wrt comments by antarus
and dol-sen
erm, i thought the previous docstring was correct. it followed
PEP257 while this new one is like javadoc or something.
It is the existing format that has been around in portage for years.
There is even a page for it:
http://www.gentoo.org/proj/en/portage/doc/policies/docstring-spec.xml
It is also the style that epydoc recognizes.
-utilities.format_qa_output(f, stats, fails, dofull, dofail,
options, qawarnings)
+if options.output_style == 'column':
+ utilities.format_qa_output_column(f, stats, fails, dofull,
dofail, options, qawarnings)
+else:
+ utilities.format_qa_output(f, stats, fails, dofull,
dofail, options, qawarnings)
use a func pointer instead.
format_outputs = {
'column': utilities.format_qa_output_column,
'default': utilities.format_qa_output,
}
format_output = format_outputs.get(options.output_style,
format_outputs['default'])
format_output(f, stats, fails, dofull, dofail, options, qawarnings)
yeah, make it so. Good spot, Mike
Committed, thanks for the spot.
Since Mike was too slow in replying, make another commit to change
it.
+ formatter.add_literal_data(NumberOf + category
+ )
prefer to use % rather than + like so:
'NumberOf %s ' % category
+ formatter.add_literal_data(%s % number)
well actually, for simple additions like that, string1 + string2, it is
actually faster.
But for multiple additions, %s is much better, faster. Also if the
string is translated, then use %s regardless. That way the %s can be
moved around for the translation.
str(number)
-mike
[gentoo-portage-dev] [PATCH 0/2] Refactor repoman QA handling
This series of patches alters repoman's QA output to be much more usable. The first patch changes all checks to output a list of either length 1 or 2, splitting the file with the error from the error itself. This will be helpful for making machine-parseable output in the future. The second both makes the variables used to build the output much more consistent and makes the output itself more consistent by removing a few instances where the full path was printed rather than the relative path. This will change the existing repoman output format and potentially break scripts which rely on the old and inconsistent behavior. Chris Reffett (2): Split output for repoman checks into file and message Repoman check code cleanup bin/repoman | 232 +++ pym/repoman/utilities.py | 18 +++- 2 files changed, 128 insertions(+), 122 deletions(-) -- 1.8.5.3
Re: [gentoo-portage-dev] [PATCH 1/2] Split output for repoman checks into file and message
On Wed, 19 Feb 2014 13:10:04 -0500
Chris Reffett creff...@gentoo.org wrote:
This wraps the output of emerge checks so that a list of length 1-2 is
generated. The first element is the file, the second element (optional)
is a more descriptive error message. This change will help us eventually
introduce more machine-readable output formats.
---
bin/repoman | 232
+++
pym/repoman/utilities.py | 18 +++-
2 files changed, 128 insertions(+), 122 deletions(-)
diff --git a/bin/repoman b/bin/repoman
index 92b..3d5dde4 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -1402,7 +1402,7 @@ for x in effective_scanlist:
repoman_settings['PORTAGE_QUIET'] = '1'
if not portage.digestcheck([], repoman_settings, strict=1):
stats[manifest.bad] += 1
- fails[manifest.bad].append(os.path.join(x,
'Manifest'))
+ fails[manifest.bad].append([os.path.join(x,
'Manifest')])
repoman_settings.pop('PORTAGE_QUIET', None)
This looks so,so to me. I think you are much better off using a
namedtuple class for these errors instead. They have built-in
formatted printing, etc.
from collections import namedtuple
and you can have pre-defined named tuple classes that have the error
message already embedded.
for some examples see the pyGPG ones I created for gpg data mining:
https://github.com/dol-sen/pyGPG/blob/master/pyGPG/legend.py
NOTE: CLASSES is a list of tuples that have the info to define the
classes that subclass the namedtuple class. They are initialized by
the code at the bottom of the page when the page is first loaded into
memory. This format saves writing out each individual class by hand
and makes changes easy. It also reduced the size of the file to about
1/3. I have done similar to tis in 3 modules in the catalys rewrite
as well.
The data is then available in many ways and will have a consistent
output.
--
Brian Dolbec dolsen
Re: [gentoo-portage-dev] [PATCH 1/2] Split output for repoman checks into file and message
On Wed, 19 Feb 2014 10:33:15 -0800
Brian Dolbec dol...@gentoo.org wrote:
On Wed, 19 Feb 2014 13:10:04 -0500
Chris Reffett creff...@gentoo.org wrote:
This wraps the output of emerge checks so that a list of length 1-2 is
generated. The first element is the file, the second element (optional)
is a more descriptive error message. This change will help us eventually
introduce more machine-readable output formats.
---
bin/repoman | 232
+++
pym/repoman/utilities.py | 18 +++-
2 files changed, 128 insertions(+), 122 deletions(-)
diff --git a/bin/repoman b/bin/repoman
index 92b..3d5dde4 100755
--- a/bin/repoman
+++ b/bin/repoman
@@ -1402,7 +1402,7 @@ for x in effective_scanlist:
repoman_settings['PORTAGE_QUIET'] = '1'
if not portage.digestcheck([], repoman_settings, strict=1):
stats[manifest.bad] += 1
- fails[manifest.bad].append(os.path.join(x,
'Manifest'))
+ fails[manifest.bad].append([os.path.join(x,
'Manifest')])
repoman_settings.pop('PORTAGE_QUIET', None)
This looks so,so to me. I think you are much better off using a
namedtuple class for these errors instead. They have built-in
formatted printing, etc.
from collections import namedtuple
and you can have pre-defined named tuple classes that have the error
message already embedded.
for some examples see the pyGPG ones I created for gpg data mining:
https://github.com/dol-sen/pyGPG/blob/master/pyGPG/legend.py
NOTE: CLASSES is a list of tuples that have the info to define the
classes that subclass the namedtuple class. They are initialized by
the code at the bottom of the page when the page is first loaded into
memory. This format saves writing out each individual class by hand
and makes changes easy. It also reduced the size of the file to about
1/3. I have done similar to tis in 3 modules in the catalys rewrite
as well.
The data is then available in many ways and will have a consistent
output.
for smaller simpler examples:
http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=blob;f=catalyst/hash_utils.py;h=cd31ad3edbdd412c0da4d968596777a71fbd4beb;hb=refs/heads/3.0
http://git.overlays.gentoo.org/gitweb/?p=proj/catalyst.git;a=blob;f=catalyst/contents.py;h=1a9ed28a58cc63ed954ca8bdbcd1b594e8a7f2e5;hb=refs/heads/3.0
--
Brian Dolbec dolsen
Re: [gentoo-portage-dev] [PATCH] portdbapi: Add cache to avoid repeated failing os.access calls
Am 18.02.2014 22:35, schrieb David James: + cache_key = (mycpv, mytree, myrepo) + try: + return self._findname2_cache[cache_key] + except KeyError: + self._findname2_cache[cache_key] = (None, 0) To me, it seems potentially error-prone to cache a (potentially) incorrect value and then correct it later. It is. The problem are all these returns. The whole thing would need to be reorganized to fix this. I'd rather go for a memoize decorator and leave that thing alone. If I just could find a usable one. Would it be possible to refactor your patch so that we only cache the value when we know the final answer? + except TypeError: In what circumstances does it happen that mytree / myrepo are unhashable types? Can you add a comment to explain this? That's my mistake. I was under the impression that mytree is actually mytreeS and would accept a list. I'll remove the except TypeError: in a future version of the patch.
[gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.
---
pym/portage/emaint/modules/merges/__init__.py | 20
pym/portage/emaint/modules/merges/merges.py | 70 +++
2 files changed, 90 insertions(+)
create mode 100644 pym/portage/emaint/modules/merges/__init__.py
create mode 100644 pym/portage/emaint/modules/merges/merges.py
diff --git a/pym/portage/emaint/modules/merges/__init__.py
b/pym/portage/emaint/modules/merges/__init__.py
new file mode 100644
index 000..2cd79af
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Scan for failed merges and fix them.
+
+
+
+module_spec = {
+ 'name': 'merges',
+ 'description': __doc__,
+ 'provides':{
+ 'module1': {
+ 'name': merges,
+ 'class': MergesHandler,
+ 'description': __doc__,
+ 'functions': ['check', 'fix',],
+ 'func_desc': {}
+ }
+ }
+ }
diff --git a/pym/portage/emaint/modules/merges/merges.py
b/pym/portage/emaint/modules/merges/merges.py
new file mode 100644
index 000..b243082
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/merges.py
@@ -0,0 +1,70 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage import os
+from portage.const import PRIVATE_PATH, VDB_PATH
+from portage.util import writemsg
+
+import shutil
+import sys
+
+if sys.hexversion = 0x300:
+ # pylint: disable=W0622
+ long = int
+
+class MergesHandler(object):
+
+ short_desc = Remove failed merges
+
+ def name():
+ return merges
+ name = staticmethod(name)
+
+ def __init__(self):
+ self._eroot = portage.settings['EROOT']
+ self._vardb = portage.db[self._eroot][vartree].dbapi
+ self._vardb_path = os.path.join(self._eroot, VDB_PATH) +
os.path.sep
+
+ def can_progressbar(self, func):
+ return True
+
+ def _failed_packages(self, onProgress):
+ for cat in os.listdir(self._vardb_path):
+ pkgs = os.listdir(self._vardb_path + cat)
+ maxval = len(pkgs)
+ for i, pkg in enumerate(pkgs):
+ if onProgress:
+ onProgress(maxval, i+1)
+
+ if '-MERGING-' in pkg:
+ yield cat + os.path.sep + pkg
+
+ def check(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ failed_pkgs = []
+ for pkg in self._failed_packages(onProgress):
+ failed_pkgs.append(pkg)
+
+ errors = ['%s' failed to merge. % x for x in failed_pkgs]
+ return errors
+
+ def fix(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ tracking_path = os.path.join(self._eroot, PRIVATE_PATH,
'failed-merges');
+ try:
+ with open(tracking_path, 'w') as tracking_file:
+ for failed_pkg in
self._failed_packages(onProgress):
+ tracking_file.write(failed_pkg
+ '\n')
+ pkg_path = self._vardb_path +
failed_pkg
+ # Delete failed merge directory
+ # XXX: Would be a good idea to
attempt try removing
+ # package contents to prevent
orphaned files
+ shutil.rmtree(pkg_path)
+ # Re-emerge package
+ pkg_name = '=' +
failed_pkg.replace('-MERGING-', '')
+
features='FEATURES=-collision-detect -protect-owned'
+ emerge_cmd=emerge --verbose
--oneshot --complete-graph=y
+ os.system('%s %s %s' %
(features, emerge_cmd, pkg_name))
+ except Exception as ex:
+ writemsg('Unable to fix failed package: %s' % str(ex))
--
1.8.3.2
Re: [gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.
On Wed, Feb 19, 2014 at 12:31 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:
---
pym/portage/emaint/modules/merges/__init__.py | 20
pym/portage/emaint/modules/merges/merges.py | 70
+++
2 files changed, 90 insertions(+)
create mode 100644 pym/portage/emaint/modules/merges/__init__.py
create mode 100644 pym/portage/emaint/modules/merges/merges.py
diff --git a/pym/portage/emaint/modules/merges/__init__.py
b/pym/portage/emaint/modules/merges/__init__.py
new file mode 100644
index 000..2cd79af
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Scan for failed merges and fix them.
+
Scan for failed merges fix them.
+
+
+module_spec = {
+ 'name': 'merges',
+ 'description': __doc__,
+ 'provides':{
'module1' ?
+ 'module1': {
+ 'name': merges,
+ 'class': MergesHandler,
+ 'description': __doc__,
+ 'functions': ['check', 'fix',],
+ 'func_desc': {}
+ }
+ }
+ }
diff --git a/pym/portage/emaint/modules/merges/merges.py
b/pym/portage/emaint/modules/merges/merges.py
new file mode 100644
index 000..b243082
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/merges.py
@@ -0,0 +1,70 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage import os
+from portage.const import PRIVATE_PATH, VDB_PATH
+from portage.util import writemsg
+
+import shutil
+import sys
+
+if sys.hexversion = 0x300:
+ # pylint: disable=W0622
+ long = int
What is this little guy? Can we just do this in a library someplace?
+
+class MergesHandler(object):
+
+ short_desc = Remove failed merges
+
@staticmethod decorator?
+ def name():
+ return merges
+ name = staticmethod(name)
+
+ def __init__(self):
Generally you want to be able to change these later, so you might do
something like:
def __init__(self, eroot=None, vardb=None, vardb_path=None):
self._eroot = error or portage.settings['EROOT']
... and so forth.
Also..why can't self._vardb_path be queried from the vardb?
+ self._eroot = portage.settings['EROOT']
+ self._vardb = portage.db[self._eroot][vartree].dbapi
+ self._vardb_path = os.path.join(self._eroot, VDB_PATH) +
os.path.sep
+
+ def can_progressbar(self, func):
+ return True
+
+ def _failed_packages(self, onProgress):
+ for cat in os.listdir(self._vardb_path):
os.listdir(os.path.join(...)) ?
+ pkgs = os.listdir(self._vardb_path + cat)
+ maxval = len(pkgs)
+ for i, pkg in enumerate(pkgs):
+ if onProgress:
+ onProgress(maxval, i+1)
+
+ if '-MERGING-' in pkg:
+ yield cat + os.path.sep + pkg
+
+ def check(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ failed_pkgs = []
+ for pkg in self._failed_packages(onProgress):
+ failed_pkgs.append(pkg)
+
+ errors = ['%s' failed to merge. % x for x in failed_pkgs]
+ return errors
+
+ def fix(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ tracking_path = os.path.join(self._eroot, PRIVATE_PATH,
'failed-merges');
+ try:
+ with open(tracking_path, 'w') as tracking_file:
is this unicode safe?
+ for failed_pkg in
self._failed_packages(onProgress):
+
tracking_file.write(failed_pkg + '\n')
+ pkg_path =
self._vardb_path + failed_pkg
os.path.join(...)
+ # Delete failed merge
directory
+ # XXX: Would be a good
idea to attempt try removing
+ # package contents to
prevent orphaned files
# XXX is terrible style. I realize a bunch of code does that, and it is
stupid.
# use
# TODO: foo
+ shutil.rmtree(pkg_path)
+ # Re-emerge package
+ pkg_name = '=' +
failed_pkg.replace('-MERGING-', '')
+
features='FEATURES=-collision-detect -protect-owned'
+ emerge_cmd=emerge
--verbose --oneshot
Re: [gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.
On Wed, 19 Feb 2014 12:31:44 -0800
Pavel Kazakov nullishz...@gentoo.org wrote:
---
pym/portage/emaint/modules/merges/__init__.py | 20
pym/portage/emaint/modules/merges/merges.py | 70
+++
2 files changed, 90 insertions(+)
create mode 100644 pym/portage/emaint/modules/merges/__init__.py
create mode 100644 pym/portage/emaint/modules/merges/merges.py
diff --git a/pym/portage/emaint/modules/merges/__init__.py
b/pym/portage/emaint/modules/merges/__init__.py
new file mode 100644
index 000..2cd79af
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Scan for failed merges and fix them.
+
+
+
+module_spec = {
+ 'name': 'merges',
+ 'description': __doc__,
+ 'provides':{
+ 'module1': {
+ 'name': merges,
+ 'class': MergesHandler,
+ 'description': __doc__,
+ 'functions': ['check', 'fix',],
+ 'func_desc': {}
+ }
+ }
+ }
diff --git a/pym/portage/emaint/modules/merges/merges.py
b/pym/portage/emaint/modules/merges/merges.py
new file mode 100644
index 000..b243082
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/merges.py
@@ -0,0 +1,70 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage import os
+from portage.const import PRIVATE_PATH, VDB_PATH
+from portage.util import writemsg
+
+import shutil
+import sys
+
+if sys.hexversion = 0x300:
+ # pylint: disable=W0622
+ long = int
+
+class MergesHandler(object):
+
+ short_desc = Remove failed merges
+
+ def name():
+ return merges
+ name = staticmethod(name)
+
+ def __init__(self):
+ self._eroot = portage.settings['EROOT']
+ self._vardb = portage.db[self._eroot][vartree].dbapi
+ self._vardb_path = os.path.join(self._eroot, VDB_PATH) +
os.path.sep
+
+ def can_progressbar(self, func):
+ return True
+
+ def _failed_packages(self, onProgress):
+ for cat in os.listdir(self._vardb_path):
+ pkgs = os.listdir(self._vardb_path + cat)
+ maxval = len(pkgs)
+ for i, pkg in enumerate(pkgs):
+ if onProgress:
+ onProgress(maxval, i+1)
+
+ if '-MERGING-' in pkg:
+ yield cat + os.path.sep + pkg
+
+ def check(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ failed_pkgs = []
+ for pkg in self._failed_packages(onProgress):
+ failed_pkgs.append(pkg)
+
+ errors = ['%s' failed to merge. % x for x in failed_pkgs]
+ return errors
+
+ def fix(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ tracking_path = os.path.join(self._eroot, PRIVATE_PATH,
'failed-merges');
+ try:
+ with open(tracking_path, 'w') as tracking_file:
+ for failed_pkg in
self._failed_packages(onProgress):
+ tracking_file.write(failed_pkg
+ '\n')
+ pkg_path = self._vardb_path +
failed_pkg
+ # Delete failed merge directory
+ # XXX: Would be a good idea to
attempt try removing
+ # package contents to prevent
orphaned files
+ shutil.rmtree(pkg_path)
+ # Re-emerge package
+ pkg_name = '=' +
failed_pkg.replace('-MERGING-', '')
+
features='FEATURES=-collision-detect -protect-owned'
+ emerge_cmd=emerge --verbose
--oneshot --complete-graph=y
+ os.system('%s %s %s' %
(features, emerge_cmd, pkg_name))
+ except Exception as ex:
+ writemsg('Unable to fix failed package: %s' % str(ex))
Really
? don't use os.system() It is nearly obsolete. use subprocess.call()
or Popen(). call() is a direct sub. Use Popen for piping output...
Also, it would be better to call emerge with all pkgs in one command.
I know it will rarely be more than 1, maybe 2 but, emerge is slow
enough just intitializing.
You might also want to turn off the progressbar for fix unless you
intend to pipe
Re: [gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.
On Wed, 19 Feb 2014 14:32:02 -0800
Alec Warner anta...@gentoo.org wrote:
On Wed, Feb 19, 2014 at 12:31 PM, Pavel Kazakov nullishz...@gentoo.orgwrote:
---
pym/portage/emaint/modules/merges/__init__.py | 20
pym/portage/emaint/modules/merges/merges.py | 70
+++
2 files changed, 90 insertions(+)
create mode 100644 pym/portage/emaint/modules/merges/__init__.py
create mode 100644 pym/portage/emaint/modules/merges/merges.py
diff --git a/pym/portage/emaint/modules/merges/__init__.py
b/pym/portage/emaint/modules/merges/__init__.py
new file mode 100644
index 000..2cd79af
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/__init__.py
@@ -0,0 +1,20 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Scan for failed merges and fix them.
+
Correct ^^
Scan for failed merges fix them.
No, your grammar is wrong
+
+
+module_spec = {
+ 'name': 'merges',
+ 'description': __doc__,
+ 'provides':{
'module1' ?
It is just an identifier, not used other than to group together
everything for that module. The plugin system can handle multiple
sub-modules within the same general module directory. See the
emaint/modules/move/__init__.py. Move supplies 2 submodules.
The new websync sync module (emerge-webrsync replacement) we started
roughing in also has 2 sub-modules, the 1st will be the module that
runs the original bash script.
The 2nd will be the new python converted code to replace it.
+ 'module1': {
+ 'name': merges,
+ 'class': MergesHandler,
+ 'description': __doc__,
+ 'functions': ['check', 'fix',],
+ 'func_desc': {}
+ }
+ }
+ }
diff --git a/pym/portage/emaint/modules/merges/merges.py
b/pym/portage/emaint/modules/merges/merges.py
new file mode 100644
index 000..b243082
--- /dev/null
+++ b/pym/portage/emaint/modules/merges/merges.py
@@ -0,0 +1,70 @@
+# Copyright 2005-2014 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import portage
+from portage import os
+from portage.const import PRIVATE_PATH, VDB_PATH
+from portage.util import writemsg
+
+import shutil
+import sys
+
+if sys.hexversion = 0x300:
+ # pylint: disable=W0622
+ long = int
What is this little guy? Can we just do this in a library someplace?
+
+class MergesHandler(object):
+
+ short_desc = Remove failed merges
+
@staticmethod decorator?
Yeah, that is copying legacy emaint code from the original module
classes.
+ def name():
+ return merges
+ name = staticmethod(name)
+
+ def __init__(self):
Generally you want to be able to change these later, so you might do
something like:
def __init__(self, eroot=None, vardb=None, vardb_path=None):
self._eroot = error or portage.settings['EROOT']
... and so forth.
The emaint code was not setup to handle init variable assignments.
None of the original module classes used any. The plugin system
doesn't care. But the TaskHandler class in main.py would need to be
modified. Also, all modules must use the same method of initializing,
regardless whether they need it or not. In the new sync modules all
relevant data is passed in using kwargs, then it saves any info it
needs.
Also..why can't self._vardb_path be queried from the vardb?
+ self._eroot = portage.settings['EROOT']
+ self._vardb = portage.db[self._eroot][vartree].dbapi
+ self._vardb_path = os.path.join(self._eroot, VDB_PATH) +
os.path.sep
+
+ def can_progressbar(self, func):
+ return True
+
+ def _failed_packages(self, onProgress):
+ for cat in os.listdir(self._vardb_path):
os.listdir(os.path.join(...)) ?
+ pkgs = os.listdir(self._vardb_path + cat)
+ maxval = len(pkgs)
+ for i, pkg in enumerate(pkgs):
+ if onProgress:
+ onProgress(maxval, i+1)
+
+ if '-MERGING-' in pkg:
+ yield cat + os.path.sep + pkg
+
+ def check(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+ failed_pkgs = []
+ for pkg in self._failed_packages(onProgress):
+ failed_pkgs.append(pkg)
+
+ errors = ['%s' failed to merge. % x for x in failed_pkgs]
+ return errors
+
+ def fix(self, **kwargs):
+ onProgress = kwargs.get('onProgress', None)
+
Re: [gentoo-portage-dev] [PATCH] Add an emaint module that can scan for failed merges and that can fix failed merges.
On 02/19/2014 02:50 PM, Brian Dolbec wrote: Really ? don't use os.system() It is nearly obsolete. use subprocess.call() or Popen(). call() is a direct sub. Use Popen for piping output... Good point, I'll switch over. Also, it would be better to call emerge with all pkgs in one command. I know it will rarely be more than 1, maybe 2 but, emerge is slow enough just intitializing. Yep, makes sense. You might also want to turn off the progressbar for fix unless you intend to pipe emerge's output and hide it. I might be inclined to just run emerge in minimum output mode. It will display what it is doing and any more failed builds/merges. I know I had the other modules output the data without displaying and let the cli handle any display. We should be able to do similar to the progressbar, and connect the emerge stdout pipe to a display code. That way any other frontend could do with the output what they like. I'll disable the progressbar for fix(), and look more into minimizing emerge output. Regards, Pavel
