Re: [gentoo-dev] Adding large files to the mirrors?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hiya, First off, thanks everybody for the suggestions. It seems like people aren't keen to mirror the large tables on our mirrors, but no one's mentioned a reason for not doing so. Is there an infra reason behind that or did everyone just take my caution and run with it? On 16/10/13 02:40, Chí-Thanh Christopher Nguyễn wrote: I suggest that you add a large USE flag, and if it is enabled, download and install the whole thing. If disabled, just print the URLs in a log message, so the user can download themselves if they wish. So downloading them manually is a pain (the larger tables aren't in a single zip, they're split amongst 12 files for each table), and the ebuild to do the downloading is already built. I'll include a postinst note indicating that the tables are getting stored twice, and I should even be able to give them an indication of how much space they're wasting. I'll also provide a URL they can visit if they want to download manually instead. The question then becomes, do I split the ebuild into two (giving us three ebuilds for what is otherwise a tiny package) just so some SRCs are mirror-restricted and others aren't? All of that hinges on whether our servers can handle the extra size... Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJeWR4ACgkQu7rWomwgFXq8bQCfRde/Tg7sAirqT05d5spckC+s fUIAoIH1SlXvLmM3CqM0x1vQN0oPdiKi =qqsa -END PGP SIGNATURE-
[gentoo-dev] Re: Adding large files to the mirrors?
Mike Auty posted on Wed, 16 Oct 2013 10:15:10 +0100 as excerpted: First off, thanks everybody for the suggestions. It seems like people aren't keen to mirror the large tables on our mirrors, but no one's mentioned a reason for not doing so. Is there an infra reason behind that or did everyone just take my caution and run with it? Based on previous related discussion, it's simply a reluctance to throw that much extra resource requirements at the folks who are after all /volunteering/ to host gentoo mirrors. We're invited guests whose continued presence is at their pleasure, and we shouldn't be taking undue advantage of the privileges we're given or the resources required to support us as such. If it's a core requirement, no problem. But the further from core requirement it gets the more sensitive infra seems to be about non- trivial space requirements, and any way you look at it, 30 gigs of rainbow tables are both non-core and non-trivial, especially with the split package and no-mirror on the big package solution, so... Beyond that, I suppose it'd be a topic for the mirror list. -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
Re: [gentoo-dev] Adding large files to the mirrors?
Il 16/10/2013 11:15, Mike Auty ha scritto: Hiya, [snip] So downloading them manually is a pain (the larger tables aren't in a single zip, they're split amongst 12 files for each table), and the ebuild to do the downloading is already built. I'll include a postinst note indicating that the tables are getting stored twice, and I should even be able to give them an indication of how much space they're wasting. I'll also provide a URL they can visit if they want to download manually instead. The question then becomes, do I split the ebuild into two (giving us three ebuilds for what is otherwise a tiny package) just so some SRCs are mirror-restricted and others aren't? All of that hinges on whether our servers can handle the extra size... Mike 5:) Add a bash script that can download all the files and mention it in postinst. the script will download all the files in current directory and put them in the correct place.
[gentoo-dev] Guidelines for using slot and sub-slot dependencies
Hello, all. Considering the past experiences wrt slot, sub-slot and slot operator dependencies, I'd like to write some kind of guidelines for using them. I'd like to know your opinion on my 'draft' and possibly someone more fluent in English to put them somewhere on the Wiki. First of all, please remember that slots and sub-slots are just generic tools. How they end up being used is up to the maintainer's discretion. Therefore, always check how the package is slotted and sub-slotted before depending on it. If it doesn't use sub-slots yet, use your common sense or -- when in doubt -- ask the maintainer. The goal of slot (and rarely sub-slot) dependencies is to limit the number of package versions that can satisfy a dependency. The goal of slot operator is to bind the package to the currently installed version of a dependency and rebuild it when uninstalling that version. So, the following questions need to be answered: 1. can any (current or possibly future) version of the package satisfy your dependency? 2. does your package need to rebuilt when slot/sub-slot changes? Of course, the answers depend on what exact purpose slots and sub-slots serve on a package. 1. You almost always want to depend on a specific slot of a library. This came up during the multilib work. If we are to support prebuilt binary software, we need to be able to install the specific versions of dependent libraries. As a result, so-called binary-only (runtime, compat) slots are introduced. An example of this is media-video/ffmpeg. It has a :0 slot that corresponds to a complete ffmpeg install, and a :0.10 slot that installs the libraries only. So every package that is being built against ffmpeg, need to depend on slot :0. It is likely that many more packages will grow binary-only slots in the future. Therefore, it seems reasonable to depend on slot :0 (or a similar single slot) already. Please specifically note that this not only applies to linking to a library but also to using the tools provided by it during run-time. So if your package uses 'ffmpeg' command-line tool, and depends on 'media-video/ffmpeg', portage will be able to uninstall the 'ffmpeg' executable and leave your package silently broken. 2. You most often want := deps when linking to libraries. Just check first how the library is sub-slotted. The commonly repeated example, app-text/poppler, uses sub-slot for its low-level library only. The high-level libraries that most programs use have stable APIs and aren't supposed to have := deps. If a particular package does not use sub-slots yet, use your common sense. If it installs a single library only, you can use := dep. If it installs libraries that changes their ABIs in approximately same intervals, you can use the := dep. Please remember to combine this with (1), so you'll often have :0= as the actual dep. 3. You *can* depend on a specific sub-slot. Most people seem not to know that but you can depend on sub-slots the same way you depend on slots. This makes it possible to have sub-slot versioned virtuals. For example, virtual/udev-206-r3 (sub-slot 0) depends on: || ( ... =sys-apps/systemd-206:0/1[...] =sys-apps/systemd-206:0/0[...] ) which means that systemd sub-slots 1 0 have the same ABI of libudev. Packages can depend on virtual/udev:= and expect the rebuild to be propagated. 4. You *can not* trust portage's --dynamic-deps anymore. This one is fairly important. In the past, we often assumed that portage will 'update' deps from ebuilds automatically. This is no longer correct if sub-slots are used -- since portage simply can't know which sub-slot of the dependency was used to build the package. This makes the point (1) above pretty important (see: the python-exec:2 issues). Since we don't have any proper tools to update dependencies of installed packages (there's a future-eapi bug for that [1]), we need to try harder to get them right. [1]:https://bugs.gentoo.org/show_bug.cgi?id=486778 -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Guidelines for using slot and sub-slot dependencies
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/16/2013 02:59 PM, Michał Górny wrote: -- when in doubt -- ask the maintainer. We should just require maintainers to document what their slots and subslots are for in the ebuild. 4. You *can not* trust portage's --dynamic-deps anymore. This one is fairly important. In the past, we often assumed that portage will 'update' deps from ebuilds automatically. This is no longer correct if sub-slots are used -- since portage simply can't know which sub-slot of the dependency was used to build the package. so we should always revbump when introducing subslots? -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/ iQEcBAEBAgAGBQJSXpBtAAoJEFpvPKfnPDWzMsoH/0CLsYBCRPk3vChM4tywIdBI sIGEzzmvYeQcHqt4yZNfrSDZF1ACxNk3mrJsyW3E4NLVXU8m13FKh26yPyxpLXiX BHZJ/+Yb4aiRRBNMY/39tI1MXm1QeVUmuYreVIeS0llpnfOKUFityJUwSKcoda96 sDxUcVwSLq+gPExKOpMV3aEc2moVUWWz/VPFaSRiKBDhbw61cfd+Vcg0pRWCmoT3 w5h1jPFF7z07RDtoz1dGJ0X3Ss4thd10pO2/bACSvuldgTCKbmcU3m42TWcGn+ya ZIyRtOu3QLKZTPuP2N/S1K8A6qR/gqz/OOxKZztxTVvy2DgTuwXYyVhSsXuPJ1c= =bF2i -END PGP SIGNATURE-
Re: [gentoo-dev] Guidelines for using slot and sub-slot dependencies
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 On 16/10/13 09:11 AM, hasufell wrote: On 10/16/2013 02:59 PM, Michał Górny wrote: 4. You *can not* trust portage's --dynamic-deps anymore. This one is fairly important. In the past, we often assumed that portage will 'update' deps from ebuilds automatically. This is no longer correct if sub-slots are used -- since portage simply can't know which sub-slot of the dependency was used to build the package. so we should always revbump when introducing subslots? Yes, but that's been recommended since the onset of EAPI=5 and sub-slot/slot-operator deps, so hopefully this is nothing new. -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iF4EAREIAAYFAlJeu24ACgkQ2ugaI38ACPAYEwD/S5I90mGXS3+z3jdflC5YoKi/ VUaL67/Q2SowHBqa+hcA/jtLhOwC7TCGansk5EYNpcCzMhDF1ALh49K18kGUldae =Nbuy -END PGP SIGNATURE-
Re: [gentoo-dev] Guidelines for using slot and sub-slot dependencies
Dnia 2013-10-16, o godz. 15:11:09 hasufell hasuf...@gentoo.org napisał(a): -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 10/16/2013 02:59 PM, Michał Górny wrote: -- when in doubt -- ask the maintainer. We should just require maintainers to document what their slots and subslots are for in the ebuild. I agree. However, this particular sentence was referring to packages that don't have sub-slots yet. 4. You *can not* trust portage's --dynamic-deps anymore. This one is fairly important. In the past, we often assumed that portage will 'update' deps from ebuilds automatically. This is no longer correct if sub-slots are used -- since portage simply can't know which sub-slot of the dependency was used to build the package. so we should always revbump when introducing subslots? If you want them retroactively, yes. Of course, sometimes you may just assume users will get them with the next version/revision. -- Best regards, Michał Górny signature.asc Description: PGP signature
Re: [gentoo-dev] Re: Adding large files to the mirrors?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 16/10/13 12:01, Duncan wrote: If it's a core requirement, no problem. But the further from core requirement it gets the more sensitive infra seems to be about non- trivial space requirements, and any way you look at it, 30 gigs of rainbow tables are both non-core and non-trivial, especially with the split package and no-mirror on the big package solution, so... No, that's fine. I wanted to check, because I don't take the resources for granted, but I do see us as providing a mirroring service for open source projects (to a degree), and no one so far had put forward an opinion about how infra would feel about it. Anyway, Francesco made an excellent point, and I think I'll be going with the additional bash script for grabbing the large tables... Thanks for you help everyone! 5:) Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJfGKkACgkQu7rWomwgFXo7dwCeIgp2WTS5vaBpG+zNTzdw0Q6g 4ocAoLUlnjIcZzcWzJ8Va1P5mdx65moP =Y1m5 -END PGP SIGNATURE-
Re: [gentoo-dev] Adding large files to the mirrors?
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Hiya, On 16/10/13 12:54, Francesco R. wrote: Il 16/10/2013 11:15, Mike Auty ha scritto: Add a bash script that can download all the files and mention it in postinst. the script will download all the files in current directory and put them in the correct place. That's an absolutely awesome suggestion, I'll do that! Thanks! 5:) Mike 5:) -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.22 (GNU/Linux) iEYEARECAAYFAlJfGK8ACgkQu7rWomwgFXrUIACfTTUcmqbApW4CqMlNd3YLC5yp pLwAn0hUHOEoG+noOMtgHqhBtKFb6tbT =E9tY -END PGP SIGNATURE-
[gentoo-portage-dev] [PATCH] xattr: centralize the various shims in one place
Rather than each module implementing its own shim around the various
methods for accessing extended attributes, start a dedicated module
that exports a consistent API.
---
bin/xattr-helper.py | 11 +--
pym/portage/util/_xattr.py | 189 +++
pym/portage/util/movefile.py | 99 ++-
3 files changed, 213 insertions(+), 86 deletions(-)
create mode 100644 pym/portage/util/_xattr.py
diff --git a/bin/xattr-helper.py b/bin/xattr-helper.py
index 6d99521..69b83f7 100755
--- a/bin/xattr-helper.py
+++ b/bin/xattr-helper.py
@@ -17,16 +17,7 @@ import re
import sys
from portage.util._argparse import ArgumentParser
-
-if hasattr(os, getxattr):
-
- class xattr(object):
- get = os.getxattr
- set = os.setxattr
- list = os.listxattr
-
-else:
- import xattr
+from portage.util._xattr import xattr
_UNQUOTE_RE = re.compile(br'\\[0-7]{3}')
diff --git a/pym/portage/util/_xattr.py b/pym/portage/util/_xattr.py
new file mode 100644
index 000..0e594f9
--- /dev/null
+++ b/pym/portage/util/_xattr.py
@@ -0,0 +1,189 @@
+# Copyright 2010-2013 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Portability shim for xattr support
+
+Exported API is the xattr object with get/get_all/set/remove/list operations.
+
+See the standard xattr module for more documentation.
+
+
+import contextlib
+import os
+import subprocess
+
+from portage.exception import OperationNotSupported
+
+
+class _XattrGetAll(object):
+ Implement get_all() using list()/get() if there is no easy bulk
method
+
+ @classmethod
+ def get_all(cls, item, nofollow=False, namespace=None):
+ return [(name, cls.get(item, name, nofollow=nofollow,
namespace=namespace))
+ for name in cls.list(item, nofollow=nofollow,
namespace=namespace)]
+
+
+class _XattrSystemCommands(_XattrGetAll):
+ Implement things with getfattr/setfattr
+
+ @staticmethod
+ def _parse_output(output):
+ for line in proc.stdout.readlines():
+ if line.startswith('#'):
+ continue
+ line = line.rstrip()
+ if not line:
+ continue
+ # The line will have the format:
+ # user.name0=value0
+ yield line.split('=', 1)
+
+ @classmethod
+ def get(item, name, nofollow=False, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['getfattr', '--absolute-names', '-n', name, item]
+ if nofollow:
+ cmd += ['-h']
+ proc = subprocess.call(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+ proc.wait()
+
+ value = None
+ for _, value in cls._parse_output(proc.stdout):
+ break
+
+ proc.stdout.close()
+ return value
+
+ @staticmethod
+ def set(item, name, value, flags=0, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['setfattr', '-n', name, '-v', value, item]
+ subprocess.call(cmd)
+
+ @staticmethod
+ def remove(item, name, nofollow=False, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['setfattr', '-x', name, item]
+ subprocess.call(cmd)
+
+ @classmethod
+ def list(cls, item, nofollow=False, namespace=None, _names_only=True):
+ cmd = ['getfattr', '-d', '--absolute-names', item]
+ if nofollow:
+ cmd += ['-h']
+ cmd += ['-m', ('^%s[.]' % namespace) if namespace else '']
+ proc = subprocess.call(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+ proc.wait()
+
+ ret = []
+ if namespace:
+ namespace = '%s.' % namespace
+ for name, value in cls._parse_output(proc.stdout):
+ if namespace:
+ if name.startswith(namespace):
+ name = name[len(namespace):]
+ else:
+ continue
+ if _names_only:
+ ret.append(name)
+ else:
+ ret.append((name, value))
+
+ proc.stdout.close()
+ return ret
+
+ @classmethod
+ def get_all(cls, item, nofollow=False, namespace=None):
+ cls.list(item, nofollow=nofollow, namespace=namespace,
_names_only=False)
+
+
+class _XattrStub(_XattrGetAll):
+ Fake
Re: [gentoo-portage-dev] [PATCH] xattr: centralize the various shims in one place
2013-10-16 23:03 Mike Frysinger napisał(a):
Rather than each module implementing its own shim around the various
methods for accessing extended attributes, start a dedicated module
that exports a consistent API.
Some things are incompatible with Python 3.
See other comments below.
---
bin/xattr-helper.py | 11 +--
pym/portage/util/_xattr.py | 189
+++
pym/portage/util/movefile.py | 99 ++-
3 files changed, 213 insertions(+), 86 deletions(-)
create mode 100644 pym/portage/util/_xattr.py
diff --git a/bin/xattr-helper.py b/bin/xattr-helper.py
index 6d99521..69b83f7 100755
--- a/bin/xattr-helper.py
+++ b/bin/xattr-helper.py
@@ -17,16 +17,7 @@ import re
import sys
from portage.util._argparse import ArgumentParser
-
-if hasattr(os, getxattr):
-
- class xattr(object):
- get = os.getxattr
- set = os.setxattr
- list = os.listxattr
-
-else:
- import xattr
+from portage.util._xattr import xattr
_UNQUOTE_RE = re.compile(br'\\[0-7]{3}')
diff --git a/pym/portage/util/_xattr.py b/pym/portage/util/_xattr.py
new file mode 100644
index 000..0e594f9
--- /dev/null
+++ b/pym/portage/util/_xattr.py
@@ -0,0 +1,189 @@
+# Copyright 2010-2013 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+Portability shim for xattr support
+
+Exported API is the xattr object with get/get_all/set/remove/list operations.
+
+See the standard xattr module for more documentation.
+
+
+import contextlib
+import os
+import subprocess
+
+from portage.exception import OperationNotSupported
+
+
+class _XattrGetAll(object):
+ Implement get_all() using list()/get() if there is no easy bulk
method
+
+ @classmethod
+ def get_all(cls, item, nofollow=False, namespace=None):
+ return [(name, cls.get(item, name, nofollow=nofollow,
namespace=namespace))
+ for name in cls.list(item, nofollow=nofollow,
namespace=namespace)]
+
+
+class _XattrSystemCommands(_XattrGetAll):
+ Implement things with getfattr/setfattr
+
+ @staticmethod
+ def _parse_output(output):
+ for line in proc.stdout.readlines():
NameError: global name 'proc' is not defined
+ if line.startswith('#'):
+ continue
+ line = line.rstrip()
+ if not line:
+ continue
+ # The line will have the format:
+ # user.name0=value0
+ yield line.split('=', 1)
+
+ @classmethod
+ def get(item, name, nofollow=False, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['getfattr', '--absolute-names', '-n', name, item]
+ if nofollow:
+ cmd += ['-h']
+ proc = subprocess.call(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+ proc.wait()
AttributeError: 'int' object has no attribute 'wait'
+
+ value = None
+ for _, value in cls._parse_output(proc.stdout):
+ break
+
+ proc.stdout.close()
+ return value
+
+ @staticmethod
+ def set(item, name, value, flags=0, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['setfattr', '-n', name, '-v', value, item]
Calling setfattr once per attribute is slower than once per file (as is now).
+ subprocess.call(cmd)
+
+ @staticmethod
+ def remove(item, name, nofollow=False, namespace=None):
+ if namespace:
+ name = '%s.%s' % (namespace, name)
+ cmd = ['setfattr', '-x', name, item]
+ subprocess.call(cmd)
+
+ @classmethod
+ def list(cls, item, nofollow=False, namespace=None, _names_only=True):
+ cmd = ['getfattr', '-d', '--absolute-names', item]
+ if nofollow:
+ cmd += ['-h']
+ cmd += ['-m', ('^%s[.]' % namespace) if namespace else '']
+ proc = subprocess.call(cmd, stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
+ proc.wait()
AttributeError: 'int' object has no attribute 'wait'
+
+ ret = []
+ if namespace:
+ namespace = '%s.' % namespace
+ for name, value in cls._parse_output(proc.stdout):
+ if namespace:
+ if name.startswith(namespace):
+ name = name[len(namespace):]
+ else:
+ continue
+ if _names_only:
+ ret.append(name)
+
Re: [gentoo-portage-dev] [PATCH] xattr: centralize the various shims in one place
On Wednesday 16 October 2013 20:02:50 Arfrever Frehtes Taifersar Arahesis wrote: 2013-10-16 23:03 Mike Frysinger napisał(a): Rather than each module implementing its own shim around the various methods for accessing extended attributes, start a dedicated module that exports a consistent API. Some things are incompatible with Python 3. See other comments below. i can run a linter on the code (probably should make this a git hook). i'm interested more in review on the bigger picture. also, please snip context you aren't actually replying to. it makes it much harder to read replies when you don't do that. + @staticmethod + def set(item, name, value, flags=0, namespace=None): + if namespace: + name = '%s.%s' % (namespace, name) + cmd = ['setfattr', '-n', name, '-v', value, item] Calling setfattr once per attribute is slower than once per file (as is now). true, however: - setfattr is the fallback logic, not the primary code path - the # of attributes per file is low (rarely more than 2) -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] xattr: centralize the various shims in one place
On Wednesday 16 October 2013 22:51:17 Mike Frysinger wrote: On Wednesday 16 October 2013 20:02:50 Arfrever Frehtes Taifersar Arahesis wrote: 2013-10-16 23:03 Mike Frysinger napisał(a): Rather than each module implementing its own shim around the various methods for accessing extended attributes, start a dedicated module that exports a consistent API. Some things are incompatible with Python 3. See other comments below. i can run a linter on the code (probably should make this a git hook). i'm interested more in review on the bigger picture. also, none of your comments were py3 issues that i saw -mike signature.asc Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] prepstrip: rewrite in python
On Wednesday 16 October 2013 19:26:04 Arfrever Frehtes Taifersar Arahesis
wrote:
2013-10-16 23:34 Mike Frysinger napisał(a):
+ raise subprocess.CalledProcessError('command failed
(ret=%i): %s' % (
+ proc.returncode, ' '.join(map(repr, cmd
A generator expression would look better than map().
i don't see how
i fixed the rest and ran it through a linter
-mike
signature.asc
Description: This is a digitally signed message part.
Re: [gentoo-portage-dev] [PATCH] xattr: centralize the various shims in one place
2013-10-17 04:53 Mike Frysinger napisał(a): On Wednesday 16 October 2013 22:51:17 Mike Frysinger wrote: On Wednesday 16 October 2013 20:02:50 Arfrever Frehtes Taifersar Arahesis wrote: 2013-10-16 23:03 Mike Frysinger napisał(a): Rather than each module implementing its own shim around the various methods for accessing extended attributes, start a dedicated module that exports a consistent API. Some things are incompatible with Python 3. See other comments below. i can run a linter on the code (probably should make this a git hook). i'm interested more in review on the bigger picture. also, none of your comments were py3 issues that i saw I said other comments, so I meant comments not related to incompatibility with Python 3. About incompatibility with Python 3: - subprocess.check_output(), subprocess.Popen().stdout.read(), subprocess.Popen().stderr.read() return bytes, which is incorrectly compared with str in your patches. - dict.iteritems() was renamed to dict.items() (and its return type was changed from dictionary-itemiterator to dict_items, but it does not matter here). - Queue module was renamed to queue. - cStringIO module should not be used. io module is a replacement available since Python 2.6. - Maybe other problems... -- Arfrever Frehtes Taifersar Arahesis signature.asc Description: This is a digitally signed message part.
