date:20150116

[gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)

2015-01-16 Thread Zac Medico

This adds support for using a binary package's compression header to
determine the compression type, providing forward-compatibility for
xz and gzip decompression. The file name extension is disregared, so
that it will be possible to use a compression-independent file naming
scheme in the future (see bug #150031 for discussion about proposed
file naming schemes).

Currently, only decompression is supported. It's useful to provide
forward-compatibility now, so that binhost clients will be prepared
to handle future binhost servers that use xz or gzip compression.

X-Gentoo-Bug: 142579
X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=142579
---
 pym/_emerge/BinpkgExtractorAsync.py   | 25 ++---
 pym/portage/util/compression_probe.py | 68 +++
 2 files changed, 89 insertions(+), 4 deletions(-)
 create mode 100644 pym/portage/util/compression_probe.py

diff --git a/pym/_emerge/BinpkgExtractorAsync.py 
b/pym/_emerge/BinpkgExtractorAsync.py
index be74c2f..6aaa448 100644
--- a/pym/_emerge/BinpkgExtractorAsync.py
+++ b/pym/_emerge/BinpkgExtractorAsync.py
@@ -1,8 +1,13 @@
 # Copyright 1999-2013 Gentoo Foundation
 # Distributed under the terms of the GNU General Public License v2
 
+import logging
+
 from _emerge.SpawnProcess import SpawnProcess
 import portage
+from portage.localization import _
+from portage.util.compression_probe import (compression_probe,
+   _decompressors)
 import signal
 import subprocess
 
@@ -20,19 +25,31 @@ class BinpkgExtractorAsync(SpawnProcess):
if b--xattrs in output:
tar_options = --xattrs
 
-   # Add -q to bzip2 opts, in order to avoid trailing garbage 
after
-   # EOF ignored warning messages due to xpak trailer.
+   decomp_cmd = _decompressors.get(
+   compression_probe(self.pkg_path))
+   if decomp_cmd is None:
+   self.scheduler.output(!!! %s\n %
+   _(File compression header unrecognized: %s) %
+   self.pkg_path, log_path=self.logfile,
+   background=self.background, level=logging.ERROR)
+   self.returncode = 1
+   self._async_wait()
+   return
+
+   # Add -q to decomp_cmd opts, in order to avoid trailing garbage
+   # after EOF ignored warning messages due to xpak trailer.
# SIGPIPE handling (128 + SIGPIPE) should be compatible with
# assert_sigpipe_ok() that's used by the ebuild unpack() helper.
self.args = [self._shell_binary, -c,
-   (${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} 
-d} -cq -- %s | tar -xp %s -C %s -f - ;  + \
+   (%s -cq -- %s | tar -xp %s -C %s -f - ;  + \
p=(${PIPESTATUS[@]}) ;  + \
if [[ ${p[0]} != 0  ${p[0]} != %d ]] ; then  % (128 
+ signal.SIGPIPE) + \
echo bzip2 failed with status ${p[0]} ; exit ${p[0]} ; 
fi ;  + \
if [ ${p[1]} != 0 ] ; then  + \
echo tar failed with status ${p[1]} ; exit ${p[1]} ; 
fi ;  + \
exit 0 ;) % \
-   (portage._shell_quote(self.pkg_path),
+   (decomp_cmd,
+   portage._shell_quote(self.pkg_path),
tar_options,
portage._shell_quote(self.image_dir))]
 
diff --git a/pym/portage/util/compression_probe.py 
b/pym/portage/util/compression_probe.py
new file mode 100644
index 000..1dc3547
--- /dev/null
+++ b/pym/portage/util/compression_probe.py
@@ -0,0 +1,68 @@
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import errno
+import re
+import sys
+
+if sys.hexversion = 0x300:
+   basestring = str
+
+from portage import _encodings, _unicode_encode
+from portage.exception import FileNotFound, PermissionDenied
+
+_decompressors = {
+   bzip2: ${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d},
+   gzip: gzip -d,
+   xz: xz -d,
+}
+
+_compression_re = re.compile(b'^(' +
+   b'(?Pbzip2\x42\x5a\x68\x39)|' +
+   b'(?Pgzip\x1f\x8b)|' +
+   b'(?Pxz\xfd\x37\x7a\x58\x5a\x00))')
+
+def compression_probe(f):
+   
+   Identify the compression type of a file. Returns one of the
+   following identifier strings:
+
+   bzip2
+   gzip
+   xz
+
+   @param f: a file path, or file-like object
+   @type f: str or file
+   @return: a string identifying the compression type, or None if the
+   compression type is unrecognized
+   @rtype str or None
+   
+
+   open_file = isinstance(f, basestring)
+   if open_file:
+   try:
+   f =

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

2015-01-16 Thread Sergei Trofimovich

On Mon,  1 Dec 2014 22:28:34 +0100
Michał Górny mgo...@gentoo.org wrote:

 Disallow calling most of the ebuild helpers in global scope since they
 are meaningless in that context. Most of them are also prohibited by PMS
 for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
 that restriction.
 ---
  bin/eapi.sh | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)
 
 diff --git a/bin/eapi.sh b/bin/eapi.sh
 index e0ade02..5ab92f4 100644
 --- a/bin/eapi.sh
 +++ b/bin/eapi.sh
 @@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
  }
  
  ___eapi_disallows_helpers_in_global_scope() {
 - [[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
 + [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
  }
  
  ___eapi_unpack_is_case_sensitive() {

Looks like it cries on ebuilds with unset EAPI in main tree (attached).
Can be easily fixed in tree by adding EAPI=0, but better handle it
here: ${EAPI-0} (untested!)

It also changed binutils: it used to be slotted only in presence of USE flag.
Now it's always slotted even with USE=-multislot (at least eix thinks so).

-- 

  Sergei
 [31;01m*[0m ERROR: dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo failed (depend 
phase):
 [31;01m*[0m   use() calls are not allowed in global scope
 [31;01m*[0m 
 [31;01m*[0m Call stack:
 [31;01m*[0m   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jdk-bin/ibm-jdk-bin-1.6.0.9_p1.ebuild'
 [31;01m*[0m ERROR: dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo failed (depend 
phase):
 [31;01m*[0m   use() calls are not allowed in global scope
 [31;01m*[0m   ibm-jdk-bin-1.6.0.9_p1.ebuild, line  50:  Called use 'x86'
 [31;01m*[0m 
 [31;01m*[0m Call stack:
 [31;01m*[0m   ebuild.sh, line  47:  Called die
 [31;01m*[0m The specific snippet of code:
 [31;01m*[0m   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jdk-bin/ibm-jdk-bin-1.6.0.9_p2.ebuild'
 [31;01m*[0m  # These functions die because calls to them during the 
depend phase
 [31;01m*[0m ERROR: dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo failed (depend 
phase):
 [31;01m*[0m 
 [31;01m*[0m   use() calls are not allowed in global scope
 [31;01m*[0m 
 [31;01m*[0m   ibm-jdk-bin-1.6.0.9_p2.ebuild, line  50:  Called use 'x86'
 [31;01m*[0m Call stack:
 [31;01m*[0m If you need support, post the output of `emerge --info 
'=dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo'`,
 [31;01m*[0m the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jdk-bin-1.6.0.9_p1::gentoo'`.
 [31;01m*[0m   ebuild.sh, line  47:  Called die
 [31;01m*[0m The specific snippet of code:
 [31;01m*[0m Working directory: '/usr/lib64/python3.3/site-packages'
 [31;01m*[0m ERROR: dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo failed (depend 
phase):
 [31;01m*[0m S: 
'/tmp/portage-tmpdir/portage/dev-java/ibm-jdk-bin-1.6.0.9_p1/work/ibm-jdk-bin-1.6.0.9_p1'
 [31;01m*[0m   use() calls are not allowed in global scope
 [31;01m*[0m 
 [31;01m*[0m  # These functions die because calls to them during the 
depend phase
 [31;01m*[0m Call stack:
 [31;01m*[0m 
 [31;01m*[0m If you need support, post the output of `emerge --info 
'=dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo'`,
 [31;01m*[0m the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jdk-bin-1.6.0.9_p2::gentoo'`.
 [31;01m*[0m   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jre-bin/ibm-jre-bin-1.6.0.9_p2.ebuild'
 [31;01m*[0m   ebuild.sh, line 584:  Called source 
'/gentoo-32k/gentoo-x86/dev-java/ibm-jre-bin/ibm-jre-bin-1.6.0.9_p1.ebuild'
 [31;01m*[0m Working directory: '/usr/lib64/python3.3/site-packages'
 [31;01m*[0m S: 
'/tmp/portage-tmpdir/portage/dev-java/ibm-jdk-bin-1.6.0.9_p2/work/ibm-jdk-bin-1.6.0.9_p2'
 [31;01m*[0m   ibm-jre-bin-1.6.0.9_p2.ebuild, line  31:  Called use 'x86'
 [31;01m*[0m   ibm-jre-bin-1.6.0.9_p1.ebuild, line  31:  Called use 'x86'
 [31;01m*[0m   ebuild.sh, line  47:  Called die
 [31;01m*[0m   ebuild.sh, line  47:  Called die
 [31;01m*[0m The specific snippet of code:
 [31;01m*[0m The specific snippet of code:
 [31;01m*[0m  # These functions die because calls to them during the 
depend phase
 [31;01m*[0m  # These functions die because calls to them during the 
depend phase
 [31;01m*[0m 
 [31;01m*[0m 
 [31;01m*[0m If you need support, post the output of `emerge --info 
'=dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo'`,
 [31;01m*[0m If you need support, post the output of `emerge --info 
'=dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo'`,
 [31;01m*[0m the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jre-bin-1.6.0.9_p2::gentoo'`.
 [31;01m*[0m the complete build log and the output of `emerge -pqv 
'=dev-java/ibm-jre-bin-1.6.0.9_p1::gentoo'`.
 [31;01m*[0m Working directory:

Re: [gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)

2015-01-16 Thread Brian Dolbec

On Fri, 16 Jan 2015 00:30:31 -0800
Zac Medico zmed...@gentoo.org wrote:

 This adds support for using a binary package's compression header to
 determine the compression type, providing forward-compatibility for
 xz and gzip decompression. The file name extension is disregared, so
 that it will be possible to use a compression-independent file naming
 scheme in the future (see bug #150031 for discussion about proposed
 file naming schemes).
 
 Currently, only decompression is supported. It's useful to provide
 forward-compatibility now, so that binhost clients will be prepared
 to handle future binhost servers that use xz or gzip compression.
 
 X-Gentoo-Bug: 142579
 X-Gentoo-Bug-URL: https://bugs.gentoo.org/show_bug.cgi?id=142579
 ---


oooh, now that is sooo much better :D

Merge approved!


(P.S.  too pretty to erase the code below ;) 



  pym/_emerge/BinpkgExtractorAsync.py   | 25 ++---
  pym/portage/util/compression_probe.py | 68
 +++ 2 files changed, 89
 insertions(+), 4 deletions(-) create mode 100644
 pym/portage/util/compression_probe.py
 
 diff --git a/pym/_emerge/BinpkgExtractorAsync.py
 b/pym/_emerge/BinpkgExtractorAsync.py index be74c2f..6aaa448 100644
 --- a/pym/_emerge/BinpkgExtractorAsync.py
 +++ b/pym/_emerge/BinpkgExtractorAsync.py
 @@ -1,8 +1,13 @@
  # Copyright 1999-2013 Gentoo Foundation
  # Distributed under the terms of the GNU General Public License v2
  
 +import logging
 +
  from _emerge.SpawnProcess import SpawnProcess
  import portage
 +from portage.localization import _
 +from portage.util.compression_probe import (compression_probe,
 + _decompressors)
  import signal
  import subprocess
  
 @@ -20,19 +25,31 @@ class BinpkgExtractorAsync(SpawnProcess):
   if b--xattrs in output:
   tar_options = --xattrs
  
 - # Add -q to bzip2 opts, in order to avoid trailing
 garbage after
 - # EOF ignored warning messages due to xpak trailer.
 + decomp_cmd = _decompressors.get(
 + compression_probe(self.pkg_path))
 + if decomp_cmd is None:
 + self.scheduler.output(!!! %s\n %
 + _(File compression header
 unrecognized: %s) %
 + self.pkg_path, log_path=self.logfile,
 + background=self.background,
 level=logging.ERROR)
 + self.returncode = 1
 + self._async_wait()
 + return
 +
 + # Add -q to decomp_cmd opts, in order to avoid
 trailing garbage
 + # after EOF ignored warning messages due to xpak
 trailer. # SIGPIPE handling (128 + SIGPIPE) should be compatible with
   # assert_sigpipe_ok() that's used by the ebuild
 unpack() helper. self.args = [self._shell_binary, -c,
 -
 (${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d} -cq -- %s |
 tar -xp %s -C %s -f - ;  + \
 + (%s -cq -- %s | tar -xp %s -C %s -f - ;  +
 \ p=(${PIPESTATUS[@]}) ;  + \
   if [[ ${p[0]} != 0  ${p[0]} != %d ]] ;
 then  % (128 + signal.SIGPIPE) + \ echo bzip2 failed with status
 ${p[0]} ; exit ${p[0]} ; fi ;  + \ if [ ${p[1]} != 0 ] ; then  + \
   echo tar failed with status ${p[1]} ; exit
 ${p[1]} ; fi ;  + \ exit 0 ;) % \
 - (portage._shell_quote(self.pkg_path),
 + (decomp_cmd,
 + portage._shell_quote(self.pkg_path),
   tar_options,
   portage._shell_quote(self.image_dir))]
  
 diff --git a/pym/portage/util/compression_probe.py
 b/pym/portage/util/compression_probe.py new file mode 100644
 index 000..1dc3547
 --- /dev/null
 +++ b/pym/portage/util/compression_probe.py
 @@ -0,0 +1,68 @@
 +# Copyright 2015 Gentoo Foundation
 +# Distributed under the terms of the GNU General Public License v2
 +
 +import errno
 +import re
 +import sys
 +
 +if sys.hexversion = 0x300:
 + basestring = str
 +
 +from portage import _encodings, _unicode_encode
 +from portage.exception import FileNotFound, PermissionDenied
 +
 +_decompressors = {
 + bzip2:
 ${PORTAGE_BUNZIP2_COMMAND:-${PORTAGE_BZIP2_COMMAND} -d},
 + gzip: gzip -d,
 + xz: xz -d,
 +}
 +
 +_compression_re = re.compile(b'^(' +
 + b'(?Pbzip2\x42\x5a\x68\x39)|' +
 + b'(?Pgzip\x1f\x8b)|' +
 + b'(?Pxz\xfd\x37\x7a\x58\x5a\x00))')
 +
 +def compression_probe(f):
 + 
 + Identify the compression type of a file. Returns one of the
 + following identifier strings:
 +
 + bzip2
 + gzip
 + xz
 +
 + @param f: a file path, or file-like object
 + @type f: str or file
 + @return: a string identifying the compression type, or None
 if the
 + compression type is unrecognized
 + @rtype str or None
 + 
 +
 + open_file = isinstance(f, basestring)
 + if open_file:
 +

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

2015-01-16 Thread Zac Medico

On 01/16/2015 02:14 AM, Sergei Trofimovich wrote:
 On Mon,  1 Dec 2014 22:28:34 +0100
 Michał Górny mgo...@gentoo.org wrote:
 
 Disallow calling most of the ebuild helpers in global scope since they
 are meaningless in that context. Most of them are also prohibited by PMS
 for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
 that restriction.
 ---
  bin/eapi.sh | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/bin/eapi.sh b/bin/eapi.sh
 index e0ade02..5ab92f4 100644
 --- a/bin/eapi.sh
 +++ b/bin/eapi.sh
 @@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
  }
  
  ___eapi_disallows_helpers_in_global_scope() {
 -[[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
 +[[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
  }
  
  ___eapi_unpack_is_case_sensitive() {
 
 Looks like it cries on ebuilds with unset EAPI in main tree (attached).
 Can be easily fixed in tree by adding EAPI=0, but better handle it
 here: ${EAPI-0} (untested!)

Using ${EAPI-0} would not be the correct fix, because portage is
supposed to export the EAPI that was previously determined via the
_parse_eapi_ebuild_head function (preprocessor).
-- 
Thanks,
Zac

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

2015-01-16 Thread Zac Medico

On 01/16/2015 10:18 AM, Zac Medico wrote:
 On 01/16/2015 02:14 AM, Sergei Trofimovich wrote:
 On Mon,  1 Dec 2014 22:28:34 +0100
 Michał Górny mgo...@gentoo.org wrote:

 Disallow calling most of the ebuild helpers in global scope since they
 are meaningless in that context. Most of them are also prohibited by PMS
 for all EAPIs, so EAPI 6 sounds like a good moment to finally enforce
 that restriction.
 ---
  bin/eapi.sh | 2 +-
  1 file changed, 1 insertion(+), 1 deletion(-)

 diff --git a/bin/eapi.sh b/bin/eapi.sh
 index e0ade02..5ab92f4 100644
 --- a/bin/eapi.sh
 +++ b/bin/eapi.sh
 @@ -159,7 +159,7 @@ ___eapi_helpers_can_die() {
  }
  
  ___eapi_disallows_helpers_in_global_scope() {
 -   [[ ${1-${EAPI}} =~ ^(4-python|5-progress)$ ]]
 +   [[ ! ${1-${EAPI}} =~ ^(0|1|2|3|4|4-slot-abi|5|5-hdepend)$ ]]
  }
  
  ___eapi_unpack_is_case_sensitive() {

 Looks like it cries on ebuilds with unset EAPI in main tree (attached).
 Can be easily fixed in tree by adding EAPI=0, but better handle it
 here: ${EAPI-0} (untested!)
 
 Using ${EAPI-0} would not be the correct fix, because portage is
 supposed to export the EAPI that was previously determined via the
 _parse_eapi_ebuild_head function (preprocessor).

Actually, ebuild.sh unsets the EAPI just before it sources the ebuild,
and then it does this after it sources the ebuild:

  [ ${EAPI+set} = set ] || EAPI=0

So, for any code that is called while the ebuild is being sourced, using
${EAPI-0} would be correct.
-- 
Thanks,
Zac

[gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

Re: [gentoo-portage-dev] [PATCH] BinpkgExtractorAsync: xz and gzip decompression (142579)

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

Re: [gentoo-portage-dev] [PATCH 16/18] Disallow helpers in global scope in EAPI 6

5 matches

Site Navigation

Mail list logo

Footer information