[gentoo-portage-dev] [PATCH v6] Support escaping network-sandbox through SOCKSv5 proxy
Add a minimal SOCKSv5-over-UNIX-socket proxy to Portage, and start it
whenever ebuilds are started with network-sandbox enabled. Pass the
socket address in PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables.
The proxy can be used to escape the network sandbox whenever network
access is really desired, e.g. in distcc.
The proxy is based on asynchronous I/O using the asyncio module.
Therefore, it requires the asyncio module that is built-in in Python 3.4
and available stand-alone for Python 3.3. Escaping the sandbox is not
supported with older versions of Python.
The proxy supports connecting to IPv6 IPv4 TCP hosts. UDP and socket
binding are not supported. SOCKSv5 authentication schemes are not
supported (UNIX sockets provide a security layer).
---
bin/save-ebuild-env.sh | 5 +-
bin/socks5-server.py | 227 +
man/ebuild.5 | 5 +
man/make.conf.5| 7 +
.../package/ebuild/_config/special_env_vars.py | 2 +-
pym/portage/package/ebuild/doebuild.py | 11 +
pym/portage/util/socks5.py | 81
7 files changed, 335 insertions(+), 3 deletions(-)
create mode 100644 bin/socks5-server.py
create mode 100644 pym/portage/util/socks5.py
diff --git a/bin/save-ebuild-env.sh b/bin/save-ebuild-env.sh
index c6bffb5..477ed28 100644
--- a/bin/save-ebuild-env.sh
+++ b/bin/save-ebuild-env.sh
@@ -92,7 +92,7 @@ __save_ebuild_env() {
# portage config variables and variables set directly by portage
unset ACCEPT_LICENSE BAD BRACKET BUILD_PREFIX COLS \
- DISTCC_DIR DISTDIR DOC_SYMLINKS_DIR \
+ DISTCC_DIR DISTCC_SOCKS5_PROXY DISTDIR DOC_SYMLINKS_DIR \
EBUILD_FORCE_TEST EBUILD_MASTER_PID \
ECLASS_DEPTH ENDCOL FAKEROOTKEY \
GOOD HILITE HOME \
@@ -105,7 +105,8 @@ __save_ebuild_env() {
PORTAGE_DOHTML_WARN_ON_SKIPPED_FILES \
PORTAGE_NONFATAL PORTAGE_QUIET \
PORTAGE_SANDBOX_DENY PORTAGE_SANDBOX_PREDICT \
- PORTAGE_SANDBOX_READ PORTAGE_SANDBOX_WRITE PREROOTPATH \
+ PORTAGE_SANDBOX_READ PORTAGE_SANDBOX_WRITE \
+ PORTAGE_SOCKS5_PROXY PREROOTPATH \
QA_INTERCEPTORS \
RC_DEFAULT_INDENT RC_DOT_PATTERN RC_ENDCOL RC_INDENTATION \
ROOT ROOTPATH RPMDIR TEMP TMP TMPDIR USE_EXPAND \
diff --git a/bin/socks5-server.py b/bin/socks5-server.py
new file mode 100644
index 000..71e6b01
--- /dev/null
+++ b/bin/socks5-server.py
@@ -0,0 +1,227 @@
+#!/usr/bin/env python
+# SOCKSv5 proxy server for network-sandbox
+# Copyright 2015 Gentoo Foundation
+# Distributed under the terms of the GNU General Public License v2
+
+import asyncio
+import errno
+import os
+import socket
+import struct
+import sys
+
+
+class Socks5Server(object):
+
+ An asynchronous SOCKSv5 server.
+
+
+ @asyncio.coroutine
+ def handle_proxy_conn(self, reader, writer):
+
+ Handle incoming client connection. Perform SOCKSv5 request
+ exchange, open a proxied connection and start relaying.
+
+ @param reader: Read side of the socket
+ @type reader: asyncio.StreamReader
+ @param writer: Write side of the socket
+ @type writer: asyncio.StreamWriter
+
+
+ try:
+ # SOCKS hello
+ data = yield from reader.readexactly(2)
+ vers, method_no = struct.unpack('!BB', data)
+
+ if vers != 0x05:
+ # disconnect on invalid packet -- we have no
clue how
+ # to reply in alien :)
+ writer.close()
+ return
+
+ # ...and auth method list
+ data = yield from reader.readexactly(method_no)
+ for method in data:
+ if method == 0x00:
+ break
+ else:
+ # no supported method
+ method = 0xFF
+
+ # auth reply
+ repl = struct.pack('!BB', 0x05, method)
+ writer.write(repl)
+ yield from writer.drain()
+ if method == 0xFF:
+ writer.close()
+ return
+
+ # request
+ data = yield from reader.readexactly(4)
+ vers, cmd, rsv, atyp = struct.unpack('!', data)
+
+ if vers != 0x05 or rsv != 0x00:
+ # disconnect on malformed packet
+
Re: [gentoo-portage-dev] [PATCH] test_compile_modules: skip files that require newer Python version
Dnia 2015-01-31, o godz. 09:18:06 Michał Górny mgo...@gentoo.org napisał(a): Support skipping Python modules and scripts that require newer Python version than currently used during compile tests. Add a metadata db that can be used to store additional information about Python files, and store the required language version there. Pushed as agreed during the meeting. -- Best regards, Michał Górny pgpO98hSdRn7q.pgp Description: OpenPGP digital signature
Re: [gentoo-portage-dev] [PATCH v6] Support escaping network-sandbox through SOCKSv5 proxy
Dnia 2015-02-01, o godz. 09:53:59 Michał Górny mgo...@gentoo.org napisał(a): Add a minimal SOCKSv5-over-UNIX-socket proxy to Portage, and start it whenever ebuilds are started with network-sandbox enabled. Pass the socket address in PORTAGE_SOCKS5_PROXY and DISTCC_SOCKS_PROXY variables. The proxy can be used to escape the network sandbox whenever network access is really desired, e.g. in distcc. The proxy is based on asynchronous I/O using the asyncio module. Therefore, it requires the asyncio module that is built-in in Python 3.4 and available stand-alone for Python 3.3. Escaping the sandbox is not supported with older versions of Python. The proxy supports connecting to IPv6 IPv4 TCP hosts. UDP and socket binding are not supported. SOCKSv5 authentication schemes are not supported (UNIX sockets provide a security layer). And pushed following earlier confirmation :). -- Best regards, Michał Górny pgphKZ6_l9hRL.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
Dnia 2015-02-01, o godz. 14:39:30 Matt Turner matts...@gentoo.org napisał(a): On Sun, Feb 1, 2015 at 2:17 PM, Michał Górny mgo...@gentoo.org wrote: 2. app-emulation/bochs got local USE=3dnow and USE=avx since the flag control support of emulating the two instruction sets rather than using host CPU instruction sets. Wouldn't it be simpler to use CPU_FLAGS_X86 for these and simply disable the emulation code if they're set? That seems better to me. To be honest, I don't know how this works exactly, but I'd suspect that disabling the options will cause the emulated CPU not to have those instructions at all. -- Best regards, Michał Górny pgpTQQrEal1P4.pgp Description: OpenPGP digital signature
Re: [gentoo-dev] [PATCH] git-r3.eclass: respect EVCS_UMASK
Dnia 2015-02-01, o godz. 23:18:01
Ulrich Mueller u...@gentoo.org napisał(a):
@@ -309,8 +320,14 @@ _git-r3_set_gitdir() {
addwrite ${EGIT3_STORE_DIR}
if [[ ! -d ${GIT_DIR} ]]; then
+ local restore_umask=:
+ if [[ ${EVCS_UMASK} ]]; then
+ restore_umask=$(umask -p)
I think that -p is GNU-ism.
+ umask ${EVCS_UMASK} || die Bad options to umask:
${EVCS_UMASK}
+ fi
mkdir ${GIT_DIR} || die
git init --bare || die
+ ${restore_umask} || die
And this has ugly implicit pattern expansion. Don't do such things.
Ever. Even if you want to split commands.
And before anyone suggests that, I asked ulm not to use
eumask_push/pop. If you ask why, read the eutils code and you'll
understand.
fi
}
@@ -508,6 +525,11 @@ git-r3_fetch() {
# try to fetch from the remote
local r success
+ local restore_umask=:
+ if [[ ${EVCS_UMASK} ]]; then
+ restore_umask=$(umask -p)
+ umask ${EVCS_UMASK} || die Bad options to umask:
${EVCS_UMASK}
+ fi
for r in ${repos[@]}; do
einfo Fetching ${r} ...
@@ -668,6 +690,7 @@ git-r3_fetch() {
break
fi
done
+ ${restore_umask} || die
[[ ${success} ]] || die Unable to fetch from any of EGIT_REPO_URI
# submodules can reference commits in any branch
--
Best regards,
Michał Górny
pgpkvpAn5DKAi.pgp
Description: OpenPGP digital signature
[gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
Hi, developers. Just a quick note: the CPU_FLAGS_X86 conversion of the Gentoo repository is complete now. More specifically: 1. all packages (except app-emulation/bochs) use new CPU_FLAGS_X86 flags. Whenever appropriate, the used flags were fixed to match common naming. 2. app-emulation/bochs got local USE=3dnow and USE=avx since the flag control support of emulating the two instruction sets rather than using host CPU instruction sets. 3. Global definitions of old USE flags were removed to make sure repoman will complain when someone mistakenly tries to use them. This hopefully will help getting overlays in shape. If someone wants a bit of a challenge, I've assembled list of packages potentially using old flags in overlays [1], grouped by layman overlay name. I will probably start mailing the overlay owners or filing bugs myself in a few days, after my initial overlay QA runs are done. At this point, I think we can also start discussing introducing CPU_FLAGS_* for other architectures. [1]:http://dev.gentoo.org/~mgorny/tmp/cpuflags-in-overlays.txt -- Best regards, Michał Górny pgpaRjd9J9vr9.pgp Description: OpenPGP digital signature
[gentoo-dev] [PATCH] git-r3.eclass: respect EVCS_UMASK
Attached patch makes git-r3.eclass respect EVCS_UMASK.
See the variable's description in the patch itself (stolen from
subversion.eclass) for the use case.
Ulrich
Index: git-r3.eclass
===
RCS file: /var/cvsroot/gentoo-x86/eclass/git-r3.eclass,v
retrieving revision 1.47
diff -u -B -p -r1.47 git-r3.eclass
--- git-r3.eclass 28 Jul 2014 14:13:50 - 1.47
+++ git-r3.eclass 1 Feb 2015 21:58:52 -
@@ -131,6 +131,17 @@ fi
# @DESCRIPTION:
# If non-empty, this variable prevents any online operations.
+# @ECLASS-VARIABLE: EVCS_UMASK
+# @DEFAULT_UNSET
+# @DESCRIPTION:
+# Set this variable to a custom umask. This is intended to be set by
+# users. By setting this to something like 002, it can make life easier
+# for people who do development as non-root (but are in the portage
+# group), and then switch over to building with FEATURES=userpriv.
+# Or vice-versa. Shouldn't be a security issue here as anyone who has
+# portage group write access already can screw the system over in more
+# creative ways.
+
# @ECLASS-VARIABLE: EGIT_BRANCH
# @DEFAULT_UNSET
# @DESCRIPTION:
@@ -309,8 +320,14 @@ _git-r3_set_gitdir() {
addwrite ${EGIT3_STORE_DIR}
if [[ ! -d ${GIT_DIR} ]]; then
+ local restore_umask=:
+ if [[ ${EVCS_UMASK} ]]; then
+ restore_umask=$(umask -p)
+ umask ${EVCS_UMASK} || die Bad options to umask:
${EVCS_UMASK}
+ fi
mkdir ${GIT_DIR} || die
git init --bare || die
+ ${restore_umask} || die
fi
}
@@ -508,6 +525,11 @@ git-r3_fetch() {
# try to fetch from the remote
local r success
+ local restore_umask=:
+ if [[ ${EVCS_UMASK} ]]; then
+ restore_umask=$(umask -p)
+ umask ${EVCS_UMASK} || die Bad options to umask:
${EVCS_UMASK}
+ fi
for r in ${repos[@]}; do
einfo Fetching ${r} ...
@@ -668,6 +690,7 @@ git-r3_fetch() {
break
fi
done
+ ${restore_umask} || die
[[ ${success} ]] || die Unable to fetch from any of EGIT_REPO_URI
# submodules can reference commits in any branch
pgpFWwUePCn2c.pgp
Description: PGP signature
Re: [gentoo-dev] Re: news item: nfsmount renamed nfsclient
All, here is the third iteration of this news item. Unless there are objections, this will go in the tree sometime after 13:00 utc on 2015-02-02. William Title: nfs service changes Author: William Hubbs willi...@gentoo.org Content-Type: text/plain Posted: 2015-02-02 Revision: 1 News-Item-Format: 1.0 Display-If-Installed: net-fs/nfs-utils-1.3.1-r1 The upgrade to nfs-utils-1.3.1-r1 includes significant service changes both for OpenRC and systemd users. OpenRC users: The OpenRC service which handled mounting nfs file systems has been changed to only start the nfs client daemons and renamed to nfsclient. Because of this change, if you use OpenRC and mount nfs file systems, you need to perform the following steps: Add nfsclient to the runlevel nfsmount was in before. For example, if nfsmount was in the default runlevel, run this command: rc-update add nfsclient default If you use a permanent network connection to the server, make sure netmount is in the same runlevel as nfsclient. If not, it is recommended that net-fs/autofs be set up to handle your network mounts. Systemd users: The nfs systemd units have been renamed. If you are exporting nfs mounts, you should enable the rpcbind and nfs-server services. If you are mounting nfs mounts systemd should automatically detect this and start the nfs-client service. More Information: The following wiki page has more information about nfs file systems: http://wiki.gentoo.org/wiki/NFSv4 signature.asc Description: Digital signature
Re: [gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
On 02/01/2015 02:39 PM, Matt Turner wrote: On Sun, Feb 1, 2015 at 2:17 PM, Michał Górny mgo...@gentoo.org wrote: 2. app-emulation/bochs got local USE=3dnow and USE=avx since the flag control support of emulating the two instruction sets rather than using host CPU instruction sets. Wouldn't it be simpler to use CPU_FLAGS_X86 for these and simply disable the emulation code if they're set? That seems better to me. CPU_FLAGS is intended to control which machine code instructions are allowed to be used in the compiled executables. If we use it for anything else that does not fit this precise definition, then it will be inconsistent and lead to confusion. -- Thanks, Zac
Re: [gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
On 01.02.2015 23:17, Michał Górny wrote: Hi, developers. Just a quick note: the CPU_FLAGS_X86 conversion of the Gentoo repository is complete now. Cool! Thanks for fixing the freeverb3 ebuild, too. Best, Sebastian
[gentoo-dev] Automated Package Removal and Addition Tracker, for the week ending 2015-02-01 23:59 UTC
The attached list notes all of the packages that were added or removed from the tree, for the week ending 2015-02-01 23:59 UTC. Removals: app-emacs/cedet 2015-01-28 22:17:57 ulm app-vim/svncommand 2015-01-30 20:07:19 radhermit app-vim/cvscommand 2015-01-30 20:07:55 radhermit Additions: dev-util/trace-cmd 2015-01-27 02:56:08 chutzpah net-libs/iojs 2015-01-27 03:01:33 patrick dev-python/bleach 2015-01-27 05:23:18 radhermit dev-python/readme 2015-01-27 05:25:22 radhermit www-client/vivaldi 2015-01-27 13:05:05 jer media-libs/libpagemaker 2015-01-27 20:34:04 jlec dev-python/jenkinsapi 2015-01-28 08:27:18 idella4 dev-python/httmock 2015-01-28 08:47:51 idella4 dev-python/jenkins-webapi 2015-01-29 09:19:38 idella4 sec-policy/selinux-git 2015-01-29 10:13:51 perfinion x11-drivers/xf86-video-opentegra2015-01-29 17:04:48 chithanh dev-java/cssparser 2015-01-30 20:31:22 monsieurp app-emulation/docker-compose2015-01-31 16:49:42 alunduil dev-python/oslo-context 2015-01-31 18:10:40 prometheanfire dev-python/oslo-middleware 2015-01-31 18:18:41 prometheanfire dev-haskell/tasty-kat 2015-01-31 20:23:45 qnikst dev-perl/Monitoring-Plugin 2015-01-31 23:03:02 mjo -- Robin Hugh Johnson Gentoo Linux Developer E-Mail : robb...@gentoo.org GnuPG FP : 11AC BA4F 4778 E3F6 E4ED F38E B27B 944E 3488 4E85 Removed Packages: app-emacs/cedet,removed,ulm,2015-01-28 22:17:57 app-vim/svncommand,removed,radhermit,2015-01-30 20:07:19 app-vim/cvscommand,removed,radhermit,2015-01-30 20:07:55 Added Packages: dev-util/trace-cmd,added,chutzpah,2015-01-27 02:56:08 net-libs/iojs,added,patrick,2015-01-27 03:01:33 dev-python/bleach,added,radhermit,2015-01-27 05:23:18 dev-python/readme,added,radhermit,2015-01-27 05:25:22 www-client/vivaldi,added,jer,2015-01-27 13:05:05 media-libs/libpagemaker,added,jlec,2015-01-27 20:34:04 dev-python/jenkinsapi,added,idella4,2015-01-28 08:27:18 dev-python/httmock,added,idella4,2015-01-28 08:47:51 dev-python/jenkins-webapi,added,idella4,2015-01-29 09:19:38 sec-policy/selinux-git,added,perfinion,2015-01-29 10:13:51 x11-drivers/xf86-video-opentegra,added,chithanh,2015-01-29 17:04:48 dev-java/cssparser,added,monsieurp,2015-01-30 20:31:22 app-emulation/docker-compose,added,alunduil,2015-01-31 16:49:42 dev-python/oslo-context,added,prometheanfire,2015-01-31 18:10:40 dev-python/oslo-middleware,added,prometheanfire,2015-01-31 18:18:41 dev-haskell/tasty-kat,added,qnikst,2015-01-31 20:23:45 dev-perl/Monitoring-Plugin,added,mjo,2015-01-31 23:03:02 Done.
Re: [gentoo-dev] arm64
Thanks! The issue and fix are clear by now (for details: http://sourceforge.net/p/uriparser/bugs/24/). So I don't need shell access any more, at least not in this context. Best, Sebastian On 25.01.2015 18:49, Tom Gall wrote: Least speaking for myself I can help you out starting Feb 15th, presuming all the stars are in alignment. If someone else doesn’t help you before, please mark it on your calendar and bug me again then cause I’m sure I’ll forget! Best, Tom On Jan 25, 2015, at 11:43 AM, Sebastian Pipping sp...@gentoo.org wrote: Hi! I got a bug report for arm64 against the test suite of uriparser. If I could get a temporary arm64 shell somewhere, that could help me understand the issue. Best, Sebastian
Re: [gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
On Sun, Feb 1, 2015 at 3:28 PM, Zac Medico zmed...@gentoo.org wrote: On 02/01/2015 02:39 PM, Matt Turner wrote: On Sun, Feb 1, 2015 at 2:17 PM, Michał Górny mgo...@gentoo.org wrote: 2. app-emulation/bochs got local USE=3dnow and USE=avx since the flag control support of emulating the two instruction sets rather than using host CPU instruction sets. Wouldn't it be simpler to use CPU_FLAGS_X86 for these and simply disable the emulation code if they're set? That seems better to me. CPU_FLAGS is intended to control which machine code instructions are allowed to be used in the compiled executables. If we use it for anything else that does not fit this precise definition, then it will be inconsistent and lead to confusion. I'd interpreted Michał's statement to mean that bochs would support 3DNow/AVX regardless and that the flag simply determined whether it used the CPU's native support or built emulation code. Looking at the code, I don't believe that is the case. It appears that the flag simply determines whether bochs can execute those instructions at all.
Re: [gentoo-dev] CPU_FLAGS_X86 gentoo repository migration complete
On Sun, Feb 1, 2015 at 2:17 PM, Michał Górny mgo...@gentoo.org wrote: 2. app-emulation/bochs got local USE=3dnow and USE=avx since the flag control support of emulating the two instruction sets rather than using host CPU instruction sets. Wouldn't it be simpler to use CPU_FLAGS_X86 for these and simply disable the emulation code if they're set? That seems better to me.
Re: [gentoo-dev] [PATCH] git-r3.eclass: respect EVCS_UMASK
On Sun, 1 Feb 2015, Michał Górny wrote:
@@ -309,8 +320,14 @@ _git-r3_set_gitdir() {
addwrite ${EGIT3_STORE_DIR}
if [[ ! -d ${GIT_DIR} ]]; then
+local restore_umask=:
+if [[ ${EVCS_UMASK} ]]; then
+restore_umask=$(umask -p)
I think that -p is GNU-ism.
Of course it is. But it is a shell builtin command, and we require
bash for ebuilds, not POSIX shell.
+umask ${EVCS_UMASK} || die Bad options to umask:
${EVCS_UMASK}
+fi
mkdir ${GIT_DIR} || die
git init --bare || die
+${restore_umask} || die
And this has ugly implicit pattern expansion. Don't do such things.
Ever. Even if you want to split commands.
Please show me how this could possibly cause any problem here.
restore_umask can only have well-defined values, either : or the
output of umask -p which is intended to be used this way.
Ulrich
pgpVXXg52qAub.pgp
Description: PGP signature
[gentoo-dev] Re: news item: nfsmount renamed nfsclient
William Hubbs posted on Sun, 01 Feb 2015 17:16:30 -0600 as excerpted: here is the third iteration of this news item. Unless there are objections, this will go in the tree sometime after 13:00 utc on 2015-02-02. LGTM. =:^) -- Duncan - List replies preferred. No HTML msgs. Every nonfree program has a lord, a master -- and if you use the program, he is your master. Richard Stallman
