Re: [gentoo-portage-dev] Error logs
On 12/30/18 9:51 PM, Pratyush Das wrote: > For oracle-jdk - https://pastebin.com/iT1XsfUv > In this case the log tells you to download jdk-8u192-linux-x64.tar.gz > from > http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html > and place it in your distfiles directory. But it does not tell you where > your distfiles directory is. (This is more descriptive than cudnn and > one could run a recursive bash script to find the distfiles directory > but instead of saying distfiles directory, if it printed out the actual > directory location it would be far more convenient) > > Regards, I'm sorry to say that it's intentionally vague. Since ebuilds run in a environment standardized environment [1], the true location of your distfiles directory is not strictly knowable within ebuild scope. To make things more interesting, there are plans to roll out a split distfiles directory structure [2]. [1] https://wiki.gentoo.org/wiki/Project:Package_Manager_Specification [2] https://www.gentoo.org/glep/glep-0075.html -- Thanks, Zac signature.asc Description: OpenPGP digital signature
Re: [gentoo-portage-dev] Error logs
For oracle-jdk - https://pastebin.com/iT1XsfUv In this case the log tells you to download jdk-8u192-linux-x64.tar.gz from http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html and place it in your distfiles directory. But it does not tell you where your distfiles directory is. (This is more descriptive than cudnn and one could run a recursive bash script to find the distfiles directory but instead of saying distfiles directory, if it printed out the actual directory location it would be far more convenient) Regards, On Mon, 31 Dec 2018 at 01:45, M. J. Everitt wrote: > On 30/12/18 19:58, Pratyush Das wrote: > > Hi, > > In reference to this forum post - > https://forums.gentoo.org/viewtopic-t-1091144-highlight-.html , I was > told that several users found wrong information in the portage error logs > pointing them to the wrong directory for distfiles to be downloaded and > placed in for the emerge to work when the tree variable was displayed to > the user, for the user to place the downloaded files in. > Instead of not using the tree variable to point to the distfiles location > (and having a less descriptive error message), was using some other form of > keeping a track of where the files are stored discussed? Or figuring out > why the variable pointed to the wrong directory? > > I could take a look at it if that is a feature you would be open to > having. > > Regards, > > -- > Pratyush Das(Reik) > > Can you highlight some more examples (good & bad) as even the forum post > doesn't really go into much detail about what's present and missing? I can > see that dev-libs/cudnn is poor - usually LICENSE-restricted files give > some sensible directions, iirc. Not sure quite why this package is an > exception if it's problematic in being mirrored ... > Regards, > > Michael/veremitz. > -- Pratyush Das(Reik)
Re: [gentoo-portage-dev] Error logs
On 30/12/18 19:58, Pratyush Das wrote: > Hi, > > In reference to this forum post > - https://forums.gentoo.org/viewtopic-t-1091144-highlight-.html , I was > told that several users found wrong information in the portage error logs > pointing them to the wrong directory for distfiles to be downloaded and > placed in for the emerge to work when the tree variable was displayed to > the user, for the user to place the downloaded files in. > Instead of not using the tree variable to point to the distfiles location > (and having a less descriptive error message), was using some other form > of keeping a track of where the files are stored discussed? Or figuring > out why the variable pointed to the wrong directory? > > I could take a look at it if that is a feature you would be open to having. > > Regards, > > -- > Pratyush Das(Reik) Can you highlight some more examples (good & bad) as even the forum post doesn't really go into much detail about what's present and missing? I can see that dev-libs/cudnn is poor - usually LICENSE-restricted files give some sensible directions, iirc. Not sure quite why this package is an exception if it's problematic in being mirrored ... Regards, Michael/veremitz. signature.asc Description: OpenPGP digital signature
[gentoo-portage-dev] Error logs
Hi, In reference to this forum post - https://forums.gentoo.org/viewtopic-t-1091144-highlight-.html , I was told that several users found wrong information in the portage error logs pointing them to the wrong directory for distfiles to be downloaded and placed in for the emerge to work when the tree variable was displayed to the user, for the user to place the downloaded files in. Instead of not using the tree variable to point to the distfiles location (and having a less descriptive error message), was using some other form of keeping a track of where the files are stored discussed? Or figuring out why the variable pointed to the wrong directory? I could take a look at it if that is a feature you would be open to having. Regards, -- Pratyush Das(Reik)
Re: [gentoo-portage-dev] [PATCH] process.spawn: validate unshare calls (bug 673900)
On 12/30/18 1:06 AM, Zac Medico wrote:
> On 12/30/18 12:44 AM, Michał Górny wrote:
>> On Sat, 2018-12-29 at 04:49 -0800, Zac Medico wrote:
>>> In order to prevent failed unshare calls from corrupting the state
>>> of an essential process, validate the relevant unshare call in a
>>> short-lived subprocess. An unshare call is considered valid if it
>>> successfully executes in a short-lived subprocess.
>>>
>>> Bug: https://bugs.gentoo.org/673900
>>> Signed-off-by: Zac Medico
>>> ---
>>> lib/portage/process.py | 130 +
>>> 1 file changed, 106 insertions(+), 24 deletions(-)
>>>
>>> diff --git a/lib/portage/process.py b/lib/portage/process.py
>>> index ce3e42a8f..4bd6a4192 100644
>>> --- a/lib/portage/process.py
>>> +++ b/lib/portage/process.py
>>> @@ -6,6 +6,7 @@
>>> import atexit
>>> import errno
>>> import fcntl
>>> +import multiprocessing
>>> import platform
>>> import signal
>>> import socket
>>> @@ -338,11 +339,29 @@ def spawn(mycommand, env=None, opt_name=None,
>>> fd_pipes=None, returnpid=False,
>>> fd_pipes[1] = pw
>>> fd_pipes[2] = pw
>>>
>>> - # This caches the libc library lookup in the current
>>> - # process, so that it's only done once rather than
>>> + # This caches the libc library lookup and _unshare_validator results
>>> + # in the current process, so that it's only done once rather than
>>> # for each child process.
>>> + unshare_flags = 0
>>> if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
>>> - find_library("c")
>>> + # from /usr/include/bits/sched.h
>>> + CLONE_NEWNS = 0x0002
>>> + CLONE_NEWIPC = 0x0800
>>> + CLONE_NEWPID = 0x2000
>>> + CLONE_NEWNET = 0x4000
>>> +
>>> + if unshare_net:
>>> + unshare_flags |= CLONE_NEWNET
>>> + if unshare_ipc:
>>> + unshare_flags |= CLONE_NEWIPC
>>> + if unshare_mount:
>>> + # NEWNS = mount namespace
>>> + unshare_flags |= CLONE_NEWNS
>>> + if unshare_pid:
>>> + # we also need mount namespace for slave /proc
>>> + unshare_flags |= CLONE_NEWPID | CLONE_NEWNS
>>> +
>>> + _unshare_validator.instance.validate(unshare_flags)
>>>
>>> # Force instantiation of portage.data.userpriv_groups before the
>>> # fork, so that the result is cached in the main process.
>>> @@ -358,7 +377,7 @@ def spawn(mycommand, env=None, opt_name=None,
>>> fd_pipes=None, returnpid=False,
>>> _exec(binary, mycommand, opt_name, fd_pipes,
>>> env, gid, groups, uid, umask, cwd,
>>> pre_exec, close_fds,
>>> unshare_net, unshare_ipc,
>>> unshare_mount, unshare_pid,
>>> - cgroup)
>>> + unshare_flags, cgroup)
>>> except SystemExit:
>>> raise
>>> except Exception as e:
>>> @@ -430,7 +449,7 @@ def spawn(mycommand, env=None, opt_name=None,
>>> fd_pipes=None, returnpid=False,
>>> def _exec(binary, mycommand, opt_name, fd_pipes,
>>> env, gid, groups, uid, umask, cwd,
>>> pre_exec, close_fds, unshare_net, unshare_ipc, unshare_mount,
>>> unshare_pid,
>>> - cgroup):
>>> + unshare_flags, cgroup):
>>>
>>> """
>>> Execute a given binary with options
>>> @@ -466,6 +485,8 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
>>> @type unshare_mount: Boolean
>>> @param unshare_pid: If True, PID ns will be unshared from the spawned
>>> process
>>> @type unshare_pid: Boolean
>>> + @param unshare_flags: Flags for the unshare(2) function
>>> + @type unshare_flags: Integer
>>> @param cgroup: CGroup path to bind the process to
>>> @type cgroup: String
>>> @rtype: None
>>> @@ -527,26 +548,14 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
>>> if filename is not None:
>>> libc = LoadLibrary(filename)
>>> if libc is not None:
>>> - # from /usr/include/bits/sched.h
>>> - CLONE_NEWNS = 0x0002
>>> - CLONE_NEWIPC = 0x0800
>>> - CLONE_NEWPID = 0x2000
>>> - CLONE_NEWNET = 0x4000
>>> -
>>> - flags = 0
>>> - if unshare_net:
>>> - flags |= CLONE_NEWNET
>>> - if unshare_ipc:
>>> - flags |= CLONE_NEWIPC
>>> - if unshare_mount:
>>> - # NEWNS = mount namespace
>>> - flags |= CLONE_NEWNS
>>> - if unshare_pid:
>>> -
Re: [gentoo-portage-dev] [PATCH] process.spawn: validate unshare calls (bug 673900)
On 12/30/18 12:44 AM, Michał Górny wrote:
> On Sat, 2018-12-29 at 04:49 -0800, Zac Medico wrote:
>> In order to prevent failed unshare calls from corrupting the state
>> of an essential process, validate the relevant unshare call in a
>> short-lived subprocess. An unshare call is considered valid if it
>> successfully executes in a short-lived subprocess.
>>
>> Bug: https://bugs.gentoo.org/673900
>> Signed-off-by: Zac Medico
>> ---
>> lib/portage/process.py | 130 +
>> 1 file changed, 106 insertions(+), 24 deletions(-)
>>
>> diff --git a/lib/portage/process.py b/lib/portage/process.py
>> index ce3e42a8f..4bd6a4192 100644
>> --- a/lib/portage/process.py
>> +++ b/lib/portage/process.py
>> @@ -6,6 +6,7 @@
>> import atexit
>> import errno
>> import fcntl
>> +import multiprocessing
>> import platform
>> import signal
>> import socket
>> @@ -338,11 +339,29 @@ def spawn(mycommand, env=None, opt_name=None,
>> fd_pipes=None, returnpid=False,
>> fd_pipes[1] = pw
>> fd_pipes[2] = pw
>>
>> -# This caches the libc library lookup in the current
>> -# process, so that it's only done once rather than
>> +# This caches the libc library lookup and _unshare_validator results
>> +# in the current process, so that it's only done once rather than
>> # for each child process.
>> +unshare_flags = 0
>> if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
>> -find_library("c")
>> +# from /usr/include/bits/sched.h
>> +CLONE_NEWNS = 0x0002
>> +CLONE_NEWIPC = 0x0800
>> +CLONE_NEWPID = 0x2000
>> +CLONE_NEWNET = 0x4000
>> +
>> +if unshare_net:
>> +unshare_flags |= CLONE_NEWNET
>> +if unshare_ipc:
>> +unshare_flags |= CLONE_NEWIPC
>> +if unshare_mount:
>> +# NEWNS = mount namespace
>> +unshare_flags |= CLONE_NEWNS
>> +if unshare_pid:
>> +# we also need mount namespace for slave /proc
>> +unshare_flags |= CLONE_NEWPID | CLONE_NEWNS
>> +
>> +_unshare_validator.instance.validate(unshare_flags)
>>
>> # Force instantiation of portage.data.userpriv_groups before the
>> # fork, so that the result is cached in the main process.
>> @@ -358,7 +377,7 @@ def spawn(mycommand, env=None, opt_name=None,
>> fd_pipes=None, returnpid=False,
>> _exec(binary, mycommand, opt_name, fd_pipes,
>> env, gid, groups, uid, umask, cwd,
>> pre_exec, close_fds,
>> unshare_net, unshare_ipc,
>> unshare_mount, unshare_pid,
>> -cgroup)
>> +unshare_flags, cgroup)
>> except SystemExit:
>> raise
>> except Exception as e:
>> @@ -430,7 +449,7 @@ def spawn(mycommand, env=None, opt_name=None,
>> fd_pipes=None, returnpid=False,
>> def _exec(binary, mycommand, opt_name, fd_pipes,
>> env, gid, groups, uid, umask, cwd,
>> pre_exec, close_fds, unshare_net, unshare_ipc, unshare_mount,
>> unshare_pid,
>> -cgroup):
>> +unshare_flags, cgroup):
>>
>> """
>> Execute a given binary with options
>> @@ -466,6 +485,8 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
>> @type unshare_mount: Boolean
>> @param unshare_pid: If True, PID ns will be unshared from the spawned
>> process
>> @type unshare_pid: Boolean
>> +@param unshare_flags: Flags for the unshare(2) function
>> +@type unshare_flags: Integer
>> @param cgroup: CGroup path to bind the process to
>> @type cgroup: String
>> @rtype: None
>> @@ -527,26 +548,14 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
>> if filename is not None:
>> libc = LoadLibrary(filename)
>> if libc is not None:
>> -# from /usr/include/bits/sched.h
>> -CLONE_NEWNS = 0x0002
>> -CLONE_NEWIPC = 0x0800
>> -CLONE_NEWPID = 0x2000
>> -CLONE_NEWNET = 0x4000
>> -
>> -flags = 0
>> -if unshare_net:
>> -flags |= CLONE_NEWNET
>> -if unshare_ipc:
>> -flags |= CLONE_NEWIPC
>> -if unshare_mount:
>> -# NEWNS = mount namespace
>> -flags |= CLONE_NEWNS
>> -if unshare_pid:
>> -# we also need mount namespace for
>> slave /proc
>> -
Re: [gentoo-portage-dev] [PATCH] process.spawn: validate unshare calls (bug 673900)
On Sat, 2018-12-29 at 04:49 -0800, Zac Medico wrote:
> In order to prevent failed unshare calls from corrupting the state
> of an essential process, validate the relevant unshare call in a
> short-lived subprocess. An unshare call is considered valid if it
> successfully executes in a short-lived subprocess.
>
> Bug: https://bugs.gentoo.org/673900
> Signed-off-by: Zac Medico
> ---
> lib/portage/process.py | 130 +
> 1 file changed, 106 insertions(+), 24 deletions(-)
>
> diff --git a/lib/portage/process.py b/lib/portage/process.py
> index ce3e42a8f..4bd6a4192 100644
> --- a/lib/portage/process.py
> +++ b/lib/portage/process.py
> @@ -6,6 +6,7 @@
> import atexit
> import errno
> import fcntl
> +import multiprocessing
> import platform
> import signal
> import socket
> @@ -338,11 +339,29 @@ def spawn(mycommand, env=None, opt_name=None,
> fd_pipes=None, returnpid=False,
> fd_pipes[1] = pw
> fd_pipes[2] = pw
>
> - # This caches the libc library lookup in the current
> - # process, so that it's only done once rather than
> + # This caches the libc library lookup and _unshare_validator results
> + # in the current process, so that it's only done once rather than
> # for each child process.
> + unshare_flags = 0
> if unshare_net or unshare_ipc or unshare_mount or unshare_pid:
> - find_library("c")
> + # from /usr/include/bits/sched.h
> + CLONE_NEWNS = 0x0002
> + CLONE_NEWIPC = 0x0800
> + CLONE_NEWPID = 0x2000
> + CLONE_NEWNET = 0x4000
> +
> + if unshare_net:
> + unshare_flags |= CLONE_NEWNET
> + if unshare_ipc:
> + unshare_flags |= CLONE_NEWIPC
> + if unshare_mount:
> + # NEWNS = mount namespace
> + unshare_flags |= CLONE_NEWNS
> + if unshare_pid:
> + # we also need mount namespace for slave /proc
> + unshare_flags |= CLONE_NEWPID | CLONE_NEWNS
> +
> + _unshare_validator.instance.validate(unshare_flags)
>
> # Force instantiation of portage.data.userpriv_groups before the
> # fork, so that the result is cached in the main process.
> @@ -358,7 +377,7 @@ def spawn(mycommand, env=None, opt_name=None,
> fd_pipes=None, returnpid=False,
> _exec(binary, mycommand, opt_name, fd_pipes,
> env, gid, groups, uid, umask, cwd,
> pre_exec, close_fds,
> unshare_net, unshare_ipc,
> unshare_mount, unshare_pid,
> - cgroup)
> + unshare_flags, cgroup)
> except SystemExit:
> raise
> except Exception as e:
> @@ -430,7 +449,7 @@ def spawn(mycommand, env=None, opt_name=None,
> fd_pipes=None, returnpid=False,
> def _exec(binary, mycommand, opt_name, fd_pipes,
> env, gid, groups, uid, umask, cwd,
> pre_exec, close_fds, unshare_net, unshare_ipc, unshare_mount,
> unshare_pid,
> - cgroup):
> + unshare_flags, cgroup):
>
> """
> Execute a given binary with options
> @@ -466,6 +485,8 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
> @type unshare_mount: Boolean
> @param unshare_pid: If True, PID ns will be unshared from the spawned
> process
> @type unshare_pid: Boolean
> + @param unshare_flags: Flags for the unshare(2) function
> + @type unshare_flags: Integer
> @param cgroup: CGroup path to bind the process to
> @type cgroup: String
> @rtype: None
> @@ -527,26 +548,14 @@ def _exec(binary, mycommand, opt_name, fd_pipes,
> if filename is not None:
> libc = LoadLibrary(filename)
> if libc is not None:
> - # from /usr/include/bits/sched.h
> - CLONE_NEWNS = 0x0002
> - CLONE_NEWIPC = 0x0800
> - CLONE_NEWPID = 0x2000
> - CLONE_NEWNET = 0x4000
> -
> - flags = 0
> - if unshare_net:
> - flags |= CLONE_NEWNET
> - if unshare_ipc:
> - flags |= CLONE_NEWIPC
> - if unshare_mount:
> - # NEWNS = mount namespace
> - flags |= CLONE_NEWNS
> - if unshare_pid:
> - # we also need mount namespace for
> slave /proc
> - flags |= CLONE_NEWPID | CLONE_NEWNS
> -
>
