Re: [Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

2017-05-22 Thread Graham Christensen 
I think it is worth investigating using C or something for wrappers, as
wrappers on macOS can't be used in shebangs. Wrapped ruby executables have
to be unwrapped to use.

Would using C solve your problems as well?
On Mon, May 22, 2017 at 11:00 AM Freddy Rietdijk 
wrote:

> The problem with such an environment variable is that it's inherited by
>> child
>> processes, who might get confused. (E.g. some code might run both in a
>> wrapped
>> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
>> unambiguously.)
>
>
> In the case of Python we might be able to inject some code where we unset
> the variable. Currently we inject the path to the script, but this is wrong
> in case a wrapper is used.
>
> I think in general a) wrappers should be avoided; and b) nested wrappers
>> should
>> *definitely* be avoided. Wrappers cannot really be avoided for the
>> "delayed
>> composition" use case (like firefox-wrapper), but in such a case perhaps
>> the
>> inner wrapper can be eliminated. For example, if the inner wrapper is
>> used to
>> set $PYTHONPATH, this can be moved into the underlying Python script by
>> setting
>> sys.path at the start.
>
>
> When possible I would prefer to avoid wrappers as well. Unfortunately,
> there's not really a way around it with Python (2.x).
> Sometimes we need to add a program to PATH and for that we need a wrapper.
> When we add the wrapper, the name is wrong. if the wrapper is added in the
> same derivation, then we can insert sys.path like we do now.
>
> However, often we also need composition: creating an environment
> consisting of the interpreter and multiple packages. In that case, they
> need to be able to find each other, and that means again wrappers (although
> with 3.x we might get away with pyvenv.cfg).
>
> On Mon, May 22, 2017 at 2:36 PM, Eelco Dolstra <
> eelco.dols...@logicblox.com> wrote:
>
>> Hi,
>>
>> On 05/19/2017 01:35 PM, Freddy Rietdijk wrote:
>>
>> > Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME`
>> that the
>> > scripts check for. To prevent creating yet another `wrapProgram`
>> variant I
>> > propose we extend `wrapProgram` to always set this variable.
>>
>> The problem with such an environment variable is that it's inherited by
>> child
>> processes, who might get confused. (E.g. some code might run both in a
>> wrapped
>> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
>> unambiguously.)
>>
>> But maybe $NIX_PROGRAM_NAME can be used only in the case where one wrapper
>> script calls another, so the final wrapper can unset it. However, if we
>> have
>> wrapper scripts detecting that the wrapped program is also a wrapper
>> script, we
>> might as well eliminate the execve into the next wrapper (thus preserving
>> the
>> original argv[0]).
>>
>> > This, however, doesn't solve the issue with nested wrappers yet
>>
>> I think in general a) wrappers should be avoided; and b) nested wrappers
>> should
>> *definitely* be avoided. Wrappers cannot really be avoided for the
>> "delayed
>> composition" use case (like firefox-wrapper), but in such a case perhaps
>> the
>> inner wrapper can be eliminated. For example, if the inner wrapper is
>> used to
>> set $PYTHONPATH, this can be moved into the underlying Python script by
>> setting
>> sys.path at the start.
>>
>> --
>> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>>
>
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

2017-05-22 Thread Freddy Rietdijk 
>
> The problem with such an environment variable is that it's inherited by
> child
> processes, who might get confused. (E.g. some code might run both in a
> wrapped
> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
> unambiguously.)


In the case of Python we might be able to inject some code where we unset
the variable. Currently we inject the path to the script, but this is wrong
in case a wrapper is used.

I think in general a) wrappers should be avoided; and b) nested wrappers
> should
> *definitely* be avoided. Wrappers cannot really be avoided for the "delayed
> composition" use case (like firefox-wrapper), but in such a case perhaps
> the
> inner wrapper can be eliminated. For example, if the inner wrapper is used
> to
> set $PYTHONPATH, this can be moved into the underlying Python script by
> setting
> sys.path at the start.


When possible I would prefer to avoid wrappers as well. Unfortunately,
there's not really a way around it with Python (2.x).
Sometimes we need to add a program to PATH and for that we need a wrapper.
When we add the wrapper, the name is wrong. if the wrapper is added in the
same derivation, then we can insert sys.path like we do now.

However, often we also need composition: creating an environment consisting
of the interpreter and multiple packages. In that case, they need to be
able to find each other, and that means again wrappers (although with 3.x
we might get away with pyvenv.cfg).

On Mon, May 22, 2017 at 2:36 PM, Eelco Dolstra 
wrote:

> Hi,
>
> On 05/19/2017 01:35 PM, Freddy Rietdijk wrote:
>
> > Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME`
> that the
> > scripts check for. To prevent creating yet another `wrapProgram` variant
> I
> > propose we extend `wrapProgram` to always set this variable.
>
> The problem with such an environment variable is that it's inherited by
> child
> processes, who might get confused. (E.g. some code might run both in a
> wrapped
> and unwrapped context, so it wouldn't be able to rely on $NIX_PROGRAM_NAME
> unambiguously.)
>
> But maybe $NIX_PROGRAM_NAME can be used only in the case where one wrapper
> script calls another, so the final wrapper can unset it. However, if we
> have
> wrapper scripts detecting that the wrapped program is also a wrapper
> script, we
> might as well eliminate the execve into the next wrapper (thus preserving
> the
> original argv[0]).
>
> > This, however, doesn't solve the issue with nested wrappers yet
>
> I think in general a) wrappers should be avoided; and b) nested wrappers
> should
> *definitely* be avoided. Wrappers cannot really be avoided for the "delayed
> composition" use case (like firefox-wrapper), but in such a case perhaps
> the
> inner wrapper can be eliminated. For example, if the inner wrapper is used
> to
> set $PYTHONPATH, this can be moved into the underlying Python script by
> setting
> sys.path at the start.
>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> ___
> nix-dev mailing list
> nix-dev@lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

Re: [Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

2017-05-22 Thread Freddy Rietdijk 
If a Python script is run, then `sys.argv[0]` is set to the script name (or
path, depending on platform). Some scripts use this name or path to start a
new instance of itself. Unfortunately, exec -a doesn't set `sys.argv[0]`.
Therefore, when running a wrapped Python script, `sys.argv[0]` points not
to the wrapped script, but to the original one.

On Mon, May 22, 2017 at 9:41 AM, zimbatm  wrote:

> Hi Freddy,
>
> Can you expand a bit on why this is a problem? If a program is executed
> using python I would expect this to be reflected in the process name. Argv0
> would contain a path to python and argv1 the path to the script.
>
> Cheers,
> z
>
> On Fri, 19 May 2017, 12:35 Freddy Rietdijk, 
> wrote:
>
>> Hi,
>>
>> When we create a wrapper we use `exec -a` so that the wrapped program
>> uses the name of the original program. This isn't respected by all
>> programs, e.g. Python ignores this. If I am correct this is the case with
>> all interpreters. How do we handle this with other interpreters?
>>
>> Currently we 'solve' the issue for Python by patching the scripts setting
>> `sys.argv[0]` to the original script name, including the full path. This is
>> sufficient for scripts that call themselves, however, it is troublesome
>> when such a script is wrapped in another derivation, because when the
>> script now calls itself, it will call the unwrapped version.
>>
>> Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME`
>> that the scripts check for. To prevent creating yet another `wrapProgram`
>> variant I propose we extend `wrapProgram` to always set this variable.
>>
>> This, however, doesn't solve the issue with nested wrappers yet, because
>> if we would use `set`, it would still refer to the name passed by the
>> innermost wrapper. Either we try and update wrappers, or, which has my
>> preference, we do not use `set` but use `prefix` and make the script pick
>> the last item from the list.
>>
>> What do you think? I like to also refer to the following gist where I've
>> been documenting some of the issues and how we can rewrite the Python infra
>> to solve those issues.
>> https://gist.github.com/FRidh/c3e974e34eac738405af699786b462d5
>>
>>
>> Freddy
>> ___
>> nix-dev mailing list
>> nix-dev@lists.science.uu.nl
>> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>>
>
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev

[Nix-dev] wrapProgram: also pass name as NIX_PROGRAM_NAME

2017-05-19 Thread Freddy Rietdijk 
Hi,

When we create a wrapper we use `exec -a` so that the wrapped program uses
the name of the original program. This isn't respected by all programs,
e.g. Python ignores this. If I am correct this is the case with all
interpreters. How do we handle this with other interpreters?

Currently we 'solve' the issue for Python by patching the scripts setting
`sys.argv[0]` to the original script name, including the full path. This is
sufficient for scripts that call themselves, however, it is troublesome
when such a script is wrapped in another derivation, because when the
script now calls itself, it will call the unwrapped version.

Therefore, I propose we set an environment variable `NIX_PROGRAM_NAME` that
the scripts check for. To prevent creating yet another `wrapProgram`
variant I propose we extend `wrapProgram` to always set this variable.

This, however, doesn't solve the issue with nested wrappers yet, because if
we would use `set`, it would still refer to the name passed by the
innermost wrapper. Either we try and update wrappers, or, which has my
preference, we do not use `set` but use `prefix` and make the script pick
the last item from the list.

What do you think? I like to also refer to the following gist where I've
been documenting some of the issues and how we can rewrite the Python infra
to solve those issues.
https://gist.github.com/FRidh/c3e974e34eac738405af699786b462d5


Freddy
___
nix-dev mailing list
nix-dev@lists.science.uu.nl
https://mailman.science.uu.nl/mailman/listinfo/nix-dev