Re: A define program using blocks for execline?
Yes, it can be done with current execline tools through options like
-s in define and importas, but I feel something like this would be
clearer:
block-define var { 1 2 3 }
printf "%s\n" "This is ${var}"
Does this already exist?
Not really, but that sounds like a possible addition, the model sounds
sane. Thanks for the suggestion, I'll think about it.
--
Laurent
A define program using blocks for execline?
One feature I sometimes miss in execline is being able to define an
array in a simple, direct way.
Yes, it can be done with current execline tools through options like
-s in define and importas, but I feel something like this would be
clearer:
block-define var { 1 2 3 }
printf "%s\n" "This is ${var}"
Does this already exist?
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
On Thu, Jun 29, 2023 at 11:04:33AM +, Laurent Bercot wrote: > Indeed, /dev should work; but using it still makes me queasier than > crafting a nonexistent path. Actually deliberately supplying a directory to posix_spawn*() is a slightly deeper test, as the path exists but is unexecutable. But surely this does not constitute a decisive advantage. > Of course, it doesn't matter for glibc, and it doesn't matter for > s6 which needs fork anyway. And chances are that platforms that > implement posix_spawn() with internals that are *not* fork+exec will > not make it return before the spawning has really succeeded. But still, > it's nice to make sure it can be used wherever it exists. Now I see, thanks... > If you don't like the workaround, nobody's preventing you from using > --with-sysdep-posixspawn=no manually. ;) Yes, and I am keeping my origin solution at work intact :) -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
Actually I mean a *directory* that is guaranteed to exist (and meanwhile unexecutable): so /dev here. Indeed, /dev should work; but using it still makes me queasier than crafting a nonexistent path. The mkstemp thing works, so, not going to change it to save a couple of syscalls in a configure test. :) Well I was intending to suggest that we simpliy avoided posix_spawn*() where it disagreed with posix_spawn(3p); that is to say simply replacing all previous `#ifdef HASPOSIXSPAWN' conditions with `#if (defined HASPOSIXSPAWN) && (!defined SKALIBS_HASPOSIXSPAWNEARLYRETURN)'. After all it seems to me child_spawn*() is not used that prevalently, so the performance penalty is really minor; of course, feel free to correct me. Yes, falling back to fork+exec when posix_spawn is bad is an option, and I would probably have done just that if I hadn't been pointed to the existence of waitid() to achieve the "test whether a child is dead without reaping it" thing, without which there can be no workaround. But posix_spawn is more than a performance thing. The point of this interface is that its implementation doesn't have to be vfork+exec internally; it was precisely designed to allow spawning processes on nommu machines, where vfork and fork are basically impossible. So, using posix_spawn wherever possible helps with portability as well. Of course, it doesn't matter for glibc, and it doesn't matter for s6 which needs fork anyway. And chances are that platforms that implement posix_spawn() with internals that are *not* fork+exec will not make it return before the spawning has really succeeded. But still, it's nice to make sure it can be used wherever it exists. If you don't like the workaround, nobody's preventing you from using --with-sysdep-posixspawn=no manually. ;) -- Laurent
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
On Thu, Jun 29, 2023 at 08:21:19AM +, Laurent Bercot wrote: > POSIX doesn't mandate any path other than /dev/null and /dev/console > and I'd rather not try executing them, who knows what weird permissions > they may have on obscure OSes. Actually I mean a *directory* that is guaranteed to exist (and meanwhile unexecutable): so /dev here. > I agree it's a lot of work for not much, but as you said, the > behaviour is arguably conformant, and your experience proves that old > glibcs are still around, so I'd rather make posix_spawn usable where > it exists instead of placing the burden of --with-sysdep-posixspawn=no > on users who have a bad version. Well I was intending to suggest that we simpliy avoided posix_spawn*() where it disagreed with posix_spawn(3p); that is to say simply replacing all previous `#ifdef HASPOSIXSPAWN' conditions with `#if (defined HASPOSIXSPAWN) && (!defined SKALIBS_HASPOSIXSPAWNEARLYRETURN)'. After all it seems to me child_spawn*() is not used that prevalently, so the performance penalty is really minor; of course, feel free to correct me. And I probably also need to bring the incorrectness of posix_spawn(3) to the attention of its (Linux manpages) maintainers. -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
Fixes pushed to git, thanks! When given an unexecutable path, child_spawn() returns 0, but errno is unset... that's on purpose. Unfortunately, in the parent there is no way to know the child's execve() error code; all we have is the exit status, 127, and we cannot report the reason for the failure. Rather than set errno to something that may be wrong and prompt the caller to take inadequate measures, I'd rather set it to 0, which glibc reports as "success" but really means "no error information" except in a few, well-known contexts; and let the caller deal with the lack of more accurate reporting. I know it's not satisfying, but we can't do any better. I have realised that a simpler unexecutable path can be, for example, /etc (is it mandated in POSIX?); this can save the mkstemp() call in the sysdep test. POSIX doesn't mandate any path other than /dev/null and /dev/console and I'd rather not try executing them, who knows what weird permissions they may have on obscure OSes. It's a sysdep test, it's not performance-critical; I'd rather use mkstemp() to be *sure* we have a path that does not exist. (Of course the user could always race the program, but we're not trying to harden against stupidity here.) (And frankly I personally do not really find it much worthwhile to introduce this amount of complexity for the broken dependency of a quite minor performance optimisation...) I agree it's a lot of work for not much, but as you said, the behaviour is arguably conformant, and your experience proves that old glibcs are still around, so I'd rather make posix_spawn usable where it exists instead of placing the burden of --with-sysdep-posixspawn=no on users who have a bad version. As shown by the qemu bug I linked above, this impacts s6-svscan, which relies on correct child_spawn() reporting when running custom signal handlers, so not working around bad posix_spawn QoI may lead to buggy signal management in s6-svscan, and nobody wants that. A cursory web search appears to say that glibc-2.27 is when they fixed the posix_spawn QoI; 2.17 being bad is consistent with that. But I can't be bothered to go spelunk in glibc code to check and/or bisect, so if someone could confirm, thank you, otherwise, no big deal. -- Laurent
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
I have realised that a simpler unexecutable path can be, for example, /etc (is it mandated in POSIX?); this can save the mkstemp() call in the sysdep test. (And frankly I personally do not really find it much worthwhile to introduce this amount of complexity for the broken dependency of a quite minor performance optimisation...) -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Re: posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
On Wed, Jun 28, 2023 at 09:40:48PM +, Laurent Bercot wrote: > I pushed a workaround to the skalibs git. > Could you please try a build on a machine that exhibits the early > return behaviour and tell me if > - the behaviour is correctly detected by ./configure (the last sysdep) Yes, after changing `attrp' in tryposixspawnearlyreturn.c to `attr'. > - the child_spawn*() family of functions now works properly even on > this machine? Partially, after an additional WEXITED is OR-ed with the options for waitid() in child_spawn_workaround.c. When given an unexecutable path, child_spawn() returns 0, but errno is unset. > Also, can you please tell me what version of glibc these distribution > versions are running? glibc-2.17-317.el7.x86_64, from CentOS 7.9.2009. -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
posix_spawn (was: Bugs with execline/s6 documentation and skalibs functions using posix_spawn())
I pushed a workaround to the skalibs git. Could you please try a build on a machine that exhibits the early return behaviour and tell me if - the behaviour is correctly detected by ./configure (the last sysdep) - the child_spawn*() family of functions now works properly even on this machine? Also, can you please tell me what version of glibc these distribution versions are running? Thanks! -- Laurent
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
On Wed, Jun 28, 2023 at 01:33:57AM +, Laurent Bercot wrote: > They're not the only possible behaviours: for instance, [1] shows that > under some buggy qemu, posix_spawn() always returns early. That seems to at least inadvertently comply with posix_spawn(3)... -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
Actually I copied the fragment of posix_spawn(3) from a Devuan Chimaera machine, so the problem may be not specific to CentOS 7. I did not test CentOS 6 or other distro (version)s, for example; but on Rocky Linux 8, which I unfortunately also need to support at work, the behaviour is as expected. Attached is a simple test. It may be a bug in some old glibc, then. If we assume posix_spawn(3) and posix_spawn(3p) were the only possible behaviours (which is frankly not that reliable, judging from how neither manpage noted the violation of conformance), then the two behaviours could be distinguished with the attached test. They're not the only possible behaviours: for instance, [1] shows that under some buggy qemu, posix_spawn() always returns early. But that behaviour can also be caught by the same workaround as the glibc behaviour you're observing, so it's fine. Since the bug is more widespread than "one old version of one distro", is visible in production environments used at large, and seems constrained to "posix_spawn succeeds even if exec fails", which is testable, I'll add a sysdep to detect it and a workaround in child_spawn*, but it will mean additional manual --with-sysdep-foobar=blah noise for skalibs cross-builds. [1]: https://skarnet.org/lists/skaware/1658.html -- Laurent
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
On Tue, Jun 27, 2023 at 09:27:17PM +, Laurent Bercot wrote:
> Testing the behaviour may be challenging, however, because I suspect
> the CentOS 7 implementation of posix_spawn() is just racy, and they
> simply documented that they don't care.
Actually I copied the fragment of posix_spawn(3) from a Devuan Chimaera
machine, so the problem may be not specific to CentOS 7. I did not test
CentOS 6 or other distro (version)s, for example; but on Rocky Linux 8,
which I unfortunately also need to support at work, the behaviour is
as expected. Attached is a simple test.
>From the posix_spawn(3), which had the similar statement across the
three different versions above, there is also a statement two paragraphs
later: "The posix_spawn() and posix_spawnp() functions fail only in the
case where the underlying fork(2), vfork(2) or clone(2) call fails".
So if the manpage is followed to the word, the CentOS 7 behaviour is
expected; if posix_spawn(3p) is followed instead, any exec() error must
also be caught and result in returning 0, which we really expect.
If we assume posix_spawn(3) and posix_spawn(3p) were the only possible
behaviours (which is frankly not that reliable, judging from how
neither manpage noted the violation of conformance), then the two
behaviours could be distinguished with the attached test.
--
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
/* ISC license. */
#include
#include
#include
#include
#include
pid_t child_spawn0 (char const *prog, char const *const *argv, char const *const *envp)
{
pid_t pid ;
posix_spawnattr_t attr ;
int e ;
e = posix_spawnattr_init() ;
if (e) goto err ;
e = posix_spawnp(, prog, 0, , (char *const *)argv, (char *const *)envp) ;
posix_spawnattr_destroy() ;
if (e) goto err ;
return pid ;
err:
errno = e ;
return 0 ;
}
int main (int argc, char const *const *argv, char const *const *envp)
{
if (argc < 2) return 1 ;
printf ("%d\n", child_spawn0(argv[1], [1], envp)) ;
return 0 ;
}
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
Testing the behaviour may be challenging, however, because I suspect the CentOS 7 implementation of posix_spawn() is just racy, and they simply documented that they don't care. Thinking about it more, I'm afraid it's not a testable behaviour. Not only isn't there any way to force the race since it entirely happens inside a libc function, but also, the test would require running code on the build machine, which doesn't work for cross-builds and people would have to manually set the sysdep anyway. It seems like --with-sysdep-posixspawn=no, as you did, is the easiest workaround. -- Laurent
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
As a more general fix, I think tryposixspawn.c should at least try spawning a probably unexecutable path (like the one above) as well, which corrects the sysdep on systems where the expected conformance is broken. Adding a sysdep to detect that case is a good idea indeed! Rather than pretending it doesn't exist, though, I'd rather add a different sysdep that tests its behaviour, so it can still be used with the proper workaround. Testing the behaviour may be challenging, however, because I suspect the CentOS 7 implementation of posix_spawn() is just racy, and they simply documented that they don't care. -- Laurent
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
On Tue, Jun 27, 2023 at 08:50:35AM +, Laurent Bercot wrote:
> I believe the correct setting is actually KillMode=mixed; and the
> ExecStop= line is incorrect as well since ExecStop expects a synchronous
> command, not an asynchronous one. Better let systemd just send a SIGTERM
> to s6-svscan, wait for the supervision tree to exit on its own, and
> SIGKILL the stragglers. I pushed a fix accordingly.
Yes, I did my change in a haste and did not notice the `mixed' option
which is indeed better. As you see, systemd waits for the stragglers
so that is at least correct, although the timeout (150s by default on
CentOS 7) is really exorbitant.
> Yeah, well, tough for non-conforming systems.
> That said, I also pushed a change last week that should have fixed
> this issue as a side effect, so it's all good. If you feel like it,
> you can try the s6-svscan version in the latest s6 git. :)
s6-svscan is quite a minor issue; I added .s6-svscan/SIGTERM for all
those scandirs which may unfortunately need to run on CentOS 7, as an
extra line of defence -- in addition to `--with-sysdep-posixspawn=no'
to skalibs. The latter also fixes other programs that involve
child_spawn*(), including home-made ones (iotrap, discussed a few days
ago on the supervision list) and upstream ones (for instance trap).
I personally think `--with-sysdep-posixspawn=no' is the really correct
fix on CentOS 7, which fixes silent failures with eg.:
#!/bin/execlineb -P
trap {
SIGHUP { echo test }
} /dev/xyzzy
As a more general fix, I think tryposixspawn.c should at least try
spawning a probably unexecutable path (like the one above) as well,
which corrects the sysdep on systems where the expected conformance
is broken.
--
My current OpenPGP key:
RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30)
7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
* In `trap.html', there is a reference to the removed `timeout' keyword. Fixed. * In `s6-svscan-not-1.html', the systemd unit (traumatic experience with it, as you may easily expect) lacks a `KillMode = process'. I believe the correct setting is actually KillMode=mixed; and the ExecStop= line is incorrect as well since ExecStop expects a synchronous command, not an asynchronous one. Better let systemd just send a SIGTERM to s6-svscan, wait for the supervision tree to exit on its own, and SIGKILL the stragglers. I pushed a fix accordingly. * The child_spawn*() family of functions, depending on using posix_spawn or not, exhibit different behaviours on CentOS 7 (trauma again), as posix_spawnp() may return 0 with argv pointing to unexecutable paths. This, for example, results in s6-svscan not exiting on SIGTERM when .s6-svscan/SIGTERM is absent. The behaviour of posix_spawnp() on CentOS 7 does not conform to posix_spawn(3p), but is documented in posix_spawn(3): "Even when these functions return a success status, the child process may still fail for a plethora of reasons related to its pre-exec() initialization. In addition, the exec(3) may fail." Yeah, well, tough for non-conforming systems. That said, I also pushed a change last week that should have fixed this issue as a side effect, so it's all good. If you feel like it, you can try the s6-svscan version in the latest s6 git. :) > --with-sysdep-devurandom Also fixed. Thanks a lot for these reports! -- Laurent
Re: Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
On Tue, Jun 27, 2023 at 10:16:10AM +0800, Casper Ti. Vector wrote: > * The child_spawn*() family of functions, [...] Well, just found another one when handling the above issue: * `flags.html' from skalibs references `--enable-sysdep-devurandom', which should now be `--with-sysdep-devurandom'. -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
Bugs with execline/s6 documentation and skalibs functions using posix_spawn()
* In `trap.html', there is a reference to the removed `timeout' keyword. * In `s6-svscan-not-1.html', the systemd unit (traumatic experience with it, as you may easily expect) lacks a `KillMode = process'. * The child_spawn*() family of functions, depending on using posix_spawn or not, exhibit different behaviours on CentOS 7 (trauma again), as posix_spawnp() may return 0 with argv pointing to unexecutable paths. This, for example, results in s6-svscan not exiting on SIGTERM when .s6-svscan/SIGTERM is absent. The behaviour of posix_spawnp() on CentOS 7 does not conform to posix_spawn(3p), but is documented in posix_spawn(3): "Even when these functions return a success status, the child process may still fail for a plethora of reasons related to its pre-exec() initialization. In addition, the exec(3) may fail." -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2024.09.30) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
New URLs for the updates to s6-man-pages and execline-man-pages
Apologies, new URLs for the updates to s6-man-pages and execline-man-pages: https://git.sr.ht/~flexibeast/s6-man-pages/archive/v2.11.3.2.4.tar.gz https://git.sr.ht/~flexibeast/execline-man-pages/archive/v2.9.3.0.5.tar.gz Alexis.
s6-man-pages and execline-man-pages now on sr.ht; new releases of both
Hi all, Both s6-man-pages and execline-man-pages are now on sr.ht; their repos on GitHub have been set to read-only. With that, all the -man-pages projects are now on sr.ht. There are also new releases for s6-man-pages and execline-man-pages, which add the s6-why.7 and execline-why.7 pages, respectively: https://git.sr.ht/~flexibeast/s6-man-pages/archive/v2.11.3.2.3.tar.gz https://git.sr.ht/~flexibeast/execline-man-pages/archive/v2.9.3.0.4.tar.gz Alexis.
Re: [execline] Conditional export
On Thu, Mar 23, 2023 at 03:19:21PM +, Laurent Bercot wrote:
>
> I'm going to regret this.
>
> ifthenelse -s { eltest -f ${FILE} } { export EXISTS ${FILE} } { }
> env
This is something I was originally looking for but did not know it is
possible. Anyway, I'll stick with backitck -x, it is well documented and
few extra calls on the path are not an issue.
Thanks!
Petr
Re: [execline] Conditional export
I'm going to regret this.
ifthenelse -s { eltest -f ${FILE} } { export EXISTS ${FILE} } { }
env
This construct is purposefully not documented, because it breaks
syntactic and logic assumptions that are true in the rest of execline.
But it can simplify your life in a handful of cases, like this one.
What it does: it will *prepend the rest of your script* with the
contents of the second or the third block, depending on whether the
test in the first block is true.
Do not overuse it. Do not ask for support about it. If it makes your
script easier to maintain, enjoy. If you start feeling like a sorcerer
and are tempted to explore what kind of magical feats you can accomplish
with it, don't - it will not end well.
--
Laurent
Re: [execline] Conditional export
On Thu, Mar 23, 2023 at 09:58:00AM -0300, Carlos Eduardo wrote:
> I'd
>
> define FILE file
> backtick -x EXISTS { if { eltest -f $FILE } s6-echo -- $FILE }
> env
>
> -x removes EXISTS from the environment if the "if" fails.
Thanks a lot! I should have read the documentation more carefully, I
completely overlooked "or if prog1... crashes or exits nonzero" part of
backtick -x :(
> (still impressed by the creative use of importas -us).
>
> Em qui., 23 de mar. de 2023 às 09:44, Petr Vaněk escreveu:
> >
> > Hi execline enthusiasts,
> >
> > I am trying to write an execline script which conditionally exports
> > variable only if a given file exists. I am basically trying to mimic
> > this shell script:
> >
> > ```
> > #! /bin/sh
> >
> > FILE="file"
> >
> > if [ -f "${FILE}" ]; then
> > EXISTS="${FILE}"
> > fi
> >
> > export EXISTS
> >
> > env
> > ```
> >
> > I have come up with following solution, which does what I want but I am
> > wondering if there is a better way of expressing this in execline?
> >
> > ```
> > #! /bin/execlineb -P
> >
> > define FILE "file"
> >
> > backtick EXPORTER {
> > foreground {
> > if { eltest -f "${FILE}" }
> > echo export EXISTS "${FILE}"
> > }
> > }
> >
> > importas -us EXPORTER EXPORTER "${EXPORTER}"
> >
> > env
> > ```
> >
> > Cheers,
> > Petr
Re: [execline] Conditional export
I'd
define FILE file
backtick -x EXISTS { if { eltest -f $FILE } s6-echo -- $FILE }
env
-x removes EXISTS from the environment if the "if" fails.
(still impressed by the creative use of importas -us).
Em qui., 23 de mar. de 2023 às 09:44, Petr Vaněk escreveu:
>
> Hi execline enthusiasts,
>
> I am trying to write an execline script which conditionally exports
> variable only if a given file exists. I am basically trying to mimic
> this shell script:
>
> ```
> #! /bin/sh
>
> FILE="file"
>
> if [ -f "${FILE}" ]; then
> EXISTS="${FILE}"
> fi
>
> export EXISTS
>
> env
> ```
>
> I have come up with following solution, which does what I want but I am
> wondering if there is a better way of expressing this in execline?
>
> ```
> #! /bin/execlineb -P
>
> define FILE "file"
>
> backtick EXPORTER {
> foreground {
> if { eltest -f "${FILE}" }
> echo export EXISTS "${FILE}"
> }
> }
>
> importas -us EXPORTER EXPORTER "${EXPORTER}"
>
> env
> ```
>
> Cheers,
> Petr
[execline] Conditional export
Hi execline enthusiasts,
I am trying to write an execline script which conditionally exports
variable only if a given file exists. I am basically trying to mimic
this shell script:
```
#! /bin/sh
FILE="file"
if [ -f "${FILE}" ]; then
EXISTS="${FILE}"
fi
export EXISTS
env
```
I have come up with following solution, which does what I want but I am
wondering if there is a better way of expressing this in execline?
```
#! /bin/execlineb -P
define FILE "file"
backtick EXPORTER {
foreground {
if { eltest -f "${FILE}" }
echo export EXISTS "${FILE}"
}
}
importas -us EXPORTER EXPORTER "${EXPORTER}"
env
```
Cheers,
Petr
Re: [PATCH] execline: multicall: make sort independent of locale
Can you please tell me what locale you're using, for testing purposes ? -- Laurent
Re: [PATCH] execline: multicall: make sort independent of locale
reset LC_ALL to avoid locale dependent sorting. This is critical to ensure bsort() works reliably. In my locale "execline-cd" was sorted after "execlineb" ... lol. Changing the sorting order for ASCII characters is probably the most insane misdesign in locales. Good catch! Thanks for the patch. Going to apply with a slight modification: making the change global to the whole script, for easier maintainability. -- Laurent
Re: single-binary for execline programs?
Yes, this is only possible because you did a very good job in the first place. Good work! This cannot be said enough. Thanks. I managed to de-global the arrays in trap.c, so now the only unavoidable global is in forstdin: a pointer to a structure accessed by a signal handler. You'd think with all the siginfo stuff, POSIX would have thought of mandating a void * auxiliary pointer you'd give to sigaction() and that would be stored and provided to the signal handler, but no, there's just no room to pass user data other than globally. Yet another example of wonderful, user-friendly design. But yeah, 8 bytes of bss/data for the whole thing is pretty good, the crt and the libc are basically the only static RAM users, so there's nothing more to do here. I was also curious about starting time and should have done that in my previous mail, it's a bit slower as expected. Yeah, a 0.2 ms difference is fine, I think. :P But I'm not sure if it's possible to get an accurate benchmark, because the cost of 4-5 strcmp()s are negligible before the cost of the execve's in the first place. I suspect at least half of the difference comes from mapping a bigger executable. I think the main reason to like shared libraries as a distribution is that if you upgrade it, you get the upgrade for all programs that depend on it -- which isn't reall a problem for this. Oh, absolutely, and that's why it's hard to advocate static linking to distributions. It's a very reasonable argument for dynamic linking. At the risk of repeating myself, I'll be happy to help with anything related to this -- that's the least I can do given I brought it up. Thank you. I might seriously take you up on that offer further down the road. :) But really, since the "cat everything together" method works in this case, there's not much more to do except pay attention when writing or editing normal programs in the future. I pushed "multicall-strip" and "multicall-install" targets in git, and documented the setup in the INSTALL file. As experimental, because although I *think* everything is working, there may still be some interaction I've missed. -- Laurent
Re: single-binary for execline programs?
Laurent Bercot wrote on Wed, Feb 01, 2023 at 06:59:04PM +:
> > Look, here's a trivial, suboptimal wrapper, far from pretty:
> > (...)
> > (look, I said it wasn't pretty -- there are at least a dozen of problems
> > with this, but nothing a day of work I offered to do can't fix; I wrote
> > this because it was faster than talking without a concrete example to
> > have some figures, and that took me less time than the rest of this mail)
>
> Damn you to the nine circles of Hell, one by one, slowly, then all of
> them at the same time.
> You piqued my curiosity, so I did it, and I spent the day making it work.
I'm sorry -- I also know the feeling of doing myself something someone
suggested :-D
If that's any consolation, I'm also here testing it at 5AM...
> That said, you were right: that's some pretty hefty saving of disk
> space. The execline binary is 169kB, statically linked against musl on
> x86_64. That's neat. I expected it to be at least twice bigger. And
> the data/bss isn't too bad either: only 2 pages. But that's because
> execline programs use very little global memory in the first place -
> the only places where globals are used is when state needs to be
> accessed by signal handlers, and there's nothing I can do about that in
> short-lived programs. (Long-lived programs use a selfpipe, so only
> one int of global data is ever needed for them.)
Yes, this is only possible because you did a very good job in the first
place. Good work! This cannot be said enough.
> So, all in all, much better results than I expected, it was a pleasant
> surprise. Still, concatenating all the code feels really clunky, and
> a real multicall program needs to be designed for this from the start,
> which won't happen for execline in the foreseeable future, so this is
> as much as you get for now.
>
> If you're interested in hacking the thing, the magic happens in
> tools/gen-multicall.sh.
Will have a further look tomorrow.
I was also curious about starting time and should have done that in my
previous mail, it's a bit slower as expected.
Running 'execline-cd /' 50 times followed by 'true', I get:
$ hyperfine --warmup 5 -N 'cd / cd / ... true'
(multi)
Time (mean ± σ): 21.9 ms ± 2.1 ms[User: 5.0 ms, System: 16.7 ms]
Range (min … max):17.2 ms … 26.7 ms148 runs
(original)
Time (mean ± σ): 21.7 ms ± 1.9 ms[User: 5.0 ms, System: 16.5 ms]
Range (min … max):16.8 ms … 24.9 ms120 runs
so the original binary is slightly faster to load in a benchmark; but
I'm curious at the benefits one would get from not having to look for
and read multiple binaries in more real cases...
Either way it probably will be hard to notice.
> > libexecline is statically linked, so these pages aren't shared afaik?
>
> That's right, I forgot it was always statically linked. If it helps,
> changing ${LIBEXECLINE} to -lexecline in the src/execline/deps-exe
> files, then running ./tools/gen-deps.sh > package/deps.mak, should
> allow you to link against a dynamic libexecline. Can you do it, see
> how much space you gain? That's a configuration I would definitely
> support, even if it's slower - people usually love shared libraries.
I think the main reason to like shared libraries as a distribution is
that if you upgrade it, you get the upgrade for all programs that depend
on it -- which isn't reall a problem for this.
Regardless curiosity triumphs, the gains aren't as big as I thought: text
going from 231914 to 186776, which is still twice over the multi binary
(75066 in this configuration).
The rest will be C runtime...
(data is slighly lower (~400 bytes), bss slightly bigger (~50 bytes), but
not different enough to matter)
> Sigh. I shouldn't feel that way, and any potential improvement should
> be a source of joy, not dread - but really I wish the results weren't
> so good. Now Pandora's box has been opened and everyone will want to
> use the multicall exclusively, so at some point I'll have to support it,
> i.e. ensure it's actually correct and enhance its maintainability.
> And that means a lot more work. :(
At the risk of repeating myself, I'll be happy to help with anything
related to this -- that's the least I can do given I brought it up.
I understand you probably consider it's faster to do it yourself and
it's your baby, but this is probably orthogonal enough to be worth
delegating a bit. Well, ultimately it's up to you :)
Cheers,
--
Dominique
Re: single-binary for execline programs?
allow you to link against a dynamic libexecline. Can you do it, see how much space you gain? That's a configuration I would definitely support, even if it's slower - people usually love shared libraries. I'm tired. This configuration is obviously already supported, and no need to patch. You just need to ./configure --disable-allstatic. Compared to fully static binaries with musl, a fully static multicall is a size gain of 87%. Compared to fully dynamic binaries, it's a gain of 69%. Still impressive. A fully dynamic multicall binary is only 80 kB on x86_64, but it's a pretty stupid configuration because it's the only user of libexecline so nothing is gained by sharing it. There still may be a case for sharing libskarnet; but toolchains make it so difficult to link against some libraries statically and some others dynamically that supporting that configuration is just not worth it. -- Laurent
Re: single-binary for execline programs?
Look, here's a trivial, suboptimal wrapper, far from pretty:
> (...)
(look, I said it wasn't pretty -- there are at least a dozen of problems
with this, but nothing a day of work I offered to do can't fix; I wrote
this because it was faster than talking without a concrete example to
have some figures, and that took me less time than the rest of this mail)
Damn you to the nine circles of Hell, one by one, slowly, then all of
them at the same time.
You piqued my curiosity, so I did it, and I spent the day making it
work.
The execline git now has a 'multicall' make target. It will make an
"execline" binary that has *everything* in it. You can symlink it to
the name of an execline program and it will do what you expect. You can
also call the subcommand as argv[1]: "execline exit 3" will exit 3.
No install targets, no automatic stripping, no symlinks, nothing.
I don't want to officially support this configuration, because I *know*
it will be a time sink - every ricer on the planet will want me to
change
something. So you get the binary for your own enjoyment, and that's it.
Have fun. If it breaks, you get to keep both pieces.
It's really rough: it only marginally improves on your model, fixing
the most glaring problems. The only fancy thing it does is find the
applet via bsearch(), because that's easy and it saves about 20 strcmp()
per call. Apart from that, it's super dumb.
That said, you were right: that's some pretty hefty saving of disk
space. The execline binary is 169kB, statically linked against musl on
x86_64. That's neat. I expected it to be at least twice bigger. And
the data/bss isn't too bad either: only 2 pages. But that's because
execline programs use very little global memory in the first place -
the only places where globals are used is when state needs to be
accessed by signal handlers, and there's nothing I can do about that in
short-lived programs. (Long-lived programs use a selfpipe, so only
one int of global data is ever needed for them.)
So, all in all, much better results than I expected, it was a pleasant
surprise. Still, concatenating all the code feels really clunky, and
a real multicall program needs to be designed for this from the start,
which won't happen for execline in the foreseeable future, so this is
as much as you get for now.
If you're interested in hacking the thing, the magic happens in
tools/gen-multicall.sh.
libexecline is statically linked, so these pages aren't shared afaik?
That's right, I forgot it was always statically linked. If it helps,
changing ${LIBEXECLINE} to -lexecline in the src/execline/deps-exe
files, then running ./tools/gen-deps.sh > package/deps.mak, should
allow you to link against a dynamic libexecline. Can you do it, see
how much space you gain? That's a configuration I would definitely
support, even if it's slower - people usually love shared libraries.
I really don't see what's different between e.g. execline and coreutils,
who apparently thought it was worth it;
coreutils also thought it was worth it to implement true --help and
true --version, so I'll leave to your imagination how much I value their
technical judgment.
The only way to know for sure whether it will be worth it is to stop
speculating and start profiling, which is what I did. And it appears
the results are interesting, so, that's great!
Sigh. I shouldn't feel that way, and any potential improvement should
be a source of joy, not dread - but really I wish the results weren't
so good. Now Pandora's box has been opened and everyone will want to
use the multicall exclusively, so at some point I'll have to support it,
i.e. ensure it's actually correct and enhance its maintainability.
And that means a lot more work. :(
But, unfortunately for you, the full openrc suite is 2.2MB (5 on arm
with bloated aarch64), which is a bit less than the s6 suite :-D
No, that's fair. It's true that s6 takes a bit more disk space.
Where OpenRC loses is RAM and CPU, because it does everything in
shell scripts. And shell scripts definitely win on disk space. :)
--
Laurent
Re: single-binary for execline programs?
Laurent Bercot wrote on Wed, Feb 01, 2023 at 10:41:39AM +:
> > I'd go out a limb and say if you only support single-binary mode, some
> > of the code could be simplified further by sharing some argument
> > handling, but it's hard to do simpler than your exlsn_main wrapper so
> > it'll likely be identical with individual programs not changing at all,
> > with just an extra shim to wrap them all; it's not like busybox where
> > individual binaries can be selected so a static wrapper would be dead
> > simple.
>
> I doubt much sharing would be possible.
>
> The main problem I have with multicall is that the program's
> functionality changes depending on argv[0]. You need to first select
> on argv[0], and *then* you can parse options and handle arguments.
> Note that each exlsn_* function needs its own call to subgetopt_r(),
> despite the options being very similar because they all fill an
> eltransforminfo_t structure.
Yes, as I've mentioned you've already done a great job at sharing as
much as possible.
I'm not expecting any change to the program.
Look, here's a trivial, suboptimal wrapper, far from pretty:
$ cd execline; ./configure
$ cd src
$ grep -l 'int main' */*.c | while read f; do
b=${f##*/}; b=${b%.c}; b=${b//-/_}
sed -i -e 's/int main/int main_'"$b"'/' "$f"
echo "$b" >> programs
done
$ {
cat <
int main (int argc, const char **argv, char const *const *envp)
{
const char *app = strrchr(argv[0], '/');
if (app) app++; else app=argv[0];
#define APP(name) if (strcmp(app, #name) == 0) return main_##name(argc, argv,
envp);
EOF
sed -e 's/.*/APP(&)/' < programs
cat < wrapper.c
$ gcc -O2 -I$PWD/include -I$PWD/include-local \
*/*.c wrapper.c -o wrapper \
-lskarnet -Wno-implicit-function-declaration
$ ln wrapper execline_cd
$ ln wrapper if
$ ./if true '' ./execline_cd / ls
(look, I said it wasn't pretty -- there are at least a dozen of problems
with this, but nothing a day of work I offered to do can't fix; I wrote
this because it was faster than talking without a concrete example to
have some figures, and that took me less time than the rest of this mail)
$ size wrapper
textdata bss dec hex filename
9716728601136 101163 18b2b wrapper (glibc)
9852928361264 102629 190e5 wrapper (musl)
> Having a shim over *all* the execline programs would be that,
> multiplied by the number of programs; at the source level, there would
> not be any significant refactoring, because each program is pretty much
> its own thing. An executable is its own atomic unit, more or less.
>
> If anything, execline is the package that's the *least* adapted to
> multicall because of this. There is no possible sharing between
> "if" and "piperw", for instance, because these are two small units with
> very distinct functionality. The only way to make execline suited to
> multicall would be to entirely refactor the code of the executables and
> make a giant library, à la busybox. And I am familiar enough with
> analyzing and patching busybox that I certainly do not want to add that
> kind of maintenance nightmare to execline.
I don't think any more refactoring would be useful, I don't see the
problem of looking at argv[0] first independantly... And gcc still found
quite a bit to share as the sum of all text segments of all binaries
goes to ~235000; many binaries really do sum up.
(And that's before ELF/ld overhead)
> Anything that can be shared in execline is pretty much already shared
> in libexecline. If you build execline with full shared libraries, you
> get as much code sharing as is reasonably accessible without a complete
> rearchitecture.
libexecline is statically linked, so these pages aren't shared afaik?
My understanding is that if any symbol from a compilation unit (a .lo in
the .a) are used, the whole unit is going to be duplicated there, and
runtime has no way of figuring that out.
Of course, C runtime also probably amounts for a part of that
difference.
> The "one unique binary" argument applies better to some of my other
> software; for instance, the latest s6-instance-* additions to s6.
> I considered making a unique "s6-instance" binary, with varying
> functionality depending on an argv[1] subcommand; I eventually decided
> against it because it would have broken UI consistency with the rest of
> s6, but it would have been a reasonable choice for this set of programs -
> which are already thin wrappers around library calls and share a lot
> of code. Same thing with s6-fdholder-*.
> execline binaries, by contrast, are all over the place, and *not* good
> candidates for multicall.
I really don't see what's different between e.g. execline and coreutils,
who apparently th
Re: single-binary for execline programs?
I believe I did my homework looking first -- are there other discussion channels than this list that one should be aware of? The lists are definitely the only place you *should* be aware of, but there are a lot of informal spaces where discussions happen, because not everyone is as well-behaved as you are :) Github issues, webforums of other projects, IRC channels, etc. The important stuff normally only happens here, but I'm getting user feedback from several sources. I'd go out a limb and say if you only support single-binary mode, some of the code could be simplified further by sharing some argument handling, but it's hard to do simpler than your exlsn_main wrapper so it'll likely be identical with individual programs not changing at all, with just an extra shim to wrap them all; it's not like busybox where individual binaries can be selected so a static wrapper would be dead simple. I doubt much sharing would be possible. The main problem I have with multicall is that the program's functionality changes depending on argv[0]. You need to first select on argv[0], and *then* you can parse options and handle arguments. Note that each exlsn_* function needs its own call to subgetopt_r(), despite the options being very similar because they all fill an eltransforminfo_t structure. Having a shim over *all* the execline programs would be that, multiplied by the number of programs; at the source level, there would not be any significant refactoring, because each program is pretty much its own thing. An executable is its own atomic unit, more or less. If anything, execline is the package that's the *least* adapted to multicall because of this. There is no possible sharing between "if" and "piperw", for instance, because these are two small units with very distinct functionality. The only way to make execline suited to multicall would be to entirely refactor the code of the executables and make a giant library, à la busybox. And I am familiar enough with analyzing and patching busybox that I certainly do not want to add that kind of maintenance nightmare to execline. Anything that can be shared in execline is pretty much already shared in libexecline. If you build execline with full shared libraries, you get as much code sharing as is reasonably accessible without a complete rearchitecture. Any significant disk space you would gain in a multicall binary compared to a bunch of dynamically linked executables would come from the deduplication of unavoidable ELF boilerplate and C run-time, and that's basically it. The "one unique binary" argument applies better to some of my other software; for instance, the latest s6-instance-* additions to s6. I considered making a unique "s6-instance" binary, with varying functionality depending on an argv[1] subcommand; I eventually decided against it because it would have broken UI consistency with the rest of s6, but it would have been a reasonable choice for this set of programs - which are already thin wrappers around library calls and share a lot of code. Same thing with s6-fdholder-*. execline binaries, by contrast, are all over the place, and *not* good candidates for multicall. Hmm, I'd need to do some measurements, but my impression would be that since the overall size is smaller it should pay off for any pipeline calling more than a handful of binaries, as you'll benefit from running the same binary multiple times rather than having to look through multiple binaries (even without optimizing the execs out). Yes, you might win a few pages by sharing the text, but I'm more concerned about bss and data. Although I take some care in minimizing globals, I know that in my typical small programs, it won't matter if I add an int global, because the amount of global data I need will never reach 4k, so it won't map an extra page. When you start aggregating applets, the cost of globals skyrockets. You need to pay extra attention to every piece of data. Let me bring the example of busybox again: vda, the maintainer, does an excellent job of keeping the bss/data overhead low (only 2 pages of global private/dirty), but that's at the price of keeping it front and center, always, when reviewing and merging patches, and nacking stuff that would otherwise be a significant improvement. It's *hard*, and hampers code agility in a serious way. I don't want that. Sure, you can say that globals are a bad idea anyway, but a lot of programs need *some* state, if local to a TU - and the C and ELF models make it so that TU-local variables still end up in the global data section. Even almost 1MB (the x86_64 version that doesn't have the problem, package currently 852KB installed size + filesystem overhead..) is still something I consider big for the systems I'm building, even without the binutils issue it's getting harder to fit in a complete rootfs in 100MB. I will never understand how disk space is an issue for exec
Re: single-binary for execline programs?
alice wrote on Wed, Feb 01, 2023 at 07:22:14AM +0100: > > Just looking at the s6 suite (s6, s6-rc, execline, skalibs, > > s6-linux-init) I'm looking at a 3MB increase (because I won't be able to > > get rid of openrc for compatibility with user scripts it'll have to live > > in compat hooks...) ; being able to shave ~700KB of that would be > > very interesting for me (number from linking all .c together with a > > dummy main wrapper, down 148KB) > > (s6-* dozen of binaries being another similar target and would shave a > > bit more as well, build systems being similar I was hoping it could go > > next if this had been well received) > > out of (somewhat off-topic) curiosity, what is the layout here? > the general answer to such a case generally is: > "sure, it's 3MB. but it's a well-implemented well-oiled well-used 3MB, and the > 'business software' is hundreds of times that", but maybe this is something > special? > > given the (below) talk of inexperienced users, it makes me wonder if > everything > is in this 100mb, or if it's only a reserved rootfs for you while the rest if > customer-used. Exactly, we have a double-copy rootfs that we try to keep as small as possible and update atomically; then the rest of the eMMC/SD card is left for user containers in a separate partition. (I actually lied a bit here because the container runtime itself depends on podman, which is a huge go binary that in itself doubles the rootfs size ; and we cut some slack space that made the alpine 3.17 update possible, but I'm definitely counting each MB at this point, it's getting difficult to just install a debug kernel now... I don't want to waste anyone's time, which is why I offered to do it, but reviews still take time and as said previously I won't push further. I'd suggest "there's more information here", but it's all in Japanese: https://armadillo.atmark-techno.com/guide/armadillo-base-os You'd probably learn more from the rootfs[1] and update scripts[2] directly: [1] (rootfs content binary, 57MB) https://armadillo.atmark-techno.com/files/downloads/armadillo-iot-g4/baseos/baseos-x2-3.17.1-at.2.tar.zst [2] (rootfs and image builder) https://armadillo.atmark-techno.com/files/downloads/armadillo-iot-g4/tool/build-rootfs-v3.17-at.2.tar.gz [3] https://github.com/atmark-techno/mkswu/ There's probably plenty to improve and I never got any external feedback, so feel free to break everything and ask if you have time, I'd be curious if you can make sense of it without the japanese docs) > > > You're not the only one who is uncomfortable with it, but it's really a > > > perception thing. There has never been a problem caused by it. Shells > > > don't get confused. External tools don't get confused. On this aspect, > > > Unix is a lot more correct and resilient than you give it credit for. :) > > > > Shells and external tools would definitely be fine, they're not looking > > there in the first place. > > I think you're underestimating what users who haven't used a unix before > > can do though; I can already picture some rummaging in /bin and > > wondering why posix-cd "doesn't work" or something... We get impressive > > questions sometimes. > > i definitely feel for you there (in regards to inexperienced user questions), > but i'd say that generally very low level (relatively) systems integration > software is not the place where "inexperienced user support" is in-scope. the > easiest answer is that if somebody runs into such a scenario, they'll just > have > to learn the answer to the question, not have an "answer" pre-implemented for > them via a workaround (such as this one) that removes it. > > that is to say, resolving this (question-)case specifically would not be a > benefit for execline itself. Right, this is also completely off topic, I probably shouldn't have started digging this hole :) I'm not arguing for removing the installed tools here, just saying I will likely get some questions about it once we make the switch. Nothing that cannot be dealt with, but I am a greedy dev so I started asking for too much. -- Dominique Martinet | Asmadeus
Re: single-binary for execline programs?
On Wed Feb 1, 2023 at 6:58 AM CET, Dominique Martinet wrote: > Laurent Bercot wrote on Wed, Feb 01, 2023 at 04:49:47AM +: > > > It should be fairly easy to do something like coreutils' > > > --enable-single-binary without much modification > > > > The subject has come up a few times recently, > > I believe I did my homework looking first -- are there other discussion > channels than this list that one should be aware of? > > > so, at the risk of being > > blunt, I will make it very clear and definitive, for future reference: > > > > No. It will not happen. > > Well, thanks for the clear answer, I'm glad I asked first! > > I'm a sore loser though, so I'll develop a bit more below. You've > probably got better to do so feel free to just say you're not changing > your mind or pointing me at the other discussions and I'll stop bugging > you. > > > The fact that toolchains are becoming worse and worse is not imputable > > to execline, or to the way I write or package software. It has always > > been possible, and reasonable, to provide a lot of small binaries. > > Building a binary is not inherently more complicated today than it was > > 20 years ago. There is no fundamental reason why this should change; the > > only reason why people are even thinking this is that there is an > > implicit assumption that software always becomes better with time, and > > using the latest versions is always a good idea. I am guilty of this > > too. > > > > This assumption is true when it comes to bugs, but it becomes false if > > the main functionality of a project is impacted. > > If a newer version of binutils is unable to produce reasonably small > > binaries, to the point that it incites software developers to change > > their packaging to accommodate the tool, then it's not an improvement, > > it's a recession. And the place to fix it is binutils. > > I definitely agree with this, I reported the problem in the bz I linked, > and the reception has been rather good -- I trust we'll get back to > smaller binaries in the next version or otherwise near future. > > > Multicall binaries have costs, mostly maintainability costs. > > Switching from a multiple binaries model to a multicall binary model > > because the tooling is making the multiple binaries model unusably > > expensive is basically moving the burden from the tooling to the > > maintainer. Here's a worse tool, do more effort to accommodate it! > > I guess it isn't completely free, but it certainly isn't heavy if the > abstraction isn't done too badly. > > I'd go out a limb and say if you only support single-binary mode, some > of the code could be simplified further by sharing some argument > handling, but it's hard to do simpler than your exlsn_main wrapper so > it'll likely be identical with individual programs not changing at all, > with just an extra shim to wrap them all; it's not like busybox where > individual binaries can be selected so a static wrapper would be dead > simple. > > > Additionally to maintainability costs, multicall binaries also have a > > small cost in CPU usage (binary starting time) and RAM usage (larger > > mappings, fewer memory optimizations) compared to multiple binaries. > > These costs are paid not by the maintainer, but by the users. > > Hmm, I'd need to do some measurements, but my impression would be that > since the overall size is smaller it should pay off for any pipeline > calling more than a handful of binaries, as you'll benefit from running > the same binary multiple times rather than having to look through > multiple binaries (even without optimizing the execs out). > > Memory in particular ought to be shared for r-x pages, or there's some > problem with the system. I'm not sure if it'll lazily load only the > pages it requires for execution or if some readahead will read it all > (it probably should), but once it's read it shouldn't take space > multiple times, so multiple binaries is likely to take more space when > you include vfs cache as soon as you call a few in a row; memory usage > should be mostly identical to disk usage in practice. > > Anyway, I'll concede that in doubt, let's call it a space vs. speed > tradeoff where I'm favoring space. > > > Well, no. If having a bunch of execline binaries becomes more expensive > > in disk space because of an "upgrade" in binutils, that is a binutils > > problem, and the place to fix it is binutils. > > I shouldn't have brought up the binutils bug. > Even almost 1MB (the x86_64 version that doesn't have the problem, > package currently 852KB installed size + filesystem overhead.
Re: single-binary for execline programs?
Laurent Bercot wrote on Wed, Feb 01, 2023 at 04:49:47AM +: > > It should be fairly easy to do something like coreutils' > > --enable-single-binary without much modification > > The subject has come up a few times recently, I believe I did my homework looking first -- are there other discussion channels than this list that one should be aware of? > so, at the risk of being > blunt, I will make it very clear and definitive, for future reference: > > No. It will not happen. Well, thanks for the clear answer, I'm glad I asked first! I'm a sore loser though, so I'll develop a bit more below. You've probably got better to do so feel free to just say you're not changing your mind or pointing me at the other discussions and I'll stop bugging you. > The fact that toolchains are becoming worse and worse is not imputable > to execline, or to the way I write or package software. It has always > been possible, and reasonable, to provide a lot of small binaries. > Building a binary is not inherently more complicated today than it was > 20 years ago. There is no fundamental reason why this should change; the > only reason why people are even thinking this is that there is an > implicit assumption that software always becomes better with time, and > using the latest versions is always a good idea. I am guilty of this > too. > > This assumption is true when it comes to bugs, but it becomes false if > the main functionality of a project is impacted. > If a newer version of binutils is unable to produce reasonably small > binaries, to the point that it incites software developers to change > their packaging to accommodate the tool, then it's not an improvement, > it's a recession. And the place to fix it is binutils. I definitely agree with this, I reported the problem in the bz I linked, and the reception has been rather good -- I trust we'll get back to smaller binaries in the next version or otherwise near future. > Multicall binaries have costs, mostly maintainability costs. > Switching from a multiple binaries model to a multicall binary model > because the tooling is making the multiple binaries model unusably > expensive is basically moving the burden from the tooling to the > maintainer. Here's a worse tool, do more effort to accommodate it! I guess it isn't completely free, but it certainly isn't heavy if the abstraction isn't done too badly. I'd go out a limb and say if you only support single-binary mode, some of the code could be simplified further by sharing some argument handling, but it's hard to do simpler than your exlsn_main wrapper so it'll likely be identical with individual programs not changing at all, with just an extra shim to wrap them all; it's not like busybox where individual binaries can be selected so a static wrapper would be dead simple. > Additionally to maintainability costs, multicall binaries also have a > small cost in CPU usage (binary starting time) and RAM usage (larger > mappings, fewer memory optimizations) compared to multiple binaries. > These costs are paid not by the maintainer, but by the users. Hmm, I'd need to do some measurements, but my impression would be that since the overall size is smaller it should pay off for any pipeline calling more than a handful of binaries, as you'll benefit from running the same binary multiple times rather than having to look through multiple binaries (even without optimizing the execs out). Memory in particular ought to be shared for r-x pages, or there's some problem with the system. I'm not sure if it'll lazily load only the pages it requires for execution or if some readahead will read it all (it probably should), but once it's read it shouldn't take space multiple times, so multiple binaries is likely to take more space when you include vfs cache as soon as you call a few in a row; memory usage should be mostly identical to disk usage in practice. Anyway, I'll concede that in doubt, let's call it a space vs. speed tradeoff where I'm favoring space. > Well, no. If having a bunch of execline binaries becomes more expensive > in disk space because of an "upgrade" in binutils, that is a binutils > problem, and the place to fix it is binutils. I shouldn't have brought up the binutils bug. Even almost 1MB (the x86_64 version that doesn't have the problem, package currently 852KB installed size + filesystem overhead..) is still something I consider big for the systems I'm building, even without the binutils issue it's getting harder to fit in a complete rootfs in 100MB. Just looking at the s6 suite (s6, s6-rc, execline, skalibs, s6-linux-init) I'm looking at a 3MB increase (because I won't be able to get rid of openrc for compatibility with user scripts it'll have to live in compat hooks...) ; being able to shave ~700KB of that would be very interesting for me (number from linking all .c tog
Re: single-binary for execline programs?
In particular there's a "feature" with recent binutils that makes every binary be at least 64KB on arm/aarch64[1], so the execline package is a whopping 3.41MB[2] there (... and still 852KB on x86_64[3]) -- whereas just doing a dummy sed to avoid conflict on main and bundling all .c together in a single binary yields just 148KB (x86_64 but should be similar on all archs -- we're talking x20 bloat from aarch64/armv7 sizes! Precious memory and disk space!) > (...) It should be fairly easy to do something like coreutils' --enable-single-binary without much modification The subject has come up a few times recently, so, at the risk of being blunt, I will make it very clear and definitive, for future reference: No. It will not happen. The fact that toolchains are becoming worse and worse is not imputable to execline, or to the way I write or package software. It has always been possible, and reasonable, to provide a lot of small binaries. Building a binary is not inherently more complicated today than it was 20 years ago. There is no fundamental reason why this should change; the only reason why people are even thinking this is that there is an implicit assumption that software always becomes better with time, and using the latest versions is always a good idea. I am guilty of this too. This assumption is true when it comes to bugs, but it becomes false if the main functionality of a project is impacted. If a newer version of binutils is unable to produce reasonably small binaries, to the point that it incites software developers to change their packaging to accommodate the tool, then it's not an improvement, it's a recession. And the place to fix it is binutils. The tooling should be at the service of programmers, not the other way around. It is a similar issue when glibc makes it expensive in terms of RAM to run a large number of copies of the same process. Linux, like other Unix-like kernels, is very efficient at this, and shares everything that can be shared, but glibc performs *a lot* of private mappings that incur considerable overhead. (See the thread around this message: https://skarnet.org/lists/supervision/2804.html for an example.) Does that mean that running 100 copies of the same binary is a bad model? No, it just means that glibc is terrible at that and needs improvement. Back in the day when Solaris was relevant, it had an incredibly expensive implementation of fork(), which made it difficult, especially with the processing power of 1990s-era Sun hardware, to write servers that forked and still served a reasonable number of connections. It led to emerging "good practices", that were taught by my (otherwise wonderful) C/Unix programming teacher, and that were: fork as little as possible, use a single process to do everything. And that's how most userspace on Solaris worked indeed. It did a lot of harm to the ecosystem, turning programs into giant messes because people did not want to use the primitives that were available to them for fear of inefficiency, and jumping through hoops to work around it at the expense of maintainability. Switching to Linux and its efficient fork() was a relief. Multicall binaries have costs, mostly maintainability costs. Switching from a multiple binaries model to a multicall binary model because the tooling is making the multiple binaries model unusably expensive is basically moving the burden from the tooling to the maintainer. Here's a worse tool, do more effort to accommodate it! Additionally to maintainability costs, multicall binaries also have a small cost in CPU usage (binary starting time) and RAM usage (larger mappings, fewer memory optimizations) compared to multiple binaries. These costs are paid not by the maintainer, but by the users. Everyone loses. Well, no. If having a bunch of execline binaries becomes more expensive in disk space because of an "upgrade" in binutils, that is a binutils problem, and the place to fix it is binutils. In the long run this could also provide a workaround for conflicting names, cf. old 2016 thread[4], if we'd prefer either running the appropriate main directly or re-exec'ing into the current binary after setting argv[0] appropriately for "builtins". There have been no conflicts since "import". I do not expect more name conflicts in the future, and in any case, that is not an issue that multicall binaries can solve any better than multiple binaries. These are completely orthogonal things. (I assume you wouldn't like the idea of not installing the individual commands, but that'd become a possibility as well. I'm personally a bit uncomfortable having something in $PATH for 'if' and other commands that have historically been shell builtins, but have a different usage for execline...) You're not the only one who is uncomfortable with it, but it's really a perception thing. There has never been a problem caused by it. Shells
single-binary for execline programs?
Hello, I'm currently having a fresh look at s6 on alpine (thanks for the recent work with dynamic services! Looking forward to seeing it mature!). One thing that surprised me is how many small executables the programs come with. In particular there's a "feature" with recent binutils that makes every binary be at least 64KB on arm/aarch64[1], so the execline package is a whopping 3.41MB[2] there (... and still 852KB on x86_64[3]) -- whereas just doing a dummy sed to avoid conflict on main and bundling all .c together in a single binary yields just 148KB (x86_64 but should be similar on all archs -- we're talking x20 bloat from aarch64/armv7 sizes! Precious memory and disk space!) I'm expectant that binutils will improve that ultimately, but it never hurts to aim higher :) [1] https://sourceware.org/bugzilla/show_bug.cgi?id=28824 [2] https://pkgs.alpinelinux.org/package/edge/main/aarch64/execline [3] https://pkgs.alpinelinux.org/package/edge/main/x86_64/execline It should be fairly easy to do something like coreutils' --enable-single-binary without much modification, for example compile each executable with -Dmain=main_$program and have a small wrapper that forwards to main_$argv0 (or just rename if that becomes the default behaviour right away, that'd be even simpler). I would be happy to contribute to that if you're not against the idea. In the long run this could also provide a workaround for conflicting names, cf. old 2016 thread[4], if we'd prefer either running the appropriate main directly or re-exec'ing into the current binary after setting argv[0] appropriately for "builtins". (I assume you wouldn't like the idea of not installing the individual commands, but that'd become a possibility as well. I'm personally a bit uncomfortable having something in $PATH for 'if' and other commands that have historically been shell builtins, but have a different usage for execline...) [4] https://skarnet.org/lists/skaware/0737.html Cheers, -- Dominique
Re: [execline] foreground/background suggestion
I didn't think about that (facepalm)... Thank you! That'll do.
Re: [execline] foreground/background suggestion
Hello, You may want to use if: https://skarnet.org/software/execline/if.html --- Best regards, Peter
[execline] foreground/background suggestion
Hi, I've been using execline to script a build system, it has been great as
it's quite direct and does it job very well; and with its use I've come to
believe that it could be a huge convience to add a flag, or an alternative
binary, to foreground/background which would exit in case of "failure" (could
be like forx -o/-x).
I usually do this:
foreground { command }
importas -iu status ?
ifelse -Xn { test "${status}" -eq 0 } { exit $status }
[...]
I don't know if it's possible (I'm not too aware of the internals), but I
believe that, being possible, it would be worth to add a flag or something,
because it would be like its usual behaviour (as any command failing would
break the chain) and would avoid the need to create many "statusX".
Unless I've been stupid and there's a better way to do this.
Re: execline: getpid doesn't actually recognize -P and -p
I guess that because they are not mentioned in the call to subgetopt_r(): Thanks! Fixed in current git. New version coming at some point in the future, some changes are pending so a bugfix release would be more work than it's probably worth. -- Laurent
Re: execline: Minor typo in el_transform.html?
Chomping is off by default, or if you give the -c switch. Should that 'Chomping' be 'Crunching'? Indeed, good catch. Fixed, thanks! -- Laurent
execline: Minor typo in el_transform.html?
Hello, Beneath the crunch heading, first paragraph, last sentence, the document currently reads: Chomping is off by default, or if you give the -c switch. Should that 'Chomping' be 'Crunching'? i.e.: -C/-c crunching on/off -n/-N chomping on/off
Re: Implementing a helper for systemd's LISTEN_* environment variables using execline
socket-helper { name1 /run/name1.socket name2 /run/name2.socket } prog
Currently socket-helper is a shell script, as I haven't found a way to
generate the unknown number of `s6-ipcserver-socketbinder /path/sock
fdmove N 0` using just execline. Is this socket-helper (or an
easier-to-implement analogue of it) possible in execline? I know
`getpid` allows a one-line implementation of the first environment
variable, but I'm at a loss on how to implement the last two.
Do you mean you have a variable number of (name, socket) pairs?
In any case, it is certainly *possible* to implement something like
this in execline, because execline is Turing-complete, but it doesn't
mean that it's easy or even a good idea. To handle a variable number
of arguments, you need to iterate over them, and maintaining a state
is difficult in execline, so it would very likely be more complicated
than is worth.
I'd advise you to stick with your shell script. To be honest, if I were
writing such a helper, I'd make it a small C program.
The best course of action would naturally be to convince the gpg-agent
authors that socket pre-opening (for which LISTEN_FDS is a terrible
interface but that's not our concern here) has nothing to do with
foregrounding, and they should entirely decouple these.
--
Laurent
Implementing a helper for systemd's LISTEN_* environment variables using execline
I'm supervising the gpg-agent daemon using s6. Unfortunately, the only
official way of getting it to cooperate with a supervision suite is by
pre-opening sockets and passing information about them through
systemd's LISTEN_* (LISTEN_PID, LISTEN_FDS, LISTEN_FDNAMES)
environment variables. It'll only open the sockets itself if you pass
an argument that will background it. Thankfully it does not link
against libsystemd, and therefore s6 has all the tools needed to
fulfill this requirement.
The run script ended up being quite verbose due to all the
s6-ipcserver-socketbinder'ing, so I thought of writing a helper. The
usage I envisioned would be:
socket-helper { name1 /run/name1.socket name2 /run/name2.socket } prog
Currently socket-helper is a shell script, as I haven't found a way to
generate the unknown number of `s6-ipcserver-socketbinder /path/sock
fdmove N 0` using just execline. Is this socket-helper (or an
easier-to-implement analogue of it) possible in execline? I know
`getpid` allows a one-line implementation of the first environment
variable, but I'm at a loss on how to implement the last two.
Re: execline: case: Subexpression matching does not work?
When I try to run the example program in the documentation for case, the default case is executed every time: ... yeah, that's because the example program in the documentation was wrong. ^^' (I wrote it with BREs in mind, then made EREs the default, and ended up with something that works in neither case. Yay.) Fixed now. Remove all the backslashes in the example and in the output: EREs treat parentheses as special characters without escaping. Thanks for the report! I had been secretly hoping for exactly this functionality in execline for a good while now, so I would appreciate it if I could have a version of the case command that works properly! The case command should definitely work properly. Just don't trust regular expressions you see in the wild, and especially not from me. ;) -- Laurent
execline: case: Subexpression matching does not work?
Hello,
When I try to run the example program in the documentation for case, the
default case is executed every time:
```
$ cat >match
#!/bin/execlineb -S1
emptyenv
case -N -- $1
{
"\\([fo]+\\)bar\\(baz\\)" { /usr/bin/env }
}
^D
$ match fbarbaz
$
```
I see this in the release version execline as compiled on my own, as well as in
the version currently in the Void Linux repositories, which uses the release
version of the software and skalibs.
As far as I can tell, matching still works correctly when *not* using
subexpressions:
```
$ execlineb -c 'emptyenv case -N -- apple { "a.*" { /usr/bin/env } }'
#=0
0=a.*
$
```
Though I have hardly tested thoroughly.
I had been secretly hoping for exactly this functionality in execline for a
good while now, so I would appreciate it if I could have a version of the case
command that works properly!
signature.asc
Description: PGP signature
[announce] {s6,s6-networking,execline}-man-pages updated
"Laurent Bercot" writes:
New versions of all the skarnet.org packages are available.
The {s6,s6-networking,execline}-man-pages repos have now been
updated accordingly:
*
https://github.com/flexibeast/s6-man-pages/releases/tag/v2.11.0.0.1
*
https://github.com/flexibeast/s6-networking-man-pages/releases/tag/v2.5.0.0.1
*
https://github.com/flexibeast/execline-man-pages/releases/tag/v2.8.1.0.1
Alexis.
Re: [execline] [RFC] Allow `foreground` to handle signals
El dom, 27 jun 2021 a las 17:13, Laurent Bercot ()
escribió:
> It sounds like your previous foreground process group (i.e. the
> processes that launch your script) attempted to write to, or read from,
> the terminal while your editor was running. That is pretty weird.
> Replace -g with -f, see what happens. If your editor is stopped, just
> remove the options entirely and run the editor in a new session; it
> looks like your caller program is buggy.
>
That is indeed the problem: git output to console a message to tell you
it's waiting for the editor to stop to commit.
The fix was to:
```
foreground {
s6-setsid -g
editor ${file}
}
foreground {
s6-setsid -g
spellcheck ${file}
}
s6-setsid -g
vcs-commit ${file}
```
Re: [execline] [RFC] Allow `foreground` to handle signals
Isn't job control the task of commands such as `foreground` and `background`? No, it's not. "foreground" means that the script waits for the process being spawned to exit before resuming; "background" means that the parent and the child execute in parallel. This has nothing to do with job control. Is your opinion that this modified foreground is outside the scope of execline, ant thus there should it be a toolset for job control? Yes, and maybe. Job control is out of scope of regular execline binaries, and I don't think that execline can bring much value over a shell when interactivity is involved, so I'm in no hurry to write a toolset for job control - but if you have worthwhile ideas for it, feel free to submit them. Thanks, this is half the answer, now the editor receives the signals, the other half of the answer is spwaning the next process when the editor exits. As of now, on the terminal one needs to use `fg` to continue the process. It sounds like your previous foreground process group (i.e. the processes that launch your script) attempted to write to, or read from, the terminal while your editor was running. That is pretty weird. Replace -g with -f, see what happens. If your editor is stopped, just remove the options entirely and run the editor in a new session; it looks like your caller program is buggy. -- Laurent
Re: [execline] [RFC] Allow `foreground` to handle signals
El dom, 27 jun 2021 a las 6:33, Laurent Bercot ()
escribió:
> execline was not designed to handle interactive scripts: you need
> some extra tooling in the presence of job control.
>
Isn't job control the task of commands such as `foreground` and
`background`?
Is your opinion that this modified foreground is outside the scope of
execline,
ant thus there should it be a toolset for job control?
To achieve that, replace "foreground { editor ${file} }" with
> "foreground { s6-setsid -g editor ${file} }".
>
Thanks, this is half the answer, now the editor receives the signals,
the other half of the answer is spwaning the next process
when the editor exits.
As of now, on the terminal one needs to use `fg` to continue the process.
Re: [execline] [RFC] Allow `foreground` to handle signals
On bash, the editor receives ^C and ignores the signal properly.
On execline, foreground receives ^C and closes the whole script,
maybe causing me to lose the work done so far.
execline was not designed to handle interactive scripts: you need
some extra tooling in the presence of job control.
Your problem here is that you have an interactive program, editor,
that can receive job control commands while running in the same
process group as the whole script, so job control commands, which
send signals to the process group, affect the whole script instead of
just the editor process. You need to run editor in another process
group (or another session).
To achieve that, replace "foreground { editor ${file} }" with
"foreground { setsid editor ${file} }".
That will create a new session for editor to run in. If you have
s6 installed, you can also use s6-setsid -g instead - that will create
a new process group without creating a new session.
Hope this helps,
--
Laurent
{s6,s6-networking,execline}-man-pages updated
Hi all, Unfortunately, due to my health issues, i'd not been able to get to new releases as soon as i'd have liked, but thankfully Colin Booth stepped up with PRs, which i've now merged. :-) Thanks Colin! https://github.com/flexibeast/s6-man-pages/releases/tag/v2.10.0.3.1 https://github.com/flexibeast/s6-networking-man-pages/releases/tag/v2.4.1.1.1 https://github.com/flexibeast/execline-man-pages/releases/tag/v2.8.0.1.1 Alexis.
Re: Execline Help: Need the Return Value and the Stdout of a Program
Thanks for the quick responses!
On Tue, Feb 23, 2021, at 03:38, Alex Raschi wrote:
> If i understood correctly, this should do what you want:
This was exactly it. Thanks for the help.
On Tue, Feb 23, 2021, at 06:31, Laurent Bercot wrote:
> As an addition, if you use execline-2.8.* then the -i option
> to backtick is the default behaviour, and there is a -E option to
> perform the 'importas' part automatically,
I like this option: it would really cut down on the boilerplate of
specifying the variable name 3 times. I tried to use it at first,
but my distro-supplied version is too old to have it, unfortunately.
> You could also add "unexport ?" between the foreground block and
> the s6-svscan execution, to ensure the ? environment variable
> added by foreground does not spill into the whole supervision tree.
Very good point; I will do so.
> {Debian-specific things}
Indeed I am using Debian's packaged version. I had noticed that the
various commands were not on the default PATH, and I had manually
added /usr/lib/execline/bin to it. My installation (execline 2.5.0.1-3
on Debian Buster) doesn't have the wrapper in /usr/bin/execlineb.
I thought that was strange, as it means the packaged version of
execline is broken unless you make edits to the PATH...
Thanks,
Scott
Re: Execline Help: Need the Return Value and the Stdout of a Program
#!/usr/lib/execline/bin/execlineb -P
foreground {
backtick -i file_loc {
fdmove -c 2 1 xmlstarlet sel -t -v //File/Path /etc/some-config.xml
}
importas -iu file_loc file_loc
if -n { test -e $file_loc }
create-file
}
/usr/bin/s6-svscan /service
LGTM. As an addition, if you use execline-2.8.* then the -i option
to backtick is the default behaviour, and there is a -E option to
perform the 'importas' part automatically, so the script could become:
#!/usr/lib/execline/bin/execlineb -P
foreground {
backtick -E file_loc {
fdmove -c 2 1 xmlstarlet sel -t -v //File/Path /etc/some-config.xml
}
if -n { test -e $file_loc }
create-file
}
/usr/bin/s6-svscan /service
You could also add "unexport ?" between the foreground block and
the s6-svscan execution, to ensure the ? environment variable
added by foreground does not spill into the whole supervision tree.
As another note, "foreground", "backtick", and "fdmove" need to be
in the caller's PATH in order to be found. If your execlineb binary
is in /usr/lib/execline/bin/execlineb, then chances are the other
binaries are as well, so you should make sure /usr/lib/execline/bin
is in your PATH prior to running this script. As far as I know,
only Debian and Ubuntu install execline there; if you're using the
Debian or Ubuntu packages, /usr/bin/execlineb is a wrapper that adds
/usr/lib/execline/bin to your PATH then execs into the real execlineb;
so you should use the #!/usr/bin/execlineb shebang.
(I do not condone the Debian package's way of installing execline,
but they found a way to make it work without breaking the other
skarnet.org packages, so it's all I can ask for and the rest is
their policy to decide on.)
--
Laurent
Re: Execline Help: Need the Return Value and the Stdout of a Program
On Tue, Feb 23, 2021 at 03:11:18AM -0500, Scott Colby wrote:
> Hello,
>
> I am having some difficulty translating the following shell script
> to execline:
>
> #!/usr/bin/env sh
> file_loc=$(xmlstarlet sel -t -v '//File/Path' /etc/some-config.xml 2>&1)
> # if //File/Path is missing from some-config.xml,
> # xmlstarlet will have exited non-zero and we skip this
> if test $? -eq 0
> then
> if ! test -e "$file_loc"
> then
> create-file
> fi
> fi
> exec /usr/bin/s6-svscan /service
>
> Here's what I have so far:
>
> #!/usr/lib/execline/bin/execlineb -P
> foreground {
> if -n {
> backtick -i file_loc {
> fdmove -c 2 1 xmlstarlet sel -t -v //File/Path /etc/some-config.xml
> }
> importas -iu file_loc file_loc test -e $file_loc
> }
> create-file
> }
> /usr/bin/s6-svscan /service
>
> The problem is, `if -n` can't differentiate between the failure of
> xmlstarlet within backtick and the failure of `test -e`. I only
> want create-file to be called if xmlstarlet succeeds and `test -e`
> fails. I've tried various permutations of wrapping backtick and
> importas with other if constructs, but couldn't find one that worked.
> Looking at the other conditional commands, maybe I could take
> advantage of ifthenelse setting $? before continuing the exec chain,
> but I'm wondering if there is a better way.
>
> How can I make such a script using execline?
>
> Thank you,
> Scott
>
Hi!
If i understood correctly, this should do what you want:
#!/usr/lib/execline/bin/execlineb -P
foreground {
backtick -i file_loc {
fdmove -c 2 1 xmlstarlet sel -t -v //File/Path /etc/some-config.xml
}
importas -iu file_loc file_loc
if -n { test -e $file_loc }
create-file
}
/usr/bin/s6-svscan /service
Move the backtick out of the if, if xmlstartlet fails backtick (-i) will
not execute the if (and importas) at all.
Execline Help: Need the Return Value and the Stdout of a Program
Hello,
I am having some difficulty translating the following shell script
to execline:
#!/usr/bin/env sh
file_loc=$(xmlstarlet sel -t -v '//File/Path' /etc/some-config.xml 2>&1)
# if //File/Path is missing from some-config.xml,
# xmlstarlet will have exited non-zero and we skip this
if test $? -eq 0
then
if ! test -e "$file_loc"
then
create-file
fi
fi
exec /usr/bin/s6-svscan /service
Here's what I have so far:
#!/usr/lib/execline/bin/execlineb -P
foreground {
if -n {
backtick -i file_loc {
fdmove -c 2 1 xmlstarlet sel -t -v //File/Path /etc/some-config.xml
}
importas -iu file_loc file_loc test -e $file_loc
}
create-file
}
/usr/bin/s6-svscan /service
The problem is, `if -n` can't differentiate between the failure of
xmlstarlet within backtick and the failure of `test -e`. I only
want create-file to be called if xmlstarlet succeeds and `test -e`
fails. I've tried various permutations of wrapping backtick and
importas with other if constructs, but couldn't find one that worked.
Looking at the other conditional commands, maybe I could take
advantage of ifthenelse setting $? before continuing the exec chain,
but I'm wondering if there is a better way.
How can I make such a script using execline?
Thank you,
Scott
{s6,s6-networking,execline}-man-pages updated
Hi all, i've now updated s6-man-pages, s6-networking-man-pages and execline-man-pages to reflect the recent releases: https://github.com/flexibeast/s6-man-pages/releases/tag/v2.10.0.2.1 https://github.com/flexibeast/s6-networking-man-pages/releases/tag/v2.4.1.0.1 https://github.com/flexibeast/execline-man-pages/releases/tag/v2.8.0.0.1 Alexis.
[announce] execline-2.8.0.0, s6-networking-2.4.1.0, +bugfix releases
Hello, New versions of some of the skarnet.org packages are available. skalibs-2.10.0.2: bugfixes execline-2.8.0.0: major version bump, but few and low-impact changes s6-2.10.0.2: bugfixes s6-linux-init-1.0.6.1: bugfixes s6-networking-2.4.1.0: minor version bump Some details: * execline-2.8.0.0 - The if program now propagates its child's exit code by default if it exits. - The backtick program's -i behaviour (exit on child failure or presence of a null character in its output) is now the default. Other behaviours in case of child failure can be obtained via -I, -x or -D options; -x is the new one. - These changes are compatible with all the common uses of if and backtick, but break compatibility in edge cases, which is why a major version bump is required. This has nothing in common with the previous major version bump, which had massive changes all over the place; this one should go smoothly, and will only impact very specific uses of backtick. execline now has man pages, thanks to the untiring flexibeast! The repository can be found here: https://github.com/flexibeast/execline-man-pages Please allow some time for the man pages to be updated to reflect the current HTML documentation. Currently, the man pages document execline-2.7.0.1; they are accurate for 2.8.0.0 except for the if and backtick changes. * s6-linux-init-1.0.6.1 - - Bugfixes. - When s6-linux-init is built with utmps, the default utmp user for s6-linux-init-maker was set to "utmp". That was a bug: now, by default, s6-linux-init-maker does not create the utmp services if the -U option is not given. If you used s6-linux-init-maker without the -U option and still need the utmps services, you should explicitly set "-U utmp". https://skarnet.org/software/s6-linux-init/ git://git.skarnet.org/s6-linux-init * s6-networking-2.4.1.0 - - Bugfixes (nothing security-related). - It is now possible to define a maximum amount of time spent in the TLS handshake no matter how s6-networking has been built. (The -K option has been implemented for the libtls backend.) - When SNI has been required, the TLS-related binaries now export the SSL_TLS_SNI_SERVERNAME option to their application; the variable contains the relevant server name. https://skarnet.org/software/s6-networking/ git://git.skarnet.org/s6-networking s6-networking has man pages as well: https://github.com/flexibeast/s6-networking-man-pages Enjoy, Bug-reports welcome. -- Laurent
[announce] execline-man-pages
Hi all, i've just finished porting the execline documentation to mdoc: https://github.com/flexibeast/execline-man-pages/releases/tag/v2.7.0.1.2 For further information, refer to the repository README: https://github.com/flexibeast/execline-man-pages Alexis.
Re: [announce] skalibs-2.10.0.1, execline-2.7.0.1, s6-2.10.0.1
Laurent Bercot writes: New skarnet.org packages are available: skalibs-2.10.0.1 execline-2.7.0.1 s6-2.10.0.1 And i've just updated the s6-man-pages repo accordingly: https://github.com/flexibeast/s6-man-pages/releases/tag/v2.10.0.1.1 Alexis.
[announce] skalibs-2.10.0.1, execline-2.7.0.1, s6-2.10.0.1
Hello, New skarnet.org packages are available: skalibs-2.10.0.1 execline-2.7.0.1 s6-2.10.0.1 Those are bugfix releases. I normally don't announce bugfix releases, but the bugs that have been fixed here are pretty visible (sorry about that!), so all users are encouraged to upgrade ASAP. https://skarnet.org/software/skalibs/ https://skarnet.org/software/execline/ https://skarnet.org/software/s6/ Enjoy, More bug-reports always welcome. -- Laurent
Re: Some doubts about execline and skalibs
Thanks to both of you for you answers!
On Thu, Dec 24, 2020 at 10:26:20PM +, Laurent Bercot wrote:
>
> Hi Alex,
>
>
> > I noticed that both skalibs and execline have -fno-stack-protector by
> > default, i haven't found anything related while searching commits or
> > mailing lists. Is this flag a left over or it's wanted?
>
> It's a nice default for code size and efficiency, when you're
> confident enough in your programming practices to assert you're not
> going to smash the stack. Call it hubris. :)
> (I have spat in the wind a lot of times, and regretted it more often
> than not, but I'm proud to say that this particular sputum has never
> hit me in the face. Yet.)
>
> It's like this because originally the skarnet.org project was made
> to run on small devices where that kind of optimization does count.
> I suspect nowadays it makes a lot less difference than it did a
> decade ago - so maybe I should just remove it and leave it to the
> toolchain's default. It's policy, after all.
>
> Anyway, it's overridable: any option you pass in CFLAGS will take
> precedence over the default flags. If the OpenBSD policy is to build
> everything with -fstack-protector, then it shouldn't be a problem to
> have it in CFLAGS.
I hadn't thought about small devices and yes this one is not a problem
but in my case i do: sed -i /-fno-stack-protector/d ${WRKSRC}/configure
because the ports system does not pass any flags about stack protector
and/or retguard, they are enabled by default.
> > Another thing i was wondering is why both packages put the shared
> > library under lib/ but the static one under lib/skalibs/ and
> > lib/execline/. If i leave libskarnet.a under lib/skalibs/, building
> > execline statically fails because it does not find libskarnet.a.
>
> I probably have already ranted about this zillions of times, but I
> cannot find a stable reference in the ML archives, so it must have
> been on #s6. Anyway, here goes:
>
> I have a pet peeve (actually, one of many): FHS and build tools like
> to treat shared libraries and static libraries the same way, when in
> reality they are *not the same kind of object at all*. Static libraries
> are build-time objects, used in development only; shared libraries are
> run-time objects that are also used at build time because reasons.
>
> To me, it's a complete misdesign that /usr/lib is used to store both
> static and shared libraries. I have a lot - *a lot* - of criticism to
> wield against FHS, but this one might lowkey take the cake. (Okay, no:
> the worst is that the FHS structure makes atomic package upgrades
> impossible. But /usr/lib is definitely top 4.)
>
> And so, since ld.so uses /usr/lib:/lib as its default path for shared
> libraries, I treat /usr/lib as a place to store shared libraries, i.e.
> run-time objects, and would rather store additional build-time objects
> somewhere else. And /usr/lib/$package didn't sound like a terrible
> option, since it's already an existing convention to store other
> read-only package-specific files such as libexec or data files.
>
> I agree that this is non-standard, and like every. single. fcking.
> time. I have deviated ever so slightly from the well-trodden path,
> despite extensively describing things in the INSTALL file, people
> bump against the obstacle. So much that it's probably not worth doing
> it in the first place, despite being cleaner in theory. You can
> probably expect the default for static libraries to change to /usr/lib
> at some point in the future, but I'm not completely ready to surrender
> that small hill yet.
>
> For now, if your configure script invocation does not change the
> defaults at all, it will automatically fill in the vpaths and linker
> invocations with the appropriate -L options, so things will work out of
> the box. But as soon as you deviate from the default, the configure
> scripts assume that you know what you are doing and have read the
> INSTALL file, and they stop holding your hand - so you need to use
> the appropriate --with-lib options in order to find all the static
> libraries a package needs.
> Alternatively, you can use --libdir=/usr/lib for all the skarnet.org
> packages, and it will nicely install everything into /usr/lib so
> things will work too.
My fault for not reading INSTALL carefully enough, this is a lesson for
the next times. Thanks for clarifying this again.
> > While execline built without warnings skalibs printed some, i think you
> > might find them useful so i'll leave them here (openbsd uses clang):
>
> Thanks. The extra parentheses have already been added in git. There is
> a new set of numbered releases planned for Very Soon (tm), likely very
> early next year; this wi
Re: Some doubts about execline and skalibs
Hi Alex, I noticed that both skalibs and execline have -fno-stack-protector by default, i haven't found anything related while searching commits or mailing lists. Is this flag a left over or it's wanted? It's a nice default for code size and efficiency, when you're confident enough in your programming practices to assert you're not going to smash the stack. Call it hubris. :) (I have spat in the wind a lot of times, and regretted it more often than not, but I'm proud to say that this particular sputum has never hit me in the face. Yet.) It's like this because originally the skarnet.org project was made to run on small devices where that kind of optimization does count. I suspect nowadays it makes a lot less difference than it did a decade ago - so maybe I should just remove it and leave it to the toolchain's default. It's policy, after all. Anyway, it's overridable: any option you pass in CFLAGS will take precedence over the default flags. If the OpenBSD policy is to build everything with -fstack-protector, then it shouldn't be a problem to have it in CFLAGS. Another thing i was wondering is why both packages put the shared library under lib/ but the static one under lib/skalibs/ and lib/execline/. If i leave libskarnet.a under lib/skalibs/, building execline statically fails because it does not find libskarnet.a. I probably have already ranted about this zillions of times, but I cannot find a stable reference in the ML archives, so it must have been on #s6. Anyway, here goes: I have a pet peeve (actually, one of many): FHS and build tools like to treat shared libraries and static libraries the same way, when in reality they are *not the same kind of object at all*. Static libraries are build-time objects, used in development only; shared libraries are run-time objects that are also used at build time because reasons. To me, it's a complete misdesign that /usr/lib is used to store both static and shared libraries. I have a lot - *a lot* - of criticism to wield against FHS, but this one might lowkey take the cake. (Okay, no: the worst is that the FHS structure makes atomic package upgrades impossible. But /usr/lib is definitely top 4.) And so, since ld.so uses /usr/lib:/lib as its default path for shared libraries, I treat /usr/lib as a place to store shared libraries, i.e. run-time objects, and would rather store additional build-time objects somewhere else. And /usr/lib/$package didn't sound like a terrible option, since it's already an existing convention to store other read-only package-specific files such as libexec or data files. I agree that this is non-standard, and like every. single. fcking. time. I have deviated ever so slightly from the well-trodden path, despite extensively describing things in the INSTALL file, people bump against the obstacle. So much that it's probably not worth doing it in the first place, despite being cleaner in theory. You can probably expect the default for static libraries to change to /usr/lib at some point in the future, but I'm not completely ready to surrender that small hill yet. For now, if your configure script invocation does not change the defaults at all, it will automatically fill in the vpaths and linker invocations with the appropriate -L options, so things will work out of the box. But as soon as you deviate from the default, the configure scripts assume that you know what you are doing and have read the INSTALL file, and they stop holding your hand - so you need to use the appropriate --with-lib options in order to find all the static libraries a package needs. Alternatively, you can use --libdir=/usr/lib for all the skarnet.org packages, and it will nicely install everything into /usr/lib so things will work too. While execline built without warnings skalibs printed some, i think you might find them useful so i'll leave them here (openbsd uses clang): Thanks. The extra parentheses have already been added in git. There is a new set of numbered releases planned for Very Soon (tm), likely very early next year; this will be a major skalibs release, which a few compatibility breaks, and every package is being updated for it, so you may want to wait for it before submitting your ports. :) src/libstddjb/child_spawn1_internal.c:35:23: warning: & has lower precedence than !=; != will be evaluated first [-Wparentheses] if (p[to & 1] != to & 1) I actually saw this one while reading the clang build logs a few months ago and realized it was a bug, so despite not being a fan of adding parentheses where they're not needed in order to silence the warnings, I have to admit that for once -Wparentheses was useful. :D (Fortunately, in practice 'to' is always 0 or 1, so the bug never manifested.) Finally is there a way to set limits like with ulimit but without using sh? Colin answered that one: s6-softlimit, in the s6 package. At some point the border between execline and s6 becomes fuzzy: there is an &quo
Re: Some doubts about execline and skalibs
Answering the part that I know the answer to. On Thu, Dec 24, 2020 at 02:38:04PM +0100, Alex Raschi wrote: > > Finally is there a way to set limits like with ulimit but without using > sh? > The command you're looking for is s6-softlimit in the s6 package. While limit setting is something that execline itself could handle that is rarely something that is done in stand-alone scripts and s6 (or some other process supervisor) is generally a requirement for most places where rlimit setting would be useful. -- Colin Booth
Some doubts about execline and skalibs
Hi,
I created 2 openbsd ports for skalibs and execline and i'm going to
submit them as soon i tested them enough, i was wondering a few things:
I noticed that both skalibs and execline have -fno-stack-protector by
default, i haven't found anything related while searching commits or
mailing lists. Is this flag a left over or it's wanted?
Another thing i was wondering is why both packages put the shared
library under lib/ but the static one under lib/skalibs/ and
lib/execline/. If i leave libskarnet.a under lib/skalibs/, building
execline statically fails because it does not find libskarnet.a.
While execline built without warnings skalibs printed some, i think you
might find them useful so i'll leave them here (openbsd uses clang):
src/libstddjb/bitarray_clearsetn.c:11:55: warning: operator '<<' has lower
precedence than '+'; '+' will be evaluated first [-Wshift-op-parentheses]
unsigned char mask = (1 << (a & 7)) - 1 ^ (1 << 1 + (b-1 & 7)) - 1 ;
~~ ~~^~~
src/libstddjb/bitarray_clearsetn.c:11:55: note: place parentheses around the
'+' expression to silence this warning
unsigned char mask = (1 << (a & 7)) - 1 ^ (1 << 1 + (b-1 & 7)) - 1 ;
^
()
1 warning generated.
src/libstddjb/child_spawn1_internal.c:35:23: warning: & has lower precedence
than !=; != will be evaluated first [-Wparentheses]
if (p[to & 1] != to & 1)
^
src/libstddjb/child_spawn1_internal.c:35:23: note: place parentheses around the
'!=' expression to silence this warning
if (p[to & 1] != to & 1)
^
( )
src/libstddjb/child_spawn1_internal.c:35:23: note: place parentheses around the
& expression to evaluate it first
if (p[to & 1] != to & 1)
^
( )
1 warning generated.
src/libstddjb/netstring_get.c:42:20: warning: incompatible pointer types
passing 'size_t *' (aka 'unsigned long *') to parameter of type 'uint64_t *'
(aka 'unsigned long long *') [-Wincompatible-pointer-types]
if (!n || n != size_scan(buf, )) return (errno = EPROTO, -1) ;
^~~~
src/include/skalibs/types.h:213:43: note: expanded from macro 'size_scan'
#define size_scan(s, u) size_scan_base(s, (u), 10)
^~~
src/include/skalibs/uint64.h:48:57: note: passing argument to parameter here
extern size_t uint64_scan_base (char const *, uint64_t *, uint8_t) ;
^
1 warning generated.
src/libstddjb/uint640_scan_base_max.c:9:1: warning: all paths through this
function will call itself [-Winfinite-recursion]
{
^
1 warning generated.
Finally is there a way to set limits like with ulimit but without using
sh?
Thanks you for your projects, execline is awesome!
Cheers, Alex
Re: execline introspection?
I have a oneshot that needs to perform, let's say, more complex operations than I'm willing to implement with execline ATM. So I want to call a regular shell script from the up/down files. However, since oneshots ignore the shebang I can neither change the interpreter, nor pass arguments to execlineb. And the working directory is that of the oneshot runner, so I also can't invoke the shell script with a relative path. It's no biggie, though. I just wanted to confirm that I need to call the script with absolute path. I confirm that it's the intended usage for oneshot scripts. In $srcdir/up, you just invoke "/path/to/your/script args...", and "script" can be any executable you want, including a shell script. The fact that the up and down scripts are interpreted with execline is an implementation detail, and the fact that it's possible to implement whole up/down scripts in those files, using execline facilities, is a nice optimization trick, but is by no means mandatory; the idea was always to have an independent script repository à la /etc/init.d to host your oneshots, and call scripts in there via their absolute paths from up/down. -- Laurent
Re: execline introspection?
Hi Laurent On 2020-10-15 Laurent Bercot wrote: >> Does execline have introspection features? I would like to determine >> the location of an execline script from within that script (like $0 >> in a shell script). > > No, it doesn't. The point was to launch the script as fast as > possible, with the shortest possible code path, so anything that's not > essential was removed. > > If it's something you need, though, I could probably add an option > that puts the full path to the script in $0. Thanks for confirming. It's not a big issue. I need to invoke an external script from a oneshot, and being able to automatically determine the parent directory would've been nice, but I can also use an absolute path. Regards Ansgar
Re: execline introspection?
Hi Dewayne
On 2020-10-16 Dewayne Geraghty wrote:
> On 16/10/2020 12:57 am, Ansgar Wiechers wrote:
>> Does execline have introspection features? I would like to determine the
>> location of an execline script from within that script (like $0 in a
>> shell script).
>>
>> I didn't see anything like that mentioned in the docs, so I assume it
>> doesn't, but maybe I've overlooked something.
>
> Perhaps this will be helpful
>
> Script content:
> #!/usr/local/bin/execlineb -S0
> foreground { echo zero is $0 }
> foreground { echo one is $1 }
Thanks for your suggestion, but unfortunately that doesn't work in my
scenario.
I have a oneshot that needs to perform, let's say, more complex
operations than I'm willing to implement with execline ATM. So I want to
call a regular shell script from the up/down files.
However, since oneshots ignore the shebang I can neither change the
interpreter, nor pass arguments to execlineb. And the working directory
is that of the oneshot runner, so I also can't invoke the shell script
with a relative path.
It's no biggie, though. I just wanted to confirm that I need to call the
script with absolute path.
Regards
Ansgar
Re: execline introspection?
Does execline have introspection features? I would like to determine the location of an execline script from within that script (like $0 in a shell script). Hi Ansgar, No, it doesn't. The point was to launch the script as fast as possible, with the shortest possible code path, so anything that's not essential was removed. If it's something you need, though, I could probably add an option that puts the full path to the script in $0. -- Laurent
execline introspection?
Hello Does execline have introspection features? I would like to determine the location of an execline script from within that script (like $0 in a shell script). I didn't see anything like that mentioned in the docs, so I assume it doesn't, but maybe I've overlooked something. Regards Ansgar Wiechers
[announce] execline-2.6.1.0
Hello, execline-2.6.1.0 is out. This version features a more expressive format for the envfile binary. Most of the files that are suitable for a systemd EnvironmentFile= directive are now parsable with envfile: double quotes are supported, backslashed newlines are supported, a few C escape sequences are supported (including octal and hexadecimal). So it is now possible to read most existing /etc/default configuration files without spawning a shell. Additionally, envfile now comes with a -I option that makes it ignore a nonexistent file, instead of failing. git://git.skarnet.org/execline https://skarnet.org/software/execline/ Enjoy, Bug-reports welcome. -- Laurent
Re: Compiling execline shared fails without --with-lib configure opt
On 12/03, Laurent Bercot wrote: > > Since I'm not using static libraries, I would not expect to need the > > "--with-lib" configure option. > > The --disable-static flag only tells the build system not to > build the static execline library. By default, executables are still > built against the static version of skalibs. > > If you want to link executables against the dynamic version of skalibs, > you need to add the --disable-allstatic option to configure. > > IOW: to make a dynamic install, you need both --disable-static > and --disable-allstatic. OK, got it; that worked. It compiled successfully, and so did s6, both without any compile warnings! Nice! Lewis
Re: Compiling execline shared fails without --with-lib configure opt
Since I'm not using static libraries, I would not expect to need the "--with-lib" configure option. The --disable-static flag only tells the build system not to build the static execline library. By default, executables are still built against the static version of skalibs. If you want to link executables against the dynamic version of skalibs, you need to add the --disable-allstatic option to configure. IOW: to make a dynamic install, you need both --disable-static and --disable-allstatic. -- Laurent
Compiling execline shared fails without --with-lib configure opt
Hello! On x86_64 RHEL 7.7 with GCC 4.8.5, configuring and compiling execline 2.5.3.0 against skalibs 2.9.1.0 (which has been compiled with the "--disable-static" configure option) with $ ./configure --prefix=/opt/local/encap/execline-2.5.3.0 --enable-shared --disable-static --with-sysdeps=/opt/local/lib/skalibs/sysdeps --with-include=/opt/local/include --with-dynlib=/opt/local/lib $ make fails as follows: exec gcc -o background -pipe -Wall -std=c99 -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -Wa,--noexecstack -ffunction-sections -fdata-sections -O2 -fomit-frame-pointer -fno-stack-protector -Wl,--sort-section=alignment -Wl,--sort-common -Wl,--hash-style=both -Wl,--gc-sections src/execline/background.o libexecline.a.xyzzy -lskarnet /usr/bin/ld: cannot find -lskarnet collect2: error: ld returned 1 exit status make: *** [background] Error 1 If I add the "--with-lib=/opt/local/lib" configure option, it succeeds. This is unexpected because the "./configure --help" output says: Dependencies: --with-sysdeps=DIRuse sysdeps in DIR [PREFIX/lib/skalibs/sysdeps] --with-include=DIRadd DIR to the list of searched directories for headers --with-lib=DIRadd DIR to the list of searched directories for static libraries --with-dynlib=DIR add DIR to the list of searched directories for shared libraries Since I'm not using static libraries, I would not expect to need the "--with-lib" configure option. Thank you, Lewis
Re: [PATCH execline] Fix CC variable in cross compile environment
Since a new release is coming soon, is there any chance to apply another cross compile patch[1]? Applied, thanks for the reminder. New versions pushed, release announcement coming soon. -- Laurent
Re: [PATCH execline] Fix CC variable in cross compile environment
Hi Laurent,
On Mon, Oct 21, 2019 at 11:14 PM Laurent Bercot wrote:
>
>
> Confirmed; I must have been confused when testing earlier.
> Fix pushed, thanks Shengjing.
> Bugfix releases coming soon.
>
Since a new release is coming soon, is there any chance to apply
another cross compile patch[1]?
[1] https://www.mail-archive.com/skaware@list.skarnet.org/msg01343.html
diff --git a/tools/gen-deps.sh b/tools/gen-deps.sh
index 6383ac2..f2f37b6 100755
--- a/tools/gen-deps.sh
+++ b/tools/gen-deps.sh
@@ -81,7 +81,7 @@ for dir in $(ls -1 src | grep -v ^include) ; do
if echo $dep | grep -q -- \\.o$ ; then
dep="src/$dir/$dep"
fi
- if echo $dep | grep -q -- '^\${.*_LIB}' ; then
+ if echo $dep | grep -q -- '^\${.*_LIB}\|^-l' ; then
libs="$libs $dep"
else
deps="$deps $dep"
--
Shengjing Zhu
Re: [PATCH execline] Fix CC variable in cross compile environment
Confirmed; I must have been confused when testing earlier. Fix pushed, thanks Shengjing. Bugfix releases coming soon. -- Laurent
Re: [PATCH execline] Fix CC variable in cross compile environment
Makefile uses CC instead of $(CROSS_COMPILE)$(CC), So CC shouldn't strip cross prefix when configure. That sounds right, but ISTR testing it when I made the change and it worked when cross-compiling. There must have been something else going on; I will test tomorrow and possibly apply. Thanks! -- Laurent
[PATCH execline] Fix CC variable in cross compile environment
This patch fixes execline, but other softwares like s6 should
be fixed as well.
With commit fc7958ecb1c4d5b06521e1ca42f0b48b514e27e1,
Makefile uses CC instead of $(CROSS_COMPILE)$(CC),
So CC shouldn't strip cross prefix when configure.
---
configure | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/configure b/configure
index ce3939d..bcac9f3 100755
--- a/configure
+++ b/configure
@@ -418,7 +418,7 @@ SYSCLOCK_LIB := ${sysclock_lib}
TIMER_LIB := ${timer_lib}
UTIL_LIB := ${util_lib}
-CC := ${CC_AUTO##${cross}}
+CC := ${CC_AUTO}
CPPFLAGS_AUTO := $CPPFLAGS_AUTO
CPPFLAGS := $CPPFLAGS $CPPFLAGS_POST
CFLAGS_AUTO := $CFLAGS_AUTO
--
2.23.0
signature.asc
Description: PGP signature
Re: execline "importas" documentation question
On Sun, Sep 16, 2018 at 08:35:57PM -0500, Brett Neumeier wrote:
> On Sun, Sep 16, 2018 at 5:49 PM Colin Booth wrote:
>
> > I think I know what is going on. I assume you're running this on the
> > commandline? If so, the shell is splitting prefix${FOO}postfix into
> > three words ("prefix", "$FOO", "postfix"), then importas is deleting the
> > middle one. If you write this as a full execline script like so:
> > $ execlineb -c 'importas FOO FOO echo prefix${FOO}postfix'
> > you'll see the full word deletion in action.
> >
>
> Ha, it was definitely a brain cloud, but it wasn't that. bash was, of
> course, replacing ${FOO} with an empty string before running the importas
> command.
>
> The expected behavior does indeed show up if you single-quote that word so
> that bash leaves it alone:
>
> $ importas FOO FOO echo 'prefix${FOO}postfix'
>
> $
>
Glad that worked for you. Generally speaking when I'm doing oneliner
execline testing I send the whole thing to execlineb -c in order to
guarantee that the shell isn't breaking anything. This matters even more
when you start using stuff that needs blocks.
--
Colin Booth
Re: execline "importas" documentation question
On Sun, Sep 16, 2018 at 5:49 PM Colin Booth wrote:
> I think I know what is going on. I assume you're running this on the
> commandline? If so, the shell is splitting prefix${FOO}postfix into
> three words ("prefix", "$FOO", "postfix"), then importas is deleting the
> middle one. If you write this as a full execline script like so:
> $ execlineb -c 'importas FOO FOO echo prefix${FOO}postfix'
> you'll see the full word deletion in action.
>
Ha, it was definitely a brain cloud, but it wasn't that. bash was, of
course, replacing ${FOO} with an empty string before running the importas
command.
The expected behavior does indeed show up if you single-quote that word so
that bash leaves it alone:
$ importas FOO FOO echo 'prefix${FOO}postfix'
$
--
Brett Neumeier (bneume...@gmail.com)
Re: execline "importas" documentation question
On Sun, Sep 16, 2018 at 04:42:10PM -0500, Brett Neumeier wrote:
> Hi,
>
> The documentation for importas says:
>
> "When *envvar* is undefined, and the -D option is not given, any variable
> substitution <http://skarnet.org/software/execline/el_substitute.html> with
> *variable* as the key will return no word; that is true even when the ${
> *variable*} form to be substituted happens in the middle of a word (with a
> prefix and/or a postfix), which means the whole world will be deleted. If
> this is not the behaviour you want, use the -D option."
>
> I'm trying to figure out what that means. When I ensure that FOO is not set
> and run:
>
> importas FOO FOO echo prefix${FOO}postfix
>
> I get the output "prefixpostfix", which is identical to what I get if I add
> a -D option with an empty word. If the whole word were deleted, I'd expect
> to get empty output. (If the whole *world* were deleted, I'd expect to be
> floating in space...)
>
I think I know what is going on. I assume you're running this on the
commandline? If so, the shell is splitting prefix${FOO}postfix into
three words ("prefix", "$FOO", "postfix"), then importas is deleting the
middle one. If you write this as a full execline script like so:
$ execlineb -c 'importas FOO FOO echo prefix${FOO}postfix'
you'll see the full word deletion in action.
Cheers!
--
Interesting. I get the expected behavior.
Colin Booth
execline "importas" documentation question
Hi,
The documentation for importas says:
"When *envvar* is undefined, and the -D option is not given, any variable
substitution <http://skarnet.org/software/execline/el_substitute.html> with
*variable* as the key will return no word; that is true even when the ${
*variable*} form to be substituted happens in the middle of a word (with a
prefix and/or a postfix), which means the whole world will be deleted. If
this is not the behaviour you want, use the -D option."
I'm trying to figure out what that means. When I ensure that FOO is not set
and run:
importas FOO FOO echo prefix${FOO}postfix
I get the output "prefixpostfix", which is identical to what I get if I add
a -D option with an empty word. If the whole word were deleted, I'd expect
to get empty output. (If the whole *world* were deleted, I'd expect to be
floating in space...)
--
Brett Neumeier (bneume...@gmail.com)
Re: execline package not installing exec
2018-09-12 12:10 GMT-03:00 Rodrigo Severo - Fábrica : > Does anybody have any idea why that could be happening? And even > better, how to fix it? Please disregard my previous message. The problem was between a checkinstall newbie and checkinstall. After a make clean; ./configure; make cycle everything got installed without problems. Sorry for the noise. Rodrigo Severo
execline package not installing exec
Hi, I have created Ubuntu packages for the following software: * execline * s6 * s6-dns * s6-networking and * skalibs I'm using checkinstall <https://wiki.debian.org/CheckInstall> to create my deb packages. I've just found out that my execline package isn't installing the "exec" executable. In fact it's missing 19 executables from the execline package. I believe it might have something to do with the fact that these executables have the same name as build-in bash commands but I'm not sure if this really is what is happening and how to prevent it. I already checked that the "exec" executable has been built and is available at the build directory. I can't find anywhere any kind or rule that would prevent "exec" from being included in the deb file. Does anybody have any idea why that could be happening? And even better, how to fix it? Regards, Rodrigo Severo
Re: execline, s6-portable-utils: print a file to stdout
Laurent Bercot writes: >>I haven’t found a way to print a file to stdout with either >>execline nor s6-portable-utils. The `s6-cat` utility only >>echoes stuff that is already coming from stdin, >>and unlike the shell, execline doesn’t have `<`. > > https://skarnet.org/software/execline/redirfd.html > > "redirfd -r 0 $file s6-cat" will print $file to stdout. oooh, that’s a powerful command. Dunno how I missed that, I think I was too focused on finding something less general. I think I might start writing a FAQ for these things. -- Written with Emacs (mu4e) on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es/ May take up to five days to read your message. If it’s urgent, call me.
Re: execline, s6-portable-utils: print a file to stdout
I haven’t found a way to print a file to stdout with either execline nor s6-portable-utils. The `s6-cat` utility only echoes stuff that is already coming from stdin, and unlike the shell, execline doesn’t have `<`. https://skarnet.org/software/execline/redirfd.html "redirfd -r 0 $file s6-cat" will print $file to stdout. -- Laurent
Re: execline: returning something from ifthenelse blocks
So I’d like to have an ifthenelse and return the same envvar from both branches. https://www.mail-archive.com/skaware@list.skarnet.org/msg00311.html -- Laurent
execline, s6-portable-utils: print a file to stdout
Sorry for my spam of mails, I want to separate my questions by topic so they will be searchable. I haven’t found a way to print a file to stdout with either execline nor s6-portable-utils. The `s6-cat` utility only echoes stuff that is already coming from stdin, and unlike the shell, execline doesn’t have `<`. Maybe some kind of output utility might be nice? The Lisp dialect Clojure has `slurp` and `spit`: https://clojuredocs.org/clojure.core/slurp https://clojuredocs.org/clojure.core/spit -- Written with Emacs (mu4e) on NixOS. Q: Why is this email five sentences or less? A: http://five.sentenc.es/ May take up to five days to read your message. If it’s urgent, call me.
execline: returning something from ifthenelse blocks
I have the following logic:
If file in variable f is a symlink, resolve it
and use the result as link target.
Otherwise, use the result as link target directly.
So I’d like to have an ifthenelse and return the
same envvar from both branches. But this is the
best I can do:
ifte {
backtick res { s6-linkname -f $f }
importas -ui res res
s6-ln $res $origin
} {
s6-ln $f $origin
}
{ s6-test -L $f }
My hope would be something like:
ifthenelse { s6-test -L $f } {
return $res somehow
} {}
s6-ln $res $origin
--
Written with Emacs (mu4e) on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es/
May take up to five days to read your message. If it’s urgent, call me.
Re: execline globbing
That brings me to another question: is there a collection of more execline utilities somewhere? I suspect the standard GNU coreutils are not quite to your liking in a lot of cases. The execline package has all the execline-specific utilities you need to perform execline scripting (although if you have a bright idea about something that's obviously missing, please share it). GNU coreutils provide a completely different functionality, i.e. basic Unix utilities - and if you don't like them, there are a ton of other implementations of the same utilities: busybox, toybox, sbase+ubase, etc. Would you be open to switch the default behavior? Maybe with a warning message on `-0` that it is the default and will be removed after a few releases. I suggest something like `-k` for “keep”. I don't think the benefits of changing the default are worth the drawbacks and the effort. Again, the current default is how shell globbing works, and I think there's value in aligning with that behaviour. As long as the user has control, there's not much incentive to change. -- Laurent
Re: execline globbing
Laurent Bercot writes:
> Your next door "echo" command will do just that (or s6-echo
> if you risk having dashes and want reliable behaviour in all cases).
That brings me to another question: is there a collection of
more execline utilities somewhere? I suspect the standard
GNU coreutils are not quite to your liking in a lot of cases.
> The shell has the exact same default behaviour. I didn't want to
> gratuitously diverge from the shell. But I agree "elglob -0" is the
> behaviour you want most of the time.
Would you be open to switch the default behavior?
Maybe with a warning message on `-0` that it is the default
and will be removed after a few releases.
I suggest something like `-k` for “keep”.
> It's not a "zero string". It's zero word. Which means the "${f}"
> argument is replaced with nothing at all, not even an empty string.
> So the tests resolve to:
> test -z
> and
> test -n
> which both return true.
Aha!
--
Written with Emacs (mu4e) on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es/
May take up to five days to read your message. If it’s urgent, call me.
Re: execline globbing
elglob -0 fs somedir/*
if { test -n $fs }
ln -t otherdir $fs
the test will fail if there’s more than one file in `somedir`.
Is there a way to put a split variable into one variable again?
Your next door "echo" command will do just that (or s6-echo
if you risk having dashes and want reliable behaviour in all cases).
elglob -0 splitfs somedir/*
backtick -n fs { echo $splitfs }
importas -u fs fs
if { test -n $fs } ...
It feels kind of clumsy to use elglob, especially because of
the default verbatim input of the pattern if no expansion is
found. I can’t imagine any use case where I’d want that,
especially not as default behaviour.
The shell has the exact same default behaviour. I didn't want to
gratuitously diverge from the shell. But I agree "elglob -0" is the
behaviour you want most of the time.
Another fun effect:
execlineb -c 'elglob -0 f doesnotexist/* if { test -z "${f}" } echo
foo'
foo
execlineb -c 'elglob -0 f doesnotexist/* if { test -n "${f}" } echo
foo'
foo
So for `test`, the ominous “zero string” of execline is both
empty and non-empty!
Is there some elaboration somewhere what this zero string is?
And how do work with it?
It's not a "zero string". It's zero word. Which means the "${f}"
argument is replaced with nothing at all, not even an empty string.
So the tests resolve to:
test -z
and
test -n
which both return true.
--
Laurent
execline globbing
When I have a glob like:
elglob -0 fs somedir/*
if { test -n $fs }
ln -t otherdir $fs
the test will fail if there’s more than one file in `somedir`.
Is there a way to put a split variable into one variable again?
It feels kind of clumsy to use elglob, especially because of
the default verbatim input of the pattern if no expansion is
found. I can’t imagine any use case where I’d want that,
especially not as default behaviour.
Another fun effect:
> execlineb -c 'elglob -0 f doesnotexist/* if { test -z "${f}" } echo foo'
foo
> execlineb -c 'elglob -0 f doesnotexist/* if { test -n "${f}" } echo foo'
foo
So for `test`, the ominous “zero string” of execline is both
empty and non-empty!
Is there some elaboration somewhere what this zero string is?
And how do work with it?
--
Written with Emacs (mu4e) on NixOS.
Q: Why is this email five sentences or less?
A: http://five.sentenc.es/
May take up to five days to read your message. If it=E2=80=99s urgent, call me.
Re: execline-in-execline
As a matter of example : this scripts "rewrites" its final command
depending on optional arguments. Its purpose is to restore select parts
of the environment variables, uid/gid and working directory, that were
saved in a directory at some point in the past.
https://github.com/Adirelle/s6rc-overlay/blob/master/src/sbin/with-contenv
Maybe it is pushing execline a bit too far but I didn't fell like coding
it in C (which I do no master).
Le 07/06/2018 à 17:59, Profpatsch a écrit :
Laurent Bercot writes:
Remember that once an execlineb script has been parsed, it's just a
command line, no more, no less. So your example script can just be
written as:
#!execline
define url example.com
s6-tcpclient $url 80
foreground { fdmove 1 7 echo -en "..." }
fdmove 0 6 cat
No second execlineb invocation necessary at all. No quoting
nightmares.
Ooh, you are right!
It’s even more elegant than I thought!
Nice.
Re: execline-in-execline
```
#!execline
define url example.com
s6-tcpclient $url 80
execlineb "
foreground {
fdmove 1 7 echo -en \"…\"
fdmove 0 6 cat
}
"
```
I have trouble understanding what you want to do in the general case,
because in this example, s6-tcpclient takes a whole command line, so
you do not have to call execlineb again.
Remember that once an execlineb script has been parsed, it's just a
command line, no more, no less. So your example script can just be
written as:
#!execline
define url example.com
s6-tcpclient $url 80
foreground { fdmove 1 7 echo -en "..." }
fdmove 0 6 cat
No second execlineb invocation necessary at all. No quoting nightmares.
--
Laurent
Re: execline
> My idea AWESOME, love it. Have already begun to write scripts with CHICKEN, WOLOG (shebang-csi, for chicken-scheme interpreter, can't notice the difference from an average ksh script so far, except in elegance, Scheme oblige) going to try Chez. /Roy -- 5 anjing menggonggong, kafilah tetap berlalu 5 . 5 l 4 c K W 4 r 3 55 the dogs are barking, the caravan moves on 5 + L1NuX 55 [illustrates useless protest, critic, or 5 sarcasm]
Re: execline
Some relatively minor points to add: On Tue, Apr 10, 2018 at 11:12:30AM +0800, Casper Ti. Vector wrote: > * Perferably, support something like scsh Though I like the idea of scsh, I still find it slightly large; I think a set of suitable macros and library functions for Chibi might be a better route. > Considering the existence of elegant yet performant self-hosting Scheme > compilers (like Chez, which self-compiles in seconds), the proposed > solution can also be simpler in terms of total complexity. By referring to Scheme compilers, I do not intend to exclude Scheme interpreters from our candidates, but to show that the total complexity of the proposed implementation can be comparable to (or even lower than) that of the current implementation. And by the way, since the chainloading procedure itself consumes very little time after all, even if the Scheme implementation is relatively slow, the overall performance of the system would be nearly unaffected. -- My current OpenPGP key: RSA4096/0x227E8CAAB7AA186C (expires: 2020.10.19) 7077 7781 B859 5166 AE07 0286 227E 8CAA B7AA 186C
