Hi Mateusz,
> First and a foremost apologies for radio silence, I got really busy
> with other stuff.
Thanks for making time to write such a detailed reply despite being busy
with other work. Sorry for taking so long to get back to you.
> I don't understand what race is supposed to be addressed either as
> none of the setup depends on the content binary.
You're right that none of the setup depends on an earlier check of
the executable. My earlier reply mixed up two different requirements:
executing whatever a pathname resolves to at exec time, and executing
a particular file selected earlier.
The posted RFC does the former. It stores the pathname without looking
up or inspecting the executable during configuration.
The caller-side race I had in mind was this:
caller another process child
| | |
| inspect file A via path | |
| configure builder | |
| | replace path |
| | with file B |
| spawn_run() | |
| | lookup -> B |
| | exec B |
This matters if the caller expects the file it inspected to be the
one executed. Opening A first, inspecting it through that fd, and
using the same fd for execution avoids the pathname substitution.
The fd pins the file's identity, not immutable contents.
> Regardless I don't think this is going to fly as it directly shafts
> NUMA setups. What's needed is a path to the binary + the PATH
> envifornment variable or a collection of paths to try out.
I went back to the spawn_template discussion and realized that you had
already raised these NUMA concerns there, including child-side lookup,
PATH search, and target-node allocation. Sorry, I should have connected
your current comment with that earlier explanation.
> At minimum this will allocate a dentry and an inode for usr, bin and
> coolprog itself.
>
> Afterwards any path lookup going through usr in that that container
> will have to fetch memory from a non-local node to traverse it, same
> thing for the bin subdirectory.
>
> This is a self-induced problem when using a fd.
I see how opening the executable in the parent can populate cold
filesystem metadata on the parent's node, even though the child will
run on a different node. Making an executable fd mandatory would
force that early lookup on callers who could otherwise leave it
to the child.
> A well-known exec-heavy workload where this is fine is compilation.
> Any big project will do, to give you one trivial example, make -j
> $BIGNUM modules in the Linux kernel directory.
>
> [...]
>
> The stock kernel with the fork
> + exec machinery is doing a shitty job here, but the new mechanism can
> assure process-specific allocation to come form the target node as
> long as the scheduler does not mess up.
The compilation example helps too, and this matters even without explicit
affinity. Moving lookup into the child does not by itself address the
initial per-task allocations.
> 1. copy in whatever data it needs from userspace
> 2. if no affinity is specified, ask the scheduler: assuming something
> forks now, where is it going to run? the answer would be speculative
> ofc
> 3. to the minimum allocations needed (task_struct etc.) while passing
> the node number from above
> 4. have the newly spawned task handle everything else
I'll look into using the scheduler's speculative placement for those
initial allocations, then leaving the remaining work to the child
where it actually runs.
> In the real world userspace notoriously performs failed execs as it
> makes its way through dirs in provided in PATH, and while you can
> patch select programs all you want, the mechanism will have to account
> for it.
>
> [...]
>
> This in particular means that a failed exec would be very expensive,
> as it would go through task creation and teardown. Meaning the child
> needs to be able to figure out where to binary to execute is. And
> user-visible part has to account for it from the get-go.
Your point about failed execs during PATH search makes sense too.
Creating and tearing down a task for each candidate would be wasteful.
I plan to have libc build a list of candidate paths without doing any
filesystem lookup, and pass that list to the kernel. The child would
apply its setup once, then try the candidates in order, continuing
after recoverable exec failures in the same task.
That keeps PATH parsing in libc and lookup in the child. Relative paths
would stay relative, so they resolve against the child's cwd after file
actions. A single pathname would just be a one-entry list. This needs
interface support, and I'll need to work through the exec cleanup and
error handling to preserve the expected posix_spawnp() behavior.
I'm still unsure whether supporting both modes is a good interface
design. An fd option seems useful for callers that need to execute
the same file they opened earlier, while child-side pathname lookup
would avoid forcing that early lookup on everyone. I'm leaning
towards keeping both, but haven't settled on that yet.
> I'm going to write more after I go through the patchset, should be next week.
Thanks again for taking the time to explain this. I look forward to
your further comments.
Regards,
Li
