I'm haven't identified the specific problem with fork support, but I did see
this in libmlx4:
mlx4_alloc_context()
{
...
context->uar = mmap(NULL, to_mdev(ibdev)->page_size, PROT_WRITE,
MAP_SHARED, cmd_fd, 0);
if (context->uar == MAP_FAILED)
goto err_free;
if (resp.bf_reg_size) {
context->bf_page = mmap(NULL, to_mdev(ibdev)->page_size,
PROT_WRITE, MAP_SHARED, cmd_fd,
to_mdev(ibdev)->page_size);
...
}
I don't know for certain that these mmap() calls cause an issue, but the
preload library socket() function calls rsocket(), which loads and initializes
libibverbs. This calls mlx4_alloc_context() before fork() has been called.
I added the following hack to the preload socket() call, which skips calling
rsocket().
if ((domain == PF_INET || domain == PF_INET6) &&
(type == SOCK_STREAM) && (!protocol || protocol == IPPROTO_TCP) &&
fork_support) {
printf("skipping rsocket call\n");
goto realsock;
}
recursive = 1;
ret = rsocket(domain, type, protocol);
recursive = 0;
if (ret >= 0) {
if (fork_support) {
rclose(ret);
realsock:
ret = real.socket(domain, type, protocol);
With the above hack, I no longer see hangs when running with fork() and the
netperf performance is split. This should delay libibverbs initializing until
after fork() has been called.
I'm trying to decide whether to keep this work-around or find a better solution.
- Sean
