It has been almost a month and I think I can see light in the tunnel. After all the patches I have sent recently (including the libc/time-related ones), I think we are at the point where we can proceed with the real upgrade.
As far as libc/ folder goes, we are down from roughly 215 C files month ago to 104 ones where only following 54 + 8 (under libc/arch) at some point came from musl and are actual candidates for some kind of upgrade: libc/arch/aarch64/setjmp/sigrtmin.c libc/arch/aarch64/setjmp/block.c libc/arch/aarch64/setjmp/siglongjmp.c libc/arch/aarch64/setjmp/sigrtmax.c libc/arch/x64/setjmp/sigrtmin.c libc/arch/x64/setjmp/block.c libc/arch/x64/setjmp/siglongjmp.c libc/arch/x64/setjmp/sigrtmax.c libc/env/__environ.c libc/env/secure_getenv.c libc/errno/strerror.c libc/internal/libc.c libc/locale/freelocale.c libc/locale/strtod_l.c libc/locale/strtof_l.c libc/locale/strtold_l.c libc/locale/uselocale.c libc/misc/ffs.c libc/network/getaddrinfo.c libc/network/gethostbyname_r.c libc/network/getifaddrs.c libc/network/getnameinfo.c libc/network/inet_addr.c libc/network/inet_aton.c libc/network/inet_ntop.c libc/network/__ipparse.c libc/prng/__rand48_step.c libc/stdio/__fdopen.c libc/stdio/fgetc.c libc/stdio/flockfile.c libc/stdio/fmemopen.c libc/stdio/__fopen_rb_ca.c libc/stdio/fputc.c libc/stdio/ftrylockfile.c libc/stdio/funlockfile.c libc/stdio/getc.c libc/stdio/__lockfile.c libc/stdio/open_memstream.c libc/stdio/open_wmemstream.c libc/stdio/putc.c libc/stdio/remove.c libc/stdio/sscanf.c libc/stdio/stderr.c libc/stdio/stdin.c libc/stdio/__stdio_read.c libc/stdio/__stdio_write.c libc/stdio/stdout.c libc/stdio/vdprintf.c libc/stdio/vfprintf.c libc/stdio/vfscanf.c libc/stdio/vfwscanf.c libc/stdio/vsnprintf.c libc/stdio/vsscanf.c libc/stdio/vswprintf.c libc/stdio/vswscanf.c libc/stdlib/strtod.c libc/stdlib/strtol.c libc/stdlib/wcstol.c libc/string/strerror_r.c libc/syslog.c libc/time/ftime.c libc/time/__tz.c The libc/arch ones should be simple makefile replacements to point to a new version of musl. Many other files (~10-20) from the 54 ones should also be as easy. Most of the files that will require manual changes are those under libc/stdio (28, mostly .lock vs .locking changes in FILE and syscalls) and libc/network. For both stdio and locale and I should also have some android-based unit tests. I have looked at locale related changes in a new version of musl and we might need to make some changes in other places of our code (linux.cc). For all the files we will have to keep changing the original musl files, I suggest we actually store the .patch files as they make it easier/quicker to see what changes we make and automatically detect we need to update those as we upgrade to future versions of musl. For that, we will have to slightly change the build process. Now the worst part to deal with might actually be the headers. Out of over ~220 *.h files under include/api, most of those symlink to musl/include folder but many are still copies from older/newer version of musl and often manually modified. Some might require simple change to symlinks, but others might require special attention. Here are all the files (not symlinks to musl/include) I identified as candidates for an upgrade: include/api/sched.h include/api/stdint.h include/api/link.h include/api/time.h include/api/mqueue.h include/api/aarch64/bits/stat.h include/api/aarch64/bits/stdint.h include/api/aarch64/bits/posix.h include/api/aarch64/bits/limits.h include/api/aarch64/bits/errno.h include/api/aarch64/bits/float.h include/api/aarch64/bits/shm.h include/api/aarch64/bits/syscall.h include/api/aarch64/bits/mman.h include/api/aarch64/bits/fenv.h include/api/aarch64/bits/statfs.h include/api/aarch64/bits/termios.h include/api/aarch64/bits/setjmp.h include/api/aarch64/bits/ipc.h include/api/aarch64/bits/fcntl.h include/api/aarch64/bits/socket.h include/api/aarch64/bits/ioctl.h include/api/aarch64/bits/endian.h include/api/aarch64/bits/signal.h include/api/pthread.h include/api/fenv.h include/api/string.h include/api/sys/stat.h include/api/sys/mount.h include/api/sys/shm.h include/api/sys/types.h include/api/sys/sem.h include/api/sys/time.h include/api/sys/mman.h include/api/sys/sysinfo.h include/api/sys/uio.h include/api/sys/statfs.h include/api/sys/eventfd.h include/api/sys/resource.h include/api/sys/sysmacros.h include/api/sys/socket.h include/api/sys/ioctl.h include/api/sys/inotify.h include/api/sys/__socket.h include/api/sys/poll.h include/api/sys/sysctl.h include/api/sys/statvfs.h include/api/printf.h include/api/x64/bits/stat.h include/api/x64/bits/stdint.h include/api/x64/bits/limits.h include/api/x64/bits/syscall.h include/api/x64/bits/mman.h include/api/x64/bits/fenv.h include/api/x64/bits/socket.h include/api/x64/bits/ioctl.h include/api/x64/bits/msg.h include/api/stdio.h include/api/stddef.h include/api/math.h include/api/assert.h include/api/net/__if.h include/api/net/if.h include/api/fcntl.h include/api/utmpx.h include/api/endian.h include/api/arpa/inet.h include/api/netinet/in.h include/api/netinet/__in.h include/api/netinet/ip_icmp.h include/api/stdlib.h include/api/paths.h include/api/libaio.h include/api/wchar.h include/api/unistd.h The main problem, I anticipate, is that there will be conflicts when we change musl/ to the new version (1.1.24) and all of a sudden most of the headers under include/api will be new and other ones will be old. This will very likely affect the compilation of my C files under new musl/, libc/ and even other ones. Shall we have two include/api folders like include/api_old and include/api (this may not work either)? Given this situation with headers, how shall we go about executing the upgrade incrementally, ideally one musl/src/ and libc/ subfolder at a time? Is this even possible? Any suggestions? Finally, how shall we name the directories? Rename current musl/ to musl.0.9.12/ and add musl/ subproject that would directly point to 1.1.14 tag of the original musl git repo (git://git.musl-libc.org/musl)? I know that right now all git subprojects point to the copies of the relevant repos in cloudous git org. Shall we copy the 1.1.14 tag of musl somewhere and make github subproject point to it? I have created at some point new https://github.com/osvunikernel org where we can create the musl 1.1.14 copy project (original musl is not on github). Any more suggestions regarding mechanics? Order of things to do - here is orignal Nadav's plan (with my some adjustments): - Bring the latest Musl version into a "musl-1.1.24" subdirectory in OSv (or rename musl to musl-0.9.12 and bring 1.1.24 into musl directory). - Leave the old "musl/" and "libc/" directories as well. - In Makefile, make a new list of objects (say, "nmusl") which will take code from musl-1.1.24/ instead of musl/. - Start to switch individual files and directories from "musl += ..." to "nmusl += ...". We can start with the math functions needed for aarch64. Eventually, everything can be converted to nmusl and hopefully, nothing or little will break and need to be fixed. - Replace our include/api with musl-1.1.24/include. Would be even better to drop include/api, and just use musl-1.1.24/include directly if we can. I think, though, this is low priority, and might take quite a bit of work ("git log include/api" shows we modified this quite a bit since we took it from musl). I'm hoping that the old header files will work correctly also for the newer musl. - I think this might be tricky one at this point Waldek -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/a49e2e1c-89fc-49b5-a87b-88b3737097e9n%40googlegroups.com.
