linguini1 opened a new issue, #18514: URL: https://github.com/apache/nuttx/issues/18514
### Is your feature request related to a problem? Please describe. The current CI has to run complex builds very frequently because our method of selecting which configurations are built is dependent on pre-determined wildcard paths which we use to configure the CI. The community wants a solution that avoids running complex builds for things like PRs to `nuttx-apps`, and which reduces our CI usage. ### Describe the solution you'd like An idea which I had been thinking of was to construct a dependency graph of the NuttX codebase. This graph associates each source file in the tree with all of the `defconfig` configurations that use it in the final image. This idea has more benefits for CI usage than our scripted approach that examines path names/Kconfig variables, etc. (and the approach in #18500) in that it doesn't require custom logic to determine the dependencies between source files and builds. It uses our existing source of truth: the build system. So long as we've set up builds for `defconfig` files, our dependency information cannot be wrong. It turns out that the CMake build system (which is a supported NuttX build system and a requirement for all new additions to NuttX) actually supports what I was hoping for: a way to query which source files are pulled into a target. There is this [`cmake-file-api`](https://cmake.org/cmake/help/latest/manual/cmake-file-api.7.html) I have written a little script I've been testing with and it's working quite well. Here it is to try (requires the `jq` command to parse JSON in the shell): ```bash #!/usr/bin/env bash # Input to this file: a board configuration like `sim:nsh` # # Output: the full list of sources that the target uses # # Pre-requisites: # - Clean build system # - jq command for JSON parsing config="$1" # TODO: Quiet CMake mkdir -p build/.cmake/api/v1/query/codemodel-v2 # Create API query cmake -S . -B build -DBOARD_CONFIG=${config} -GNinja # Generate configuration # Output list of source files sources="$(cat build/.cmake/api/v1/reply/target-*.json | jq '.sources[].path')" cmake --build build -t clean # Clean configuration echo "=== SOURCE FILES FOR CONFIG ${config} ===" echo "${sources}" ``` All we need to do to get dependency information for a configuration is run this script. It doesn't even build anything, just requires build environment configurations. Here is the result for running the script with the `sim:nsh` configuration: ```console $ tools/ci/depgraph/target_sources.sh sim:nsh -- NuttX Host Tools -- CMake C compiler: GNU -- CMake system name: Linux -- CMake host system processor: x86_64 -- Configuring done (0.0s) -- Generating done (0.0s) -- Build files have been written to: /home/linguini/coding/nuttx-space/nuttx/build/bin_host -- nuttx_add_subdirectory: Skipping cxx-oot-build -- Configuring done (0.3s) -- Generating done (0.2s) -- Build files have been written to: /home/linguini/coding/nuttx-space/nuttx/build [1/1] Cleaning all built files... Cleaning... 0 files. === SOURCE FILES FOR CONFIG sim:nsh === "build/libs/libc/misc/CMakeFiles/always_rebuild_lib_utsname" "build/libs/libc/misc/CMakeFiles/always_rebuild_lib_utsname.rule" "build/apps/dummy.c" "/home/linguini/coding/nuttx-space/apps/fsutils/mkfatfs/mkfatfs.c" "/home/linguini/coding/nuttx-space/apps/fsutils/mkfatfs/configfat.c" "/home/linguini/coding/nuttx-space/apps/fsutils/mkfatfs/writefat.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_init.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_parse.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_console.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_script.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_system.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_command.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_fscmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_proccmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_mmcmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_timcmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_envcmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_prompt.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_syscmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_dbgcmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_session.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_fsutils.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_builtin.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_fileapps.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_mntcmds.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_consolemain.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_printf.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_test.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_wait.c" "/home/linguini/coding/nuttx-space/apps/nshlib/nsh_alias.c" "/home/linguini/coding/nuttx-space/apps/system/readline/readline.c" "/home/linguini/coding/nuttx-space/apps/system/readline/readline_fd.c" "/home/linguini/coding/nuttx-space/apps/system/readline/readline_stream.c" "/home/linguini/coding/nuttx-space/apps/system/readline/readline_common.c" "/home/linguini/coding/nuttx-space/apps/builtin/builtin_list.c" "/home/linguini/coding/nuttx-space/apps/builtin/exec_builtin.c" "build/apps/builtin/builtin_list.h" "build/apps/builtin/builtin_proto.h" "build/CMakeFiles/apps_context" "build/CMakeFiles/apps_context.rule" "/home/linguini/coding/nuttx-space/apps/system/dd/dd_main.c" "/home/linguini/coding/nuttx-space/apps/system/dumpstack/dumpstack.c" "/home/linguini/coding/nuttx-space/apps/system/gcov/gcov.c" "/home/linguini/coding/nuttx-space/apps/examples/gpio/gpio_main.c" "/home/linguini/coding/nuttx-space/apps/examples/hello/hello_main.c" "/home/linguini/coding/nuttx-space/apps/system/nsh/nsh_main.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/ostest_main.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/getopt.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/libc_memmem.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/restart.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/sighelper.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/sighand.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/signest.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/sigprocmask.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/dev_null.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/setvbuf.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/waitpid.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/cancel.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/cond.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/mutex.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/timedmutex.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/sem.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/semtimed.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/barrier.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/timedwait.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/pthread_rwlock.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/pthread_rwlock_cancel.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/schedlock.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/pthread_exit.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/robust.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/wqueue.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/timedmqueue.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/mqueue.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/posixtimer.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/vfork.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/setjmp.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/nxevent.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/wdog.c" "/home/linguini/coding/nuttx-space/apps/testing/ostest/spinlock.c" "/home/linguini/coding/nuttx-space/apps/system/nsh/sh_main.c" "arch/sim/src/sim/sim_initialize.c" "arch/sim/src/sim/sim_idle.c" "arch/sim/src/sim/sim_cpuinfo.c" "arch/sim/src/sim/sim_doirq.c" "arch/sim/src/sim/sim_initialstate.c" "arch/sim/src/sim/sim_createstack.c" "arch/sim/src/sim/sim_usestack.c" "arch/sim/src/sim/sim_releasestack.c" "arch/sim/src/sim/sim_stackframe.c" "arch/sim/src/sim/sim_exit.c" "arch/sim/src/sim/sim_switchcontext.c" "arch/sim/src/sim/sim_ummheap.c" "arch/sim/src/sim/sim_uart.c" "arch/sim/src/sim/sim_copyfullstate.c" "arch/sim/src/sim/sim_registerdump.c" "arch/sim/src/sim/sim_saveusercontext.c" "arch/sim/src/sim/sim_tcbinfo.c" "arch/sim/src/sim/sim_sectionheap.c" "arch/sim/src/sim/sim_checkhostfstypes.c" "arch/sim/src/sim/sim_schedulesigaction.c" "arch/sim/src/sim/sim_sigdeliver.c" "arch/sim/src/sim/sim_fork_x86_64.S" "arch/sim/src/sim/sim_backtrace.c" "arch/sim/src/sim/sim_fork.c" "arch/sim/src/sim/sim_oneshot.c" "arch/sim/src/sim/sim_blockdevice.c" "arch/sim/src/sim/sim_deviceimage.c" "binfmt/binfmt_globals.c" "binfmt/binfmt_initialize.c" "binfmt/binfmt_register.c" "binfmt/binfmt_unregister.c" "binfmt/binfmt_loadmodule.c" "binfmt/binfmt_unloadmodule.c" "binfmt/binfmt_execmodule.c" "binfmt/binfmt_exec.c" "binfmt/binfmt_copyargv.c" "binfmt/binfmt_copyactions.c" "binfmt/binfmt_dumpmodule.c" "binfmt/binfmt_execsymtab.c" "binfmt/builtin.c" "build/dummy.c" "boards/boardctl.c" "boards/sim/sim/sim/src/sim_appinit.c" "boards/sim/sim/sim/src/sim_bringup.c" "boards/sim/sim/sim/src/sim_ioexpander.c" "build/boards/exclude_board/src/CMakeFiles/romfs_etc.dir/romfs_etc.c.o" "libs/libc/assert/lib_assert.c" "libs/libc/assert/lib_stackchk.c" "libs/libc/builtin/lib_builtin_getname.c" "libs/libc/builtin/lib_builtin_isavail.c" "libs/libc/builtin/lib_builtin_forindex.c" "libs/libc/ctype/lib_isalnum.c" "libs/libc/ctype/lib_isalpha.c" "libs/libc/ctype/lib_isascii.c" "libs/libc/ctype/lib_isblank.c" "libs/libc/ctype/lib_iscntrl.c" "libs/libc/ctype/lib_isdigit.c" "libs/libc/ctype/lib_isgraph.c" "libs/libc/ctype/lib_islower.c" "libs/libc/ctype/lib_isprint.c" "libs/libc/ctype/lib_ispunct.c" "libs/libc/ctype/lib_isspace.c" "libs/libc/ctype/lib_isupper.c" "libs/libc/ctype/lib_isxdigit.c" "libs/libc/ctype/lib_tolower.c" "libs/libc/ctype/lib_toupper.c" "libs/libc/ctype/lib_ctype.c" "libs/libc/dirent/lib_readdirr.c" "libs/libc/dirent/lib_telldir.c" "libs/libc/dirent/lib_alphasort.c" "libs/libc/dirent/lib_scandir.c" "libs/libc/dirent/lib_ftw.c" "libs/libc/dirent/lib_nftw.c" "libs/libc/dirent/lib_opendir.c" "libs/libc/dirent/lib_fdopendir.c" "libs/libc/dirent/lib_closedir.c" "libs/libc/dirent/lib_readdir.c" "libs/libc/dirent/lib_rewinddir.c" "libs/libc/dirent/lib_seekdir.c" "libs/libc/dirent/lib_dirfd.c" "libs/libc/dirent/lib_versionsort.c" "libs/libc/dlfcn/lib_dlfind_object.c" "libs/libc/errno/lib_errno.c" "libs/libc/fixedmath/lib_fixedmath.c" "libs/libc/fixedmath/lib_b16sin.c" "libs/libc/fixedmath/lib_b16cos.c" "libs/libc/fixedmath/lib_b16atan2.c" "libs/libc/fixedmath/lib_ubsqrt.c" "libs/libc/grp/lib_getgrgid.c" "libs/libc/grp/lib_getgrgidr.c" "libs/libc/grp/lib_getgrnam.c" "libs/libc/grp/lib_getgrnamr.c" "libs/libc/grp/lib_initgroups.c" "libs/libc/grp/lib_getgrbuf.c" "libs/libc/grp/lib_getgrbufr.c" "libs/libc/inttypes/lib_imaxabs.c" "libs/libc/inttypes/lib_imaxdiv.c" "libs/libc/inttypes/lib_strtoimax.c" "libs/libc/inttypes/lib_strtoumax.c" "libs/libc/libgen/lib_basename.c" "libs/libc/libgen/lib_dirname.c" "libs/libc/locale/lib_duplocale.c" "libs/libc/locale/lib_freelocale.c" "libs/libc/locale/lib_localeconv.c" "libs/libc/locale/lib_newlocale.c" "libs/libc/locale/lib_setlocale.c" "libs/libc/locale/lib_uselocale.c" "libs/libc/locale/lib_catalog.c" "libs/libc/locale/lib_gettext.c" "libs/libc/locale/lib_langinfo.c" "libs/libc/locale/lib_iconv.c" "libs/libc/locale/lib_maxlocale.c" "libs/libc/machine/sim/arch_setjmp_x86_64.S" "libs/libc/machine/arch_atomic.c" "libs/libc/misc/lib_bitmap.c" "libs/libc/misc/lib_circbuf.c" "libs/libc/misc/lib_creat.c" "libs/libc/misc/lib_mknod.c" "libs/libc/misc/lib_umask.c" "libs/libc/misc/lib_utsname.c" "libs/libc/misc/lib_getrandom.c" "libs/libc/misc/lib_xorshift128.c" "libs/libc/misc/lib_tea_encrypt.c" "libs/libc/misc/lib_tea_decrypt.c" "libs/libc/misc/lib_cxx_initialize.c" "libs/libc/misc/lib_idr.c" "libs/libc/misc/lib_impure.c" "libs/libc/misc/lib_memfd.c" "libs/libc/misc/lib_mutex.c" "libs/libc/misc/lib_fchmodat.c" "libs/libc/misc/lib_fstatat.c" "libs/libc/misc/lib_getfullpath.c" "libs/libc/misc/lib_openat.c" "libs/libc/misc/lib_mkdirat.c" "libs/libc/misc/lib_utimensat.c" "libs/libc/misc/lib_mallopt.c" "libs/libc/misc/lib_getnprocs.c" "libs/libc/misc/lib_tempbuffer.c" "libs/libc/misc/lib_umul32.c" "libs/libc/misc/lib_umul64.c" "libs/libc/misc/lib_umul32x64.c" "libs/libc/misc/lib_uadd32x64.c" "libs/libc/misc/lib_uadd64.c" "libs/libc/misc/lib_usub64x32.c" "libs/libc/misc/lib_usub64.c" "libs/libc/misc/lib_mkfifo.c" "libs/libc/misc/lib_dumpbuffer.c" "libs/libc/misc/lib_dumpvbuffer.c" "libs/libc/misc/lib_fnmatch.c" "libs/libc/misc/lib_debug.c" "libs/libc/misc/lib_crc64.c" "libs/libc/misc/lib_crc64emac.c" "libs/libc/misc/lib_crc32.c" "libs/libc/misc/lib_crc32h04c11db7.c" "libs/libc/misc/lib_crc32hf4acfb13.c" "libs/libc/misc/lib_crc16.c" "libs/libc/misc/lib_crc16ccitt.c" "libs/libc/misc/lib_crc16ibm.c" "libs/libc/misc/lib_crc16xmodem.c" "libs/libc/misc/lib_crc16h1021.c" "libs/libc/misc/lib_crc16h8005.c" "libs/libc/misc/lib_crc8.c" "libs/libc/misc/lib_crc8ccitt.c" "libs/libc/misc/lib_crc8h1d.c" "libs/libc/misc/lib_crc8h2f.c" "libs/libc/misc/lib_crc8rohc.c" "libs/libc/misc/lib_crc8table.c" "libs/libc/misc/lib_glob.c" "libs/libc/misc/lib_backtrace.c" "libs/libc/misc/lib_ftok.c" "libs/libc/misc/lib_err.c" "libs/libc/misc/lib_instrument.c" ... # Omitted for brevity ``` Notice how it also tracks `nuttx-apps` dependencies! The idea is that we can now build some graph like this: ```json { "libs/libc/misc/lib_instrument.c": ["sim:nsh", "sim:nsh_smp", ...], "arch/sim/src/sim/sim_fork.c": ["sim:nsh", "sim:nsh_smp"], ... } ``` using the output of this script combined with `tools/configure.sh -L` to get all of source -> config dependencies. If we want to avoid Python in CI this can probably be done with just a C program that has a hashmap implementation. Now when we have a change set, we use the graph to look up the files in the patch. We collect all the `sim:nsh`, etc. configs together that cover the entire change set, and we now have the exact set of builds needed to cover the patch in its entirety. No unnecessary builds! This step can run entirely on a Linux Docker file before the build steps happen, so we can use just shell scripts/Linux available tools and not have to worry about portability. ## Caveats Not everything is CMake supported quite yet, as evidenced by @simbit18's hard work copying Make stuff to CMake almost every day. The dependency graph changes when: * Build files are updated (CMakelists.txt, Kconfig) * `defconfig` files are updated This requires the graph to be re-built. It is actually a decently cheap operation from what I can tell with my local experimentation, and should be vastly shorter than a full build (which we would be saving). The graph would be version controlled in the source tree of `nuttx`, only re-built when a PR modifies one of the aforementioned files that trigger a rebuild. We'd have to ensure that the copy in `nuttx` gets updated when a rebuild is needed in a `nuttx-apps` change. Potentially this could be done by storing a version of the graph in each repo and running a `diff` on the beginning of the workflow to determine if they are out of sync, and then re-generate the `nuttx` copy if so. ### Describe alternatives you've considered This is just for discussion at the moment to introduce the idea and see what others think. It would supersede what I was trying to do in #18500 and I would proceed with testing this approach instead. If it's received positively then I would continue with local CI tests (on a special fork as suggested) and come up with some timing information about how long it takes to generate the graph, look up a patch set, etc. ### Verification - [x] I have verified before submitting the report. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
