Hello Pierre,
Thanks for sharing your progress. A few remarks below. > 3. Size of the database: > I've persisted all locally-present store items for my current Guix version > and it produced a database of 72 MiB. It compresses down to 8 MiB in zstd. > > But since we can have multiple Guix versions, this means that the > packages have one entry per store path, so we might end up with more > entries than that as the number of Guix generations grows. I'm not sure we actually need to save the full history. I think we could just store that the package X produces [Y1, Y2, ...] executable files. Then on package X update, the executable files list could be refreshed. > Question: Should we include empty directories in the database? I'm tempted > to answer no. I would also say no, and also exclude non-executable files. > Question: This bounds us to the SQLite syntax for pattern matching. Is it > a > problem? > It seems powerful enough in practice. But maybe we can use regular > expression in SQLite as well? >From the UI perspective, we already have "guix search" that expects a regex. If we were to include a "guix file-search" command, then I think it would make sense that it uses the same regex syntax. > Next points I'd like to address: > > 6. Automatically persist the database entry when building a package. > Any idea where I should plug that in? > > 7. Have substitute servers distribute database content. When the user > performs > a file search, Guix asks the substitute server for a database update. Only > the diff should be sent over the network, not the whole thing since it > might If I understand correctly, you are proposing to create local databases that would be kept in sync with a master database populated by the CI server. This seems a bit complex. What about extending Cuirass database to add the two tables you are proposing. Then, each time a package is built, if the version is updated, the "Files" table would be updated. Then we could add an HTTP interface such as "/search/file?query=libxxx" to Cuirass, that would directly query the database. In Guix itself, we could add the counterpart in the (guix ci) module. WDYT? Thanks, Mathieu