Hi, On Sat, Jun 06, 2020 at 12:57:19PM +0200, Ruben Di Battista wrote: > I'm probably saying something completely wrong, but can't you leverage > the CI system and buildbots to create a Github release that includes > the PortIndex and whatsoever and then download it from Github servers > via http? What am I missing here?
The ports tree changes multiple times a day. We currently build a new PortIndex every few hours, and the way we sync things (PortIndex together with the ports tree), we automatically ensure that the PortIndex matches the files you have on disk. Additionally, we sync the file modification time, so even if the PortIndex does not match exactly, we can easily compute on the client side which ports have not been indexed yet. This is not as simple with Git. Git sets the mtime of modified files to the date at which you did a checkout on them, not the time when they were committed. The portindex command, however, compares the PortIndex mtime with the mtimes of the portfiles. The unsolved problems are: 1. Do we really want to create a new GitHub release every few hours? 2. How does MacPorts know which ports are correctly indexed in a downloaded PortIndex, and which ones were added afterwards and thus need to be re-indexed locally? 3. How do we ensure efficient transfer of the PortIndex? The PortIndex is currently ~15M, rsync efficiently only copies the delta for us. With hosting a precompiled PortIndex on GitHub, how do we avoid downloading the entire PortIndex on every sync? I guess the solution for (2) could be to include the Git hash that was indexed, compute the modified files between that hash and HEAD, touch them, and re-run portindex. I don't have a good suggestion for (3) yet. It might be possible to generate daily PortIndex diffs, download all of those between the time of the last update and now and apply them in-order. Such a mechanism would have to work with any local modifications users might have in their tree, though. As soon as the size of the diffs is larger than just re-downloading the entire thing, we could switch to that. Technically, all of this is doable. However, somebody actually needs to sit down and implement it. -- Clemens
