Hi, Lets keep the ball rolling on this, seems to be a hard loop to close.
On Thu, 2020-10-01 at 17:20 +0100, Chandan Singh wrote: > Hi folks, > > I agree that we need to sort out this story before 2.0. Tomaz, I have > a clarifying question for your comment near the end of the message. > > On Thu, Oct 1, 2020 at 10:45 AM Tristan Van Berkom > <[email protected]> wrote: > > Hi, > > > > On Thu, 2020-10-01 at 10:38 +0100, Tomaz Canabrava wrote: > > > On Thu, Oct 1, 2020 at 10:21 AM Jürg Billeter <[email protected]> wrote: > > > > > > > Hi Tristan, > > > > > > > > On Thu, 2020-09-24 at 15:34 +0900, Tristan Van Berkom wrote: > > > > > I am suggesting today that we need to consider the ability of > > > > > disabling remote caches[0]: > > > > > - Do we want it only for the toplevel project ? > > > > > - Recursively ? with a -J/--cross-junctions option ? > > > > > - With granular selection of which junctions to disable cache > > > > > interactions for ? (with junction path specifications on the > > > > > command line ?) > > > > > > > > I currently tend towards two knobs for this: > > > > > > > > * `--no-pull`, `--no-fetch` and the equivalent options in > > > > buildstream.conf as described in #819. This would completely disable > > > > pulling artifacts and fetching sources, resp. I.e., always across > > > > junctions and also disable servers configured in buildstream.conf. > > +1 Personally I am happy with this, I would have been happy to piggy back on the `--fetchers` configuration and recognize a `0` value, but this would not distinguish between source fetching and artifact pulling. With regards to the proposal around configuration, things become less clear due to the current mess, however I tend to agree that in any scenario, these command line options should supersede all other controls, and only act unilaterally across junctions. So far, I think this means that regardless of the other more complicated proposal about configuration, we can accept adding these options and it will not cause any conflict. > > > for the usecase I need, it would need to be element-dependant. for > > > instance, I want to test the reprocibility of `python.bst`, so I need to > > > block it from using the cache and do a build. but python depends on other > > > things that I do want to use from the cache. > > > --no-pull="element1.bst, element2.bst" would do the trick. > > > > I believe that your use case is satisfied by: > > > > * Build it once > > > > * Build it repeatedly (however number of times you want to test > > reproducibility of python.bst) using `--no-pull` after the > > first build, while deleting only that artifact in between runs > > locally. > > > > I.e. you won't need to download or rebuild all of those dependencies if > > you have them already in your local cache from the last build. > > Tomaz, I think the question we need to ask is - do you expect to > repeat the build locally or remotely. What Tristan describes would > work IF both the initial build and the subsequent builds are happening > locally. If you are expecting either of them to happen remotely, I > don't think that can be solved just by BuildStream, as the RE endpoint > would have its own caching mechanism. Chandan, others... What do we foresee in terms of ability to interact with remote caches in an RE context ? I remember older discussions where Sander had the idea that `bst artifact` commands such as `list-artifacts` and `delete` etc, in general, should assume we mean the remote cache instead of the local one (I mostly recall disagreeing with this). But the point is, regardless of the particular shape of the future, we need to have a better idea of what that shape is going to be if we're going to have a meaningful conversation about how command line options and configuration is to be expressed. Even if that future has not materialized completely, we need some hints in order to design API stable interfaces in advance of things being implemented. So, I guess my question is; if we have --no-fetch / --no-pull options respected in the RE context, such that external CAS caches are explicitly disabled and only the main RE CAS is consulted, and we have a way to at least perform `bst artifact delete` on a remote CAS, e.g. a remote CAS used for the configured RE cluster... then would it still be reasonable to say that we actually *can* do this with the same user facing interfaces in an RE context ? Cheers, -Tristan
