Hi Caleb, I'm sure Ludo may have more to say, but I think I can help you with a few of your questions.
Caleb Ristvedt <[email protected]> writes: >> Though something went from with your email client. :-) > > Aye, turns out M-x epa-mail-sign doesn't work well with attachments. I > posted another message that should be signed correctly, as well as this > one. Looks like your latest email was signed correctly. I see that it was signed using your Savannah key, which has the following fingerprint: 74D6 A930 F44B 9B84 9EA5 5606 C166 AA49 5F7F 189C > - A derivation is a node in a (acyclic?) graph of "what depends on what" or, > alternately put, what is input to what. I don't think there's a simple answer for the question of "what is a derivation"? I'll explain what I know, and hopefully it'll help you. No, derivations cannot depend on one another [1]. I believe it is correct to think of a derivation as a node in an acyclic graph of inter-dependent software components; that is one way to think about derivations. Perhaps Ludo can say more about that. The word "derivation" is a bit overloaded. Fundamentally (and vaguely), it means "a component build action, which *derives* the component from its inputs" [2]. More specifically, there are various interpretations of the word "derivation", depending on the context. For example, there are structs in the Nix C++ source code that represent derivations [3]. In the Nix expression language, derivations are represented as attribute sets [4]. There are also "store derivations", which are files in the store that precisely encode the build actions that will be executed when the derivation is built [5]. Some of this carries over to Guix. The general meaning of the word "derivation" appears to be the same as in Nix: a derivation basically takes inputs and produces an output. Guix also uses "store derivations" because we use (a modified version of) the Nix daemon. Note that the Guix manual sometimes uses the word "derivation" to refer to "store derivations", which can be confusing. As you probably know, Guix eschews the Nix expression language in favor of Guile for composing components. Therefore, Guix has defined abstractions in Guile to represent derivations [6]. Guix has also defined additional abstractions to represent things like packages [7], which are higher level abstractions than derivations. There are a lot of abstractions like this in Guix, and they interact with each other in various ways. To get an idea of what I mean, you can look at the '(guix) Invoking guix graph' section in the Guix manual. Derivations are a general concept taken from Nix, but as far as I know, things like bags and packages are Guix-specific concepts. > - "Building" a derivation involves first ensuring that all of its inputs are > "built" (in the case of inputs that are derivations, this means ensuring its > output exists as a store item, and if not, producing it by this same > process. In the case of plain file inputs, this means just ensuring that > they > exist as a store item), and then preparing a build environment. The > "builder" > script is then executed in a chroot rooted under the build tree. After it > has > completed, the output (where is it put during the build process?) from > building the derivation is placed in the store (and registered). Note that sometimes the word "build" is used to refer to the act of creating a derivation, but not executing its build actions [8]. The process you've described sounds basically right to me; Ludo probably knows more. The Nix thesis might be out of date by now, but it describes (described?) these kinds of things in pretty good detail [9]. It also describes the invariants that must be upheld by the entire process. > - "preparing a build environment" involves first creating a directory >in which > the build will be run, then populating it with certain expected files and > directories (as detailed in 2.4.1 of the manual). Also, the store items > produced from building the inputs need to somehow be made available (is > there > a gnu/store/ under the build tree? I feel like I don't quite understand > where > everything goes in the build tree). Also, set any environment variables > associated with the derivation. I believe that inputs and outputs are, from the a derivation's builder's perspective, just paths in the store which can be accessed via environment variables that Nix sets up during the build. I think this is the same in Guix as in Nix. However, in Guix, I believe there are additional ways to refer to inputs when defining a builder, e.g. by using G-Expressions, which hide details like this from the user so that the user can reason about builds in terms of higher-level abstractions. As for how the store items produced from building the inputs are "made available," my understanding is that the builder just needs to write files to the special "output" path, which (I think) is in the store. That special output path is made available to the builder during the build via an environment variable (the variable is named "out" or "output"; I can't remember exactly). If you're using G-Expressions to define a builder, the output path can be accessed via "#$output" (see '(guix) G-Expressions' for details). > Assuming I've got the gist of it sort of right, that leaves one more >question: > how are hashes for the paths computed? It's not sha256 like the database > hashes, > it seems to be 160 bits in size. How are all the inputs hashed together? Are > all > the inputs put in one big directory and that is hashed in nar form, or what? > And > what about store items that don't have any inputs (for example, something > placed > in there with add-text-to-store)? I suppose in that case they are their own > input, perhaps? The hashing mechanisms, the relevant invariants, and the motivation behind these design decisions are described in the Nix thesis [10]. I think some (maybe all) of this hashing stuff is implemented in Guix in guix/derivations.scm (check the 'derivation' procedure) and guix/store.scm (check the procedures that 'derivation' calls). Hope that helps! Good luck with your project. Footnotes: [1] See footnote 4 on p. 29 of Eelco Dolstra's Ph. D. thesis. [2] See p. 27 of the thesis. There are alternate (and equally vague) definitions in the Guix and Nix manuals. They all have the same spirit: a derivation is like a function from software components to software components. [3] See Derivation in derivations.hh in the Nix source code. [4] The following blog entry explains this in detail: https://lethalman.blogspot.co.uk/2014/07/nix-pill-6-our-first-derivation.html [5] See Section 2.4 "Store derivations" in the thesis for a great explanation of store derivations, including why it is useful to represent derivations in this way. [6] See the <derivation> record type in guix/derivations.scm. Looks like Guix also implements some of the actual logic for serializing a <derivation> into a store derivation (see write-derivation in the same module). [7] See <package> guix/packages.scm. [8] See for example the docstring for the 'derivation' procedure defined in guix/derivations.scm, which claims to "build a derivation" when in fact it only creates the store derivation without executing its build actions. [9] See for example section 5.5 "Building store derivations", which describes how store derivations are built in the extensional model (the intensional model, discussed later in the thesis, has not been implemented in Nix or Guix). Figure 5.11, box #79 shows that the builder is executed in a temporary directory; presumably the builder writes the component's output directly into the output path (which is in the store). You can probably compare this to the actual build algorithm in the C++ source code to figure out what's really going on. [10] See sections 5.1 "Cryptographic hashes" and 5.2.2 "Store paths" in the Nix thesis. Various kinds of hashes are used in various places; it's all pretty complicated, but the thesis describes much of the details and motivation. There may be other relevant sections, depending on what exactly you want to know. Of course, this thesis may be out of date by now, so the C++ source code is the place to look to find actual implementation. But it's helpful to know the intent behind the code. -- Chris
signature.asc
Description: PGP signature
