Re: [GNU-linux-libre] Replacing Yocto with Guix kernel image builds: best practices
On Sun, May 17, 2020 at 10:55 PM Ricardo Wurmus wrote: > > Hi, > > [+guix-devel, -gnu-linux-libre] > > > We are now looking to build Linux kernels using Guix instead of Yocto. > We > > can't see any reason why the builds wouldn't be linux-libre. Ideally we'd > > like our effort to be accepted by upstream guix. > […] > > We'd appreciate any pointers to package definition(s) that demonstrate > best > > practices to do what we'd like: > > > > - We'd like to build custom configured kernels for each patch series in > the > > LTS 4.14.72+, 4.19+ and 5.4+. > > - Currently we have two `base` kernel configs that each 'variant' > > configuration is applied to for each of a machine 'type' (3 machine > types) > > and one of two 'arch'. > > - Currently we can generate a full kernel `.config` for a > > kernel+base+variant+arch (we are working on the best way to handle > > different machines if we are not using Yocto.) > > - We'd ideally like to generate `vmlinux`, `initrd` and `rootfs` images > for > > each. > > > > Based on Efraim's post we think the first example is the least friction - > > "including an actual .config file as a native input to our custom > kernel". > > Assume we resolve the machine definition issue. However we're puzzled > > about how to best distribute the configuration file such that a build of > > kernel x.y.z can be updated with fixes. > > You can either put your config files in a separate git repository and add > that to > the native inputs, or you can include the config files in your channel > repository (or later in Guix itself). > Thanks for the suggestion. That gives some assurance. Could you point to an existing guix (upstream) package that is a best practice example of each of those two approaches? - accessing files from a separate repo - a guix (upstream) package using other files > The constraint of users being able to use the std guix commands rather > than > > telling them to download a config file or clone a git repo and copy a > > config file is what is puzzling. Some options we thought about seem > > inelegant - hence too embarassing to mention - so we'll skip them ;) > > Leaving > > > > 1) We did wonder if channels[2] were the way to go with each kernel x.y.z > > in its own branch and config files therein. Could anyone point us to > > packages that setup and use package specific channels? > > Channels handle all the gnarly bits of interacting with git. Users who > would like to add your channel providing alternative kernels would only > need to add it to their ~/.config/guix/channels.scm file. > Can "add it to their ~/.config/guix/channels.scm file" be scripted as part of the package? Is there an example of a guix (upstream) package that does this? But since your stated goal is to add these definitions to Guix itself > you can simply add the kernel variants to linux.scm and include the > config files in the repository. > Current guix/linux.scm seems to take a different approach and cater to a different use cases - but our understanding of the code is limited and no doubt flawed. We think a three step process: 1. Get a package definition working and publicly available. 2. Make any changes to get it accepted as a separate package by guix. 3. Consider whether merging with guix/linux.scm is what linux-libre folks want > 2) Should we be aiming to provide a single package with multiple > parameters > > or is it better to provide a package for each kernel x.y.z, or some other > > partitioning. We'd likely want to script the package definition then - > > correct? > > If the main differences between kernel variants are not in the package > definition but the sources and the configuration file I’d suggest > defining a procedure that returns a package value. You can then define > multiple packages in terms of that procedure. > Thanks for that suggestion. It seems the issue to resolve is the best approach for getting those config files to the users machine. Thanks again > -- > Ricardo > -- Kind Regards Begley Brothers Inc. 1. *The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.* 2. *This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most gratef
Re: Replacing Yocto with Guix kernel image builds: best practices
On Sun, May 17, 2020 at 11:21 PM Efraim Flashner wrote: > On Sun, May 17, 2020 at 09:41:00PM +1000, Begley Brothers Inc wrote: > > Hi, > > We are now looking to build Linux kernels using Guix instead of Yocto. > We > > can't see any reason why the builds wouldn't be linux-libre. Ideally we'd > > like our effort to be accepted by upstream guix. > > > > However, being new to Guix we are still coming grips with the best > > practice(s) for what we would like to do. > > We've looked at Guix's linux.scm, and > > Efraim Flashner's post[1] is our primary reference - many thanks Efraim. > > I'd like to embarrassingly mention that I never actually booted any of > my custom kernels. My machine was a bit too slow to regularly compile > for some guess-and-check. > No worries. Something is better than nothing and that was very useful outline of possible alternatives. > 1) We did wonder if channels[2] were the way to go with each kernel x.y.z > > in its own branch and config files therein. Could anyone point us to > > packages that setup and use package specific channels? > > 2) Should we be aiming to provide a single package with multiple > parameters > > or is it better to provide a package for each kernel x.y.z, or some other > > partitioning. We'd likely want to script the package definition then - > > correct? > > I would probably start with one package each and then see how they could > be made to inherit from one another or grouped together. The way the > current make-linux-libre procedures work has grown quite a bit but they > should still mostly work as a starting point to add/remove bits for > customizing for your needs. > > I don't think I'd go with different branches. If you keep everything in > one branch then it's easier to deduplicate work between the different > variants. > That was our thought too. We're just wondering how best to get that branch content down to the users machine without them having to do anything. Thanks again. > Appreciate any comments suggestions or tips. > > > > [1]: > > > https://guix.gnu.org/blog/2019/creating-and-using-a-custom-linux-kernel-on-guix-system/ > > [2]: https://guix.gnu.org/manual/en/guix.html#Channels > > > > -- > > Kind Regards > > > > Begley Brothers Inc. > > > > > >1. *The content of this email is confidential and intended for the > >recipient specified in message only. It is strictly forbidden to > share any > >part of this message with any third party, without a written consent > of the > >sender. If you received this message by mistake, please reply to this > >message and follow with its deletion, so that we can ensure such a > mistake > >does not occur in the future.* > >2. *This message has been sent as a part of discussion between Begley > >Brothers Inc. and the addressee whose name is specified above. Should > you > >receive this message by mistake, we would be most grateful if you > informed > >us that the message has been sent to you. In this case, we also ask > that > >you delete this message from your mailbox, and do not forward it or > any > >part of it to anyone else. Thank you for your cooperation and > >understanding.* > >3. *Begley Brothers Inc. puts the security of the client at a high > >priority. Therefore, we have put efforts into ensuring that the > message is > >error and virus-free. Unfortunately, full security of the email > cannot be > >ensured as, despite our efforts, the data included in emails could be > >infected, intercepted, or corrupted. Therefore, the recipient should > check > >the email for threats with proper software, as the sender does not > accept > >liability for any damage inflicted by viewing the content of this > email.* > >4. *The views and opinions included in this email belong to their > author > >and do not necessarily mirror the views and opinions of the company. > Our > >employees are obliged not to make any defamatory clauses, infringe, or > >authorize infringement of any legal right. Therefore, the company > will not > > take any liability for such statements included in emails. In case of > any > >damages or other liabilities arising, employees are fully responsible > for > >the content of their emails.* > > -- > Efraim Flashner אפרים פלשנר > GPG key = A28B F40C 3E55 1372 662D 14F7 41AA E7DC CA3D 8351 > Confidentiality cannot be guaranteed on emails sent or received unencrypted > -- Kind Regards Begley Brothers Inc. 1. *The content of th
Re: Replacing Yocto with Guix kernel image builds: best practices
On Mon, May 18, 2020 at 1:35 AM Efraim Flashner wrote: > On Mon, May 18, 2020 at 07:23:12AM +1000, Begley Brothers Inc wrote: > > On Sun, May 17, 2020 at 11:21 PM Efraim Flashner > > wrote: > > > > > On Sun, May 17, 2020 at 09:41:00PM +1000, Begley Brothers Inc wrote: > > > > Hi, > > > > We are now looking to build Linux kernels using Guix instead of > Yocto. > > > We > > > > can't see any reason why the builds wouldn't be linux-libre. Ideally > we'd > > > > like our effort to be accepted by upstream guix. > > > > > > > > However, being new to Guix we are still coming grips with the best > > > > practice(s) for what we would like to do. > > > > We've looked at Guix's linux.scm, and > > > > Efraim Flashner's post[1] is our primary reference - many thanks > Efraim. > > > > > > I'd like to embarrassingly mention that I never actually booted any of > > > my custom kernels. My machine was a bit too slow to regularly compile > > > for some guess-and-check. > > > > > > > No worries. Something is better than nothing and that was very useful > > outline of possible alternatives. > > > > > > > > > 1) We did wonder if channels[2] were the way to go with each kernel > x.y.z > > > > in its own branch and config files therein. Could anyone point us to > > > > packages that setup and use package specific channels? > > > > 2) Should we be aiming to provide a single package with multiple > > > parameters > > > > or is it better to provide a package for each kernel x.y.z, or some > other > > > > partitioning. We'd likely want to script the package definition then > - > > > > correct? > > > > > > I would probably start with one package each and then see how they > could > > > be made to inherit from one another or grouped together. The way the > > > current make-linux-libre procedures work has grown quite a bit but they > > > should still mostly work as a starting point to add/remove bits for > > > customizing for your needs. > > > > > > I don't think I'd go with different branches. If you keep everything in > > > one branch then it's easier to deduplicate work between the different > > > variants. > > > > > > > That was our thought too. > > We're just wondering how best to get that branch content down to the > users > > machine without them having to do anything. > > > > Thanks again. > > > > One option would be to include a channels file in the repo that's setup > so that you can add to the README "this is where our customizations > live. In order to build an image, run 'git pull; guix pull > --channels=our-channel-file.scm --profile > ~/.config/begley/current; ~/.config/begley/current/bin/guix > system build image.scm'". Then you can distribute the channels file you > Yes, that is somthing like what I had in mind, but would prefer to make it frictionless. Channels have the disadvantage of requiring the user do somthing that adds no value - by that I mean we've asked them to do somthing generic, and haven't saved or gotten any benefit. For example if in creating the channel they gave data that allowed us to automagically build x,y and z when they `guix pull`, and they don't have to remeber to rebuild after `guix pull` then there has been some value add. Right now I just don't see the value we could add - but we're still coming to grips with Guix. Immediately I can see that our objection is eliminated if we build a package that adds a channel in the background. Not sure if that is possible or in the spirit of Guix - we'd be altering one of their config files. How would we get their permission? etc. etc. Thanks for the feedback. -- Kind Regards Begley Brothers Inc. 1. *The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.* 2. *This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperat
Re: [GNU-linux-libre] Replacing Yocto with Guix kernel image builds: best practices
On Mon, May 18, 2020 at 5:24 AM Ricardo Wurmus wrote: > > Begley Brothers Inc writes: > > > > You can either put your config files in a separate git repository and > add > > > that to > > > the native inputs, or you can include the config files in your channel > > > repository (or later in Guix itself). > > > > > > > Thanks for the suggestion. That gives some assurance. > > Could you point to an existing guix (upstream) package that is a best > > practice > > example of each of those two approaches? > > - accessing files from a separate repo > > - a guix (upstream) package using other files > > There are many examples in the Guix repository. One example is > java-cisd-args4j in gnu/packages/java.scm, which has “build-resources” > as a native-input, which is an SVN origin. > Thanks Ricardo, That was great - I expected external linkages like that to be rejected for inclusion in upstream, so it is nice to see - I think that approach gives us a fallback in case our preferred approach does not work out. > > Can "add it to their ~/.config/guix/channels.scm file" be scripted as > part > > of the > > package? > > Is there an example of a guix (upstream) package that does this? > > No, channel configuration is a user action. The channel would be the > thing that provides your package in the first place. > > But since you want to add your package variants to Guix itself a > discussion of channels isn’t really interesting. > Agreed. Thanks again. -- Kind Regards Begley Brothers Inc. 1. *The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.* 2. *This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperation and understanding.* 3. *Begley Brothers Inc. puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.* 4. *The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorize infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.*
Re: [GNU-linux-libre] Replacing Yocto with Guix kernel image builds: best practices
On Mon, May 18, 2020 at 2:17 AM Efraim Flashner wrote: > On Mon, May 18, 2020 at 07:40:53AM +1000, Begley Brothers Inc wrote: > > On Sun, May 17, 2020 at 10:55 PM Ricardo Wurmus > wrote: > > > > > > > > Hi, > > > > > > [+guix-devel, -gnu-linux-libre] > > > > > > > We are now looking to build Linux kernels using Guix instead of > Yocto. > > > We > > > > can't see any reason why the builds wouldn't be linux-libre. Ideally > we'd > > > > like our effort to be accepted by upstream guix. > > > […] > > > > We'd appreciate any pointers to package definition(s) that > demonstrate > > > best > > > > practices to do what we'd like: > > > > > > > > - We'd like to build custom configured kernels for each patch series > in > > > the > > > > LTS 4.14.72+, 4.19+ and 5.4+. > > > > - Currently we have two `base` kernel configs that each 'variant' > > > > configuration is applied to for each of a machine 'type' (3 machine > > > types) > > > > and one of two 'arch'. > > > > - Currently we can generate a full kernel `.config` for a > > > > kernel+base+variant+arch (we are working on the best way to handle > > > > different machines if we are not using Yocto.) > > > > - We'd ideally like to generate `vmlinux`, `initrd` and `rootfs` > images > > > for > > > > each. > > > > > > > > Based on Efraim's post we think the first example is the least > friction - > > > > "including an actual .config file as a native input to our custom > > > kernel". > > > > Assume we resolve the machine definition issue. However we're > puzzled > > > > about how to best distribute the configuration file such that a > build of > > > > kernel x.y.z can be updated with fixes. > > > > > > You can either put your config files in a separate git repository and > add > > > that to > > > the native inputs, or you can include the config files in your channel > > > repository (or later in Guix itself). > > > > > > > Thanks for the suggestion. That gives some assurance. > > Could you point to an existing guix (upstream) package that is a best > > practice > > example of each of those two approaches? > > - accessing files from a separate repo > > - a guix (upstream) package using other files > > > > Do you mean something like this? It's a container instead of a full disk > image but it seems close enough. The 'gn' namespace is local to that > repo and 'gnu' and 'guix' are from the upstream guix repo. > > http://git.genenetwork.org/guix-bioinformatics/guix-bioinformatics/src/branch/master/gn/services/bnw-container.scm Thanks for sharing that definition - very interesting. It seems gexp are something to be mastered. We're trying to disentangle ourselves from containers. Guix gets us a big step in that direction, and that definitions shows some of its power. Much appreciated. -- Kind Regards Begley Brothers Inc. 1. *The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.* 2. *This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperation and understanding.* 3. *Begley Brothers Inc. puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email.* 4. *The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorize infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.*
Guix mirrors
Hi, Over the last 24 hours I've experienced `guix pull` etc being unavailable (HTTP 504's then 502's) more than available. Is there a reason why a post receive hook can't be added to the guix repo to push to github, gitlab, etc. and in that way at least give users some protection against these outages? There is a mirror[1] possibly (unofficial?) but it looks like it is driven by some chron task. The required post receive hook is well documented[2], and not un-common amoung reputable OS projects: - Android - The Apache Software Foundation - The Chromium Project - The Eclipse Foundation - The FreeBSD Project - The Glasgow Haskell Compiler - GNOME - The Linux kernel source tree - Qt [1]: https://github.com/guix-mirror/guix [2]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperation and understanding. Begley Brothers Inc. puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email. The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorize infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.
Re: [GNU-linux-libre] Replacing Yocto with Guix kernel image builds: best practices
Hi, Thanks for all the thoughtful input. We think we have worked out a preferred path - ironically from a closer reading of the linux.scm - once we had some familiarity with Guile. We think (with doubts we'll elaborate on): 1. All config files and any such files should go in a folder `gnu/packages/aux-files/`. 2. All patches should go in `gnu/packages/patches` where they all seem to live at the toplevel. The doubts are: Is this is the 'correct' way going forward - it appears only three packages adopt this location. Are we able to store patches in `gnu/packages/patches/` - it appears other packages just drop everythin in `patches` Appreciate any thoughts. On Tue, May 19, 2020 at 2:09 AM Begley Brothers Inc wrote: > > On Mon, May 18, 2020 at 5:24 AM Ricardo Wurmus wrote: >> >> >> Begley Brothers Inc writes: >> >> > > You can either put your config files in a separate git repository and add >> > > that to >> > > the native inputs, or you can include the config files in your channel >> > > repository (or later in Guix itself). >> > > >> > >> > Thanks for the suggestion. That gives some assurance. >> > Could you point to an existing guix (upstream) package that is a best >> > practice >> > example of each of those two approaches? >> > - accessing files from a separate repo >> > - a guix (upstream) package using other files >> >> There are many examples in the Guix repository. One example is >> java-cisd-args4j in gnu/packages/java.scm, which has “build-resources” >> as a native-input, which is an SVN origin. > > > Thanks Ricardo, That was great - I expected external linkages like that to be > rejected for inclusion in upstream, so it is nice to see - I think that > approach > gives us a fallback in case our preferred approach does not work out. > >> >> > Can "add it to their ~/.config/guix/channels.scm file" be scripted as part >> > of the >> > package? >> > Is there an example of a guix (upstream) package that does this? >> >> No, channel configuration is a user action. The channel would be the >> thing that provides your package in the first place. >> >> But since you want to add your package variants to Guix itself a >> discussion of channels isn’t really interesting. > > > Agreed. > > Thanks again. > > -- > Kind Regards > > Begley Brothers Inc. > > The content of this email is confidential and intended for the recipient > specified in message only. It is strictly forbidden to share any part of this > message with any third party, without a written consent of the sender. If you > received this message by mistake, please reply to this message and follow > with its deletion, so that we can ensure such a mistake does not occur in the > future. > This message has been sent as a part of discussion between Begley Brothers > Inc. and the addressee whose name is specified above. Should you receive this > message by mistake, we would be most grateful if you informed us that the > message has been sent to you. In this case, we also ask that you delete this > message from your mailbox, and do not forward it or any part of it to anyone > else. Thank you for your cooperation and understanding. > Begley Brothers Inc. puts the security of the client at a high priority. > Therefore, we have put efforts into ensuring that the message is error and > virus-free. Unfortunately, full security of the email cannot be ensured as, > despite our efforts, the data included in emails could be infected, > intercepted, or corrupted. Therefore, the recipient should check the email > for threats with proper software, as the sender does not accept liability for > any damage inflicted by viewing the content of this email. > The views and opinions included in this email belong to their author and do > not necessarily mirror the views and opinions of the company. Our employees > are obliged not to make any defamatory clauses, infringe, or authorize > infringement of any legal right. Therefore, the company will not take any > liability for such statements included in emails. In case of any damages or > other liabilities arising, employees are fully responsible for the content of > their emails. -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future.
Re: Guix mirrors
Hi, Thanks for taking the time to respond. Sounds like good news overall On Wed, May 20, 2020 at 6:57 AM zimoun wrote: > > Hi Ricardo, > > On Wed, 20 May 2020 at 13:02, Ricardo Wurmus wrote: > > > I expect that this will change soon after some more discussion of the > > details. > > Last time we discussed that was on December [1]. What is missing is: discuss > if > > a) an "official" mirror is hard-coded in '%default-channels' > or > b) an "official" mirror is documented in the manual and it is up to > the user to setup it via ~/.config/guix/channels.scm. > > The a) annoys people who do not want the mirror, so they have to tweak > the list by themself and in the same a) is more newcomer-friendly > because it just works. > The b) annoys newcomers because it is another step and in the same > time b) is more customizable. > > Obviously, there is option c): guix-install.sh asks a question and > then do the b) dance. > > > And discuss if a channel is described as: > > (channel >(name 'guix) >(url "https://savannah…;) >(mirror (list "https://mirror-one…; > "https://mirror-two…;))) > > [ where mirror is optional ] > > or something else? There should be a built in (hard-coded) failover list - ideally three - without the user being told or asked to do anything during setup. Ideally the user could alter that list - but that seems a stage 2 feature given the curent state. Stage 3 might alllow the default list to vary by country code - driven by and env var set to two letter ISO code? I was surprised guix doesn't seem to be mirrored to gnu.io - or is that the same machine(s)? On reflection in this context 'mirror' is not descriptive in the channel config - 'failover' might better communicate the intent and context (channel (name 'guix) (url "https://savannah…;) (failover (list (url "https://mirror-one…;) (url "https://mirror-two…; where **overriding** the failover list is optional, but you cannot unset the default list HTH > Well, Ludo suggested [2] to address "Trustable guix pull" [3]; which > should be really cool! > Now, I am not sure to understand if there is no mirror mechanism > because no one took the time to implement it or if any other reasons > related to "trustable guix pull". > > > [1] https://lists.gnu.org/archive/html/guix-devel/2019-12/msg00150.html > [2] https://lists.gnu.org/archive/html/guix-devel/2019-12/msg00276.html > [3] http://issues.guix.gnu.org/issue/22883 > > > Cheers, > simon -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperation and understanding. Begley Brothers Inc. puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email. The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorize infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.
Re: Slurm with containers (i.e., orchestration)
xecute jobs using > some constraint specifiers (like network/CPU/RAM). It could be a > 'dynamic' Slurm that makes use of real machines and VMs. Or hook into > an existing cloud service. A slurm job could monitor sending a > container into a cloud service. Agreed. Those also strike us as 2nd order/stage scope elements - once orchestrated VM's and MM's are running Guix deployed OS and apps. > I think we can build this up a step at a time. It is not clear, but it does sound like you could intend to implment everything in Guix? Or take a more "build small, build modular, and build simple" approach where Guix connects some pre-existing elements? We've identified one in Tinkerbell/tink, where Guix would get the benefit of "four microservices that take you from a powered off server to a high-level execution environment running your very special custom thingamabobber" but there maybe others better suited? > > Thoughts? Very interesting. Thanks for sharing. [1]: https://github.com/tinkerbell/tink [2]: https://github.com/tinkerbell/tink/blob/master/docs/hello-world.md [3]: https://github.com/tinkerbell/tink/blob/master/docs/setup.md [4]: https://github.com/alexellis/awesome-baremetal [5]: https://www.packet.com/blog/open-sourcing-tinkerbell -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this message by mistake, we would be most grateful if you informed us that the message has been sent to you. In this case, we also ask that you delete this message from your mailbox, and do not forward it or any part of it to anyone else. Thank you for your cooperation and understanding. Begley Brothers Inc. puts the security of the client at a high priority. Therefore, we have put efforts into ensuring that the message is error and virus-free. Unfortunately, full security of the email cannot be ensured as, despite our efforts, the data included in emails could be infected, intercepted, or corrupted. Therefore, the recipient should check the email for threats with proper software, as the sender does not accept liability for any damage inflicted by viewing the content of this email. The views and opinions included in this email belong to their author and do not necessarily mirror the views and opinions of the company. Our employees are obliged not to make any defamatory clauses, infringe, or authorize infringement of any legal right. Therefore, the company will not take any liability for such statements included in emails. In case of any damages or other liabilities arising, employees are fully responsible for the content of their emails.
Re: Guix mirrors
On Wed, May 20, 2020 at 7:45 PM Begley Brothers Inc wrote: > > Thanks to @kozodev on gitlab.com, a possible workaround, available > immediately, is to document the following in the web pages and alt-F2 > help as appropriate: > > 3.6.2 Proceeding with the Installation > > 8.1 Using the Configuration System > > with (note the inital root init channel path needs to be inserted) > > ```bash > ... > (use-package-modules screen) > > ;; Your `guix system init /etc/conf.scm` or > ;; susequent `guix pull` and `guix system reconfigure` may > ;; fail because the Guix server is unavailable - you will see > ;; HTTPS errors 404, 504, 502. > ;; In this situtaion, please use one of: > ;; - "https://mirror1.com/x/y/z/guix.git; > ;; - "https://mirror2.com/a/guix.git; > ;; - "https://mirror2.com/d/e/guix.git; > ;; in the `with-output-to-file` code below. Which you should uncomment, > ;; then re-run: > ;; > ;; # guix system init /etc/conf.scm > ;; > ;; NOTE: Air-Gapped Facility Users. > ;; If you are using guix in an air-gapped facility you will > ;; need to add your Guix repository and uncomment this code > ;; *before* running: > ;; > ;; # guix system init /etc/conf.scm > ;; > ;; There is curently no other way to repoint the init phase > ;; to your air-gapped repository. > > ;;(with-output-to-file "/path/to/roots/first/init/channels.scm" > ;; (lambda () > ;;(display "(cons* (channel (name 'guix) (url > \"https://internal.net/x/y/z/guix.git\;)) %default-channels)"))) > > (operating-system > ... > ``` > > HTH? > > > On Tue, May 19, 2020 at 2:32 AM Begley Brothers Inc > wrote: > > > > Hi, > > Over the last 24 hours I've experienced `guix pull` etc being > > unavailable (HTTP 504's then 502's) more than available. > > > > Is there a reason why a post receive hook can't be added to the guix > > repo to push to github, gitlab, etc. and in that way at least give > > users some protection against these outages? > > > > There is a mirror[1] possibly (unofficial?) but it looks like it is > > driven by some chron task. > > > > The required post receive hook is well documented[2], and not > > un-common amoung reputable OS projects: > > > > - Android > > - The Apache Software Foundation > > - The Chromium Project > > - The Eclipse Foundation > > - The FreeBSD Project > > - The Glasgow Haskell Compiler > > - GNOME > > - The Linux kernel source tree > > - Qt > > > > [1]: https://github.com/guix-mirror/guix > > [2]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks > > > > -- > > Kind Regards > > > > Begley Brothers Inc. PS I'm not sure if you want to add this to the docs, but there does not seem to be dedicated functionality to securely deal with secret data in Guix. One way to work a around that is to host your own package, and add that to the channels before you run (as root): ```bash guix system init /etc/conf.csm ``` These two issues (configurable repo and secret management) are what I've encountered that blocked "reproducible operating systems" being true. Since that is a claim made on the fornt page (https://guix.gnu.org/) I think adding something like the prior suggestion to the docs mean you can plausibly claim the statement is not misleading. Otherwise maybe change the statement to "partially reproducible systems" and at the "declarative system configuration" link to section 8.1 Using the Configuration System, just add two bullet points that still in devlopment are 1) configurable init repositories 2) secret management. Its debatable whether orchestration is a necessary function for the "reproducible operating systems" claim to be considered (reasonably) true. Hashicorp/Terraform and Packet/Tinkerbell would disagree. I should note I haven't mentioned the FSF/GNU/Guix sponor and their orchestration product mainly because it is Apache Airflow adapted to a cloud use case. Apache Airflow themselves say they are more comparable to Oozie and Azkaban, so you have to do a lot of heavy lifting to get your sponsor's product - as best I know there is not OSS project upstream to your sponsors product. Happy to stand corrected. -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. This message has been sent as a part of discuss
Re: Guix mirrors
Thanks to @kozodev on gitlab.com, a possible workaround, available immediately, is to document the following in the web pages and alt-F2 help as appropriate: 3.6.2 Proceeding with the Installation 8.1 Using the Configuration System with (note the inital root init channel path needs to be inserted) ```bash ... (use-package-modules screen) ;; Your `guix system init /etc/conf.scm` or ;; susequent `guix pull` and `guix system reconfigure` may ;; fail because the Guix server is unavailable - you will see ;; HTTPS errors 404, 504, 502. ;; In this situtaion, please use one of: ;; - "https://mirror1.com/x/y/z/guix.git; ;; - "https://mirror2.com/a/guix.git; ;; - "https://mirror2.com/d/e/guix.git; ;; in the `with-output-to-file` code below. Which you should uncomment, ;; then re-run: ;; ;; # guix system init /etc/conf.scm ;; ;; NOTE: Air-Gapped Facility Users. ;; If you are using guix in an air-gapped facility you will ;; need to add your Guix repository and uncomment this code ;; *before* running: ;; ;; # guix system init /etc/conf.scm ;; ;; There is curently no other way to repoint the init phase ;; to your air-gapped repository. ;;(with-output-to-file "/path/to/roots/first/init/channels.scm" ;; (lambda () ;;(display "(cons* (channel (name 'guix) (url \"https://internal.net/x/y/z/guix.git\;)) %default-channels)"))) (operating-system ... ``` HTH? On Tue, May 19, 2020 at 2:32 AM Begley Brothers Inc wrote: > > Hi, > Over the last 24 hours I've experienced `guix pull` etc being > unavailable (HTTP 504's then 502's) more than available. > > Is there a reason why a post receive hook can't be added to the guix > repo to push to github, gitlab, etc. and in that way at least give > users some protection against these outages? > > There is a mirror[1] possibly (unofficial?) but it looks like it is > driven by some chron task. > > The required post receive hook is well documented[2], and not > un-common amoung reputable OS projects: > > - Android > - The Apache Software Foundation > - The Chromium Project > - The Eclipse Foundation > - The FreeBSD Project > - The Glasgow Haskell Compiler > - GNOME > - The Linux kernel source tree > - Qt > > [1]: https://github.com/guix-mirror/guix > [2]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks > > -- > Kind Regards > > Begley Brothers Inc. > > The content of this email is confidential and intended for the > recipient specified in message only. It is strictly forbidden to share > any part of this message with any third party, without a written > consent of the sender. If you received this message by mistake, please > reply to this message and follow with its deletion, so that we can > ensure such a mistake does not occur in the future. > This message has been sent as a part of discussion between Begley > Brothers Inc. and the addressee whose name is specified above. Should > you receive this message by mistake, we would be most grateful if you > informed us that the message has been sent to you. In this case, we > also ask that you delete this message from your mailbox, and do not > forward it or any part of it to anyone else. Thank you for your > cooperation and understanding. > Begley Brothers Inc. puts the security of the client at a high > priority. Therefore, we have put efforts into ensuring that the > message is error and virus-free. Unfortunately, full security of the > email cannot be ensured as, despite our efforts, the data included in > emails could be infected, intercepted, or corrupted. Therefore, the > recipient should check the email for threats with proper software, as > the sender does not accept liability for any damage inflicted by > viewing the content of this email. > The views and opinions included in this email belong to their author > and do not necessarily mirror the views and opinions of the company. > Our employees are obliged not to make any defamatory clauses, > infringe, or authorize infringement of any legal right. Therefore, the > company will not take any liability for such statements included in > emails. In case of any damages or other liabilities arising, employees > are fully responsible for the content of their emails. -- Kind Regards Begley Brothers Inc. The content of this email is confidential and intended for the recipient specified in message only. It is strictly forbidden to share any part of this message with any third party, without a written consent of the sender. If you received this message by mistake, please reply to this message and follow with its deletion, so that we can ensure such a mistake does not occur in the future. This message has been sent as a part of discussion between Begley Brothers Inc. and the addressee whose name is specified above. Should you receive this mess
Re: Slurm with containers (i.e., orchestration)
P.S Just to keep things interesting - GWL: A workflow management language extension for GNU Guix The Guix Workflow Language (GWL) provides a scientific computing extension to GNU Guix's declarative language for package management for the declaration of scientific workflows. https://www.guixwl.org/tutorial On Tue, May 19, 2020 at 5:33 PM Begley Brothers Inc wrote: > > On Mon, May 18, 2020 at 7:50 AM Pjotr Prins wrote: > > > > I am looking into some light-weight style orchestration. One > > We think there is such a niche, the 80/20 rule. > We think containers are too limiting and a bad idea to target - but we > use them and they have mind share. > We also have some other ideas in mind, related to this context, but > we'll keep this on-topic. > > Compromise: > Cast the issue in terms of a VM and let the 'hello world' MVP/example > be a VM that grabs a container from a registry and runs it to > completion and shutsdown. > Then demostrate the use of Guix building the VM and show the you can > not only discard the the container overhead, cruft and headaches, but > you also get a more powerful Dockerfile, and all the other Guix > features. > Then show that you can easily repurposethat VM workflow to a Metal Machine. > > Of course in real world cases there are many scenarios where people > are looking for the reverse incrementalist pathway: > 1.) Legacy-App + MM > 2.) Legacy MM + (Guix + Legacy-App) > 3.) Legacy MM + Guix + workflow > 4.) Guix + workflow + (VM or MM or both) > > > possibility is to use Slurm with Guix containers - on a cluster with > > Guix that is almost trivial (we use Guix containers a lot! They are > > great) and would also allow non-container jobs. > > Hmm, doesn't slurm break the opening objective 'light-weight'? > Maybe better to write a VM abstraction/adapter for something like > Tinkerbell/tink[1], its Apache-2.0, and some project context is > here[5]. > > Define the use case as: VM's that run a task lauched by init and shut > themselves down when done - many of course have open-ended run times. > For multiple VM use cases: > There are a multitude of distributed computing tools that Guix leaves > the user free to chose amoung to build into their VM - Guix could take > no position on whether Condor, Nomad, etc, etc., etc. are better > suited to someone's problem. > > With those constraints in mind, and lightweight being primary, then it > is simple to imagine Guix generating a VM version of such a > workflow[2]and delgating the workflow heavy lifting to > Tinkerbell/tink: > > ```bash > guix light ~/src/project/hello-world.tmpl > ``` > > > Once we have containers and Slurm it should also be possible to deploy > > slurm: -1 > containers: -1 > > > in some cloud infrastructure, provided there are no dependencies on > > I think you could get there and beyond with some relatively minor > (compared to Slurm) contributions to Tinkerbell (Apache-2.0). > This setup[3] targets an AWS instance, so you could likely leverage > `guix deploy` too. > > > the cluster itself. I think it would make a terrific BLOG story if we > > put something like that together. > > > > Bcbio describes an architecture that uses the common workflow language > > (CWL) to run pipelines with containers > > > > > > https://bcbio-nextgen.readthedocs.io/en/latest/contents/cwl.html#running-with-cromwell-local-hpc > > > > I am not promoting the use of this, but it shows that infrastructure > > exists that can deploy workflows on containers in different setups > > Again we believe if you think in terms of VM's (rather than > containers) there is a wider set of possible use cases. > If you build on Tinkerbell/tink or re-implement its logic - not clear > what you have in mind - you could also expand the Guix use cases to > workflows that include metal machine (MM) users/managers. > > > (Bcbio supports Slurm). I know the Guix infrastructure uses Guix > > deploy to achieve similar roll-outs. What that lacks is the > > orchestration mechanism itself which should handle dependencies > > between jobs (i.e. a workflow). > > We're not familiar with GNU workflow language - with that caveat: > We think that for a lightweight implementation you might be better off > defining the scope more narrowly but still expanding the universe of > Guix use cases as we outlined. > Once that is in place, experience might lead the project to be > opinionated on how 'jobs' are handled between machines (VM and MM). > > Maybe getting to the point of a Guix entry in such Awesome-Metal lists[4] > > > The GNU Workflow Language goes some > > way, but it does not handle orchestrat
