Re: GSoC 2024

[Efraim Flashner]

On Thu, Mar 07, 2024 at 02:09:32PM +, Steve George wrote:
> 
> Hi,
> 
> I had a couple of ideas - but would need help from someone to mentor
> 
> 1. Moldable development in Guix
> Exploratory REPL experience is one of the hall-marks of 'moldable' systems. 
> This shortens the development cycle and improves the ability of users to 
> explore Guix.
> 
> The best REPL experience today is through Emacs. We have a modern nREPL 
> implementation that is compatible with Guile. This needs further development 
> and the Guix client side improved.
> 
> * Develop a basic CLI Nrepl experience in guile-ares-rs 
> (https://git.sr.ht/~abcdw/guile-ares-rs)
> * Add further CLI REPL functions to Guix 
> * Stretch goal to add a Guix / Guile Scheme nrepl support to Conjure 
> (https://github.com/Olical/conjure/issues/549) 
> 
> This would need co-ordination with Andrew Tropin (abcw) and Oliver Caldwell 
> (Olical), and some help from a Guix mentor. 
> 
> 2. Improving Docker image output (guix pack)
> Docker containers are a common deployment method for applications. While they 
> may be good for deployment, they have weak reproducibilty which Guix solves. 
> Docker containers generated by Guix for deployment are large compared to 
> similar deployments using Nix or Alpine. The purpose of this project is to 
> optimise the build and deployment pipeline in Guix.
> 
> * Examine the current 'guix pack' process for optimisations
> * Optimise the build process to add docker specific capabilities like 
> multi-stage builds
> * Explore using grafts or masking to reduce final image size
> 
> ** NOTE:** I know this is a bit weak - I don't know enough about this myself 
> yet - is this even a good target - I think it's interesting for scientific 
> computing?

This would also be useful for "deploy this guix service as a docker
container".

> 3. Add sandboxing to guix packages
> Improving the security for end-users by implementing optional sandboxing for 
> desktop applications. The likes of Bubblewrap and Flatseal are available for 
> Linux. There's some existing Nix prior-art that could be a good starting 
> point (https://nixos.wiki/wiki/Firejail) and 
> (https://sr.ht/~fgaz/nix-bubblewrap/)
> 
> * Figure out which of the available options is the most sustainable
> * Integrate policys and implementation into high-profile packages
> * Stretch would be to create a Guile native library / approach
> 
> Anyone interested in these - willing to mentor/co-mentor with me?
> 
> On  4 Mar, Gábor Boskovits wrote:
> > Hello guix,
> > 
> > I coordinated with the GNU org admins, and we can still do this round,
> > but we have to go fast to make this happen. I have already taken the
> > initiative to try to get an ideas page up, now I would like to confirm
> > if the mentors from last year are still available, and that the ideas
> > are still valid.
> > 
> > Hereby I quickly collected the projects with the respective mentors,
> > please pm me your availability:
> > 
> > Decentralized substitute distribution
> > pukkamustard (pukkamustard [at] posteo [dot] net)
> > attila.lendvai (ethswarm.org, scheme)
> > 
> > Robustify long-term support for Reproducible Research
> > Simon Tournier (zimoun)
> > 
> > Develop a Web interface to configure Guix System
> > Ludovic Courtès (civodul)
> > 
> > Trusted computing: Goblins for GNU Guix
> > Christopher Webber, Ludovic Courtès and Pjotr Prins
> > 
> > Guix Data Service revision processing instrumentation and performance
> > Christopher Baines
> > 
> > Guile based build-tool
> > Pjotr Prins
> > 
> > GNU Guix system monitor
> > Pjotr Prins
> > 
> > Booting via network
> > Danny Milosavljevic
> > 
> > Syntax and semantics of systemd units in the Shepherd
> > Ludovic Courtès (civodul)
> > 
> > GNUnet integration
> > no mentor available
> > 
> > Adding modules in support of continuous integration to cuirass
> > Ludovic Courtès (civodul)
> > 
> > Continue rewrite build daemon in Guile Scheme
> > Ludovic Courtès (civodul)
> > 
> > I myself am available to co-mentor, and also to be the formal mentor
> > in case someone does not feel like doing the official dance with
> > Google. Currently I can commit to devoting two hours a week to this.
> > 
> > Regards,
> > g_bor
> > 
> 

-- 
Efraim Flashner  רנשלפ םירפא
GPG key = A28B F40C 3E55 1372 662D  14F7 41AA E7DC CA3D 8351
Confidentiality cannot be guaranteed on emails sent or received unencrypted


signature.asc
Description: PGP signature

Re: GSoC 2024

[Gábor Boskovits]

Hello,

Ekaitz Zarraga  ezt írta (időpont: 2024. márc. 7., Cs
15:16):

> I have an idea too, but I don't have the knowledge to mentor it.
>
> guix pack support for AppImages.
>

Good idea. I also don't have the technical for this. Could you add it to
the ideas page as draft and without mentor, so that we can track it?

Should you need any help feel free to contact me.

Regards,
g_bor

Re: GSoC 2024

[Steve George]

Hi,

I had a couple of ideas - but would need help from someone to mentor

1. Moldable development in Guix
Exploratory REPL experience is one of the hall-marks of 'moldable' systems. 
This shortens the development cycle and improves the ability of users to 
explore Guix.

The best REPL experience today is through Emacs. We have a modern nREPL 
implementation that is compatible with Guile. This needs further development 
and the Guix client side improved.

* Develop a basic CLI Nrepl experience in guile-ares-rs 
(https://git.sr.ht/~abcdw/guile-ares-rs)
* Add further CLI REPL functions to Guix 
* Stretch goal to add a Guix / Guile Scheme nrepl support to Conjure 
(https://github.com/Olical/conjure/issues/549) 

This would need co-ordination with Andrew Tropin (abcw) and Oliver Caldwell 
(Olical), and some help from a Guix mentor. 

2. Improving Docker image output (guix pack)
Docker containers are a common deployment method for applications. While they 
may be good for deployment, they have weak reproducibilty which Guix solves. 
Docker containers generated by Guix for deployment are large compared to 
similar deployments using Nix or Alpine. The purpose of this project is to 
optimise the build and deployment pipeline in Guix.

* Examine the current 'guix pack' process for optimisations
* Optimise the build process to add docker specific capabilities like 
multi-stage builds
* Explore using grafts or masking to reduce final image size

** NOTE:** I know this is a bit weak - I don't know enough about this myself 
yet - is this even a good target - I think it's interesting for scientific 
computing?

3. Add sandboxing to guix packages
Improving the security for end-users by implementing optional sandboxing for 
desktop applications. The likes of Bubblewrap and Flatseal are available for 
Linux. There's some existing Nix prior-art that could be a good starting point 
(https://nixos.wiki/wiki/Firejail) and (https://sr.ht/~fgaz/nix-bubblewrap/)

* Figure out which of the available options is the most sustainable
* Integrate policys and implementation into high-profile packages
* Stretch would be to create a Guile native library / approach

Anyone interested in these - willing to mentor/co-mentor with me?

On  4 Mar, Gábor Boskovits wrote:
> Hello guix,
> 
> I coordinated with the GNU org admins, and we can still do this round,
> but we have to go fast to make this happen. I have already taken the
> initiative to try to get an ideas page up, now I would like to confirm
> if the mentors from last year are still available, and that the ideas
> are still valid.
> 
> Hereby I quickly collected the projects with the respective mentors,
> please pm me your availability:
> 
> Decentralized substitute distribution
> pukkamustard (pukkamustard [at] posteo [dot] net)
> attila.lendvai (ethswarm.org, scheme)
> 
> Robustify long-term support for Reproducible Research
> Simon Tournier (zimoun)
> 
> Develop a Web interface to configure Guix System
> Ludovic Courtès (civodul)
> 
> Trusted computing: Goblins for GNU Guix
> Christopher Webber, Ludovic Courtès and Pjotr Prins
> 
> Guix Data Service revision processing instrumentation and performance
> Christopher Baines
> 
> Guile based build-tool
> Pjotr Prins
> 
> GNU Guix system monitor
> Pjotr Prins
> 
> Booting via network
> Danny Milosavljevic
> 
> Syntax and semantics of systemd units in the Shepherd
> Ludovic Courtès (civodul)
> 
> GNUnet integration
> no mentor available
> 
> Adding modules in support of continuous integration to cuirass
> Ludovic Courtès (civodul)
> 
> Continue rewrite build daemon in Guile Scheme
> Ludovic Courtès (civodul)
> 
> I myself am available to co-mentor, and also to be the formal mentor
> in case someone does not feel like doing the official dance with
> Google. Currently I can commit to devoting two hours a week to this.
> 
> Regards,
> g_bor
> 

Re: GSoC 2024

[Ekaitz Zarraga]

I have an idea too, but I don't have the knowledge to mentor it.

guix pack support for AppImages.

On 2024-03-04 20:10, Gábor Boskovits wrote:

Hello guix,

I coordinated with the GNU org admins, and we can still do this round,
but we have to go fast to make this happen. I have already taken the
initiative to try to get an ideas page up, now I would like to confirm
if the mentors from last year are still available, and that the ideas
are still valid.

Hereby I quickly collected the projects with the respective mentors,
please pm me your availability:

Decentralized substitute distribution
pukkamustard (pukkamustard [at] posteo [dot] net)
attila.lendvai (ethswarm.org, scheme)

Robustify long-term support for Reproducible Research
Simon Tournier (zimoun)

Develop a Web interface to configure Guix System
Ludovic Courtès (civodul)

Trusted computing: Goblins for GNU Guix
Christopher Webber, Ludovic Courtès and Pjotr Prins

Guix Data Service revision processing instrumentation and performance
Christopher Baines

Guile based build-tool
Pjotr Prins

GNU Guix system monitor
Pjotr Prins

Booting via network
Danny Milosavljevic

Syntax and semantics of systemd units in the Shepherd
Ludovic Courtès (civodul)

GNUnet integration
no mentor available

Adding modules in support of continuous integration to cuirass
Ludovic Courtès (civodul)

Continue rewrite build daemon in Guile Scheme
Ludovic Courtès (civodul)

I myself am available to co-mentor, and also to be the formal mentor
in case someone does not feel like doing the official dance with
Google. Currently I can commit to devoting two hours a week to this.

Regards,
g_bor

Re: GSoC 2024

[Ricardo Wurmus]

Gábor Boskovits  writes:

> Continue rewrite build daemon in Guile Scheme
> Ludovic Courtès (civodul)

Since this is an ongoing NLnet project I think we should not offer it as
a GSoC project this time.

-- 
Ricardo

Re: GSoC 2024

[Pjotr Prins]

Hi Gábor,

https://libreplanet.org/wiki/Group:Guix/GSoC-2024

and added a project. Feel free to modify.

For the others: you can visit the older ideas pages - there is also
commented out projects in some of them. If you are looking for yours.

We still have a few days, but best to get it done this week. Sounds
like we should propose a goblins project!

PJ.

On Mon, Mar 04, 2024 at 07:10:34PM +, Gábor Boskovits wrote:
> Hello guix,
> 
> I coordinated with the GNU org admins, and we can still do this round,
> but we have to go fast to make this happen. I have already taken the
> initiative to try to get an ideas page up, now I would like to confirm
> if the mentors from last year are still available, and that the ideas
> are still valid.
> 
> Hereby I quickly collected the projects with the respective mentors,
> please pm me your availability:
> 
> Decentralized substitute distribution
> pukkamustard (pukkamustard [at] posteo [dot] net)
> attila.lendvai (ethswarm.org, scheme)
> 
> Robustify long-term support for Reproducible Research
> Simon Tournier (zimoun)
> 
> Develop a Web interface to configure Guix System
> Ludovic Courtès (civodul)
> 
> Trusted computing: Goblins for GNU Guix
> Christopher Webber, Ludovic Courtès and Pjotr Prins
> 
> Guix Data Service revision processing instrumentation and performance
> Christopher Baines
> 
> Guile based build-tool
> Pjotr Prins
> 
> GNU Guix system monitor
> Pjotr Prins
> 
> Booting via network
> Danny Milosavljevic
> 
> Syntax and semantics of systemd units in the Shepherd
> Ludovic Courtès (civodul)
> 
> GNUnet integration
> no mentor available
> 
> Adding modules in support of continuous integration to cuirass
> Ludovic Courtès (civodul)
> 
> Continue rewrite build daemon in Guile Scheme
> Ludovic Courtès (civodul)
> 
> I myself am available to co-mentor, and also to be the formal mentor
> in case someone does not feel like doing the official dance with
> Google. Currently I can commit to devoting two hours a week to this.
> 
> Regards,
> g_bor
> 

-- 

Re: GSoC news

[Maxim Cournoyer]

Hi Sarthak,

Sarthak Shah  writes:

> Hello Guix,
>
> Thanks for having me onboard!
> I hope that my project will prove useful to Guix.
>
> My project is about implementing Parameterized Packages, which will add
> optional Gentoo USE flag-like functionality to Guix packages. This will not
> only help reduce package size but also provide users with greater choice
> with how they'd like to use their packages. I believe that this should also
> help somewhat relieve the issue of the denseness of guix's dependency
> graphs, which was recently being discussed here on the mailing list.
>
> Due to the nature of this project, it will involve taking a lot of inputs
> from the Guix community, and thus you can expect a regular thread or two
> with updates/feature discussions on this topic in the mailing list.
>
> Once again, thank you for this wonderful opportunity!

Welcome!  One of the things I enjoy most about the Guix community is how
lively it is.  I'd recommend the #guix IRC channel on the libera.chat
server if you haven't already, where more than 400 people regularly
discuss and try to help each others.

I wish you the best of luck in your project, and most importantly, to
have fun doing it!

-- 
Thanks,
Maxim

Re: GSoC news

[Ludovic Courtès]

Hi Sarthak,

Sarthak Shah  skribis:

> Thanks for having me onboard!
> I hope that my project will prove useful to Guix.

Welcome on board!!  Really happy to have you here.

> My project is about implementing Parameterized Packages, which will add
> optional Gentoo USE flag-like functionality to Guix packages. This will not
> only help reduce package size but also provide users with greater choice
> with how they'd like to use their packages. I believe that this should also
> help somewhat relieve the issue of the denseness of guix's dependency
> graphs, which was recently being discussed here on the mailing list.
>
> Due to the nature of this project, it will involve taking a lot of inputs
> from the Guix community, and thus you can expect a regular thread or two
> with updates/feature discussions on this topic in the mailing list.

Indeed!  Like you write, there’s a lot of potential with parameterized
packages, and I’m confident you can make great strides in this area.
Oh, and ‘--with-configure-flags’ is already a success among HPC people
I’ve talked to!  :-)

Cheers,
Ludo’.

Re: GSoC news

[Simon Tournier]

Hi Sarthak,

On ven., 05 mai 2023 at 03:35, Sarthak Shah  wrote:

> Thanks for having me onboard!

Welcome!

> I hope that my project will prove useful to Guix.

Cool!


> Due to the nature of this project, it will involve taking a lot of inputs
> from the Guix community, and thus you can expect a regular thread or two
> with updates/feature discussions on this topic in the mailing list.

Well, I do not know if pointing past discussions is helpful.  Bah, just
in case. ;-)  In addition to [1],

Parameterized packages
Tue, 14 May 2019 13:54:18 +0200
Ludovic Courtès 
id:87woitz1xx@gnu.org
https://yhetil.org/guix/87woitz1xx@gnu.org/#r

1: https://lists.gnu.org/archive/html/guix-devel/2020-11/msg00312.html


See you around.

Cheers,
simon

Re: GSoC news

[Sarthak Shah]

Hello Guix,

Thanks for having me onboard!
I hope that my project will prove useful to Guix.

My project is about implementing Parameterized Packages, which will add
optional Gentoo USE flag-like functionality to Guix packages. This will not
only help reduce package size but also provide users with greater choice
with how they'd like to use their packages. I believe that this should also
help somewhat relieve the issue of the denseness of guix's dependency
graphs, which was recently being discussed here on the mailing list.

Due to the nature of this project, it will involve taking a lot of inputs
from the Guix community, and thus you can expect a regular thread or two
with updates/feature discussions on this topic in the mailing list.

Once again, thank you for this wonderful opportunity!

Sincerely,
Sarthak (cel7t)

On Fri, 5 May, 2023, 01:29 Pjotr Prins,  wrote:

> Welcome Sarthak!
>
> On Thu, May 04, 2023 at 09:22:47PM +0200, Gábor Boskovits wrote:
> >Hello guix,
> >We have some news about GSoC, the community is accepting a new intern,
> >Sarthak. The agreed on internship project title is Parameterized
> >Packages for GNU Guix. Welcome on board, and we are looking forward to
> >working together.
> >As there was quite a lot of competition for places this year we only
> >got one internship slot assigned, but we are encouraging all
> >prospective interns to stay around, and try again in the next round.
> >The distributed substitutes proposals are still on the radar, and  it
> >would be nice to keep interested people around.
> >Regards,
> >g_bor
>
>

Re: GSoC news

[Pjotr Prins]

Welcome Sarthak!

On Thu, May 04, 2023 at 09:22:47PM +0200, Gábor Boskovits wrote:
>Hello guix,
>We have some news about GSoC, the community is accepting a new intern,
>Sarthak. The agreed on internship project title is Parameterized
>Packages for GNU Guix. Welcome on board, and we are looking forward to
>working together.
>As there was quite a lot of competition for places this year we only
>got one internship slot assigned, but we are encouraging all
>prospective interns to stay around, and try again in the next round.
>The distributed substitutes proposals are still on the radar, and  it
>would be nice to keep interested people around.
>Regards,
>g_bor

Re: [GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai]

> Haven't read the Swarm thing, going more off of the general vibe of
> these cryptocurrency related projects that keep popping up:
> Using some kind of (optional) web of trust for clients makes more sense
> to me than making people pay with cryptocurrencies.
> 
> I should be able to set up two computers on a LAN in the middle of
> nowhere without having to care about some blockchain's global
> consistency.


yes, but those are different tasks, solved by different tools.

Swarm (and IPFS, and their ilk) solve large-scale cooperation in storing 
content. and the larger the scale of cooperation, the larger the benefits 
(redundance, fault tolerance, speed due to automatic caching, etc).


> NDN can do this right and has been able to for a while.
> Personally, I would try that and other established non-ponzi
> technologies first for distributed substitutes.
> Not being "permissionless" should not be considered a must have.


no, it's not a must have. it's just one more storage backend for storing 
substitutes -- once an abstraction layer like ERIS is installed.

without using ERIS, i wouldn't advocate for adding Swarm as the sole p2p 
backend integrated into Guix. there are better candidates for such a 
hypothetical singular position, or even as the first backend to get integrated.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“The greatest crimes in the world are not committed by people breaking the 
rules. It's people who follow orders that drop bombs and massacre villages. As 
a precaution to ever committing major acts of evil it is our solemn duty never 
to do what we're told, this is the only way we can be sure.”
— Banksy, a graffiti artist from Bristol, 'Wall and Piece'

Re: [GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai]

> Yes, it is the task of P2P storage system. Is Guix one P2P storage
> solution? Or should Guix exploit already implemented P2P storage
> systems?


i automatically assumed the latter, because p2p storage is a non-trivial task 
that multiple teams are working to solve, and it's yet to be seen which one 
will work out in the long run.

BTW, due to this it's a good idea to have ERIS between Guix and the various 
storage backends.

a Guix-specific p2p storage solution would be less exposed to abuse initially, 
but probably not forever.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Heroes are heroes because they are heroic in behavior, not because they won or 
lost.”
— Nassim Taleb (1960–), 'Fooled by Randomness' (2004)

Re: [GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai]

thanks for the detailed elaboration Maxime!

prior to reading your email i was blind to the (rather obvious) fact that the 
current Guix servers are already run by someone (a peer), and they consume 
quite some resources, and it's currently financed through donations.

considering this, i now find it reasonable to have the resharing enabled by 
default. thanks for fixing this glitch in my model of reality, and sorry for 
being too dim here!


> Also, this crypto coin balance problem can be avoided by simply not
> basing your P2P system on money (crypto coins or otherwise); it's a
> problem that those systems invented for theirselves.


the root problem is efficient ways of locking out non-cooperating agents from 
the fruits of the cooperation. using a balance sheet, and monetary settlements 
above a certain threshold, is one attempt at solving this task. it's yet to be 
seen which solution will survive this evolutionary landscape.


> ... it appears that your view is that it's ok to spend resources of
> other people even without trying to reciprocate (), and that it is
> unreasonable to expect reciprocation by default?


no. i just lacked the necessary level of understanding of the terrain here.


> () I don't consider Swarm to be a P2P system -- Swarm by design and
> intentionally actively maintains a class distinction between customers
> (people paying for storage and uploading) and, let's say, entrepreneurs
> (people getting paid for storage and downloading). While sometimes a
> customer might also be an entrepreneur, by this inherent difference
> between customers and entrepreneurs in Swarm, by definition they aren't
> peers.


in the model proposed by Swarm every participant plays by the same rules. and 
on top of that, as long as someone's use of the shared resources is balanced 
with their contribution to the cooperation, then there are no monetary 
transactions involved. i don't see how this wouldn't qualify as p2p.

the only "class distinctions" i see here is the issuance of their crypto token, 
and the "unfair advantage" of the early investors and the founders (except the 
disadvantage of those who may end up losing their invested time/money if the 
project fails to deliver).


> That's ‘consent’ the same way that cookie banners without a "Reject"
> button () are consent. It's certainly ‘Informed’ and it's useful to
> warn people with low or expensive bandwidth to minimize the bandwidth
> limits in the GNUnet configuration, but to call it ‘consent’ is
> doublespeak. I would prefer to not have doublespeak and instead to be
> honest that it's a requirement for installing Guix instead of twisting
> things into ‘consent’.


i lost you here, and -- possibly due to that -- i find the doublespeak 
reference unfair.

consent means that i understand what's happening, and i agree to it, while i 
have the option to reject the situation/association without major harms to my 
interests.

if i'm aware that Guix will use my upstream (think of metered connections), and 
i install it anyway, and then i don't turn this feature off... then by those 
actions i implicitly consent to this happening.

it's somewhat tangential here, but the "reject cookies button" is not always a 
viable option. sometimes in life the only option besides agreeing to something 
offered is to "close the browser tab"; i.e. stop associating, which is not 
installing Guix in this context, which is not a major hindrance to anyone.. 
(although, this is arguable... :)

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“War is a racket. It always has been. It is possibly the oldest, easily the 
most profitable, surely the most vicious. It is the only one international in 
scope. It is the only one in which the profits are reckoned in dollars and the 
losses in lives.”
— Smedley Butler (1881–1940), 'War is a racket' (1935), US Marine major 
general (highest rank at that time)

Re: [GSoC 23] distributed substitutes, cost of storage

[Maxime Devos]

Op 04-04-2023 om 12:53 schreef Attila Lendvai:

Onderwerp:
Re: [GSoC 23] distributed substitutes, cost of storage
Van:
Attila Lendvai 
Datum:
04-04-2023 12:53

Aan:
Maxime Devos 
CC:
Vijaya Anand , pukkamustard 
, guix-devel@gnu.org




it's another question whether this mirroring should be enabled by default in 
the clients. probably it shouldn't,


It probably should -- if things aren't mirrored, then it's not p2p; you
would lose the main performance benefit of p2p systems.

More cynically, some p2p systems (e.g. GNUnet) have mechanisms to
disincentive freeloaders -- clients that aren't being peers will get
worse downloading speed.


any successful p2p solution must have an incentive system that makes attacks 
expensive (freeloading, DoS'ing, censorship, etc). arguably, the most important 
difference between the various solutions is what this incentive system looks 
like.

from a bird's eye view perspective, there are two fundamental architectures of 
p2p storage networks (that i know of):

  1) ipfs-like, or torrent-like, where the nodes register/publish what
 they have in their local store, and other nodes may request it
 from them

  2) swarm-like, where the nodes are responsible for storing whatever
 content "is" in their "neighborhood". (block hashes and node ids
 are in the same domain, so there's a distance metric between a
 block and a node). put another way: Swarm stores not only the
 metadata in the DHT, but also the data itself.

in 1) there's no need to pay for, and to upload content into the network. a 
node just registers as a source for whatever content it has locally, and then 
serves the incoming requests.

but if you have content that you want to make available in 2) then you need to 
make sure that this content gets to a set of distant nodes that will store it. 
this is very different from 1) from a game theoretic perspective, and can't be 
done without some form of payments/accounting.

in 1) it's simpler for a node to share: just give away your storage and 
bandwidth to the network.

in 2) it's more complicated, because if your node is requesting other nodes to 
do stuff, then you're spending a more complex set of resources than just your 
bandwidth, potentially including some crypto coin payments if the balance goes 
way off.


GNUnet is (1) but also more than that, because of the automatic pushing 
to other nodes.  To my understanding it's not (2), but at the same time 
your comment about (2) applies.


Also, this crypto coin balance problem can be avoided by simply not 
basing your P2P system on money (crypto coins or otherwise); it's a 
problem that those systems invented for theirselves.



but both cases are fundamentally the same: users are spending their resources, 
and i wouldn't expect that installing a linux distro will start spending my 
network bandwidth, or any other resource than my machine's local resources.


Network bandwidth (and storage) _is_ a local resource.

Also, how are you going to keep your distribution up to date or install 
new software without allowing your distribution to spend network 
bandwidth? -- For non-P2P systems, it is already the case that that 
network bandwidth is spent by the local machine, P2P systems just makes 
it more symmetrical and hence fairer.


More to the point, recalling that this is a reply to my statement that 
mirroring should be enabled by default:


>> it's another question whether this mirroring should be enabled by 
default in the clients. probably it shouldn't,

>
> It probably should -- if things aren't mirrored, then it's not p2p; you
> would lose the main performance benefit of p2p systems.
>
> More cynically, some p2p systems (e.g. GNUnet) have mechanisms to
> disincentive freeloaders -- clients that aren't being peers will get
> worse downloading speed.

... and noticing that you are making a distinction between the resources 
of the user and others:


‘users are spending _their_ sources, and i wouldn't expect that [...] 
will start spending _my_  network bandwith, [...], _my_ machine [...]’

(emphasis added)

... it appears that your view is that it's ok to spend resources of 
other people even without trying to reciprocate (*), and that it is 
unreasonable to expect reciprocation by default?


(*) I'm not claiming that not reciprocating is always bad -- it's a 
reasonable thing to not do when on a very limited plan.  Rather, the 
point is that reciprocating by default is reasonable and that in 
reasonable circumstances, not reciprocating is unreasonable.


I mean, given how you are a proponent of crypto, you appear to be a 
capitalist, so I'd think you are familiar with the idea that to use 
resources of other people, you need to compensate them (in money like 
with Swarm or in kind like with P2P systems (*)).


(*) I don't consider Swarm to be a P2P system -- Swarm _by design and 
intentionally_ actively maintains a class distinctio

Re: [GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai]

> > it's another question whether this mirroring should be enabled by default 
> > in the clients. probably it shouldn't,
>
>
> It probably should -- if things aren't mirrored, then it's not p2p; you
> would lose the main performance benefit of p2p systems.
>
> More cynically, some p2p systems (e.g. GNUnet) have mechanisms to
> disincentive freeloaders -- clients that aren't being peers will get
> worse downloading speed.


any successful p2p solution must have an incentive system that makes attacks 
expensive (freeloading, DoS'ing, censorship, etc). arguably, the most important 
difference between the various solutions is what this incentive system looks 
like.

from a bird's eye view perspective, there are two fundamental architectures of 
p2p storage networks (that i know of):

 1) ipfs-like, or torrent-like, where the nodes register/publish what
they have in their local store, and other nodes may request it
from them

 2) swarm-like, where the nodes are responsible for storing whatever
content "is" in their "neighborhood". (block hashes and node ids
are in the same domain, so there's a distance metric between a
block and a node). put another way: Swarm stores not only the
metadata in the DHT, but also the data itself.

in 1) there's no need to pay for, and to upload content into the network. a 
node just registers as a source for whatever content it has locally, and then 
serves the incoming requests.

but if you have content that you want to make available in 2) then you need to 
make sure that this content gets to a set of distant nodes that will store it. 
this is very different from 1) from a game theoretic perspective, and can't be 
done without some form of payments/accounting.

in 1) it's simpler for a node to share: just give away your storage and 
bandwidth to the network.

in 2) it's more complicated, because if your node is requesting other nodes to 
do stuff, then you're spending a more complex set of resources than just your 
bandwidth, potentially including some crypto coin payments if the balance goes 
way off.

but both cases are fundamentally the same: users are spending their resources, 
and i wouldn't expect that installing a linux distro will start spending my 
network bandwidth, or any other resource than my machine's local resources.

but this of course can change, too: maybe a future Guix release can advertise 
with big red letters on the download page that installing it will use your 
network bandwidth to serve other guix nodes, unless it is turned off. and then 
all is well WRT informed consent.

--
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“Historically, the most terrible things - war, genocide, and slavery - have 
resulted not from disobedience, but from obedience.”
— Howard Zinn (1922–2010)

Re: [GSoC 23] distributed substitutes, cost of storage

[Maxime Devos]

Op 25-03-2023 om 20:00 schreef Attila Lendvai:

welcome on board Anand!



In case a user requests for a substitute and there is a missing
block in the decoding process, a HTTP request for block would sent
to the substitute server and the server will encode the
corresponding block in real time and push it back into the
network. The block will be searched again and retrieved. > something to consider here: whose responsibility should it be that a 
block, that is missing from a p2p network, is (re-)uploaded there? the 
clients? or the current substitute server?


my gut instinct says that it's better if the clients do the (re-)upload of the 
blocks.

in this architecture the substitute server is just another storage mechanism 
along the other storage backends (although with a different reliability 
characteristics), and it's the clients that are doing the 
mirroring/spreading/distribution of the blocks among the various backends. the 
clients of course will/should keep the current substitute servers at the bottom 
of their list of backends in their configuration.

this way the load is distributed, and we don't need to add (too much) extra 
complexity to the substitute server codebase, and the actors are less tightly 
coupled.

it's another question whether this mirroring should be enabled by default in 
the clients. probably it shouldn't,


It probably should -- if things aren't mirrored, then it's not p2p; you 
would lose the main performance benefit of p2p systems.


More cynically, some p2p systems (e.g. GNUnet) have mechanisms to 
disincentive freeloaders -- clients that aren't being peers will get 
worse downloading speed.



and the project infrastructure should be running clients where it is turned on. 
altruistic third parties could also enable this mirroring feature, and donate 
their bandwidth/resources.

there's an issue with this, though:

some p2p storage backends will require some form of payment/credentials to use 
their resources. arguably, all p2p storage networks that will survive into the 
future will have some mechanism to limit the infinite abuse of their resources. 
it is to be researched how these payment mechanisms work on the various p2p 
networks, and whether it is possible that the Guix project pays for the storage 
globally, and then the random clients will have the necessary credentials to 
(re-)upload the missing blocks.


GNUnet has a built-in mechanism for mirroring and for avoiding overuse 
of resources.  From what I recall of the documentation and the GNUnet 
papers:


* The more a peer A requests stuff of peer B,
  the more peer B dislikes peer A.
* Likewise, the more peer A fulfills requests of peer B,
  the more peer B likes peer A.
* Requests by liked peers are prioritized.

(If you squint, I suppose this could be considered a form of payment, 
but no literal currencies are involved, so no need for any financing.)


Mirroring:

* When putting a resource on the network, a few copies
  are stored in the network.  (I assume this discreases the dislike of
  the peer that received the copy by the peer that sent the copy, and
  increases the dislike by the peer that sent the copy by the peer that
  receives the copy.)

* The more popular a resource is, the more replicas are stored in
  the network (I don't recall the mechanism, but IIRC this is an
  automatic process).

* Peers set a quotum on how much bytes they are willing to store;
  when exceeded, they throw out old stuff and low-priority stuff.

(The only way to be 100% sure a resource remains on the network, is to 
have a local copy in the local peer, so you can't really reliably ‘save’ 
something on the network, but you can use it as a CDN to spread the load 
and tolerate occasional downtime.)


Greetings,
Maxime


OpenPGP_0x49E3EE22191725EE.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature

Re: [GSoC 23] distributed substitutes, cost of storage

[pukkamustard]

Vijaya Anand  writes:

> Sorry for the late reply.
> So in the case we are running swarm nodes that serves the network and hence 
> help fund the substitute server, we can also use these
> to also upload eris encoded substitute blocks onto the network am I right? 
> The total cost will thus be cost to run the swarm nodes +
> storage cost of substitute blocks in the network + cost to run substitute 
> servers - money earned by running swarm nodes. But when
> we don't run swarm nodes which run to serve the network, the total cost is 
> not really affect right as the cost to run swarm nodes will
> be lessened and no money will be earned. 
> So in the context of fallback mechanism the user client can send request to 
> the substitute server for the missing block and the
> substitute server will serve the eris encoded data block back to the user 
> (using HTTP). The responsibility of uploading these missing
> blocks back to the network is of the third party nodes (which are running to 
> serve the desired content to the network). But how do we
> send the message to the node to report the missing block on the network? Can 
> it be done by the user itself?

Some remarks:

- Maybe we don't need to do too much of an economic cost estimation, the
  current p2p networks (blocks over HTTP and IPFS) as well as ones that
  will quite possibly work soonish (GNUnet) do not incur any additional
  monetary storage cost. I think we should focus on resource usage
  (memory, disk, network) of users, servers and caching peers instead.

- One fallback strategy we absolutely need is to use get a substitute
  using the existing mechanism (substitute is a single nar file that is
  retrieved over HTTP without ERIS or any other p2p stuff).

- I like the principle "Think globally, act locally". Maybe users
  downloading substitutes who want to improve access to substitutes over
  p2p should only try to do so withing the scope of what they can do
  locally. I.e. by making the blocks available on the p2p network from
  the local machine. For IPFS and GNUnet this works very well and
  elimintates necessity of more RPC endpoints.

> "i can dream about a future where there's a social network that is based on 
> digital signatures and encryption, and my Guix client
> authorizes compiled binaries based on some weighted transitive closure of 
> signatures of my trusted peers"Interesting! In the case
> of accessing Guix substitutes from p2p network, we ensure authorization by 
> Guix team by making sure the urn of the substitute is the
> urn mentioned in the narinfo (which we get from a trusted source like the 
> substitute server). So in the case of accessing some random
> compiled binary from the network, we just need to verify the authority of the 
> document providing the urn of the content?

This is a very nice vision, that I share! However, maybe we should keep
it out of scope from the GSoC project and rely on the existing signature
mechanism for authenticity.

A web-of-trust like system for substitute system would be an excellent
and very interesting follow-up project.

-pukkamustard

> On Mon, 27 Mar 2023 at 2:49 AM Attila Lendvai  wrote:
>
>  > Also I didn't really think about the point about having to pay for
>  > the p2p services at some point of time.
>
>  a quick note here: i forgot to mention that e.g. the Swarm Foundation has 
> programs for supporting opensource projects. so,
>  chances are high that the storage needs for Guix would be paid for by the 
> foundation.
>
>  > In this case we will have to pay for the storage of substitutes both
>  > on the p2p storage backend as well as for storage in the substitute
>  > server am I right?
>
>  well, the substitute servers are currently already operated (and paid for) 
> by the Guix team. i don't think that p2p storage solutions
>  have reached a point of maturity that we could rely on them alone. there 
> should definitely be some time where both
>  infrastructures are running in parallel. somewhere down the road a choice 
> could be made to stop running the current substitute
>  servers, but we are far away from that.
>
>  also, running swarm nodes that serve the network can earn money. so, the 
> cost of running enough swarm nodes to pay for the
>  storage needs of Guix on the swarm network should be in the same ballpark of 
> the costs of running the current substitute servers.
>  the storage price will be market based (this part is just being rolled out 
> on the live network), so it's reasonable to expect that
>  people will fire up nodes if the storage price go well above the VPS costs.
>
>  and not all p2p storage networks are made equal. e.g. IPFS is only a 
> registry of who is serving what. if you want to keep your data
>  alive on IPFS, then you need to run some nodes and make sure that they are 
> serving the content that you care about... and bear
>  the costs of running these nodes. i.e. the DoS attack surface of IPFS is 
> much smaller. (IPFS stores only the 

Re: [GSoC 23] distributed substitutes, cost of storage

[pukkamustard]

Andreas Enge  writes:

> Hello,
>
> Am Wed, Mar 29, 2023 at 01:49:23AM +0530 schrieb Vijaya Anand:
>> In the case of accessing Guix substitutes from p2p
>> network, we ensure authorization by Guix team by making sure the urn of the
>> substitute is the urn mentioned in the narinfo
>
> no, currently substitutes are authenticated by a digital signature with one
> of the substitute servers (the user has control over which signing keys are
> accepted, see /etc/guix/acl). It happens after the download.
>

Slight ellaboration:

Currently the official Guix substitute servers provide a signed Narinfo
that contains the SHA256 sum of the substitute. The SHA256 sum of a
downloaded substitute is checked to match what is in the signed
Narinfo.

With the ERIS patches (https://issues.guix.gnu.org/52555) the signed
Narinfo also contains the ERIS URN. When getting a substitute this
signed ERIS URN is used. Decoding content from an ERIS URN guarantees
integrity, thus we also have authenticity.

Nevertheless, we still compute the SHA256 sum and check it. This is not
really necessary for ensuring authenticity but, imho, good practice for
now to be really sure we only use authenticated substitutes. Especially
when developing transparent fallback mechanisms that might go back to
just downloading the entire substitute from HTTP.

-pukkamustard

Re: [GSoC 23] distributed substitutes, cost of storage

[Andreas Enge]

Hello,

Am Wed, Mar 29, 2023 at 01:49:23AM +0530 schrieb Vijaya Anand:
> In the case of accessing Guix substitutes from p2p
> network, we ensure authorization by Guix team by making sure the urn of the
> substitute is the urn mentioned in the narinfo

no, currently substitutes are authenticated by a digital signature with one
of the substitute servers (the user has control over which signing keys are
accepted, see /etc/guix/acl). It happens after the download.

And see
   
https://guix.gnu.org/en/manual/devel/en/guix.html#Substitute-Server-Authorization
 .

Andreas

Re: [GSoC 23] distributed substitutes, cost of storage

[Vijaya Anand]

Hi,

Sorry for the late reply.
So in the case we are running swarm nodes that serves the network and hence
help fund the substitute server, we can also use these to also upload eris
encoded substitute blocks onto the network am I right? The total cost will
thus be cost to run the swarm nodes + storage cost of substitute blocks in
the network + cost to run substitute servers - money earned by running
swarm nodes. But when we don't run swarm nodes which run to serve the
network, the total cost is not really affect right as the cost to run swarm
nodes will be lessened and no money will be earned.
So in the context of fallback mechanism the user client can send request to
the substitute server for the missing block and the substitute server will
serve the eris encoded data block back to the user (using HTTP). The
responsibility of uploading these missing blocks back to the network is of
the third party nodes (which are running to serve the desired content to
the network). But how do we send the message to the node to report the
missing block on the network? Can it be done by the user itself?

"i can dream about a future where there's a social network that is based on
digital signatures and encryption, and my Guix client authorizes compiled
binaries based on some weighted transitive closure of signatures of my
trusted peers"Interesting! In the case of accessing Guix substitutes
from p2p network, we ensure authorization by Guix team by making sure the
urn of the substitute is the urn mentioned in the narinfo (which we get
from a trusted source like the substitute server). So in the case of
accessing some random compiled binary from the network, we just need to
verify the authority of the document providing the urn of the content?

"i value consensual relationships, and that implies that the other party is
well informed. and clogging someone's network bandwidth is not an expected
behavior from installing a linux distribution."
I agree with this point and also since there are already specialised nodes
doing the work of uploading blocks onto the network I guess for now it is
better to assign the task to them itself. Also I will give a read about how
network's charge for uploading data onto them. As you had mentioned before
if the payment is associated with a block id, then maybe we could have any
random client upload data onto the network (if of course we are able to
ship in with Guix itself, the configuration to upload data onto the said
network).

Vijaya Anand




On Mon, 27 Mar 2023 at 2:49 AM Attila Lendvai  wrote:

> > Also I didn't really think about the point about having to pay for
> > the p2p services at some point of time.
>
>
> a quick note here: i forgot to mention that e.g. the Swarm Foundation has
> programs for supporting opensource projects. so, chances are high that the
> storage needs for Guix would be paid for by the foundation.
>
>
> > In this case we will have to pay for the storage of substitutes both
> > on the p2p storage backend as well as for storage in the substitute
> > server am I right?
>
>
> well, the substitute servers are currently already operated (and paid for)
> by the Guix team. i don't think that p2p storage solutions have reached a
> point of maturity that we could rely on them alone. there should definitely
> be some time where both infrastructures are running in parallel. somewhere
> down the road a choice could be made to stop running the current substitute
> servers, but we are far away from that.
>
> also, running swarm nodes that serve the network can earn money. so, the
> cost of running enough swarm nodes to pay for the storage needs of Guix on
> the swarm network should be in the same ballpark of the costs of running
> the current substitute servers. the storage price will be market based
> (this part is just being rolled out on the live network), so it's
> reasonable to expect that people will fire up nodes if the storage price go
> well above the VPS costs.
>
> and not all p2p storage networks are made equal. e.g. IPFS is only a
> registry of who is serving what. if you want to keep your data alive on
> IPFS, then you need to run some nodes and make sure that they are serving
> the content that you care about... and bear the costs of running these
> nodes. i.e. the DoS attack surface of IPFS is much smaller. (IPFS stores
> only the metadata in the DHT (i.e. where is what), while Swarm stores there
> the data itself -- they are different architectures with different features)
>
> (i need to learn more about GNUnet)
>
>
> > So ideally we will want to eliminate the usage of these substitute
> > servers and shift totally to p2p services and in this case we will
> > have to shift the responsibility of re-uploading the blocks to the
> > clients itself.
>
>
> yep, that's my way of thinking, too.
>
> note that 'client' here has two meanings:
>
>  1) some part of the codebase
>
>  2) a program that is running on the computers of the Guix users
>
> i was using it in the 

Re: [GSoC 23] distributed substitutes, cost of storage

[Attila Lendvai]

> Also I didn't really think about the point about having to pay for
> the p2p services at some point of time.


a quick note here: i forgot to mention that e.g. the Swarm Foundation has 
programs for supporting opensource projects. so, chances are high that the 
storage needs for Guix would be paid for by the foundation.


> In this case we will have to pay for the storage of substitutes both
> on the p2p storage backend as well as for storage in the substitute
> server am I right?


well, the substitute servers are currently already operated (and paid for) by 
the Guix team. i don't think that p2p storage solutions have reached a point of 
maturity that we could rely on them alone. there should definitely be some time 
where both infrastructures are running in parallel. somewhere down the road a 
choice could be made to stop running the current substitute servers, but we are 
far away from that.

also, running swarm nodes that serve the network can earn money. so, the cost 
of running enough swarm nodes to pay for the storage needs of Guix on the swarm 
network should be in the same ballpark of the costs of running the current 
substitute servers. the storage price will be market based (this part is just 
being rolled out on the live network), so it's reasonable to expect that people 
will fire up nodes if the storage price go well above the VPS costs.

and not all p2p storage networks are made equal. e.g. IPFS is only a registry 
of who is serving what. if you want to keep your data alive on IPFS, then you 
need to run some nodes and make sure that they are serving the content that you 
care about... and bear the costs of running these nodes. i.e. the DoS attack 
surface of IPFS is much smaller. (IPFS stores only the metadata in the DHT 
(i.e. where is what), while Swarm stores there the data itself -- they are 
different architectures with different features)

(i need to learn more about GNUnet)


> So ideally we will want to eliminate the usage of these substitute
> servers and shift totally to p2p services and in this case we will
> have to shift the responsibility of re-uploading the blocks to the
> clients itself.


yep, that's my way of thinking, too.

note that 'client' here has two meanings:

 1) some part of the codebase

 2) a program that is running on the computers of the Guix users

i was using it in the first sense.

without a functional Web of Trust solution, the Guix team will have to run 
nodes that compile packages, sign them with their PGP keys, and make them 
available somewhere. currently it's published through a HTTP based service that 
we call 'substitute servers'. this GSoC project is about adding more storage 
backends.

but those backends don't solve the problem that the Guix users need to trust 
someone with a private key who compiles and signs packages, regadless of the 
transport mechanism that gets the packages to the clients.

i can dream about a future where there's a social network that is based on 
digital signatures and encryption, and my Guix client authorizes compiled 
binaries based on some weighted transitive closure of signatures of my trusted 
peers... but we are not there yet. for now it's either trusting the Guix team's 
signature, or setting up your own substitute servers and build workers (or 
trusting someone else, but i'm not aware of any third party offering 
substitutes).


> Also if we dont keep the re-uploading blocks option as default for
> the users, won't users usually not choose to enable it? Maybe we can
> keep it on as default and resource conscious users can choose to
> turn it off? Please let me know your thoughts on these points and I
> will change the implementation point of my proposal accordingly.


first, it's a philosophical question: i value consensual relationships, and 
that implies that the other party is well informed. and clogging someone's 
network bandwidth is not an expected behavior from installing a linux 
distribution.

there's also a technical issue: these p2p backends will need to be configured. 
i have my doubts that we could ship a default config with Guix with which these 
p2p backends could just work out of the box, but... let's hope that i'm wrong!

and then there's the issue of payments: it's not obvious that a random client 
can just upload binaries into a p2p storage network. on some p2p networks 
someone needs to pay for that, and i don't yet understand well enough how the 
data is authorized through payments (they are called postage stamps in swarm).

a good, high level comparison of p2p storage solutions would be useful.

-- 
• attila lendvai
• PGP: 963F 5D5F 45C7 DFCD 0A39
--
“There is something in the human spirit that will survive and prevail, there is 
a tiny and brilliant light burning in the heart of man that will not go out no 
matter how dark the world becomes.”
— Leo Tolstoy (1828–1910)

Re: [GSoC 23] distributed substitutes, cost of storage

[Vijaya Anand]

Hi Attila

Thanks for the welcome!
I agree that the responsibility of re-uploading the blocks back to the
network should be with the clients rather than the substitute server. Also
I didn't really think about the point about having to pay for the p2p
services at some point of time. In this case we will have to pay for the
storage of substitutes both on the p2p storage backend as well as for
storage in the substitute server am I right? So ideally we will want to
eliminate the usage of these substitute servers and shift totally to p2p
services and in this case we will have to shift the responsibility of
re-uploading the blocks to the clients itself.
Also if we dont keep the re-uploading blocks option as default for the
users, won't users usually not choose to enable it? Maybe we can keep it on
as default and resource conscious users can choose to turn it off? Please
let me know your thoughts on these points and I will change the
implementation point of my proposal accordingly.

Thank you
Vijaya Anand

On Sun, 26 Mar 2023 at 00:30, Attila Lendvai  wrote:

> welcome on board Anand!
>
>
> > In case a user requests for a substitute and there is a missing
> > block in the decoding process, a HTTP request for block would sent
> > to the substitute server and the server will encode the
> > corresponding block in real time and push it back into the
> > network. The block will be searched again and retrieved.
>
>
> something to consider here: whose responsibility should it be that a
> block, that is missing from a p2p network, is (re-)uploaded there? the
> clients? or the current substitute server?
>
> my gut instinct says that it's better if the clients do the (re-)upload of
> the blocks.
>
> in this architecture the substitute server is just another storage
> mechanism along the other storage backends (although with a different
> reliability characteristics), and it's the clients that are doing the
> mirroring/spreading/distribution of the blocks among the various backends.
> the clients of course will/should keep the current substitute servers at
> the bottom of their list of backends in their configuration.
>
> this way the load is distributed, and we don't need to add (too much)
> extra complexity to the substitute server codebase, and the actors are less
> tightly coupled.
>
> it's another question whether this mirroring should be enabled by default
> in the clients. probably it shouldn't, and the project infrastructure
> should be running clients where it is turned on. altruistic third parties
> could also enable this mirroring feature, and donate their
> bandwidth/resources.
>
> there's an issue with this, though:
>
> some p2p storage backends will require some form of payment/credentials to
> use their resources. arguably, all p2p storage networks that will survive
> into the future will have some mechanism to limit the infinite abuse of
> their resources. it is to be researched how these payment mechanisms work
> on the various p2p networks, and whether it is possible that the Guix
> project pays for the storage globally, and then the random clients will
> have the necessary credentials to (re-)upload the missing blocks.
>
> this architecture shouldn't be impossible, because the content is
> authenticated by its hash, and if the payment/authorization mechanism is
> based on the hashes of the blocks (probably), then any client could
> (re-)upload a missing block that was already paid for.
>
> i'll look into this, especially in the context of Swarm.
>
> meta: i think such specific discussions should be kept off-list, but the
> financing of the storage fees is probably something that should be known
> about more widely.
>
> --
> • attila lendvai
> • PGP: 963F 5D5F 45C7 DFCD 0A39
> --
> Every lie is a debt to the truth.
>
>

Re: GSoC 2023

[Simon Tournier]

Hi,

On Wed, 08 Mar 2023 at 18:32, Simon Tournier  wrote:

> and probably more in the bug
> tracker. 

For instance, you might be interested by:

https://issues.guix.gnu.org/issue/43442#9
https://issues.guix.gnu.org/issue/43442#11

as discussed in the recent message [1].

1: https://lists.gnu.org/archive/html/guix-devel/2023-03/msg00175.html

Feel free to ask questions in guix-devel mailing list or IRC libera.chat
#guix or #guix-hpc.

Cheers,
simon

Re: GSoC 2023

[Simon Tournier]

Hi,

On Mon, 06 Mar 2023 at 23:44, Karim Taha  wrote:

> Hello all, I'm Karim Taha, a senior computer engineering student at
> Cairo University. I'm interested in the Guix project list for the 2023 GSoC
> program. I would like to work on ' Robustify long-term support for
> Reproducible Research' project', I can write c/c++, python, haskell, elisp
> and scheme. My main OS is Arch linux. I would like to know if there are any
> required skills. I hope you all have a nice day.

Thanks for your interest.

If you can become familiar with the details, I can co-mentor it—which
means we’d need to find a co-mentor.  :-)

I would suggest “poking around” generally speaking, which means first
installing Guix.  A good first contribution is often a package: one that
you’re missing, that needs an update, etc.

Then I’d encourage you to look more closely at the area you’re interest
in.  Well, from my point of view, a good way for diving into is by
investigating #51726 [1] #48540 [2] and probably more in the bug
tracker. 

1: http://issues.guix.gnu.org/issue/51726
2: http://issues.guix.gnu.org/issue/48540


Feel free to ask questions in guix-devel mailing list or IRC libera.chat
#guix or #guix-hpc.

Cheers,
simon

Re: [GSOC 2020] network-boot-service

[Brice Waegeneire]

Hello Vagrant,

On 2020-07-02 16:34, Vagrant Cascadian wrote:

On 2020-07-02, Brice Waegeneire wrote:

To support the widest hardware and boot options possible I went with
iPXE
as a chainloader. Meaning that any machine doing a PXE boot (or with
builtin iPXE with restricted feature set) will load the iPXE 
bootloader

first, which will then properly load the initrd.


iPXE is really cool! The main downside is I don't think the support for
non-x86 architectures is there, but would be happy to find out
otherwise.



It looks like it does support arm 32 and 64 bits EFI:
https://ipxe.org/appnote/buildtargets. Anyway dnsmasq can be configured 
to
provide PXE entries by on the client's architecture or feature it 
support.
For example using network-boot-service, a iPXE client without http 
support

will chainload Guix's iPXE which has http support.

Currently I'm working on the initrd part to add NFS mounting 
capability

to
it. At this point I'm blocked by building a lightweight staticly built
'nfs-utils' package to be included in the initrd. It's current total
size
219.2 MiB, I manage to reduced it to 162.2 MiB which is still one 
order

of
magnitude larger than my initrd at 19 MiB.

My issue building a static 'nfs-utils' is that it can't find
'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
getrpcbynumber are available”. This function should be provided by the
libc
or by libtirpc if it's not that first one. The problem is that
'libtirpc'
don't build it's own 'getrpcbynumber' because it find one in libc but
nfs-utils can't find it...


Have you looked into using klibc (or maybe busybox) instead? They might
be smaller and have enough capability to mount NFS.


Non I did not, thanks for the tip. Busybox is smaller that was I
expecting, 1.0 MiB by itself, but it doesn't seems it would follow the
Guix way of doing most of the work in Guile (and using boot-to-Guile™)
and it's unique binary forbid us to extract only the NFS part. As for
klibc, it seems to be what NixOS[1] is using and TIL that it also the
the original source of Arch's mkinitcpio-nfs-utils which is 
unmaintained.

So klibc may work here, I'll try packaging it.


Some other distros are using the kernel parameter 'nfsroot'[2], but we
probably don't want to use it since it can't be used together with an
initrd and it also mean we need built-in modules for NFS and network
card
driver in the kernel.


Debian at least implements support for the nfsroot kernel parameter by
emulating the behavior in the initrd, using kernel modules instead of
built-ins. Seems like Guix could do the same?


Do you happen to know which package implement it in Debian? Yes, there
will be a need to pass parameters to the initrd so we better use the
common syntax.


If you're trying to use user-space NFS, I suspect you'll run into a lot
of issues... Debian at least dropped support for parts of the 
user-space

NFS stack years ago as it had huge numbers of bugs and little activity
upstream.


What do you mean by “user-space NFS”, be it nfs-utils, busybox or klibc,
they all are user-space tools to used to mount NFS shares?
Do you have some reference about this, I would like to learn more about
it. It's true that I did not find anyone trying to build a static
nfs-utils...


Great to hear about progress on this work!


live well,
  vagrant


[1]: 
https://github.com/NixOS/nixpkgs/commit/d25c1a8fdc383b8997f6e7b4e1479875df1f06b2
[2]: 
https://github.com/NixOS/nixpkgs/blob/84cf00f98031e93f389f1eb93c4a7374a33cc0a9/pkgs/os-specific/linux/mkinitcpio-nfs-utils/default.nix


- Brice

Re: [GSOC 2020] network-boot-service

[Vagrant Cascadian]

On 2020-07-02, Brice Waegeneire wrote:
> To support the widest hardware and boot options possible I went with 
> iPXE
> as a chainloader. Meaning that any machine doing a PXE boot (or with
> builtin iPXE with restricted feature set) will load the iPXE bootloader
> first, which will then properly load the initrd.

iPXE is really cool! The main downside is I don't think the support for
non-x86 architectures is there, but would be happy to find out
otherwise.


> Currently I'm working on the initrd part to add NFS mounting capability 
> to
> it. At this point I'm blocked by building a lightweight staticly built
> 'nfs-utils' package to be included in the initrd. It's current total 
> size
> 219.2 MiB, I manage to reduced it to 162.2 MiB which is still one order 
> of
> magnitude larger than my initrd at 19 MiB.
>
> My issue building a static 'nfs-utils' is that it can't find
> 'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
> getrpcbynumber are available”. This function should be provided by the 
> libc
> or by libtirpc if it's not that first one. The problem is that 
> 'libtirpc'
> don't build it's own 'getrpcbynumber' because it find one in libc but
> nfs-utils can't find it...

Have you looked into using klibc (or maybe busybox) instead? They might
be smaller and have enough capability to mount NFS.


> Some other distros are using the kernel parameter 'nfsroot'[2], but we
> probably don't want to use it since it can't be used together with an
> initrd and it also mean we need built-in modules for NFS and network 
> card
> driver in the kernel.

Debian at least implements support for the nfsroot kernel parameter by
emulating the behavior in the initrd, using kernel modules instead of
built-ins. Seems like Guix could do the same?

If you're trying to use user-space NFS, I suspect you'll run into a lot
of issues... Debian at least dropped support for parts of the user-space
NFS stack years ago as it had huge numbers of bugs and little activity
upstream.


Great to hear about progress on this work!


live well,
  vagrant


signature.asc
Description: PGP signature

Re: [GSOC 2020] network-boot-service

[Danny Milosavljevic]

Hi Brice,

On Thu, 02 Jul 2020 10:11:28 +
Brice Waegeneire  wrote:
> My issue building a static 'nfs-utils' is that it can't find
> 'getrpcbynumber{,_r}' “configure: error: Neither getrpcbynumber_r nor
> getrpcbynumber are available”. This function should be provided by the 
> libc
> or by libtirpc if it's not that first one. The problem is that 
> 'libtirpc'
> don't build it's own 'getrpcbynumber' because it find one in libc but
> nfs-utils can't find it...

Since glibc:static doesn't seem to contain it
(/gnu/store/rj4il3jjqg23cm3a0h17yzq3b5wa8llm-glibc-2.31-static/lib$
 objdump -t *.a |grep -i getrpcby returns nothing), we could make our
own variant of libtirpc that does enable their version.

> You can find the part of my work which is committed in the
> 'wip-network-boot' at https://git.sr.ht/~bricewge/guix.

Thanks!

I've checked it out, looks good so far!


pgpjOQbXaIJzM.pgp
Description: OpenPGP digital signature

Re: [GSoC 2020] [Proposal draft #1] Clojars importer for Guix

[Pjotr Prins]

Hi all,

There are some project proposals on GNU Guix that have no mentors
assigned including Leandro's. Please announce so now on GNU GSoC if you
are interested. 

Pj.

On Fri, Mar 27, 2020 at 04:47:41PM -0300, Leandro Doctors wrote:
> On Fri, 27 Mar 2020 at 01:09, Leandro Doctors  wrote:
> > Below, I share my second draft.
> 
> I've just realized I haven't used the template from the GNU GSoC
> Guidelines page[1]...
> Sorry for that. I will use that template for my next draft.
> 
> Best,
> Leandro
> 
> [1]: GNU guidelines for Summer of Code projects:
> https://www.gnu.org/software/soc-projects/guidelines.html
> 

Re: [GSOC 2020] Booting via network

[Vincent Legoll]

Hello,

On 15/04/2020 20:57, Brice Waegeneire wrote:

On 2020-03-30 22:10, Vincent Legoll wrote:

that's a great project, I hope to be able to lend a hand,
here and there...


Looks like you already started by packaging iPXE. :)
Thanks!



That one was a "release early", but I can't promise you

a "relase often"... ;-)


--

Vincent Legoll

Re: iPXE network booting (was Re: [GSOC 2020] Booting via network)

[Brice Waegeneire]

Hello Giovanni,

On 2020-04-10 13:44, Giovanni Biscuolo wrote:


[...]

I never used iPXE but... please consider using iPXE (if possible) for
Guix network booting and consider that this feature is a prerequisite
for seamless remote desktop with Guix (using x2go or xrdp like the new
LTSP is doing [1]) in addition to "diskless fat clients"; a very cool
feature, I think :-D


Since the proposal has already been submitted I can't promise to add 
iPXE

support in the context of this GSoC, however I will keep it in mind when
implementing PXE booting. If I'm able to finish it early I'll look into
adding iPXE booting as a bonus.


In addition to LAN booting, iPXE supports booting from:

* a web server via HTTP/HTTPS
* an iSCSI SAN
* a Fibre Channel SAN via FCoE
* an AoE SAN
* a wireless network
* a wide-area network
* an Infiniband network

inlcuding "code signing" to verify the authenticity and integrity of
files downloaded by iPXE.

Users will have many interesting, configurable [2] and secure ways to
boot Guix with iPXE :-D (imagine booting from a remote host connected
via a wireguard network connection... could it be possible?!?)


Wow, that sounds cool. I would like to find out if it's possible too.



[...]

HTH! Thanks, Gio'


Yes it is helping, really, that was dense with information.

Cheers,
- Brice

Re: [GSOC 2020] Booting via network

[Brice Waegeneire]

Hello Vagrant,

On 2020-03-30 23:16, Vagrant Cascadian wrote:
I was just thinking Guix needed network boot support yesterday! Happy 
to

hear you're looking into it...

I've been the LTSP maintainer in Debian for almost 15 years, so have a
good amount of experience with network booting. LTSP was rewritten from
scratch last year and there might be elements of it useful to you:

  https://ltsp.org

None of it is scheme code, but there are possibly some useful ideas in
there you could make use of. One of the big changes is making extensive
use of iPXE, though that might need some further auditing to meet the
FSDG (Free Software Distribution Guidelines?) for inclusion into Guix.


I see the rewrite has been done during a GSoC too; it's interesting that 
it

has mostly been done in shell. It looks like a interesting project, I'll
keep reading...

- Brice

Re: [GSOC 2020] Booting via network

[Brice Waegeneire]

Hello Vincent,

On 2020-03-30 22:10, Vincent Legoll wrote:

Hello,

that's a great project, I hope to be able to lend a hand,
here and there...


Looks like you already started by packaging iPXE. :)
Thanks!

- Brice

Re: iPXE network booting (was Re: [GSOC 2020] Booting via network)

[Vincent Legoll]

Hello,


in order to start the discussion and the project, what

about the patch in [1] ?


It's marked RFC as there are a few debatable points.


And it can be enhanced (building the efi firmwares, etc.)


[1] http://debbugs.gnu.org/cgi/bugreport.cgi?bug=40579


--

Vincent Legoll

iPXE network booting (was Re: [GSOC 2020] Booting via network)

[Giovanni Biscuolo]

Hello Brice and Vagrant

Vagrant Cascadian  writes:

> On 2020-03-30, Brice Waegeneire wrote:
>> I know it's quite late to submit a GSOC proposal but here it's.
>> I would like to work on the project suggested by Danny to
>> add PXE support to Guix. Which has been requested several
>> times on IRC and in the ML. This would get us a step closer
>> to provisioning bare bone machines directly from Guix.

Great feature, I hope you are not too late

[...]

>   https://ltsp.org

Thanks Vagrant for your work with LTSP in Debian!!!

I'm an _enthusiastic_ user of LTSP (LTSP5 now, but soon I'll experiment
20.04) and I'll be very happy to test (and help as I can) develop this
Guix feature (network booting, I mean).

I never used iPXE but... please consider using iPXE (if possible) for
Guix network booting and consider that this feature is a prerequisite
for seamless remote desktop with Guix (using x2go or xrdp like the new
LTSP is doing [1]) in addition to "diskless fat clients"; a very cool
feature, I think :-D

In addition to LAN booting, iPXE supports booting from:

* a web server via HTTP/HTTPS
* an iSCSI SAN
* a Fibre Channel SAN via FCoE
* an AoE SAN
* a wireless network
* a wide-area network
* an Infiniband network

inlcuding "code signing" to verify the authenticity and integrity of
files downloaded by iPXE.

Users will have many interesting, configurable [2] and secure ways to
boot Guix with iPXE :-D (imagine booting from a remote host connected
via a wireguard network connection... could it be possible?!?)

> None of it is scheme code, but there are possibly some useful ideas in
> there you could make use of. One of the big changes is making extensive
> use of iPXE, though that might need some further auditing to meet the
> FSDG (Free Software Distribution Guidelines?) for inclusion into Guix.

Vagrant plz do you have some specific potential issue in mind?

iPXE AFAIU is completely free software https://ipxe.org/licensing , it
also contains a tool that produces a detailed license analysis for each
ROM file.

On Guix iPXE could be used in "chainloading mode" [3] if the network
card already have a PXE implementation or - for advenced users - could
replace the network card ROM [4]: Guix service configuration should then
allow disabling chainloading for advanced users.

iPXE is still not packaged for Guix but it should not be hard to package
since AFAIU it uses standard GNU build tools and deps are all already
packaged (not sure about mkisofs and syslinux):

https://ipxe.org/download:
--8<---cut here---start->8---

[...]

build it using:

  cd ipxe/src
  make

You will need to have at least the following packages installed in order to 
build iPXE:

gcc (version 3 or later)
binutils (version 2.18 or later)
make
perl
liblzma or xz header files
mtools
mkisofs (needed only for building .iso images)
syslinux (for isolinux, needed only for building .iso images)

[...]

--8<---cut here---end--->8---

Making a iPXE ISO image could be useful to boot from CD-ROM/USB on
machines lacking NIC supporting PXE (do they still exist?)


HTH! Thanks, Gio'




[1] https://github.com/ltsp/community/issues/4: «thin client support is
now reduced to "remote desktop with xfreerdp / x2go / VNC".»
Exept VNC (that I do not consider useful in this scenario), x2go and
xrdp are still not packaged in Guix but we can work it out

[2] https://ipxe.org/embed and https://ipxe.org/scripting (including
dynamic scripts)

[3] https://ipxe.org/download#chainloading_from_an_existing_pxe_rom

[4] https://ipxe.org/howto/romburning

-- 
Giovanni Biscuolo

Xelera IT Infrastructures


signature.asc
Description: PGP signature

Re: [GSoC 2020] [Final Proposal] Clojars importer for Guix

[Pierre Neidhardt]

Hi Leandro,

A quick message to let you know that I'm very excited by your project!
I've recently started developing with Clojure and I'd love to see a
Clojars importer become reality!

Good luck!

-- 
Pierre Neidhardt
https://ambrevar.xyz/


signature.asc
Description: PGP signature

Re: [GSoC 2020] [Final Proposal] Clojars importer for Guix

[Leandro Doctors]

On Tue, 31 Mar 2020 at 15:06, Leandro Doctors  wrote:
> Below, I share the final version I uploaded to the GSoC website.

Another bug: I didn't mention how I plan to publicly report my
progress over the course of my work. Besides the normal interaction
via this list, I will send (at least) three progress reports. One
progress report to this list (with a post on the Guix blog) after each
Evaluation. Additionally, I may also send irregular updates when I
achieve important milestones.

Sorry for this oversight,
Leandro

Re: [GSOC 2020] Final proposal

[Alberto EFG]

Ricardo Wurmus writes:

> Hi Alberto,
>
>> Hello, this is my final proposal for GSoC, there is still time to do
>> some changes in case anyone has feedback, otherwise this is it :)
>
> Thank you for this thorough proposal!
>
> For 8.5 it would be good to provide criteria that allow us to evaluate
> your progress.  Being able to look at the code is obviously a
> precondition, but what we really want to know is what features will have
> to have been completed (and to what degree) at the time of the
> evaluation.
>
> For some of the deliverables and goals it would be good to see a little
> more detail if possible.  You mention systemd.device and systemd.socket
> files, but I can’t imagine what your goals with regards to the Shepherd
> are — is it to implement a similar feature (what exactly would that
> be?), or support for reading these files…?

Thank you so much for all the feedback!! I add my proposal with the
suggested changes.

I tried to be more specific with the goals, and the milestones. I added
a section, so number 8.5 is now 8.6, however I tried to be more specific
in the timeline than in 8.5.

Ludovic told me to focus my main goal on improving Shepherd internals,
rather than the systemd unit files parsing as that would be the "icing on
the cake", I hope that it is reflected on my proposal.

There is still a really short amount of time for last minute changes,
but this will probably be it.


proposal-final.pdf
Description: Adobe PDF document

Re: [GSOC 2020] Booting via network

[Vagrant Cascadian]

On 2020-03-30, Brice Waegeneire wrote:
> I know it's quite late to submit a GSOC proposal but here it's.
> I would like to work on the project suggested by Danny to
> add PXE support to Guix. Which has been requested several
> times on IRC and in the ML. This would get us a step closer
> to provisioning bare bone machines directly from Guix.

I was just thinking Guix needed network boot support yesterday! Happy to
hear you're looking into it...

I've been the LTSP maintainer in Debian for almost 15 years, so have a
good amount of experience with network booting. LTSP was rewritten from
scratch last year and there might be elements of it useful to you:

  https://ltsp.org

None of it is scheme code, but there are possibly some useful ideas in
there you could make use of. One of the big changes is making extensive
use of iPXE, though that might need some further auditing to meet the
FSDG (Free Software Distribution Guidelines?) for inclusion into Guix.


live well,
  vagrant


signature.asc
Description: PGP signature

Re: [GSOC 2020] Booting via network

[Vincent Legoll]

Hello,

that's a great project, I hope to be able to lend a hand,
here and there...

-- 
Vincent Legoll

Re: [GSOC 2020] Final proposal

[Ricardo Wurmus]

Hi Alberto,

> Hello, this is my final proposal for GSoC, there is still time to do
> some changes in case anyone has feedback, otherwise this is it :)

Thank you for this thorough proposal!

For 8.5 it would be good to provide criteria that allow us to evaluate
your progress.  Being able to look at the code is obviously a
precondition, but what we really want to know is what features will have
to have been completed (and to what degree) at the time of the
evaluation.

For some of the deliverables and goals it would be good to see a little
more detail if possible.  You mention systemd.device and systemd.socket
files, but I can’t imagine what your goals with regards to the Shepherd
are — is it to implement a similar feature (what exactly would that
be?), or support for reading these files…?

--
Ricardo

Re: [GSoC 2020] [Proposal draft #1] Clojars importer for Guix

[Leandro Doctors]

On Fri, 27 Mar 2020 at 01:09, Leandro Doctors  wrote:
> Below, I share my second draft.

I've just realized I haven't used the template from the GNU GSoC
Guidelines page[1]...
Sorry for that. I will use that template for my next draft.

Best,
Leandro

[1]: GNU guidelines for Summer of Code projects:
https://www.gnu.org/software/soc-projects/guidelines.html

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Leandro Doctors]

On Mon, 23 Mar 2020 at 14:00, Julien Lepiller  wrote:
> Le 23 mars 2020 12:06:12 GMT-04:00, Leandro Doctors  a 
> écrit :
> >Even though I don't have data to back up this claim, in the
> >Guix-Jupyter Scicloj video session [1] they mention that, in the case
> >of Clojure, most packages usually include sources.
> I'll check, thanks :)

If you only are interested in the Clojure-related comments, only watch
the last 25' or so from the video.


> >I guess this means I should focus on Clojars?
> From what you say, yes.

> Does it sound doable?

I think it does. It will be a Clojars importer, then :-)
We'll see how far can I take the idea...


Best,
Leandro

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Julien Lepiller]

Le 23 mars 2020 12:06:12 GMT-04:00, Leandro Doctors  a 
écrit :
>On Mon, 23 Mar 2020 at 12:39, Julien Lepiller 
>wrote:
>
>> Hi Leandro!
>
>Hi, Julien,
>
>Thanks a lot for your feedback! I add my comments below.
>
>
>> I'm glad you're interested in JVM languages.
>
>In fact, I am only interested in Clojure... However, given that
>Clojure is hosted on the JVM, it seems very difficult to me not to
>think in JVM terms when describing a "Clojure importer". This is the
>main reason behind the proposal name change from my previous emails.
>
>
>> I cannot find a way to I am currently trying to build a proper
>maven-build-system (it's a matter of days now).
>
>Again, I am only speaking in JVM terms because of Clojure's hosted
>nature.

Actually, I'm interested in android apps, but maven ended up being a 
requirement ^^"

>
>
>> Some of the issues we have with JVM languages and that you should
>take into account  for your proposal:
>[...]
>
>Thanks for this!
>
>
>> - maven allows programmers to upload a "source" jar, but it is meant
>for debugging, contains generated sources and might be incomplete.
>
>Even though I don't have data to back up this claim, in the
>Guix-Jupyter Scicloj video session [1] they mention that, in the case
>of Clojure, most packages usually include sources.
>
>[1] Guix-Jupyter Scicloj video session from January 9th, 2020.
>https://scicloj.github.io/posts/2020-03-07-guix-jupyter/

I'll check, thanks :)

>
>
>> - maven doesn't link to source repositories
>
>You're right.
>
>However, Clojars (the primary hub for Clojure packages) does. Maybe
>implementing Clojars support could be a first step?
>(I still have to assess this, as there are many Clojure packages that
>are published primarily in Maven Central...)
>
>
>> - how to find a source repository from a groupId and an artifactId is
>not clear to me.
>
>Considering the previous points, this seems mostly a Java-specific
>problem?
>
>
>> Considering that, I don't know if your project is doable. At least,
>all other importers are able to give you a working source, but if it's
>> really impossible, having a list of dependencies would already be
>great :)
>
>I agree with you in that this does seem non-trivial...
>Perhaps, this could be a research-oriented project? I mean, to see
>whether it is feasible to (eventually) support the JVM, and the major
>milestone being a minimalistic POC implementation for Clojars support?
>
>
>> We have a lot of java packages, but none of them are usable with
>maven yet. I've worked on those that are directly required to run
>> maven and its common plugins already, and will send patches as soon
>as everything works. There will need to be a huge work on
>> these packages to rebuild them, probably with the maven-build-system.
>
>I guess this means I should focus on Clojars?

From what you say, yes. A clojars importer would be great. If/when we find 
other ways to be able to import other JVM packages, we'll be able to improve 
upon that work :). Maybe the importer could work in that way:

Import from Clojars, if it fails, import from central, without source 
information. Basically, try multiple repos, from the most specific (that have 
source info) to more generic (that at least haveqversion and dependency 
information).

If you leave some space to be able to plug in other sources (eg. when we have 
scala, for sbt repos), I think you'll have a substantial contribution :). Does 
it sound doable?

>
>
>Best,
>Leandro

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Leandro Doctors]

On Mon, 23 Mar 2020 at 12:39, Julien Lepiller  wrote:

> Hi Leandro!

Hi, Julien,

Thanks a lot for your feedback! I add my comments below.


> I'm glad you're interested in JVM languages.

In fact, I am only interested in Clojure... However, given that
Clojure is hosted on the JVM, it seems very difficult to me not to
think in JVM terms when describing a "Clojure importer". This is the
main reason behind the proposal name change from my previous emails.


> I cannot find a way to I am currently trying to build a proper 
> maven-build-system (it's a matter of days now).

Again, I am only speaking in JVM terms because of Clojure's hosted nature.


> Some of the issues we have with JVM languages and that you should take into 
> account  for your proposal:
[...]

Thanks for this!


> - maven allows programmers to upload a "source" jar, but it is meant for 
> debugging, contains generated sources and might be incomplete.

Even though I don't have data to back up this claim, in the
Guix-Jupyter Scicloj video session [1] they mention that, in the case
of Clojure, most packages usually include sources.

[1] Guix-Jupyter Scicloj video session from January 9th, 2020.
https://scicloj.github.io/posts/2020-03-07-guix-jupyter/


> - maven doesn't link to source repositories

You're right.

However, Clojars (the primary hub for Clojure packages) does. Maybe
implementing Clojars support could be a first step?
(I still have to assess this, as there are many Clojure packages that
are published primarily in Maven Central...)


> - how to find a source repository from a groupId and an artifactId is not 
> clear to me.

Considering the previous points, this seems mostly a Java-specific problem?


> Considering that, I don't know if your project is doable. At least, all other 
> importers are able to give you a working source, but if it's
> really impossible, having a list of dependencies would already be great :)

I agree with you in that this does seem non-trivial...
Perhaps, this could be a research-oriented project? I mean, to see
whether it is feasible to (eventually) support the JVM, and the major
milestone being a minimalistic POC implementation for Clojars support?


> We have a lot of java packages, but none of them are usable with maven yet. 
> I've worked on those that are directly required to run
> maven and its common plugins already, and will send patches as soon as 
> everything works. There will need to be a huge work on
> these packages to rebuild them, probably with the maven-build-system.

I guess this means I should focus on Clojars?


Best,
Leandro

Re: [GSoC 2020] [Proposal draft #0] Initial JVM support for Guix

[Julien Lepiller]

Le 23 mars 2020 11:21:06 GMT-04:00, Leandro Doctors  a 
écrit :
>On Fri, 20 Mar 2020 at 21:41, Leandro Doctors 
>wrote:
>>  On Tue, 3 Mar 2020 at 19:32, zimoun 
>wrote:
>> > Based on your interests (Clojure, Leiningen, etc.), you should
>> > consider something around a Clojure "importer".
>>
>> I am preparing my proposal. I will send it in the next few days.
>
>Hello, everybody!
>
>This is my first draft (at the bottom of this message). Please note
>that the document is in a very early stage, so at this point there are
>still many questions unanswered (to which I have added some notes
>below).
>
>Your feedback will be crucial in answering questions and evolving the
>attached draft into a full proposal.
>
>Best,
>Leandro
>
>
>** Notes **
>- Whereas I may switch to org-mode, I am currently using LyX for
>writing the proposal.
>- I thought plain text was the best for getting feedback. If you think
>another format is better (Markdown, LaTeX, PDF...) please let me know.
>Pandoc is my friend :-)
>- I watched the whole Guix-Jupyter Scicloj video session from last
>January 9th, 2020.
>https://scicloj.github.io/posts/2020-03-07-guix-jupyter/
>- From what I see, in Guix there are compilers and importers (I'm in
>the process of getting familiar to this terminology).
>- There is already a compiler for Clojure, and Clojure has been
>packaged into Guix. However, there is no importer for Clojure
>packages...
>- I agree with some comments from the talk: given that Clojure is a
>hosted language, merely importing "Clojure packages" is impossible. In
>this case, as Clojure is hosted in the JVM, we should aim to importing
>Maven (an eventually, also Clojars) packages. So, adding "Clojure
>support" would mean "adding JVM support".
>- I guess that supporting tools/deps.edn would enable supporting Maven
>dependencies?
>- Packaging clojupyter would be a potential task to consider during
>the first coding period (maybe even before?)
>
>
>
>
> Draft **
>
>
>Initial JVM support for Guix
>
>Leandro Doctors
>
>
>1 Overview
>
>Add support for importing JVM packages (jar files) into Guix.
>
>
>2 Status Quo
>
>Guix supports importing package metadata from multiple sources.
>Currently, these sources are as diverse as GNU packages,
>repositories such as the Python Package Index and the
>Comprehensive R Archive Network, and JSON files. However, there
>is another source not yet supported by Guix: JVM-based languages.
>Currently, Guix does not support importing from any JVM-based
>language, such as Java, Clojure. Considering Java is the most
>used programming language, this would be a very valuable addition
>for Guix.
>
>
>3 Status Desideravit
>
>
>3.1 Objectives
>
>1. Add a JVM importer.
>
>2. Also support Clojure jars.
>
>
>3.2 Benefits
>
>• Gain access to the JVM environment.
>
>
>4 Implementation Plan
>
>
>4.1 Stages & Deliverables
>
>
>
>4.2 Timeline & Milestones
>
>
>Note: I have considered 5-days weeks for all periods, so there
>can be slack time if needed.
>
>1. Student Application Period (March 16th - 31st) (2 weeks)
>
>  • Start flicking through Guix's code. [done]
>  • Set up a development environment. [done]
>  • Start learning the basics about Guix's internal processes
>(release management, developer interactions, codes of
>conduct...).
>  • Start reading Guix documentation. [in progress]
>  • Start exploring possible approaches to implement proposed
>features. [in progress]
>
>2. Application Review Period (March 31st - April 27th) (4 weeks)
>
>  • Open PRs with small code and/or documentation glitches
>discovered during the first step of this list.
>  • Finish reading introductory material.
>  • Start experimenting with possible approaches to implement
>proposed features.
>  • Finish learning the basics about Guix internal processes
>(release management, developer interactions, codes of
>conduct...).
>  • Continue hacking into Guix's codebase to get to know it
>better.
>  • Engage with the Community and develop possible features not
>initially considered.
>
>
>3. Student Projects Announced (April 27th) (1 day)
>
>4. Community Bonding (April 27th - May 18th) (3 weeks)
>
>  • Continue hacking into Guix's codebase to get to know it
>better.
>  • Finish experimenting with possible approaches found during
>the Application Review period with which to implement
>proposed features.
>  • Explore options to implement proposed features.
>  • Re-assessment of implementation difficulty proposed features.
>
>5. Coding #1 (May 18th - June 12th) (4 working weeks)
>
>  Implement Stage #1:
>
>  (a) Weeks 1 & 2: 
>i. 
>
>Milestones:
>i. M1.1: 
>
>  (b) Weeks 2 & 3
>
>  (c) Week 4
>
>6. Partial Evaluation #1 (June 15th - 19th) (1 buffer week)
>
>  (a) Week 5: (Buffer)
>
>Milestones:
>i. (finish unfinished milestones)
>
>7. Coding #2 (June 23rd - July 10th) (3 working weeks)
>
>  Implement Stage #2:
>
>  (a) Week 6
>
>  (b) Week 7
>
>  

"Technical preview" / "Beta" features in Guix? (it was Re: [GSoC 2020] Clojure importer for Guix?)

[Leandro Doctors]

On Sat, 21 Mar 2020 at 14:56, Pjotr Prins  wrote:
> On Fri, Mar 20, 2020 at 09:41:05PM -0300, Leandro Doctors wrote:
> > I am also checking Jelle Licht's 2016 GSoC project (the npm importer).
> > From what I understand, Jelle's code was never merged back into guix.
> > As I'm trying to learn from the past, could anyone please ellaborate
> > on why was that code never merged?
> Circular dependencies on a large scale. We don't really have an answer
> for that.

I see.

That being considered, why not enable that importer as "technical
preview" or "beta"? This would enable other people to realize the code
already exists, and enable potential contributors to eventually
improve the feature.

Think would be something similar to Debian's "llibgstreamer-plugins-bad1.0-0":

"GStreamer Bad Plug-ins is a set of plug-ins that aren't up to par compared
to the rest. They might be close to being good quality, but they're missing
something - be it a good code review, some documentation, a set of tests, a
real live maintainer, or some actual wide use. [...] The API is not
guaranteed to be stable."

https://packages.debian.org/sid/libgstreamer-plugins-bad1.0-0


Or is it just that I'm missing something from the documentation?

--Leandro

Re: [GSoC 2020] Clojure importer for Guix?

[Pjotr Prins]

On Fri, Mar 20, 2020 at 09:41:05PM -0300, Leandro Doctors wrote:
> I am also checking Jelle Licht's 2016 GSoC project (the npm importer).
> From what I understand, Jelle's code was never merged back into guix.
> As I'm trying to learn from the past, could anyone please ellaborate
> on why was that code never merged?

Circular dependencies on a large scale. We don't really have an answer
for that.

Pj.

Re: [GSoC 2020] Clojure importer for Guix?

[Ludovic Courtès]

Hi,

Leandro Doctors  skribis:

> I am preparing my proposal. I will send it in the next few days.

Nice!

> I am currently checking current guix importers, and their documentation.
> In particular, I have been reading the source code from the Python and
> Haskell importers.
>
> I am also checking Jelle Licht's 2016 GSoC project (the npm importer).
> From what I understand, Jelle's code was never merged back into guix.
> As I'm trying to learn from the past, could anyone please ellaborate
> on why was that code never merged?
>
> Regarding packaging clojupyter... I understand that I will need at
> least one full application to eventually test the importer. That being
> said... is there any particular value in packaging clojupyter? Could
> you please name a few "essential" Clojure projects you would like to
> see packaged for guix?

I think Clojupyter would be nice to have.  One could then run:

  guix environment --ad-hoc jupyter clojupyer -- jupyter notebook

to spawn a Clojure-enabled notebook.

Thanks,
Ludo’.

Re: [GSOC 2020] Introduction and asking for feedback

[Ludovic Courtès]

Hi Alberto,

Blackbeard  skribis:

> I want to apply to Google Summer of Code. The ideas I am most interested are
> a) for GNU Guix: 'Content-addressed protocol for substitutes' and b) for
> GNU Shepherd:  "Syntax and semantics of systemd units in the Shepherd",
> because I have a feeling any of this two would improve the experience
> of most
> Guix users.
>
> However, I would like to ask for feedback in which might be a better option,
> I would like to chose a project that I can get help and the community is
> interested in.
>
> Any help with the project and proposal would be much appreciated.

I’ll answer because I’m listed as mentor at
, though honestly, I
think this would have to be someone else or you’ll get very frustrated.
:-)

For both projects, I encourage you to follow the links to get a better
understanding of what the project means.  For project (a), please also
familiarize yourself with substitutes, how they work, and perhaps have a
look at the (guix scripts substitute) and (guix scripts publish) modules
(they may be intimidating at first, so don’t expect to grasp every
detail from day 1, that’s fine!).

For project (b), I’d encourage you to take a look at the Shepherd: it’s
a small project and relatively easy to follow, I think.  If you’re not
familiar with systemd units, you can read

and find what mechanisms are missing from the Shepherd’s API at
.
Pick a specific feature (e.g., “socket activation”), and think about
what it would take to implement it in the Shepherd.

>From there, you can go ahead and ask specific questions about things you
don’t understand here or on IRC.

Happy hacking!  :-)

Ludo’.

Re: [GSOC 2020] Guix Deploy, round 2!

[Ludovic Courtès]

Hi!

Gábor Boskovits  skribis:

> Ludovic Courtès  ezt írta (időpont: 2020. márc. 8., Vas
> 23:10):

[...]

>> What I miss the most, especially on the build farm, is the ability to
>> tell ‘guix deploy’ which services to restart upon completion.
>> Currently, like ‘guix system reconfigure’, it conservatively doesn’t
>> restart any running services.  However, often, you’d like it to run
>> “herd restart X” upon completion.
>>
>> Another thing discussed at the Guix Days, but more relevant to more
>> advanced use cases, is the ability to define “roles”: often you’d rather
>> want to think in terms of the services machines offer and abstract over
>> the actual machines.
>>
>
> These are both great ideas. It would be also nice to access these in a
> single machine setup. I don't know where to implement this, it might make
> sense to add these to the common part of deploy and reconfigure. IIRC we
> also discussed the idea of a local deployer to be able handle the deploy
> node the same way as the rest.

If you talking about the first point above, I think it’s not that
difficult.  As I see it, we could provide a command-line option and/or a
 option (?) specifying which services should be restarted right
away.  Restarting can be implemented as in (guix scripts system
reconfigure).

> Regarding the roles thing it would be nice to get a discussion going
> regarding the interface, so that we have an idea how it should look
> like. Wdyt?

Yes, that would be nice!  There were good ideas discussed at the Guix
Days and knowledgeable people, but I can’t see the notes at

and I’m not sure who was involved.  There was probably Chris Marusich,
who else?  People, please step up!  :-)

Ludo’.

Re: [GSOC 2020] Guix Deploy, round 2!

[Gábor Boskovits]

Hello,

Ludovic Courtès  ezt írta (időpont: 2020. márc. 8., Vas
23:10):

> Hi Chris!
>
> Christopher Lemmer Webber  skribis:
>
> > Let me also put out a goal for the Guix community: I think we'll see
> > Guix Deploy as a success if a bunch of us can switch over to using it on
> > our own servers (that includes me).  To that end, I'd love to know, how
> > many people are doing so, or have attempted to do so?  What features
> > would you like/need?
>
> We’re using it for the build farm, and I’m also using it for a couple of
> single servers.  It’s great!  :-)
>
> What I miss the most, especially on the build farm, is the ability to
> tell ‘guix deploy’ which services to restart upon completion.
> Currently, like ‘guix system reconfigure’, it conservatively doesn’t
> restart any running services.  However, often, you’d like it to run
> “herd restart X” upon completion.
>
> Another thing discussed at the Guix Days, but more relevant to more
> advanced use cases, is the ability to define “roles”: often you’d rather
> want to think in terms of the services machines offer and abstract over
> the actual machines.
>

These are both great ideas. It would be also nice to access these in a
single machine setup. I don't know where to implement this, it might make
sense to add these to the common part of deploy and reconfigure. IIRC we
also discussed the idea of a local deployer to be able handle the deploy
node the same way as the rest. Regarding the roles thing it would be nice
to get a discussion going regarding the interface, so that we have an idea
how it should look like. Wdyt?

>
> Ludo’.
>
Best regards,
g_bor

>
>

Re: [GSOC 2020] Guix Deploy, round 2!

[Ludovic Courtès]

Hi Chris!

Christopher Lemmer Webber  skribis:

> Let me also put out a goal for the Guix community: I think we'll see
> Guix Deploy as a success if a bunch of us can switch over to using it on
> our own servers (that includes me).  To that end, I'd love to know, how
> many people are doing so, or have attempted to do so?  What features
> would you like/need?

We’re using it for the build farm, and I’m also using it for a couple of
single servers.  It’s great!  :-)

What I miss the most, especially on the build farm, is the ability to
tell ‘guix deploy’ which services to restart upon completion.
Currently, like ‘guix system reconfigure’, it conservatively doesn’t
restart any running services.  However, often, you’d like it to run
“herd restart X” upon completion.

Another thing discussed at the Guix Days, but more relevant to more
advanced use cases, is the ability to define “roles”: often you’d rather
want to think in terms of the services machines offer and abstract over
the actual machines.

Ludo’.

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[Gábor Boskovits]

Hello Leandro,

Leandro Doctors  ezt írta (időpont: 2020. márc. 3., Ke
23:45):

> On Tue, 3 Mar 2020 at 19:32, zimoun  wrote:
> > Based on your interests (Clojure, Leiningen, etc.), you should
> > consider something around a Clojure "importer". (Note that I am
> > ignorant about the Java ecosystem.)
>
> Thanks for the idea, Simon!
>
>
> > Currently, it is possible to import Python packages from PyPI, or
> > Haskell packages from Hackage, or etc. see [1] and something about
> > Clojure seems missing. I am quoting Ludo from [2]:
>
> Thanks for the pointers!
> I will check these projects and let you know.
>
>
> On another side, I will be offline for a fer days, so I expect to
> start working on my proposal draft from March 16th onwards.
> Would it be OK if I publish a project draft on
> GoogleDocs/GitHub/GitLab in (let's say) MarkDown by March 20th, so
> people can easily comment on it? I think getting feedback on a
> document via email may be quite difficult...
> Please, state your preference.
>

Another option would be to add it to the ideas page in a comment. When it
is ready, it would be easy to publish it by uncommenting. That way the
ideas would stay om the same place.

Best regards,
g_bor

>
> Best,
> Leandro
>
>

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[Leandro Doctors]

On Tue, 3 Mar 2020 at 19:32, zimoun  wrote:
> Based on your interests (Clojure, Leiningen, etc.), you should
> consider something around a Clojure "importer". (Note that I am
> ignorant about the Java ecosystem.)

Thanks for the idea, Simon!


> Currently, it is possible to import Python packages from PyPI, or
> Haskell packages from Hackage, or etc. see [1] and something about
> Clojure seems missing. I am quoting Ludo from [2]:

Thanks for the pointers!
I will check these projects and let you know.


On another side, I will be offline for a fer days, so I expect to
start working on my proposal draft from March 16th onwards.
Would it be OK if I publish a project draft on
GoogleDocs/GitHub/GitLab in (let's say) MarkDown by March 20th, so
people can easily comment on it? I think getting feedback on a
document via email may be quite difficult...
Please, state your preference.

Best,
Leandro

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[zimoun]

Dear Leandro,

On Wed, 26 Feb 2020 at 23:42, Leandro Doctors  wrote:

> I am interested in the Scheme-based ideas. (I have recently
> rediscovered LISP, and I am a Clojure fan.) However, I haven't found
> any indication on how to proceed in the Ideas page [1].

Elsewhere, you have been discussing a proposal of about "Guile build
tool"; which IMHO a large and hard task.

Based on your interests (Clojure, Leiningen, etc.), you should
consider something around a Clojure "importer". (Note that I am
ignorant about the Java ecosystem.)

Currently, it is possible to import Python packages from PyPI, or
Haskell packages from Hackage, or etc. see [1] and something about
Clojure seems missing. I am quoting Ludo from [2]:

--8<---cut here---start->8---
Now we should keep in touch with the Clojure folks to work on an
importer (I learned about “tools.deps”) and to get Clojupyter packaged…
--8<---cut here---end--->8---

and then Jupyter is a "poor-man web-browser" version of Org-mode (very
opinionated! ;-)) and Clojupyter [3] is one of its kernel.


[1] https://guix.gnu.org/manual/devel/en/html_node/Invoking-guix-import.html
[2] https://lists.gnu.org/archive/html/guix-devel/2020-01/msg00189.html
[3] https://github.com/clojupyter/clojupyter


Well, hope that helps as feedback.

Thanks,
simon

Re: [GSOC 2020] Idea: Guile based build-tool

[Leandro Doctors]

Thanks for the link, Simon!
--Leandro

On Mon, 2 Mar 2020, 07:03 zimoun,  wrote:

> Dear,
>
> Perhaps, some inspiration about build systems in this paper [1] by
> Haskellers.
>
> [1]
> https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems-final.pdf
>
>
> All the best,
> simon
>

Re: [GSOC 2020] Idea: Guile based build-tool

[zimoun]

Dear,

Perhaps, some inspiration about build systems in this paper [1] by Haskellers.

[1] 
https://www.microsoft.com/en-us/research/uploads/prod/2018/03/build-systems-final.pdf


All the best,
simon

Re: [GSOC 2020] Guix Deploy, round 2!

[Gábor Boskovits]

Hello Chris,

Christopher Lemmer Webber  ezt írta (időpont: 2020.
febr. 29., Szo 17:55):

> Hello,
>
> I'd be interested in doing a round 2 of mentoring for Guix Deploy.
> The previous student (Jakob L. Kreuze) will not be a GSoC student this
> year but is willing to co-mentor.  (Maybe David Thompson is as well but
> I won't speak for him since we haven't discussed it yet.)
>

Wonderful idea, please add it to the GSoC 2020 ideas page if you haven't
done so.


> I don't know what the situation with Outreachy slots are, but I'd also
> be happy to mentor this for there as well, obviously.
>

The proposal deadline for this round has passed, so we can't do that right
now. If we still have work to do in this area ( I believe we will) when the
next round comes around, I will be happy to help in forming the proposal. I
will soon publish a blog post with the relevant deadlines for both program.


> I think there's probably more that can be done to improve environments.
> This includes initial bootstrapping of environments, making more
> environments available, etc.
>
> Let me also put out a goal for the Guix community: I think we'll see
> Guix Deploy as a success if a bunch of us can switch over to using it on
> our own servers (that includes me).  To that end, I'd love to know, how
> many people are doing so, or have attempted to do so?  What features
> would you like/need?
>

I believe this is a great initiative. Thanks for bringing this up.

>
>  - Chris
>

Best regards,
g_bor

>
>

Re: [GSOC 2020] Idea: Guile based build-tool

[Christopher Lemmer Webber]

There have been some conversations about using Guix's tooling as a build
tool in the past.

Personally, what I would prefer to see is a Guile-based build tool that
is standalone; an alternative for autotools basically.  That then could
be used *in combination with* Guix quite nicely... use the Guile-based
build tool for describing how to build your packages, use "guix
environment" to pull in all the dependencies and create the dev
environment, and of course this tool would then be used for package
definitions too.

However I think this is not a small task, and would take someone who is
willing to do a fair amount of research, or is already familiar with the
problem domain.  It may be hard to finish in a single summer.


Leandro Doctors writes:

> Hi, all,
>
> I am starting to do some research on this idea.
>
> Even though I haven't checked Guix's code yet (I plan to flickr it in
> the upcoming days), I am thinking on getting some inspiration from the
> architecture of similar Clojure-focused tools (which, as I stated
> earlier, is what I know).
>
> I know about three such tools: leiningen[1], boot[2], and clojure.deps[3].
>
> I plan to flickr over their code, and see what architectural decitions
> could be proposed.
> What do you think about this?
>
> Best,
> Leandro
>
> [1] leiningen: https://github.com/technomancy/leiningen
> [2] boot: https://github.com/boot-clj/boot
> [3] clojure.deps: https://github.com/clojure/tools.deps.alpha

Re: [GSOC 2020] Improving the manual?

[Leandro Doctors]

On Thu, 27 Feb 2020 at 21:39, sirgazil  wrote:
> On Thu, 27 Feb 2020 at 21:37, zimoun  wrote:

>> The source of the manual lives under doc/ of what is cloned above;

> The documentation is in the same repository of the guix source code (see the 
> doc directory).

Thanks, guys. I see it now. I had missed it.

And thanks for the pointer to the "contributing" section. Even thought
I had skimmed it, as I could not find the doc sources, I thought the
process for the docs was different.

Best,
Leandro

Re: [GSOC 2020] Improving the manual?

[sirgazil]

  On Thu, 27 Feb 2020 19:22:17 -0500 Leandro Doctors  
wrote 
 > On Thu, 27 Feb 2020 at 11:44, zimoun  wrote:
 > > Are you using Guix yet?
 > >
 > > If no, the first easy step is to install the "package manager".
 > [...]
 > 
 > Thanks for the instructions, Simon!
 > I have just successfully installed Guix :-)
 > 
 > 
 > > If yes, the easy way to start is:
 > > git clone https://git.savannah.gnu.org/git/guix.git
 > 
 > > from the note in the manual
 > > http://guix.gnu.org/manual/devel/en/html_node/Binary-Installation.html#Binary-Installation
 > 
 > While cloning the guix repo above, I found a minor omission in the
 > documentation. However, I could not find its repo in the list from the
 > Savannah group [1] in order to fix it and create a pull/merge request
 > or patch...
 > 
 > [1]: https://savannah.gnu.org/git/?group=guix
 > 
 > 
 > Could anyone please point me to in the manual repo?

The documentation is in the same repository of the guix source code (see the 
doc directory).

https://git.savannah.gnu.org/cgit/guix.git/tree/doc


 > Besides... Where
 > could I find documentation on the workflow for working on the manual
 > proper so I can actually send my patch? I could not find it in the
 > online manual version [2]...

I don't think there are specific steps to work in the manual, but a general 
section on Contributing (see 
https://guix.gnu.org/manual/en/html_node/Contributing.html#Contributing).

I guess you could create a patch and send it to guix-patches as decribed in 
https://guix.gnu.org/manual/en/html_node/Submitting-Patches.html#Submitting-Patches.

Re: [GSOC 2020] Improving the manual?

[zimoun]

On Fri, 28 Feb 2020 at 01:22, Leandro Doctors  wrote:

> > If yes, the easy way to start is:
> > git clone https://git.savannah.gnu.org/git/guix.git
>
> > from the note in the manual
> > http://guix.gnu.org/manual/devel/en/html_node/Binary-Installation.html#Binary-Installation
>
> While cloning the guix repo above, I found a minor omission in the
> documentation. However, I could not find its repo in the list from the
> Savannah group [1] in order to fix it and create a pull/merge request
> or patch...
>
> [1]: https://savannah.gnu.org/git/?group=guix

Hum? I am not sure to understand what you are asking.

The source of the manual lives under doc/ of what is cloned above; i.e.,

https://git.savannah.gnu.org/cgit/guix.git/tree/doc


The submission of patches is done via "git format-patch" and by email
via "git send-email --to=guix-patc...@gnu.org" (package
git:send-email).


> Could anyone please point me to in the manual repo? Besides... Where
> could I find documentation on the workflow for working on the manual
> proper so I can actually send my patch? I could not find it in the
> online manual version [2]...
>
> [2]: https://guix.gnu.org/manual/en/

Maybe you could be interested by this section of the manual:

https://guix.gnu.org/manual/devel/en/guix.html#Contributing


Hope that helps.

All the best,
simon

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[zimoun]

Hi Leandro,

On Thu, 27 Feb 2020 at 12:54, Leandro Doctors  wrote:

> I will clone Guix's repo, read the code and try building it, and get
> back with more specific details, if that's OK with you.

Are you using Guix yet?

If no, the first easy step is to install the "package manager". It
works on any recent Linux distro and it will not interfere with it.
Using the shell script helper, it is quick; and nothing to do. See:
https://git.savannah.gnu.org/cgit/guix.git/plain/etc/guix-install.sh
from the note in the manual
http://guix.gnu.org/manual/devel/en/html_node/Binary-Installation.html#Binary-Installation

If yes, the easy way to start is:

git clone https://git.savannah.gnu.org/git/guix.git
cd guix
guix environment guix
./bootstrap
./configure --localstatedir=/var/
make

then you can tweak any scheme file and recompile with make. And to use
this checkout version, you can just run the ./pre-env-inst hook.

 ./pre-inst-env guix help


Hope that helps.

All the best,
simon

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[Leandro Doctors]

On Thu, 27 Feb 2020 at 08:52, Leandro Doctors  wrote:
> I will clone Guix's repo, read the code and try building it, and get
> back with more specific details, if that's OK with you.

I checked the Guix group page [1], and found several repos...

[1] https://savannah.gnu.org/git/?group=guix


Could anyone please advise me which one(s) to clone first in to get started?


Best,
Leandro

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[Leandro Doctors]

On Thu, 27 Feb 2020 at 05:16, Ricardo Wurmus  wrote:
> Usually we just discuss things right here on the mailing list.

That is a great starting pointer, Ricardo. Thanks! :-)


> A draft for what exactly?

Well, as I said before, I have only started reading the Ideas page.
All I know right now is that I would like to work on a Scheme-related
project :-)

I will clone Guix's repo, read the code and try building it, and get
back with more specific details, if that's OK with you.

Best,
Leandro

Re: [GSOC 2020] Discussing GNU Guix project ideas?

[Ricardo Wurmus]

Hi Leandro,

> My name is Leandro Doctors. I am seriously considering applying to
> GSoC 2020 with GNU Guix.

Oh, that’s nice!

> I am interested in the Scheme-based ideas. (I have recently
> rediscovered LISP, and I am a Clojure fan.) However, I haven't found
> any indication on how to proceed in the Ideas page [1].

Is there any project in particular that you find interesting?

> Could you please point me in the right direction? I mean, is there a
> place where I should publish a draft to get feedback (like, a gist or
> similar), or should I just send emails here?

A draft for what exactly?  Usually we just discuss things right here on
the mailing list.  The mentors of the proposed projects are also reading
this list.

--
Ricardo

Re: GSoC 2020

[Ludovic Courtès]

Hi Gábor,

Gábor Boskovits  skribis:

> The organization application is now open for GSoC 2020. I expect that
> we soon hear something from the GNU coordinator. I will keep you
> updated.

Thanks a lot for keeping an eye on it in addition to Outreachy!

Ludo’.

Re: GSoC 2019 Recap

[Ludovic Courtès]

Hello!

Alex Sassmannshausen  skribis:

> Christopher Lemmer Webber  writes:
>
>> Hi Jakob,
>>
>> I've already said this off-list, but I feel it's worth repeating
>> on-list: thank you for making "guix deploy" a reality.  So many of us
>> have wanted it for so long, but you did the really important thing: you
>> put in the work, and clearly with great care.  We're glad to have you as
>> part of our community; please do stick around! :)
>>
>>  - Chris
>
> +1 to this!

I’m late to the party but same here, a big +1 from me, Jakob!

The result is impressive in terms of code, and it’s been a pleasure
interacting with you on this.

Thank you!

Ludo’.

Re: GSoC 2019 Recap

[Alex Sassmannshausen]

Christopher Lemmer Webber  writes:

> Hi Jakob,
>
> I've already said this off-list, but I feel it's worth repeating
> on-list: thank you for making "guix deploy" a reality.  So many of us
> have wanted it for so long, but you did the really important thing: you
> put in the work, and clearly with great care.  We're glad to have you as
> part of our community; please do stick around! :)
>
>  - Chris

+1 to this!

Alex

Re: GSoC 2019 Recap

[Christopher Lemmer Webber]

Hi Jakob,

I've already said this off-list, but I feel it's worth repeating
on-list: thank you for making "guix deploy" a reality.  So many of us
have wanted it for so long, but you did the really important thing: you
put in the work, and clearly with great care.  We're glad to have you as
part of our community; please do stick around! :)

 - Chris

Re: GSoC 2019

[Björn Höfling]

Hi Gábor,

On Fri, 25 Jan 2019 09:11:04 +0100
Gábor Boskovits  wrote:

> Hello,
> 
> Ludo created the GSoC ideas page for this year:
> https://libreplanet.org/wiki/Group:Guix/GSoC-2019
> 
> It is based on last year's page.

There is also the Outreachy May-August 2019 period:

https://www.outreachy.org/communities/cfp/

with the following timeline:

Jan. 1, 2019Outreachy call for participating communities opens. Mentor and 
volunteer sign up opens.
Feb. 18, 2019   Outreachy application period opens for applicants and most FOSS 
communities are listed
March 5, 2019   Last date for FOSS communities to sign up to participate
March 12, 2019  Last date to add new Outreachy internship projects

 
Is Guix taking part there? If so, is this handled via the same page to
collect ideas?

Björn


pgp__ghjn320k.pgp
Description: OpenPGP digital signature

Re: GSoC 2019

[Ludovic Courtès]

Hello Gábor,

Gábor Boskovits  skribis:

> Ludo created the GSoC ideas page for this year:
> https://libreplanet.org/wiki/Group:Guix/GSoC-2019
>
> It is based on last year's page.
>
> I removed the Cuirass Web Interface project, as it was completed. We
> can discuss a project to improve it, but that would need repharsing
> anyway. Would anyone like to propose something related to that?

There’s more work that could be done, it’d be interesting!

> I also removed the Daemon Rewrite project, as reepca is working on
> that as an University project.

Oh, that’s nice!  Reepca, do get in touch with us!  The more we talk,
the better.  :-)

> I would like to ask people who offered to be mentors last year to
> review, and notify us if they can't mentor. I intend to notify the
> gnu-soc mailing list about the new page no eariler than 2019.01.30,
> 12:00 UTC.

Thanks for taking care of it.  Many people here have expertise on
various subdomains of the code base, so I hope you people won’t be
afraid to step up and propose a project!

Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Gábor Boskovits  writes:

> P
>
> Clément Lassieur  ezt írta (időpont: 2018. júl. 29.,
> V 14:01):
>
>> Danny Milosavljevic  writes:
>>
>> > Hi Tatiana,
>> >
>> > On Sun, 8 Jul 2018 21:48:32 +0200
>> > Tatiana Sholokhova  wrote:
>> >
>> >> Do you have ideas on how to
>> >> implement tuple comparison and other routines in SQL and guile in a
>> >> convenient and flexible way?
>> >
>> > sqlite3 supports row values, so the comparison can be
>> > written like this:
>> >
>> >   select * from foo where (a,b,c) = (2,'foo',3);
>> >
>> > It even supports NULLs for wildcards, though it's a little more
>> complicated:
>> >
>> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>> >
>> > The sqlite C interface doesn't support parameter bindings for the entire
>> > row, though, so you'd have to specify 3 values.
>> >
>> > This works:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
>> 3 ");")
>> >
>> > but this doesn't work, unfortunately:
>> >
>> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>> >
>> > See also https://www.sqlite.org/rowvalue.html
>>
>> With the '<' operator, it doesn't give the results we are looking for, I
>> think.
>>
>> For example:
>>
>> select (0,1) < (1,0); -- returns 1
>> select (0,0) < (0,1); -- returns 1
>>
>
> This is working as expected. Actually this:
> (a,b)<(c,d) is a shortcut for a
> In both cases, we'd want it to return 0.
>>
>
> How do we use it? Why this is the expected result?
>
>
>> I think we should use:
>>
>> select (0 < 1) and (1 < 0); -- returns 0
>> select (0 < 0) and (0 < 1); -- returns 0
>>
>
> Could you please clarify which numbers are the placeholders for which
> quantities?
>
>>
>> instead, for the pagination borders code.

Actually, forget what I said, I was wrong ;-)

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

P

Clément Lassieur  ezt írta (időpont: 2018. júl. 29.,
V 14:01):

> Danny Milosavljevic  writes:
>
> > Hi Tatiana,
> >
> > On Sun, 8 Jul 2018 21:48:32 +0200
> > Tatiana Sholokhova  wrote:
> >
> >> Do you have ideas on how to
> >> implement tuple comparison and other routines in SQL and guile in a
> >> convenient and flexible way?
> >
> > sqlite3 supports row values, so the comparison can be
> > written like this:
> >
> >   select * from foo where (a,b,c) = (2,'foo',3);
> >
> > It even supports NULLs for wildcards, though it's a little more
> complicated:
> >
> >   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
> >
> > The sqlite C interface doesn't support parameter bindings for the entire
> > row, though, so you'd have to specify 3 values.
> >
> > This works:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" ","
> 3 ");")
> >
> > but this doesn't work, unfortunately:
> >
> >   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
> >
> > See also https://www.sqlite.org/rowvalue.html
>
> With the '<' operator, it doesn't give the results we are looking for, I
> think.
>
> For example:
>
> select (0,1) < (1,0); -- returns 1
> select (0,0) < (0,1); -- returns 1
>

This is working as expected. Actually this:
(a,b)<(c,d) is a shortcut for a

How do we use it? Why this is the expected result?


> I think we should use:
>
> select (0 < 1) and (1 < 0); -- returns 0
> select (0 < 0) and (0 < 1); -- returns 0
>

Could you please clarify which numbers are the placeholders for which
quantities?

>
> instead, for the pagination borders code.
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Danny Milosavljevic  writes:

> Hi Tatiana,
>
> On Sun, 8 Jul 2018 21:48:32 +0200
> Tatiana Sholokhova  wrote:
>
>> Do you have ideas on how to
>> implement tuple comparison and other routines in SQL and guile in a
>> convenient and flexible way?
>
> sqlite3 supports row values, so the comparison can be
> written like this:
>
>   select * from foo where (a,b,c) = (2,'foo',3);
>
> It even supports NULLs for wildcards, though it's a little more complicated:
>
>   select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;
>
> The sqlite C interface doesn't support parameter bindings for the entire
> row, though, so you'd have to specify 3 values.
>
> This works:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 
> ");")
>
> but this doesn't work, unfortunately:
>
>   (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")
>
> See also https://www.sqlite.org/rowvalue.html

With the '<' operator, it doesn't give the results we are looking for, I
think.

For example:

select (0,1) < (1,0); -- returns 1
select (0,0) < (0,1); -- returns 1

In both cases, we'd want it to return 0.

I think we should use:

select (0 < 1) and (1 < 0); -- returns 0
select (0 < 0) and (0 < 1); -- returns 0

instead, for the pagination borders code.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Amirouche Boubekki]

Fwiw, you can use bootstrap without js.

Causal reply,

Amirouche

Le jeu. 19 juil. 2018 à 22:11, Tatiana Sholokhova  a
écrit :

> Hi Clément,
>
> Thank you for the clarifications on the job structure!
>
> I have read your changes to the web interface and everything looks good to
> me. Also, it works nicely on your server. So, let's prepare for the merge.
> Let me know if you want me to make some fixes before the merge. In the
> meantime, I continue working on the next features for the interface locally
> and then integrate them into the updated codebase.
>
> Best regards,
> Tatiana
>
> ср, 18 июл. 2018 г. в 12:37, Clément Lassieur :
>
>> Hi Tatiana,
>>
>> Tatiana Sholokhova  writes:
>>
>> > Could you please review the last 3 commits and maybe find some more
>> issues
>> > besides that?
>>
>> I've integrated your work onto my Cuirass instance[1], and I really like
>> it!  I had to fix a few things and adapt it[2] so that it works with
>> multiple inputs.
>>
>> I will do a review as soon as possible, and then we can merge it.  I'm a
>> bit late: going through the whole conversation history took more time
>> than I expected.
>>
>> Clément
>>
>> [1]: https://cuirass.lassieur.org:8081/
>> [2]: https://git.lassieur.org/cgit/cuirass.git/
>>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi Clément,

Thank you for the clarifications on the job structure!

I have read your changes to the web interface and everything looks good to
me. Also, it works nicely on your server. So, let's prepare for the merge.
Let me know if you want me to make some fixes before the merge. In the
meantime, I continue working on the next features for the interface locally
and then integrate them into the updated codebase.

Best regards,
Tatiana

ср, 18 июл. 2018 г. в 12:37, Clément Lassieur :

> Hi Tatiana,
>
> Tatiana Sholokhova  writes:
>
> > Could you please review the last 3 commits and maybe find some more
> issues
> > besides that?
>
> I've integrated your work onto my Cuirass instance[1], and I really like
> it!  I had to fix a few things and adapt it[2] so that it works with
> multiple inputs.
>
> I will do a review as soon as possible, and then we can merge it.  I'm a
> bit late: going through the whole conversation history took more time
> than I expected.
>
> Clément
>
> [1]: https://cuirass.lassieur.org:8081/
> [2]: https://git.lassieur.org/cgit/cuirass.git/
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova  writes:

> Could you please review the last 3 commits and maybe find some more issues
> besides that?

I've integrated your work onto my Cuirass instance[1], and I really like
it!  I had to fix a few things and adapt it[2] so that it works with
multiple inputs.

I will do a review as soon as possible, and then we can merge it.  I'm a
bit late: going through the whole conversation history took more time
than I expected.

Clément

[1]: https://cuirass.lassieur.org:8081/
[2]: https://git.lassieur.org/cgit/cuirass.git/

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Hi Tatiana,

Tatiana Sholokhova  writes:

> Am I right that in terms of Cuirass database derivations correspond to
> jobs?

Yes, but to be more precise, a job is a structure containing:
  - derivation
  - job-name
  - system
  - nix-name
  - eval-id

The database table called "Derivations" should be called "Jobs", so the
name is confusing indeed.

A derivation, as Ricardo explained, is a file (.drv) representing
low-level build actions and the environment in which they are performed.

At each evaluation, there is a new set of jobs returned by the
evaluator, each job having its 'eval-id' incremented.  That means that
two different jobs for the same job-name (i.e. linux-libre-4.17.6-job)
could embed the same derivation.  In that case, it's useless to build
that job in my opinion, see that bug[1].

I hope it's clearer,
Clément

[1]: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=32190

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Dear all,

Ludovic Courtès  writes:

> Hello Tatiana & all,
>
> Ricardo Wurmus  skribis:
>
>>> I am a bit confused about the database structure. As far as I understand,
>>> there are project_name (project) and branch_name (jobset) properties, but
>>> project_name is a primary key, so a project can't have several branches?
>>
>> I share your confusion.  Maybe Ludovic or Mathieu can shed some more
>> light on this.
>
> It’s confusing indeed, I think it’s a mistake that has yet to be fixed.
> Basically what we do now is that we use a different ‘repo_name’ when we
> just want to add a branch…

The notion of "project" has been removed[1].  It was previously the
specification name, which is a primary key indeed, so it didn't make
sense because one project couldn't have several branches.

Now, Hydra's jobsets are the exact same thing as Cuirass'
specifications.  So if you want to build the "master" and "core-update"
branches of Guix, you need two specifications.

However it wasn't even possible, before, to build several branches,
because specifications names were used by the evaluator: they had to be
"guix" or "guix-modular".  Since the name was a primary key, we could
only have two specifications.  It is now[2] possible, because the
evaluator uses the input name instead of the specification name.

If you think there is a need for the notion of "Project" in Cuirass, we
could add it, but it needs to be a new SQL table.  And each
specification would be associated with one project.

Clément

[1]: 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=be713f8a30788861806a74865b07403aa6774117
[2]: 
https://git.savannah.gnu.org/cgit/guix/guix-cuirass.git/commit/?id=7b2f9e0de1ad2d320973b7aea132a8afcad8bece

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Clément Lassieur]

Ricardo Wurmus  writes:

> You can download a copy of the Cuirass database as it is used on
> berlin.guixsd.org, one of the build farms of the Guix project.  I have
> copied it here:
>
> http://bootstrappable.org/cuirass.db
>
> It is 12G(!), which indicates that Cuirass adds way too many entries
> than absolutely needed.  Ludovic wrote on IRC that we don’t seem to
> check if a record already exists when two subsequent evaluations yield
> the same build.

Adding bug-g...@gnu.org to keep track of that bug.

Re: GSoC update

[Ioannis Panagiotis Koutsidis]

> Could you expound a bit?  That’s a very short summary for all the sweat
> you’ve put in it.  :-)
My apologies, at the time I sent the mail in a hurry.
Basically now instead of converting unit files to services individually it 
happens in bulk so that it can check if there is a corresponding .socket file 
per service file (and the reverse). If there is a corresponding .socket file 
then the input and output of the .service will be redirected to the result of 
(accept) on the socket corresponding to the .socket file.
It also makes the select that waits for the commands from herd to also wait for 
the sockets.


> Also, what is the patch against?  It’s not against ‘master’; I suppose
> it’s against the previous state of your own branch, do you have a copy
> of your repo on-line?
It's against the previous patch that I sent, but I can put the branch online.

> It’s OK that the thing doesn’t quite work—we knew it was not an easy
> task.  What’s disappointing though is that you didn’t come to us to
> discuss the issues until now.  GSoC is not about working in all
> loneliness separately from the rest of the group; it’s about becoming
> part of the group.
>
> On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’
> issue; I also outlined reasonable milestones (first, only use
> signalfd(2) instead of SIGCHLD, then discuss together what it would take
> to Fiberize the whole thing.)  It’s sad that you remained stuck instead
> of taking this opportunity to discuss it with us.
Until now (in general, not only during the gsoc) I tried to solve any issues 
that had arisen when I was programming by myself, so it was a bit difficult to 
change that mindset - I will try to be more communicative after this however.


> The patch changes lots of things and unfortunately, without
> explanations, I do not understand what to do with it.  Like what’s the
> new feature?  How is it used?  What implementation choices were made?
> What’s left to be done?…
The new feature is initial support of .socket unit files. It is used like:
(let* ((port1 (open-input-file "/systemd/test.service"))
   (port2 (open-input-file "/systemd/test.socket")))
  (apply register-services (unit-files->services `(("/systemd/test.service" 
"test" service ,(read-unit-file port1))
   ("/systemd/test.socket" 
"test" socket  ,(read-unit-file port2)

  (close-port port1)
  (close-port port2))

The things that are left are supporting more systemd options, and making it work 
properly.

Re: GSoC update

[Jelle Licht]

2018-07-11 0:40 GMT+02:00 Ludovic Courtès :

> Hi Ioannis,
>
> Ioannis Panagiotis Koutsidis  skribis:
>
> > This patch adds initial support for .socket unit files. It does not
> > currently work but is near completion.
>
> Could you expound a bit?  That’s a very short summary for all the sweat
> you’ve put in it.  :-)
>
> Also, what is the patch against?  It’s not against ‘master’; I suppose
> it’s against the previous state of your own branch, do you have a copy
> of your repo on-line?
>
> > During the past month I also worked on a patch that adds signalfd and
> > fiber support but these are currently way too unstable and for that
> > reason I have not included them in this patch.
>
> It’s OK that the thing doesn’t quite work—we knew it was not an easy
> task.  What’s disappointing though is that you didn’t come to us to
> discuss the issues until now.  GSoC is not about working in all
> loneliness separately from the rest of the group; it’s about becoming
> part of the group.
>
> On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’
> issue; I also outlined reasonable milestones (first, only use
> signalfd(2) instead of SIGCHLD, then discuss together what it would take
> to Fiberize the whole thing.)  It’s sad that you remained stuck instead
> of taking this opportunity to discuss it with us.
>

Ioannis, could you perhaps share some of your w.i.p. code regarding
signalfd-based signal handling in guile? Adding to what Ludo'
mentioned, I imagine you are running into some peculiarities regarding
guile's way of handling signals, so I would recommend to start
lurking on #guile if you did not do this before now, so you can interact
with the folks with the most expertise regarding the problems you
might be facing :-)

>
> > From cd260ae65056b53749e7c03f2498a28af2525934 Mon Sep 17 00:00:00 2001
> > From: Ioannis Panagiotis Koutsidis 
> > Date: Tue, 10 Jul 2018 20:03:21 +0300
> > Subject: [PATCH] .socket units
> >
> > ---
> >  modules/shepherd.scm |  44 +++--
> >  modules/shepherd/service.scm | 170 ++---
> >  modules/shepherd/systemd.scm | 354 ---
> >  3 files changed, 368 insertions(+), 200 deletions(-)
>
> The patch changes lots of things and unfortunately, without
> explanations, I do not understand what to do with it.  Like what’s the
> new feature?  How is it used?  What implementation choices were made?
> What’s left to be done?…
>
> Thank you,
> Ludo’.
>
>

Re: GSoC update

[Ludovic Courtès]

Hi Ioannis,

Ioannis Panagiotis Koutsidis  skribis:

> This patch adds initial support for .socket unit files. It does not
> currently work but is near completion.

Could you expound a bit?  That’s a very short summary for all the sweat
you’ve put in it.  :-)

Also, what is the patch against?  It’s not against ‘master’; I suppose
it’s against the previous state of your own branch, do you have a copy
of your repo on-line?

> During the past month I also worked on a patch that adds signalfd and
> fiber support but these are currently way too unstable and for that
> reason I have not included them in this patch.

It’s OK that the thing doesn’t quite work—we knew it was not an easy
task.  What’s disappointing though is that you didn’t come to us to
discuss the issues until now.  GSoC is not about working in all
loneliness separately from the rest of the group; it’s about becoming
part of the group.

On IRC Jelle and I (and possibly others) offered help on the ‘signalfd’
issue; I also outlined reasonable milestones (first, only use
signalfd(2) instead of SIGCHLD, then discuss together what it would take
to Fiberize the whole thing.)  It’s sad that you remained stuck instead
of taking this opportunity to discuss it with us.

> From cd260ae65056b53749e7c03f2498a28af2525934 Mon Sep 17 00:00:00 2001
> From: Ioannis Panagiotis Koutsidis 
> Date: Tue, 10 Jul 2018 20:03:21 +0300
> Subject: [PATCH] .socket units
>
> ---
>  modules/shepherd.scm |  44 +++--
>  modules/shepherd/service.scm | 170 ++---
>  modules/shepherd/systemd.scm | 354 ---
>  3 files changed, 368 insertions(+), 200 deletions(-)

The patch changes lots of things and unfortunately, without
explanations, I do not understand what to do with it.  Like what’s the
new feature?  How is it used?  What implementation choices were made?
What’s left to be done?…

Thank you,
Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Tatiana Sholokhova  ezt írta (időpont: 2018. júl.
8., V, 21:48):

>  Hello all!
>
> Thank you for your helpful comments and ideas!
>
> I've committed an improved version of the pagination. As you advised I
> chose and implemented (2) variant. I alter sorting order in SQL query
> depending on the type of the current page border. So, now all
> operators (gotofirst, gotolast, next and previous) are working.
>
> Also, I added pagination for builds page ("eval" query). Here I face a
> problem that Denny mentioned before.
>
>> The tuple of data cells should uniquely identify one row in the result.
>> (If it
>> didn't, you'd skip the other same-value rows when going to the next page)
>
> I order builds by stoptime and there are some builds with identical
> stoptime timestamps. I am not sure what is the best way to support
> pagination filtering by multiple columns. Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?
>
>
Please have a look at this. https://www.sqlite.org/rowvalue.html. I did not
check the sqlite version.


> Could you please review the last 3 commits and maybe find some more issues
> besides that?
>
> Best regards,
> Tatiana
>
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

On Sun, 8 Jul 2018 21:48:32 +0200
Tatiana Sholokhova  wrote:

> Do you have ideas on how to
> implement tuple comparison and other routines in SQL and guile in a
> convenient and flexible way?

sqlite3 supports row values, so the comparison can be
written like this:

  select * from foo where (a,b,c) = (2,'foo',3);

It even supports NULLs for wildcards, though it's a little more complicated:

  select * from foo where coalesce((a,b,c) = (2,NULL,3), 1) = 1;

The sqlite C interface doesn't support parameter bindings for the entire
row, though, so you'd have to specify 3 values.

This works:

  (sqlite-exec db "select * from foo where (a,b,c) = (" 2 "," "foo" "," 3 ");")

but this doesn't work, unfortunately:

  (sqlite-exec db "select * from foo where (a,b,c) = " '(2 "foo" 3) ";")

See also https://www.sqlite.org/rowvalue.html


pgpK8_s3Kc_Iz.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

 Hello all!

Thank you for your helpful comments and ideas!

I've committed an improved version of the pagination. As you advised I
chose and implemented (2) variant. I alter sorting order in SQL query
depending on the type of the current page border. So, now all
operators (gotofirst, gotolast, next and previous) are working.

Also, I added pagination for builds page ("eval" query). Here I face a
problem that Denny mentioned before.

> The tuple of data cells should uniquely identify one row in the result.
> (If it
> didn't, you'd skip the other same-value rows when going to the next page)

I order builds by stoptime and there are some builds with identical
stoptime timestamps. I am not sure what is the best way to support
pagination filtering by multiple columns. Do you have ideas on how to
implement tuple comparison and other routines in SQL and guile in a
convenient and flexible way?

Could you please review the last 3 commits and maybe find some more issues
besides that?

Best regards,
Tatiana

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Danny Milosavljevic  ezt írta (időpont: 2018. júl.
5., Cs 10:28):

> Hi Tatiana,
>
> On Wed, 4 Jul 2018 22:54:46 +0200
> Tatiana Sholokhova  wrote:
>
> > If we want to maintain a link to
> > the previous page we have to filter the database table entries with to
> > types of filters: one with lower bound on the id, and the other with the
> > upper bound.
>
> Yeah, I know what you mean.
>
> I'd suggest one of the following alternatives for implementing "Previous
> page":
>
> (1) Remember all the page boundaries in the query string (or maybe hidden
> form elements).  So instead of finding out where the beginning of the
> previous page was all over again just remember it from before.
> (2) Reverse the ordering in the query and the boundary check and run
> the query, reverse the result of the query.  Handle finished result as
> before.
> (3) Just use the browser's back button.  In fact, you can just put a
> "Previous" link that presses the back button via Javascript for the time
> being.
>
> I suggest (3) - and implement one of the others later.
>
> I agree that going with (3) now makes sense. The most flexible solution is
(2). I usually do that, as it does not rely on having the earlier pages
seen. I usually abstract this away in a cursor, which has the first, the
last, the current page first and last ids, and a gotofirst, gotolast, next
and previous operators.WDYT?

> > The current implementation of pagination works correctly but it does not
> > support link to the previous page (first, and next only).
>
> Cool!
>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Danny Milosavljevic]

Hi Tatiana,

On Wed, 4 Jul 2018 22:54:46 +0200
Tatiana Sholokhova  wrote:

> If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. 

Yeah, I know what you mean.

I'd suggest one of the following alternatives for implementing "Previous page":

(1) Remember all the page boundaries in the query string (or maybe hidden
form elements).  So instead of finding out where the beginning of the
previous page was all over again just remember it from before.
(2) Reverse the ordering in the query and the boundary check and run
the query, reverse the result of the query.  Handle finished result as before.
(3) Just use the browser's back button.  In fact, you can just put a
"Previous" link that presses the back button via Javascript for the time being.

I suggest (3) - and implement one of the others later.

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).

Cool!


pgp3fOZK322cj.pgp
Description: OpenPGP digital signature

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Jelle Licht]

2018-07-04 22:54 GMT+02:00 Tatiana Sholokhova :

> Hi all,
>
>
Hi Tatiana,


> I just committed the code I wrote trying to improve pagination. I screwed
> up a bit with the new pagination.
> The problem I encountered is following. If we want to maintain a link to
> the previous page we have to filter the database table entries with to
> types of filters: one with lower bound on the id, and the other with the
> upper bound. As we do so simultaneously with limiting the query output to
> the maximal page size, we need to change the ordering type depending on the
> type of the filter. At the moment I am not sure, what it the best way to
> implement database query with such properties. Could you please take a look
> on the commit and share your ideas on that?
>

> The current implementation of pagination works correctly but it does not
> support link to the previous page (first, and next only).
>

It has been some time since I last had a look at databases, so you have my
apologies
in advance if what I am saying does not really apply, or is even not
entirely correct.

You could perhaps have a look at reversing the sort order, and replace ">"
with "<" and "<"
with "<" in your WHERE clauses. The query to the previous page would be
similar to
retrieving the next page, but basically reversing the order you would page
through the results,
if that makes any sense.

If this works, you could also hack together a maybe-efficient query to
retrieve the items
for the last page; simply insert the maximum possible value in your query,
and retrieve the
previous page with regards to that maximum value. In the current case, you
could enter the
highest possible value any id can have.

If it is possible for new items to show up on previous pages as well (so
with id's that are lower
than the highest id), you would need to replace your sorting and filtering
to act on a composite
value of e.g. , instead of on only the id value.


>
> I have been trying to improve pagination for a while, and I now am
> thinking about starting the parallel work on the implementation of the
> features we listed before. What do you think about it?
>

> Best regards,
> Tatiana
>
> Good luck, and HTH!

- Jelle

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hi all,

I just committed the code I wrote trying to improve pagination. I screwed
up a bit with the new pagination.
The problem I encountered is following. If we want to maintain a link to
the previous page we have to filter the database table entries with to
types of filters: one with lower bound on the id, and the other with the
upper bound. As we do so simultaneously with limiting the query output to
the maximal page size, we need to change the ordering type depending on the
type of the filter. At the moment I am not sure, what it the best way to
implement database query with such properties. Could you please take a look
on the commit and share your ideas on that?

The current implementation of pagination works correctly but it does not
support link to the previous page (first, and next only).

I have been trying to improve pagination for a while, and I now am thinking
about starting the parallel work on the implementation of the features we
listed before. What do you think about it?

Best regards,
Tatiana

ср, 27 июн. 2018 г. в 21:56, Ludovic Courtès :

> Hi Tatiana,
>
> Tatiana Sholokhova  skribis:
>
> > On the last week, I had fallen out of the process. I had been having
> exams
> > at the university since the beginning of June. The last exam was
> > rescheduled and that has affected my plans. I am sorry for that. Now the
> > semester is finished and I am having much more time to focus on our
> > project. I am very happy to pass the first evaluation. It is a pleasure
> for
> > me to work with such a great team!
>
> Thank you, and thanks for letting us know.
>
> > In a few days, I am going to implement the whitelist for the static files
> > and improve pagination tool performance as we discussed before. Also, I
> > intend to make the pagination tool universal and add it to the page which
> > displays all builds of a selected evaluation.
>
> Sounds cool!  I haven’t followed your work as closely as I was hoping
> for, but it seems like we could merge it quickly, notably because
> everyone wants these enhancements.  :-)
>
> > Do you think it would be okay if I add auxiliary pagination filters to
> > the request which retrieves builds from the database?
>
> Why not.  Depending on the filter, We may need to add indexes to the
> database (see the bottom of ‘schema.sql’) so that requests aren’t too
> slow.
>
> Thank you for the update!
>
> Ludo’.
>

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Ioannis Panagiotis Koutsidis]

I am currently working on the implementation of .socket unit files, signalfd for 
signal handling, and fibers. It is mostly done, I just have to fix some issues 
that are left.


On 06/25/18 13:47, boskov...@gmail.com wrote:

Hello, could you please send us an update on your project?

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Ludovic Courtès]

Hi Tatiana,

Tatiana Sholokhova  skribis:

> On the last week, I had fallen out of the process. I had been having exams
> at the university since the beginning of June. The last exam was
> rescheduled and that has affected my plans. I am sorry for that. Now the
> semester is finished and I am having much more time to focus on our
> project. I am very happy to pass the first evaluation. It is a pleasure for
> me to work with such a great team!

Thank you, and thanks for letting us know.

> In a few days, I am going to implement the whitelist for the static files
> and improve pagination tool performance as we discussed before. Also, I
> intend to make the pagination tool universal and add it to the page which
> displays all builds of a selected evaluation.

Sounds cool!  I haven’t followed your work as closely as I was hoping
for, but it seems like we could merge it quickly, notably because
everyone wants these enhancements.  :-)

> Do you think it would be okay if I add auxiliary pagination filters to
> the request which retrieves builds from the database?

Why not.  Depending on the filter, We may need to add indexes to the
database (see the bottom of ‘schema.sql’) so that requests aren’t too
slow.

Thank you for the update!

Ludo’.

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Tatiana Sholokhova]

Hello all,

On the last week, I had fallen out of the process. I had been having exams
at the university since the beginning of June. The last exam was
rescheduled and that has affected my plans. I am sorry for that. Now the
semester is finished and I am having much more time to focus on our
project. I am very happy to pass the first evaluation. It is a pleasure for
me to work with such a great team!

In a few days, I am going to implement the whitelist for the static files
and improve pagination tool performance as we discussed before. Also, I
intend to make the pagination tool universal and add it to the page which
displays all builds of a selected evaluation. Do you think it would be okay
if I add auxiliary pagination filters to the request which retrieves builds
from the database?

Best regards,
Tatiana


2018-06-25 13:46 GMT+03:00 Gábor Boskovits :

> Can you please send us an update on your project?
>

Re: GSoC 2018 Syntax and semantics of systemd units in the Shepherd - 1st update

[Gábor Boskovits]

Hello, could you please send us an update on your project?

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Can you please send us an update on your project?

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Gábor Boskovits  writes:

> Joshua Branson  ezt írta (időpont: 2018. jún. 13., Sze 
> 15:52):
>
>  Tatiana Sholokhova  writes:
>
>  > Hello,
>  >
>  > Thank you for your reviews!
>  >
>  > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
>  Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
>  (This question comes from non guix developer by the way.  Just an 
> enthusiast).
>
>  >
>  >
>  >  Thanks again for your excellent work!
>  >
>  >  --
>  >  Ricardo
>
> Joshua, both formats have advantages, and also disadvantages. I think this 
> issue deserves its own discussion. It would be nice to collect the opinions, 
> but I think that currently we have a good
> quality and well integrated xml generator, and we do not have a html5 
> generator. Please correct me if I am wrong.

Nope you sound right.  I'm not arguing in favor or html5 over
xhtml. Both are widely supported by all browsers.  Just trying to
learn.  Thanks!

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Gábor Boskovits]

Joshua Branson  ezt írta (időpont: 2018. jún. 13.,
Sze 15:52):

> Tatiana Sholokhova  writes:
>
> > Hello,
> >
> > Thank you for your reviews!
> >
> > I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.
>
> Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
> (This question comes from non guix developer by the way.  Just an
> enthusiast).
>
> >
> >
> >  Thanks again for your excellent work!
> >
> >  --
> >  Ricardo
>
Joshua, both formats have advantages, and also disadvantages. I think this
issue deserves its own discussion. It would be nice to collect the
opinions, but I think that currently we have a good quality and well
integrated xml generator, and we do not have a html5 generator. Please
correct me if I am wrong.

>

Re: GSoC: Adding a web interface similar to the Hydra web interface

[Joshua Branson]

Tatiana Sholokhova  writes:

> Hello,
>
> Thank you for your reviews!
>
> I've just fixed codestyle issues and replaced HTML5 preamble with XHTML.

Just cause I'm curious, why XHTML instead HTML5?  Is XHTML better to parse?
(This question comes from non guix developer by the way.  Just an enthusiast).

>
>
>  Thanks again for your excellent work!
>
>  --
>  Ricardo