Re: bug#47867: [1.2.1 pre-release testing] substitute downloading and TLS errors

[pelzflorian (Florian Pelz)]

On Sat, Apr 24, 2021 at 07:13:01PM +0200, Ludovic Courtès wrote:
> I’ve pushed these as two separate patches:
> 
>   c50db7156d http-client: Remove exception mishandling in 'http-multiple-get'.
>   02d62978f4 http-client, substitute: Gracefully handle GnuTLS EAGAIN/EINTR.

I assume they will be cherry-picked to the version-1.3.0 branch and
the guix package will be updated.

> This bug should be gone now.

Yes, after and only after locally updating the guix package I can
create good installer images.

Closing.

Note though that "Guu, Jin-Cheng"  reported the TLS
errors before me, but to guix-devel and not as a bug report.

> I’ll go ahead and fix ‘write_to_session_record_port’ in GnuTLS.

Thank you for all the work!

Regards,
Florian

Re: [Outreachy] - Guix Data Service - Set a more informative page title

[Christopher Baines]

Canan Talayhan  writes:

> Thanks for your quick response. It helps a lot to me. But still, I
> have some confusion about the reproduction steps. As I understand it,
> I can reproduce the slow query just using the pure SQL queries without
> touching the code for now, right?
>
> Please find my steps below:
>
> 1. CREATE TEMPORARY TABLE temp_package_metadata (LIKE package_metadata
> INCLUDING ALL)
>
> As we expected, it creates a table but has no data. So I want to
> insert some data but colon types are integer and I don't know any
> meaningful data for these colons.

So, the question to ask here is what data would the temporary table
usually contain during loading data for a revision (when the slow query
happens). I gave more specific information about the data in the
temporary table in my previous email.

> 2. CREATE TEMPORARY TABLE temp_package_metadata AS TABLE package_metadata
>
> Then I create a copy of the package_metadata.
>
> 3. EXPLAIN ANALYZE SELECT * FROM temp_package_metadata
> This code generates 155 msec execution time, but I think it should
> take more time.

I don't see why that query should be slower, but that's also not that
relevant since I think the actual slow query in question here is quite
different.

You can find the query in the code by looking at where it does the
timing for "querying the temp_package_metadata".

I've also got information from the PostgreSQL logs, which gives this as
the query:

  SELECT package_metadata.id, package_metadata.home_page, 
package_metadata.location_id, package_metadata.license_set_id, 
package_metadata.package_description_set_id, 
package_metadata.package_synopsis_set_id FROM package_metadata INNER JOIN 
temp_package_metadata ON (package_metadata.home_page = 
temp_package_metadata.home_page OR (package_metadata.home_page IS NULL AND 
temp_package_metadata.home_page IS NULL)) AND (package_metadata.location_id = 
temp_package_metadata.location_id OR (package_metadata.location_id IS NULL AND 
temp_package_metadata.location_id IS NULL)) AND 
(package_metadata.license_set_id = temp_package_metadata.license_set_id OR 
(package_metadata.license_set_id IS NULL AND 
temp_package_metadata.license_set_id IS NULL)) AND 
(package_metadata.package_description_set_id = 
temp_package_metadata.package_description_set_id OR 
(package_metadata.package_description_set_id IS NULL AND 
temp_package_metadata.package_description_set_id IS NULL)) AND 
(package_metadata.package_synopsis_set_id = 
temp_package_metadata.package_synopsis_set_id OR 
(package_metadata.package_synopsis_set_id IS NULL AND 
temp_package_metadata.package_synopsis_set_id IS NULL))


signature.asc
Description: PGP signature

Re: Pinebook pro TODO

[Vincent Legoll]

Hello Vagrant,

Thanks a lot for the update.

I'll see what I can do.

-- 
Vincent Legoll

Re: bug#47867: [1.2.1 pre-release testing] substitute downloading and TLS errors

[Ludovic Courtès]

Hi Florian,

"pelzflorian (Florian Pelz)"  skribis:

> Success!  Thank you.

\o/

> On Fri, Apr 23, 2021 at 11:19:28AM +0200, Ludovic Courtès wrote:
>> Florian, could you try again with the attached patch?
>
> It succeeds on two full installs of Enlightenment, no errors, no
> prolonged getting stuck.

Yay.

>> I double-checked and the GnuTLS Guile bindings already
>> handle GNUTLS_E_AGAIN and GNUTLS_E_INTERRUPTED, so my guess is that this
>> was just a side effect of dealing with stale TLS sessions:
>>   https://gitlab.com/gnutls/gnutls/-/blob/master/guile/src/core.c#L1042
>
> Strange,.

My bad: I was looking at ‘read_from_session_record_port’, but the
problem is in ‘write_to_session_record_port’.  All is clear now.

I’ve pushed these as two separate patches:

  c50db7156d http-client: Remove exception mishandling in 'http-multiple-get'.
  02d62978f4 http-client, substitute: Gracefully handle GnuTLS EAGAIN/EINTR.

This bug should be gone now.

I’ll go ahead and fix ‘write_to_session_record_port’ in GnuTLS.

Thanks a lot for the quick and thorough testing!

Ludo’.

Re: Pinebook pro TODO

[Vagrant Cascadian]

On 2021-04-24, Vincent Legoll wrote:
> now that I've got my pbp running guix, I have
> more questions.
>
> The kernel is still "manjaro kernel", should
> this be changed to reflect some changes
> we have made ?

That's probably just tweaking the kernel defconfig a bit more.


> What about the panfrost patch ?
> https://issues.guix.gnu.org/40835
> https://lists.gnu.org/archive/html/guix-patches/2020-04/msg01317.html

The panfrost changes for mesa are in guix since 1.2.0rc1, commit:

  ee401ed9249fbe284ef1b9b437d39207ca88131b

I think that bug report addressed multiple other issues, so didn't
exactly seem appropriate to close.


> The wip-pbp branch:
> why does it get merges from master instead
> of being rebased (as wip-* branches can be) ?

It has some history that makes rebasing complicated that I'd rather not
loose for now. When it's ready to merge to master, I'll go through the
trouble to rebase it.


> How can one help getting things going upstream ?

I think the only things left are the 22 kernel patches in the
wip-pinebook-pro branch in
gnu/packages/patches/linux-libre-pinebook-pro-* ... many are flagged as
hacks in the comments and need to be re-implemented in a way that makes
the patches upstreamable... and then submitted upstream.

Might be able to further refactor those patches to get something useable
directly on guix master, but I haven't been comfortable pushing such
large changes to master (~2000 lines of code, ~3000 line kernel
config).

Although the patches are only applied to linux-libre-pinbook-pro and
shouldn't affect other packages at all...

There are also some changes (usb-c alternate display mode, wifi) that
may need to be removed for linux-libre; I believe linux-libre makes it
impossible to load the non-free firmware in practice, but the support
should just not be present at all.

The wip-pinebook-pro patches are stalled at 5.10, but at least 5.10 is a
long-term support kernel series:

  https://gitlab.manjaro.org/tsys/linux-pinebook-pro.git


live well,
  vagrant


signature.asc
Description: PGP signature

Re: [Outreachy] - Guix Data Service - Set a more informative page title

[Canan Talayhan]

Thanks for your quick response. It helps a lot to me. But still, I
have some confusion about the reproduction steps. As I understand it,
I can reproduce the slow query just using the pure SQL queries without
touching the code for now, right?

Please find my steps below:

1. CREATE TEMPORARY TABLE temp_package_metadata (LIKE package_metadata
INCLUDING ALL)

As we expected, it creates a table but has no data. So I want to
insert some data but colon types are integer and I don't know any
meaningful data for these colons.

2. CREATE TEMPORARY TABLE temp_package_metadata AS TABLE package_metadata

Then I create a copy of the package_metadata.

3. EXPLAIN ANALYZE SELECT * FROM temp_package_metadata
This code generates 155 msec execution time, but I think it should
take more time.

Am I still on the right track?


On Sat, Apr 24, 2021 at 2:39 PM Christopher Baines  wrote:
>
>
> Christopher Baines  writes:
>
> > The approach I'd recommend is, make yourself a realistic
> > temp_package_metadata table by populating it with all the
> > package_metadata entries for a single revision already in your local
> > database. Then construct and try the slow query, and see how long it
> > takes, and look at the query plan (run the query with EXPLAIN at the
> > start).
>
> Following up on your question on IRC about creating the temporary table,
> the code that does this is here [1]. I wouldn't suggest running the
> code, just the same SQL it uses, and use relevant values for
> temp-table-name and table-name that are appropriate for the
> package_metadata table.
>
> 1: 
> https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data-service/model/utils.scm#n313

Re: What's the meaning of the percent sign in variable names

[Luis Felipe]

‐‐‐ Original Message ‐‐‐
On Friday, April 23, 2021 2:49 PM, Leo Prikler  
wrote:

> Am Freitag, den 23.04.2021, 14:22 + schrieb Luis Felipe:
>
> > Hi,
> > Are all these constants (%base-packages, for example)? Is this a Guix
> > convention or does it come from Guile? Although looking at Guile's
> > Variable index I see many constants in uppercase, and also some
> > variables prefixed with the percent sign, while Guix' Programming
> > index doesn't seem to list any uppercase names.
>
> The tendency to prepend variables with % certainly comes from Guile or
> more generally Scheme. The upper case constants in Guile likely come
> from C, where this convention is more popular and IIUC often concerns
> integer constants, that are in some way important for the C side of
> things (think of stuff like OPEN_READ). Since that is not a concern
> for Guix, a package manager mostly written in Guile Scheme, we can
> prefix our constants with a percent sign.

Thanks, Leo.

Re: [Outreachy] - Guix Data Service - Set a more informative page title

[Christopher Baines]

Christopher Baines  writes:

> The approach I'd recommend is, make yourself a realistic
> temp_package_metadata table by populating it with all the
> package_metadata entries for a single revision already in your local
> database. Then construct and try the slow query, and see how long it
> takes, and look at the query plan (run the query with EXPLAIN at the
> start).

Following up on your question on IRC about creating the temporary table,
the code that does this is here [1]. I wouldn't suggest running the
code, just the same SQL it uses, and use relevant values for
temp-table-name and table-name that are appropriate for the
package_metadata table.

1: 
https://git.savannah.gnu.org/cgit/guix/data-service.git/tree/guix-data-service/model/utils.scm#n313


signature.asc
Description: PGP signature

Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Vincent Legoll]

On Sat, Apr 24, 2021 at 11:10 AM Christopher Baines  wrote:
> I did think about trying to include something about Cuirass, but I don't
> have a clear picture of it's scope or purpose, so I'm not really the
> right person to attempt to write authoritatively about it.

OK, fair enough, Matthieu, Ludo, or anyone else can shed some light
here ?

> I try to avoid using the CI (Continuous Integration) term as I'm not
> sure there's a shared understanding of the term (I think it's the
> practice of multiple people frequently merging their changes to some
> software they're all working on).

Yes, I know the term is overloaded, but it's easy and conveys at least
a bit of the subject at hands, so...

> Does that make any sense? Do say if you have more questions.

Yes, and I will, thanks

-- 
Vincent Legoll

Pinebook pro TODO

[Vincent Legoll]

Hello,

now that I've got my pbp running guix, I have
more questions.

The kernel is still "manjaro kernel", should
this be changed to reflect some changes
we have made ?

What about the panfrost patch ?
https://issues.guix.gnu.org/40835
https://lists.gnu.org/archive/html/guix-patches/2020-04/msg01317.html

The wip-pbp branch:
why does it get merges from master instead
of being rebased (as wip-* branches can be) ?

How can one help getting things going upstream ?

Thanks to all people involved

-- 
Vincent Legoll

Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Christopher Baines]

Vincent Legoll  writes:

> Hello,
>
> On Sat, Apr 24, 2021 at 8:28 AM Christopher Baines  wrote:
>> With some prompting, there's now a blog post about the Guix Build
>> Coordinator
>
> Nice post that explains a lot, but I'm still not so sure about the
> relations to cuirass. I'd have liked a small paragraph explaining
> what the differences are, how can they complement each other,
> kind of the CI envisionned big picture...

Thanks for the feedback Vincent :)

I did think about trying to include something about Cuirass, but I don't
have a clear picture of it's scope or purpose, so I'm not really the
right person to attempt to write authoritatively about it.

On a technical level, there's no connection, although there was talk
over the last 6 months or so about trying to have or allow Cuirass to
benefit from the Guix Build Coordinator's ability to perform builds in a
methodical manor across multiple machines. That hasn't happened yet
though, and in the mean time, Cuirass has gained it's own mechanism of
running builds on other machines.

I try to avoid using the CI (Continuous Integration) term as I'm not
sure there's a shared understanding of the term (I think it's the
practice of multiple people frequently merging their changes to some
software they're all working on).

In terms of building things for substitutes, that's one of the use cases
I had in mind when I designed the Guix Build Coordinator, and I think
that design has worked out well, so I'm still interested in trying to
get some benefits for Guix users through using the Guix Build
Coordinator to produce substitutes in a faster and more reliable way.

Does that make any sense? Do say if you have more questions.

Thanks,

Chris


signature.asc
Description: PGP signature

Update on using the Guix Build Coordinator for substitutes

[Christopher Baines]

Hey,

This is probably something I haven't spoken enough about, but my test
site (guix.cbaines.net) for building and providing substitutes has now
got to a point where I've tested most things I set out to test.

In particular, it's building the following things:

 - packages for:
   - x86_64-linux
   - i686-linux
   - aarch64-linux
   - armhf-linux
   - i586-gnu

 - cross build packages:
   - x86_64-linux -> arm-linux-gnueabihf
   - x86_64-linux -> aarch64-linux-gnu
   - x86_64-linux -> arm-linux-gnueabihf

 - channel instance derivations (guix pull)
   - x86_64-linux
   - i686-linux
   - aarch64-linux

I can't remember when this started, but I think it was mid 2020,
starting with a subset of the above, and I gradually got it building
more things.

Recent additions have been i586-gnu (the Hurd) support, as well as
mixing QEMU and native/compatible builds for aarch64-linux and
armhf-linux.

One of the things I wanted to try out with the Guix Build Coordinator
was retrying builds on failure, and I think that's a big part of why
guix.cbaines.net often will have higher substitute availability than
ci.guix.gnu.org.

I wanted to build on this to try mixing builds through QEMU for
aarch64-linux and armhf-linux with builds taking place on aarch64-linux
(with hardware that has compatibility with armhf). I don't think using
QEMU is a goal, as I think using QEMU can introduce problems, but
suplementing some ARM hardware with x86_64+QEMU can be useful for
building things for substitutes.

As the Guix Build Coordinator has the ability to tag builds and agents,
and match up tags so that specific builds only get allocated to specific
agents, this allows for spotting when aarch64-linux or armhf-linux
builds fail, and scheduling retry builds that are guarrenteed to take
place without using QEMU. Like the general approach of retrying failing
builds, this reduces the chance that a substitute won't be available
because of a specific failure, like using QEMU.

I think overall this experiment has been a success, specifically on the
aarch64-linux and armhf-linux builds, I think most things that can be
built, have been built (although I don't have a rigerous way of checking
this).

Back at the start of 2020, I think there was some optimism about using
the Guix Build Coordinator for building substitutes for ci.guix.gnu.org,
which hasn't happened yet, but I'll send a separate email about that.

Thanks,

Chris


signature.asc
Description: PGP signature

Re: New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Vincent Legoll]

Hello,

On Sat, Apr 24, 2021 at 8:28 AM Christopher Baines  wrote:
> With some prompting, there's now a blog post about the Guix Build
> Coordinator

Nice post that explains a lot, but I'm still not so sure about the
relations to cuirass. I'd have liked a small paragraph explaining
what the differences are, how can they complement each other,
kind of the CI envisionned big picture...

Regards

-- 
Vincent Legoll

Re: A "cosmetic changes" commit that removes security fixes

[Mark H Weaver]

Hi Raghav,

Raghav Gururajan  writes:

>> Thank you for these links.  From the IRC log cited above, it now appears
>> that Léo Le Bouter  bears primary responsibility
>> for these mistakes.  In particular, according to the IRC
>> logs, Léo wrote:
>> 
>> raghavgururajan: the main issues on the rebasing were about
>>security fixes on cairo, gdk-pixbuf and glib
>> I modified the cosmetic commits to remove the graft and
>>patches etc.
>> 
>>
> Yes, Leo made these changes. But I also have to acknowledge that it was 
> part of my responsibility to question any changes. My apologies for 
> that.

Thank you for this acknowledgment, Raghav.  Your recent responses in
this thread have been commendable.

 Regards,
   Mark

-- 
Support Richard Stallman against the vicious disinformation campaign
against him and the FSF.  See  for more.

Re: A "cosmetic changes" commit that removes security fixes

[Mark H Weaver]

Hi Léo,

Léo Le Bouter  writes:

> On Fri, 2021-04-23 at 15:18 -0400, Leo Famulari wrote:
>> Léo and Raghav, you need to keep learning our workflow around
>> security updates.  It's not okay to remove security patches and later
>> update a package to a fixed version in a different commit. `git
>> rebase` is the tool to learn for cases like this one.
>
> I knew about this but I didnt feel like telling Raghav to do yet
> another rebase. I felt like Raghav was taking on with so much already.

This response is what native English speakers call a "red herring"
: something that misleads or
distracts from a relevant or important question.

Why do I say that?  As I pointed out elsewhere in this thread,

  https://lists.gnu.org/archive/html/guix-devel/2021-04/msg00366.html

it appears that Raghav didn't actually remove the security fixes; it
appears that *you* did.  Therefore, I fail to see how it could have been
avoided by asking Raghav to do another rebase.

>From the IRC logs cited in the message above, it appears that you took
the liberty of modifying Raghav's "cosmetic changes" commits to also
remove security fixes in the re-indented packages, while claiming in the
summary lines that you were "ungraft[ing]" cairo and glib.

  
https://git.sr.ht/~lle-bout/guix/commit/a045a48dd961f0c5c3d536dcc3fd21d9c08d2d50
  
https://git.sr.ht/~lle-bout/guix/commit/6477daa338fbf1c9edacfc3690aca77cacfe0008

Can you please explain what went wrong here?

  Thanks,
Mark

-- 
Support Richard Stallman against the vicious disinformation campaign
against him and the FSF.  See  for more.

Re: Organizing Tui Apps

[Leo Prikler]

Am Samstag, den 24.04.2021, 01:50 + schrieb Ryan Prior:
> On April 23, 2021, Leo Famulari  wrote:
> > On Fri, Apr 23, 2021 at 10:00:14PM +0200, Leo Prikler wrote:
> > > Spreadsheets sounds fine to me, but I think the most important
> > ones
> > > (libreoffice and org-mode) are already excluded from that module
> > for
> > > obvious reasons ;)
> > > Perhaps an even more generic "office" module might be better,
> > because
> > > then we could at least add some small word processors to it,
> > WDYT? Or
> > > maybe sc-im already fits as-is into textutils if you squint hard
> > enough
> > 
> > My main objection was about moving packages. So whatever module you
> > create is fine, but let's not start moving libreoffice and org-mode
> > around.
> 
> As the person who originally suggested organizing tui-apps into a
> file, I'd envisioned it as being sort of like its own desktop
> environment. So similar to how we might put Gnome (GTK) apps and KDE
> (Qt) apps together, we might put tui (ASCII terminal) apps together.
I don't think that makes too much sense.  GNOME, MATE, XFCE, Unity,
Cinnamon etc. are all GTK-based toolkits, but their mere size warrants
to put them into their own modules (at least for some, probably).
I also don't think, that TUI apps form a cohesive interface as there
are few "collective guidelines" for how a TUI app should look like. 
See ncmpc and ncmpcpp for an example of two TUI apps serving the same
purpose but looking very different.

> I am not upset if that's unconvincing to others & will happily drop
> the proposal if it's not useful, I never intended for it to become a
> drawn out discussion. It also makes sense to me that we might
> organize things by programming language or by function, those each
> have strengths as strategies.
Supposing sc-im was programmed in Rust (it is not afaik, but suppose it
was), putting it into rust-apps would probably be fine.

Regards,
Leo

New blog post on the Guix Build Coordinator: Building derivations, how complicated can it be?

[Christopher Baines]

Hey,

With some prompting, there's now a blog post about the Guix Build
Coordinator: 

  
https://guix.gnu.org/en/blog/2021/building-derivations-how-complicated-can-it-be/

Since it doesn't have a web interface like the Guix Data Service, and
doesn't directly meet a widespread need, I think it's probably harder
than usual to understand the intent and design , but hopefully this
helps.

Thanks to Ludo for his review and feedback.

Do let me know if you notice any mistakes or have any questions!

Chris


signature.asc
Description: PGP signature