Re: Any recent changes to the arm builders?

[Florian Weimer]

* Neal Gompa:

> On Fri, Aug 20, 2021 at 12:45 AM Florian Weimer  wrote:
>>
>> * Neal Gompa:
>>
>> > On Thu, Aug 19, 2021 at 8:59 AM Dennis Gilmore  wrote:
>> >>
>> >> On Wed, Aug 18, 2021 at 3:11 PM Florian Weimer  wrote:
>> >> >
>> >> > * Dennis Gilmore:
>> >> >
>> >> > > We intentionally never looked at enabling that and always had no plans
>> >> > > to support multi-lib on Arm
>> >> >
>> >> > It's not multilib.  Buildroots aren't multilib.
>> >> >
>> >> > I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
>> >> > 32-bit Arm kernels.  It's very much a dead end.
>> >> >
>> >> > Debian uses 64-bit kernels:
>> >> >
>> >> > | Package: glibc
>> >> > | Version: 2.31-16
>> >> > | Source Version: 2.31-16
>> >> > | Distribution: sid
>> >> > | Machine Architecture: arm64
>> >> > | Host Architecture: armhf
>> >> > | Build Architecture: armhf
>> >> > | Build Type: any
>> >>
>> >> There is some magic that's needed for multi-lib that will be needed
>> >> for the AArch64 host's rpm to install the rpms into a 32bit chroot.
>> >
>> > There is no "magic" other than qemu-user-static needing to be
>> > installed. Mock will transparently handle everything just fine. It's
>> > how I build foreign architecture packages on my computer.
>>
>> No, the code would run natively, without QEMU, like i686 on an x86-64
>> kernel.  There is no emulation.
>
> That depends on the CPU. As Dennis and Peter like to remind me,
> armv7hl backwards compatibility is *optional* in AArch64 CPUs (which
> was really a stupid idea, but whatever). There are several AArch64
> CPUs that lack it. Maybe we're lucky and ours have it like the RPi CPU
> does, but not all do. My AMD Opteron A1000 board doesn't, for example.

Fedora can choose builder hardware that has native userspace support for
32-bit programs.  Today, this requirement is much easier to meet than
32-bit support in supervisor mode, particularly for hardware that one
can reasonably install in a data center.

Jeremy Linton already said as much on this thread:

| OS wise, 32-bit containers/etc work fine on 64-bit fedora
| kernels. This also solves the problem that server grade HW with 32-bit
| guest (EL1+) support is becoming rarer (basically only older A72 based
| platforms now) and would provide a path forward on more recent
| Gravaton2, Ampere, etc platforms.

Also see the table on page 3 of
.
It says “AArch32 at EL0 only” for all the newer cores.

With a 64-bit kernel, we would haven't use LPAE mode either, which I
assume is a an outlier and otherwise rarely used with 32-bit Arm
kernels.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[David Schwörer]

On 8/18/21 10:27 AM, Miro Hrončok wrote:
> On 16. 08. 21 9:00, Miro Hrončok wrote:
>> On 16. 08. 21 0:29, Kevin Fenzi wrote:
>>> On Sun, Aug 15, 2021 at 11:43:48AM +0200, Miro Hrončok wrote:
 On 14. 08. 21 18:19, Kevin Fenzi wrote:
> It makes me wonder if we should consider letting 32bit arm go...
> (insert pitchforks and torches).

 Let's propose it and see what people say? As a package maintainer, I
 would
 certainly appreciate this.

 I can draft a change proposal next week.
>>>
>>> How about we give time for the iot and arm folks who likely have not
>>> seen this yet to chime in. :)
>>
>> I was planning to ask them directly.
> 
> IoT: https://pagure.io/fedora-iot/issue/47
> 
> ARM:
> https://lists.fedoraproject.org/archives/list/a...@lists.fedoraproject.org/thread/YC2LYBJSFKDAVBUJAIFQCCBS5VLW5TUB/
> 

While I can understand that it is difficult, I do run Fedora on my
Raspberry Pi 1, which still runs perfectly fine as web and mail server.
The RPi 1 and 2 are 32-bit only, so they would be affected.

As I use them for not so intense workloads I think I could keep the old
RPis for a long time, as long as there are updates.

Only Raspberry Pi 3 or later are 64 bit.

It would be sad having to replace it or switch away from Fedora for them.

Best,
David
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Neal Gompa]

On Fri, Aug 20, 2021 at 12:45 AM Florian Weimer  wrote:
>
> * Neal Gompa:
>
> > On Thu, Aug 19, 2021 at 8:59 AM Dennis Gilmore  wrote:
> >>
> >> On Wed, Aug 18, 2021 at 3:11 PM Florian Weimer  wrote:
> >> >
> >> > * Dennis Gilmore:
> >> >
> >> > > We intentionally never looked at enabling that and always had no plans
> >> > > to support multi-lib on Arm
> >> >
> >> > It's not multilib.  Buildroots aren't multilib.
> >> >
> >> > I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
> >> > 32-bit Arm kernels.  It's very much a dead end.
> >> >
> >> > Debian uses 64-bit kernels:
> >> >
> >> > | Package: glibc
> >> > | Version: 2.31-16
> >> > | Source Version: 2.31-16
> >> > | Distribution: sid
> >> > | Machine Architecture: arm64
> >> > | Host Architecture: armhf
> >> > | Build Architecture: armhf
> >> > | Build Type: any
> >>
> >> There is some magic that's needed for multi-lib that will be needed
> >> for the AArch64 host's rpm to install the rpms into a 32bit chroot.
> >
> > There is no "magic" other than qemu-user-static needing to be
> > installed. Mock will transparently handle everything just fine. It's
> > how I build foreign architecture packages on my computer.
>
> No, the code would run natively, without QEMU, like i686 on an x86-64
> kernel.  There is no emulation.

That depends on the CPU. As Dennis and Peter like to remind me,
armv7hl backwards compatibility is *optional* in AArch64 CPUs (which
was really a stupid idea, but whatever). There are several AArch64
CPUs that lack it. Maybe we're lucky and ours have it like the RPi CPU
does, but not all do. My AMD Opteron A1000 board doesn't, for example.


-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Björn 'besser82' Esser]

Am Mittwoch, dem 18.08.2021 um 16:23 -0400 schrieb Rich Mattes:
> On 8/17/21 10:21 AM, Orion Poplawski wrote:
> > On 8/14/21 10:19 AM, Kevin Fenzi wrote:
> > > On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
> > > > Have there been any recent changes to the arm (32bit) builders? 
> > > > It 
> > > > seems
> > > > like I'm having much more issues there with builds likely
> > > > running out of
> > > > memory or similar.
> > > 
> > > Yes. They were mistakenly running the normal kernel (so they had
> > > ~3GB
> > > memory available). I moved them back to the lpae kernel (so they
> > > see
> > > 40GB memory), but this causes
> > > 
> > > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> > > 
> > > basically OOM kills kojid, which restarts kojid, which restarts
> > > the
> > > build, which kills kojid, etc...
> > > 
> > > I've tried all kinds of things here, but haven't been able to find
> > > any
> > > way to make it work. Arm folks can't duplicate it on non koji
> > > builders.
> > > I suspect the number of people using lpae on 32bit arm is... low.
> > > We could just go back to non lpae, but that breaks building some
> > > other
> > > packages (llvm fails to build for example).
> > > 
> > > It makes me wonder if we should consider letting 32bit arm go...
> > > (insert pitchforks and torches).
> > > 
> > > Anyhow, if anyone has any ideas, let me know.
> > > 
> > > kevin
> > 
> > Looks like the vtk build just as it was about to finish (after 11+
> > hours 
> > - had completed extracting debuginfo and was on to checking the
> > build 
> > root last I saw) restarted. This is pretty unworkable.
> > 
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672
> > 
> 
> I'm waiting on VTK to complete to do a rebuild of PCL.

VTK should be fine on Rawhide now.


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Neal Gompa:

> On Thu, Aug 19, 2021 at 8:59 AM Dennis Gilmore  wrote:
>>
>> On Wed, Aug 18, 2021 at 3:11 PM Florian Weimer  wrote:
>> >
>> > * Dennis Gilmore:
>> >
>> > > We intentionally never looked at enabling that and always had no plans
>> > > to support multi-lib on Arm
>> >
>> > It's not multilib.  Buildroots aren't multilib.
>> >
>> > I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
>> > 32-bit Arm kernels.  It's very much a dead end.
>> >
>> > Debian uses 64-bit kernels:
>> >
>> > | Package: glibc
>> > | Version: 2.31-16
>> > | Source Version: 2.31-16
>> > | Distribution: sid
>> > | Machine Architecture: arm64
>> > | Host Architecture: armhf
>> > | Build Architecture: armhf
>> > | Build Type: any
>>
>> There is some magic that's needed for multi-lib that will be needed
>> for the AArch64 host's rpm to install the rpms into a 32bit chroot.
>
> There is no "magic" other than qemu-user-static needing to be
> installed. Mock will transparently handle everything just fine. It's
> how I build foreign architecture packages on my computer.

No, the code would run natively, without QEMU, like i686 on an x86-64
kernel.  There is no emulation.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Orion Poplawski]

On 8/19/21 2:08 PM, Kevin Fenzi wrote:

On Thu, Aug 19, 2021 at 07:38:14AM -0600, Orion Poplawski wrote:


Well, that build unfortunately failed due to gcc getting killed (the usual
arm OOM symptom).  Looks like another try has been started.  If that fails I
think I'll need to bump down ncpus yet more (already dropped from 5 to 3).


So, it's worth noting here that the entire VM here shouldn't be out of
memory. It's the process hitting the 32bit limit. So, I don't think
lowering number of cpus is going to help if any of them are hitting the
32bit process limit. :(

kevin


Except that I would expect that to lead to a more deterministic error 
rather than the intermittent one we are seeing now.


--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems  720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Neal Gompa]

On Thu, Aug 19, 2021 at 8:59 AM Dennis Gilmore  wrote:
>
> On Wed, Aug 18, 2021 at 3:11 PM Florian Weimer  wrote:
> >
> > * Dennis Gilmore:
> >
> > > We intentionally never looked at enabling that and always had no plans
> > > to support multi-lib on Arm
> >
> > It's not multilib.  Buildroots aren't multilib.
> >
> > I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
> > 32-bit Arm kernels.  It's very much a dead end.
> >
> > Debian uses 64-bit kernels:
> >
> > | Package: glibc
> > | Version: 2.31-16
> > | Source Version: 2.31-16
> > | Distribution: sid
> > | Machine Architecture: arm64
> > | Host Architecture: armhf
> > | Build Architecture: armhf
> > | Build Type: any
>
> There is some magic that's needed for multi-lib that will be needed
> for the AArch64 host's rpm to install the rpms into a 32bit chroot.

There is no "magic" other than qemu-user-static needing to be
installed. Mock will transparently handle everything just fine. It's
how I build foreign architecture packages on my computer.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Björn 'besser82' Esser]

Am Donnerstag, dem 19.08.2021 um 07:38 -0600 schrieb Orion Poplawski:
> On 8/18/21 2:23 PM, Rich Mattes wrote:
> > On 8/17/21 10:21 AM, Orion Poplawski wrote:
> > > On 8/14/21 10:19 AM, Kevin Fenzi wrote:
> > > > On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
> > > > > Have there been any recent changes to the arm (32bit)
> > > > > builders?  It 
> > > > > seems
> > > > > like I'm having much more issues there with builds likely
> > > > > running 
> > > > > out of
> > > > > memory or similar.
> > > > 
> > > > Yes. They were mistakenly running the normal kernel (so they had
> > > > ~3GB
> > > > memory available). I moved them back to the lpae kernel (so they
> > > > see
> > > > 40GB memory), but this causes
> > > > 
> > > > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> > > > 
> > > > basically OOM kills kojid, which restarts kojid, which restarts
> > > > the
> > > > build, which kills kojid, etc...
> > > > 
> > > > I've tried all kinds of things here, but haven't been able to
> > > > find any
> > > > way to make it work. Arm folks can't duplicate it on non koji
> > > > builders.
> > > > I suspect the number of people using lpae on 32bit arm is...
> > > > low.
> > > > We could just go back to non lpae, but that breaks building some
> > > > other
> > > > packages (llvm fails to build for example).
> > > > 
> > > > It makes me wonder if we should consider letting 32bit arm go...
> > > > (insert pitchforks and torches).
> > > > 
> > > > Anyhow, if anyone has any ideas, let me know.
> > > > 
> > > > kevin
> > > 
> > > hours - had completed extracting debuginfo and was on to checking
> > > the 
> > > build root last I saw) restarted. This is pretty unworkable.
> > > 
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672
> > > 
> > 
> > I kicked this off again to see if the ARM builder kernel changes
> > help. 
> > I'm waiting on VTK to complete to do a rebuild of PCL.
> > 
> > Rich
> 
> Well, that build unfortunately failed due to gcc getting killed (the 
> usual arm OOM symptom).  Looks like another try has been started.  If 
> that fails I think I'll need to bump down ncpus yet more (already 
> dropped from 5 to 3).


Looks like this time the build is likely going to pass, as it already
reached the "extracting debuginfo" step.  =)


signature.asc
Description: This is a digitally signed message part
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Thu, Aug 19, 2021 at 07:38:14AM -0600, Orion Poplawski wrote:
> 
> Well, that build unfortunately failed due to gcc getting killed (the usual
> arm OOM symptom).  Looks like another try has been started.  If that fails I
> think I'll need to bump down ncpus yet more (already dropped from 5 to 3).

So, it's worth noting here that the entire VM here shouldn't be out of
memory. It's the process hitting the 32bit limit. So, I don't think
lowering number of cpus is going to help if any of them are hitting the
32bit process limit. :( 

kevin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Rich Mattes]

On 8/19/21 9:38 AM, Orion Poplawski wrote:

On 8/18/21 2:23 PM, Rich Mattes wrote:

On 8/17/21 10:21 AM, Orion Poplawski wrote:


Looks like the vtk build just as it was about to finish (after 11+ 
hours - had completed extracting debuginfo and was on to checking the 
build root last I saw) restarted. This is pretty unworkable.


https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672



I kicked this off again to see if the ARM builder kernel changes help. 
I'm waiting on VTK to complete to do a rebuild of PCL.


Rich


Well, that build unfortunately failed due to gcc getting killed (the 
usual arm OOM symptom).  Looks like another try has been started.  If 
that fails I think I'll need to bump down ncpus yet more (already 
dropped from 5 to 3).



There are other things you can look in to to try to reduce gcc memory 
usage.  For instance, PCL modifies the %optflags[1] to reduce the 
debuginfo level and turn off -pipe.


Rich

[1] https://src.fedoraproject.org/rpms/pcl/blob/rawhide/f/pcl.spec#_107
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Orion Poplawski]

On 8/18/21 2:23 PM, Rich Mattes wrote:

On 8/17/21 10:21 AM, Orion Poplawski wrote:

On 8/14/21 10:19 AM, Kevin Fenzi wrote:

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
Have there been any recent changes to the arm (32bit) builders?  It 
seems
like I'm having much more issues there with builds likely running 
out of

memory or similar.


Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).

Anyhow, if anyone has any ideas, let me know.

kevin


Looks like the vtk build just as it was about to finish (after 11+ 
hours - had completed extracting debuginfo and was on to checking the 
build root last I saw) restarted. This is pretty unworkable.


https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672



I kicked this off again to see if the ARM builder kernel changes help. 
I'm waiting on VTK to complete to do a rebuild of PCL.


Rich


Well, that build unfortunately failed due to gcc getting killed (the 
usual arm OOM symptom).  Looks like another try has been started.  If 
that fails I think I'll need to bump down ncpus yet more (already 
dropped from 5 to 3).



--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems  720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Dennis Gilmore]

On Wed, Aug 18, 2021 at 3:11 PM Florian Weimer  wrote:
>
> * Dennis Gilmore:
>
> > We intentionally never looked at enabling that and always had no plans
> > to support multi-lib on Arm
>
> It's not multilib.  Buildroots aren't multilib.
>
> I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
> 32-bit Arm kernels.  It's very much a dead end.
>
> Debian uses 64-bit kernels:
>
> | Package: glibc
> | Version: 2.31-16
> | Source Version: 2.31-16
> | Distribution: sid
> | Machine Architecture: arm64
> | Host Architecture: armhf
> | Build Architecture: armhf
> | Build Type: any

There is some magic that's needed for multi-lib that will be needed
for the AArch64 host's rpm to install the rpms into a 32bit chroot.

Dennis

> 
>
> Others use cross-compilers.
>
> Thanks,
> Florian
>
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Rich Mattes]

On 8/17/21 10:21 AM, Orion Poplawski wrote:

On 8/14/21 10:19 AM, Kevin Fenzi wrote:

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
Have there been any recent changes to the arm (32bit) builders?  It 
seems

like I'm having much more issues there with builds likely running out of
memory or similar.


Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).

Anyhow, if anyone has any ideas, let me know.

kevin


Looks like the vtk build just as it was about to finish (after 11+ hours 
- had completed extracting debuginfo and was on to checking the build 
root last I saw) restarted. This is pretty unworkable.


https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672



I kicked this off again to see if the ARM builder kernel changes help. 
I'm waiting on VTK to complete to do a rebuild of PCL.


Rich
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Dennis Gilmore:

> We intentionally never looked at enabling that and always had no plans
> to support multi-lib on Arm

It's not multilib.  Buildroots aren't multilib.

I'm pretty sure no one but Fedora is building 32-bit Arm binaries on
32-bit Arm kernels.  It's very much a dead end.

Debian uses 64-bit kernels:

| Package: glibc
| Version: 2.31-16
| Source Version: 2.31-16
| Distribution: sid
| Machine Architecture: arm64
| Host Architecture: armhf
| Build Architecture: armhf
| Build Type: any



Others use cross-compilers.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Justin Forbes]

On Wed, Aug 18, 2021 at 1:12 PM Kevin Fenzi  wrote:
>
> On Tue, Aug 17, 2021 at 10:32:59AM -0500, Justin Forbes wrote:
> > On Tue, Aug 17, 2021 at 10:27 AM Caolán McNamara  wrote:
> > >
> > > On Tue, 2021-08-17 at 08:21 -0600, Orion Poplawski wrote:
> > > > Looks like the vtk build just as it was about to finish (after 11+
> > > > hours ...) restarted. This is pretty unworkable.
> > > >
> > > > https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672
> > >
> > > Yeah, for libreoffice my last success took over 34 hours, it's very
> > > painful.
> > >
> > > https://koji.fedoraproject.org/koji/buildinfo?buildID=1817735
> > > Started:   Sat, 14 Aug 2021 10:36:02 UTC
> > > Completed: Sun, 15 Aug 2021 21:05:53 UTC
> > >
> > > while back in June a build was taking 8 hours, like:
> > > https://koji.fedoraproject.org/koji/buildinfo?buildID=1757639
> > > Started:   Wed, 02 Jun 2021 12:04:23 UTC
> > > Completed: Wed, 02 Jun 2021 20:14:44 UTC
> >
> >
> > Kernels are in the same boat, and I build 1-4 of them per day.
>
> Yeah, I definitely agree thats not tenable.
>
> Anyhow, I have rebooted them all to a previous lpae kernel that I
> thought they were pretty stable on. Please look and see if it's better
> now.
>

After the update and reboot, every kernel build in flight (4 of them),
all finished up at roughly the same time, a reasonable timeframe from
the reboots. All kernel builds since then have been without issue.  I
would say they are good.

Justin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Wed, Aug 18, 2021 at 10:27:42AM +0200, Miro Hrončok wrote:
> On 16. 08. 21 9:00, Miro Hrončok wrote:
> > On 16. 08. 21 0:29, Kevin Fenzi wrote:
> > > On Sun, Aug 15, 2021 at 11:43:48AM +0200, Miro Hrončok wrote:
> > > > On 14. 08. 21 18:19, Kevin Fenzi wrote:
> > > > > It makes me wonder if we should consider letting 32bit arm go...
> > > > > (insert pitchforks and torches).
> > > > 
> > > > Let's propose it and see what people say? As a package maintainer, I 
> > > > would
> > > > certainly appreciate this.
> > > > 
> > > > I can draft a change proposal next week.
> > > 
> > > How about we give time for the iot and arm folks who likely have not
> > > seen this yet to chime in. :)
> > 
> > I was planning to ask them directly.
> 
> IoT: https://pagure.io/fedora-iot/issue/47
> 
> ARM: 
> https://lists.fedoraproject.org/archives/list/a...@lists.fedoraproject.org/thread/YC2LYBJSFKDAVBUJAIFQCCBS5VLW5TUB/

FYI, I also asked in the weekly arm status meeting: 

https://meetbot.fedoraproject.org/teams/fedora_arm_and_aarch64_status_meeting/fedora_arm_and_aarch64_status_meeting.2021-08-17-15.00.log.html

TLDR version: 
...
15:36:40  So, in the devel list thread where people were mad about that 
oom / kojid/ lpae thing it was brought up that perhaps we 
shoud look at dropping 32bit arm. I get the sense personally that it's too 
soon, but do we have any idea how popular 32bit arm stuff is now? 
is iot using it for anything? or is there other big uses that should be known? 
Or has everything moved pretty solidly to 64 bit now?
15:38:42  nirik: I've had some discussions about planning to deprecate 
32-bit, largely coming from the lack of hardware on the horizon.
15:38:48  yes, we have an IoT image
15:39:03  The flock presentation had a significant number of armv7 
installs.
15:39:07  it is being discussed but there's still some interest in 
it
15:39:13  I think we would want to give it a long notice... ie, not just 
decide and next release... but think a few ahead...
15:39:31  right and I've been thinking ahead already about some of 
this
15:40:03  in the general community there's still a lot of arm32 
devices around that get used
15:40:07  ok, fair enough. Just wanted to get a sense of usage
15:40:14  and they're *cheap*
15:40:34  I personally think the lift on armv7 is going to get greater 
since !debian has basically dropped all 32-bit targets (eg x86).
...

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Tue, Aug 17, 2021 at 10:32:59AM -0500, Justin Forbes wrote:
> On Tue, Aug 17, 2021 at 10:27 AM Caolán McNamara  wrote:
> >
> > On Tue, 2021-08-17 at 08:21 -0600, Orion Poplawski wrote:
> > > Looks like the vtk build just as it was about to finish (after 11+
> > > hours ...) restarted. This is pretty unworkable.
> > >
> > > https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672
> >
> > Yeah, for libreoffice my last success took over 34 hours, it's very
> > painful.
> >
> > https://koji.fedoraproject.org/koji/buildinfo?buildID=1817735
> > Started:   Sat, 14 Aug 2021 10:36:02 UTC
> > Completed: Sun, 15 Aug 2021 21:05:53 UTC
> >
> > while back in June a build was taking 8 hours, like:
> > https://koji.fedoraproject.org/koji/buildinfo?buildID=1757639
> > Started:   Wed, 02 Jun 2021 12:04:23 UTC
> > Completed: Wed, 02 Jun 2021 20:14:44 UTC
> 
> 
> Kernels are in the same boat, and I build 1-4 of them per day.

Yeah, I definitely agree thats not tenable. 

Anyhow, I have rebooted them all to a previous lpae kernel that I
thought they were pretty stable on. Please look and see if it's better
now. 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Tue, Aug 17, 2021 at 10:51:13AM -0500, Bruno Wolff III wrote:
> On Sat, Aug 14, 2021 at 09:19:55 -0700,
>  Kevin Fenzi  wrote:
> > 
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> > 
> > basically OOM kills kojid, which restarts kojid, which restarts the
> > build, which kills kojid, etc...
> > 
> > I've tried all kinds of things here, but haven't been able to find any
> > way to make it work. Arm folks can't duplicate it on non koji builders.
> > I suspect the number of people using lpae on 32bit arm is... low.
> > We could just go back to non lpae, but that breaks building some other
> > packages (llvm fails to build for example).
> 
> If you disabled over commit, would that end up having better failures when
> there wasn't enough memory?

I'm not sure. :) That might be worth a try I guess... 

Will see if the current kernel downgrade doesn't work I can try that. 

Thanks for the idea!

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Mon, Aug 16, 2021 at 07:55:17AM +0200, Florian Weimer wrote:
> * Kevin Fenzi:
> 
> > On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:
> >> * Kevin Fenzi:
> >> 
> >> > Yes. They were mistakenly running the normal kernel (so they had ~3GB
> >> > memory available). I moved them back to the lpae kernel (so they see
> >> > 40GB memory), but this causes 
> >> >
> >> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> >> >
> >> > basically OOM kills kojid, which restarts kojid, which restarts the
> >> > build, which kills kojid, etc... 
> >> 
> >> Why aren't the builders running 64-bit kernels, like we do for x86-64?
> >
> > This is not a configuration that I think we support in any way.
> 
> Who is “we”?

The Fedora Project?
Do we have any kind of documentation on how to install or set this up?
Does anyone test it? 
If I ask questions or have problems with it, will anyone be able to
answer? 

I think the answers are: no, no, no. 

> I expect that 64-bit kernel bugs will get more attention upstream.

I'm sure. 

> > At least rpm rejects trying to install a aarch64 kernel on a 32bit
> > userspace.
> 
> The host (including kojid) should probably be 64-bit, and only the
> chroot 32-bit.  If that doesn't work, it's an RPM bug/missing feature.

Sure, but that doesn't help me much right now. 

Anyhow, will talk to arm folks and see what they suggest...

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

So, I have just rebooted all the 32bit arm builders with 
5.12.19-300.fc34.armv7hl+lpae

Please let me know if you still see any 'restarts' with this kernel /
after this time. 

kevin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Sun, Aug 15, 2021 at 06:43:29PM -0400, Demi Marie Obenour wrote:
...snip...
> 
> Mark kojid as non-killable by setting its OOM score to -1000?  Adding
> swap might also help, but then the build is by no means guaranteed to
> finish in a reasonable amount of time.

Doing this causes the vm to completely lock up and become unresponsive.
;) 

It's worth noting that the machines are not even close to being out of
memory here (at least from the general pool). It's likely some specific
"kind" of memory or way lpae accounts for things. :( 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Dennis Gilmore]

On Mon, Aug 16, 2021 at 12:55 AM Florian Weimer  wrote:
>
> * Kevin Fenzi:
>
> > On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:
> >> * Kevin Fenzi:
> >>
> >> > Yes. They were mistakenly running the normal kernel (so they had ~3GB
> >> > memory available). I moved them back to the lpae kernel (so they see
> >> > 40GB memory), but this causes
> >> >
> >> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> >> >
> >> > basically OOM kills kojid, which restarts kojid, which restarts the
> >> > build, which kills kojid, etc...
> >>
> >> Why aren't the builders running 64-bit kernels, like we do for x86-64?
> >
> > This is not a configuration that I think we support in any way.
>
> Who is “we”?
>
> I expect that 64-bit kernel bugs will get more attention upstream.
>
> > At least rpm rejects trying to install a aarch64 kernel on a 32bit
> > userspace.
>
> The host (including kojid) should probably be 64-bit, and only the
> chroot 32-bit.  If that doesn't work, it's an RPM bug/missing feature.

We intentionally never looked at enabling that and always had no plans
to support multi-lib on Arm

Dennis

> Thanks,
> Florian
> ___
> devel mailing list -- devel@lists.fedoraproject.org
> To unsubscribe send an email to devel-le...@lists.fedoraproject.org
> Fedora Code of Conduct: 
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> Do not reply to spam on the list, report it: 
> https://pagure.io/fedora-infrastructure
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Miro Hrončok]

On 16. 08. 21 9:00, Miro Hrončok wrote:

On 16. 08. 21 0:29, Kevin Fenzi wrote:

On Sun, Aug 15, 2021 at 11:43:48AM +0200, Miro Hrončok wrote:

On 14. 08. 21 18:19, Kevin Fenzi wrote:

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).


Let's propose it and see what people say? As a package maintainer, I would
certainly appreciate this.

I can draft a change proposal next week.


How about we give time for the iot and arm folks who likely have not
seen this yet to chime in. :)


I was planning to ask them directly.


IoT: https://pagure.io/fedora-iot/issue/47

ARM: 
https://lists.fedoraproject.org/archives/list/a...@lists.fedoraproject.org/thread/YC2LYBJSFKDAVBUJAIFQCCBS5VLW5TUB/


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Bruno Wolff III]

On Sat, Aug 14, 2021 at 09:19:55 -0700,
 Kevin Fenzi  wrote:


https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).


If you disabled over commit, would that end up having better failures 
when there wasn't enough memory?

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Jeremy Linton:

> Hi,
>
> On 8/17/21 2:06 AM, Florian Weimer wrote:
>> * Jeremy Linton:
>> 
>>> That said, there as you mention various rpm/package build/etc problems
>>> caused by `uname -m` returning armv8.
>> Is this something that can be changed with setarch?  It works on
>> other
>> architectures (at least on x86 and POWER).
>
> Beyond the defaults? I don't know how to pull that off. You get three
> differing unames on a armv8 machine that supports aarch64 and aarch32.
>
> 32-bit kernel armv7l
> 64-bit kernel 64-bit process, aarch64
> 64-bit kernel 32-bit process (or via setarch), armv8l
>
> Trying to force the 64-bit kernel to armv7l via compat/etc just
> results in the armv8 moniker. This is probably fixable/etc, or there
> is a way I don't know about off hand.

On x86-64, it similar, i686 cannot be downgraded:

# setarch i586 uname -m
i686

This makes me wonder if we have to fix RPM etc. instead to cope with
armv8l.  Maybe it's working already.

I do think that switching to 64-bit kernels on builders is a
prerequisite for continuing to build armhfp.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Jeremy Linton]

Hi,

On 8/17/21 2:06 AM, Florian Weimer wrote:

* Jeremy Linton:


That said, there as you mention various rpm/package build/etc problems
caused by `uname -m` returning armv8.


Is this something that can be changed with setarch?  It works on other
architectures (at least on x86 and POWER).


Beyond the defaults? I don't know how to pull that off. You get three 
differing unames on a armv8 machine that supports aarch64 and aarch32.


32-bit kernel armv7l
64-bit kernel 64-bit process, aarch64
64-bit kernel 32-bit process (or via setarch), armv8l

Trying to force the 64-bit kernel to armv7l via compat/etc just results 
in the armv8 moniker. This is probably fixable/etc, or there is a way I 
don't know about off hand.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Justin Forbes]

On Tue, Aug 17, 2021 at 10:27 AM Caolán McNamara  wrote:
>
> On Tue, 2021-08-17 at 08:21 -0600, Orion Poplawski wrote:
> > Looks like the vtk build just as it was about to finish (after 11+
> > hours ...) restarted. This is pretty unworkable.
> >
> > https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672
>
> Yeah, for libreoffice my last success took over 34 hours, it's very
> painful.
>
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1817735
> Started:   Sat, 14 Aug 2021 10:36:02 UTC
> Completed: Sun, 15 Aug 2021 21:05:53 UTC
>
> while back in June a build was taking 8 hours, like:
> https://koji.fedoraproject.org/koji/buildinfo?buildID=1757639
> Started:   Wed, 02 Jun 2021 12:04:23 UTC
> Completed: Wed, 02 Jun 2021 20:14:44 UTC


Kernels are in the same boat, and I build 1-4 of them per day.

Justin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Caolán McNamara]

On Tue, 2021-08-17 at 08:21 -0600, Orion Poplawski wrote:
> Looks like the vtk build just as it was about to finish (after 11+
> hours ...) restarted. This is pretty unworkable.
> 
> https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672

Yeah, for libreoffice my last success took over 34 hours, it's very
painful.

https://koji.fedoraproject.org/koji/buildinfo?buildID=1817735
Started:   Sat, 14 Aug 2021 10:36:02 UTC
Completed: Sun, 15 Aug 2021 21:05:53 UTC

while back in June a build was taking 8 hours, like:
https://koji.fedoraproject.org/koji/buildinfo?buildID=1757639
Started:   Wed, 02 Jun 2021 12:04:23 UTC
Completed: Wed, 02 Jun 2021 20:14:44 UTC
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Orion Poplawski]

On 8/14/21 10:19 AM, Kevin Fenzi wrote:

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:

Have there been any recent changes to the arm (32bit) builders?  It seems
like I'm having much more issues there with builds likely running out of
memory or similar.


Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).

Anyhow, if anyone has any ideas, let me know.

kevin


Looks like the vtk build just as it was about to finish (after 11+ hours 
- had completed extracting debuginfo and was on to checking the build 
root last I saw) restarted. This is pretty unworkable.


https://koji.fedoraproject.org/koji/taskinfo?taskID=73984672


--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems  720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Colin Walters:

> On Sun, Aug 15, 2021, at 6:43 PM, Demi Marie Obenour wrote:
>> 
>> Mark kojid as non-killable by setting its OOM score to -1000?  Adding
>> swap might also help, but then the build is by no means guaranteed to
>> finish in a reasonable amount of time.
>
> If Koji wasn't a clustered container system itself, but just deferred
> to e.g. Kubernetes, then both problems are solved for free - there's
> extensive support for using cgroups to correctly protect the control
> plane (kubelet, etc.) versus user workloads, *and* running 32 bit
> containers on a 64 bit host is obviously supported.

Not sure about the obviously supported part.  Are Kubernetes maintainers
really interested in maintaining configuration code for 32-bit
containers?

Does Kubernetes even offer batch processing capabilities (of the form
tarball in, tarball out) without cluster administrator privileges?

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Jeremy Linton:

> That said, there as you mention various rpm/package build/etc problems
> caused by `uname -m` returning armv8.

Is this something that can be changed with setarch?  It works on other
architectures (at least on x86 and POWER).

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Jeremy Linton]

Hi,

On 8/16/21 12:55 AM, Florian Weimer wrote:

* Kevin Fenzi:


On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:

* Kevin Fenzi:


Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...


Why aren't the builders running 64-bit kernels, like we do for x86-64?


This is not a configuration that I think we support in any way.


Who is “we”?

I expect that 64-bit kernel bugs will get more attention upstream.


At least rpm rejects trying to install a aarch64 kernel on a 32bit
userspace.


The host (including kojid) should probably be 64-bit, and only the
chroot 32-bit.  If that doesn't work, it's an RPM bug/missing feature.


OS wise, 32-bit containers/etc work fine on 64-bit fedora kernels. This 
also solves the problem that server grade HW with 32-bit guest (EL1+) 
support is becoming rarer (basically only older A72 based platforms now) 
and would provide a path forward on more recent Gravaton2, Ampere, etc 
platforms.



That said, there as you mention various rpm/package build/etc problems 
caused by `uname -m` returning armv8.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Colin Walters]

On Sun, Aug 15, 2021, at 6:43 PM, Demi Marie Obenour wrote:
> 
> Mark kojid as non-killable by setting its OOM score to -1000?  Adding
> swap might also help, but then the build is by no means guaranteed to
> finish in a reasonable amount of time.

If Koji wasn't a clustered container system itself, but just deferred to e.g. 
Kubernetes, then both problems are solved for free - there's extensive support 
for using cgroups to correctly protect the control plane (kubelet, etc.) versus 
user workloads, *and* running 32 bit containers on a 64 bit host is obviously 
supported.
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Orion Poplawski]

On 8/15/21 3:41 AM, Miro Hrončok wrote:
> On 15. 08. 21 4:13, Orion Poplawski wrote:
>> Or perhaps at least a statement that support for it is "best effort" only to
>> make it more acceptable to ExcludeArch it on some packages.
> 
> Due to the nature of our dependency chain, this would be just slower version
> of not including it. E.g. we have trouble building Python and if we
> ExcludeArch it, the distro is not working any more.
> 

Well, yes, for python it either makes sense to support arm 32-bit or just drop
it entirely from the distribution.  For 3D visualization libraries with a
relatively small dependency chain ExcludeArch seems more viable.

-- 
Orion Poplawski
IT Systems Manager 720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Miro Hrončok]

On 16. 08. 21 0:29, Kevin Fenzi wrote:

On Sun, Aug 15, 2021 at 11:43:48AM +0200, Miro Hrončok wrote:

On 14. 08. 21 18:19, Kevin Fenzi wrote:

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).


Let's propose it and see what people say? As a package maintainer, I would
certainly appreciate this.

I can draft a change proposal next week.


How about we give time for the iot and arm folks who likely have not
seen this yet to chime in. :)


I was planning to ask them directly.

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Neal Gompa:

> ARM is the only remaining non-embedded 32-bit architecture in common use.

Where do you see 32-bit Arm being used in a non-embedded way?  With
“non-embedded” I mean use for general-purpose computation, where the end
user installs software of their own choice, and not some fixed set of
applications.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Kevin Fenzi:

> On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:
>> * Kevin Fenzi:
>> 
>> > Yes. They were mistakenly running the normal kernel (so they had ~3GB
>> > memory available). I moved them back to the lpae kernel (so they see
>> > 40GB memory), but this causes 
>> >
>> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
>> >
>> > basically OOM kills kojid, which restarts kojid, which restarts the
>> > build, which kills kojid, etc... 
>> 
>> Why aren't the builders running 64-bit kernels, like we do for x86-64?
>
> This is not a configuration that I think we support in any way.

Who is “we”?

I expect that 64-bit kernel bugs will get more attention upstream.

> At least rpm rejects trying to install a aarch64 kernel on a 32bit
> userspace.

The host (including kojid) should probably be 64-bit, and only the
chroot 32-bit.  If that doesn't work, it's an RPM bug/missing feature.

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Neal Gompa]

On Sun, Aug 15, 2021 at 6:44 PM Demi Marie Obenour
 wrote:
>
> On 8/14/21 12:19 PM, Kevin Fenzi wrote:
> > On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
> >> Have there been any recent changes to the arm (32bit) builders?  It seems
> >> like I'm having much more issues there with builds likely running out of
> >> memory or similar.
> >
> > Yes. They were mistakenly running the normal kernel (so they had ~3GB
> > memory available). I moved them back to the lpae kernel (so they see
> > 40GB memory), but this causes
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> >
> > basically OOM kills kojid, which restarts kojid, which restarts the
> > build, which kills kojid, etc...
>
> Mark kojid as non-killable by setting its OOM score to -1000?  Adding
> swap might also help, but then the build is by no means guaranteed to
> finish in a reasonable amount of time.
>
> > I've tried all kinds of things here, but haven't been able to find any
> > way to make it work. Arm folks can't duplicate it on non koji builders.
> > I suspect the number of people using lpae on 32bit arm is... low.
> > We could just go back to non lpae, but that breaks building some other
> > packages (llvm fails to build for example).
> >
> > It makes me wonder if we should consider letting 32bit arm go...
> > (insert pitchforks and torches).
> >
> > Anyhow, if anyone has any ideas, let me know.
> >
> > kevin
>
> If Fedora wants to keep supporting 32-bit platforms, I believe the only
> reasonable solution is cross-compilation.  32-bit systems are nowadays
> almost always embedded, and everyone cross-compiles in the embedded
> world.
>

ARM is the only remaining non-embedded 32-bit architecture in common use.



-- 
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Demi Marie Obenour]

On 8/14/21 12:19 PM, Kevin Fenzi wrote:
> On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
>> Have there been any recent changes to the arm (32bit) builders?  It seems
>> like I'm having much more issues there with builds likely running out of
>> memory or similar.
> 
> Yes. They were mistakenly running the normal kernel (so they had ~3GB
> memory available). I moved them back to the lpae kernel (so they see
> 40GB memory), but this causes 
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> 
> basically OOM kills kojid, which restarts kojid, which restarts the
> build, which kills kojid, etc... 

Mark kojid as non-killable by setting its OOM score to -1000?  Adding
swap might also help, but then the build is by no means guaranteed to
finish in a reasonable amount of time.

> I've tried all kinds of things here, but haven't been able to find any
> way to make it work. Arm folks can't duplicate it on non koji builders. 
> I suspect the number of people using lpae on 32bit arm is... low. 
> We could just go back to non lpae, but that breaks building some other
> packages (llvm fails to build for example).
> 
> It makes me wonder if we should consider letting 32bit arm go...
> (insert pitchforks and torches). 
> 
> Anyhow, if anyone has any ideas, let me know. 
> 
> kevin

If Fedora wants to keep supporting 32-bit platforms, I believe the only
reasonable solution is cross-compilation.  32-bit systems are nowadays
almost always embedded, and everyone cross-compiles in the embedded
world.

Sincerely,

Demi


OpenPGP_0xB288B55FFF9C22C1.asc
Description: OpenPGP public key


OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Sun, Aug 15, 2021 at 11:43:48AM +0200, Miro Hrončok wrote:
> On 14. 08. 21 18:19, Kevin Fenzi wrote:
> > It makes me wonder if we should consider letting 32bit arm go...
> > (insert pitchforks and torches).
> 
> Let's propose it and see what people say? As a package maintainer, I would
> certainly appreciate this.
> 
> I can draft a change proposal next week.

How about we give time for the iot and arm folks who likely have not
seen this yet to chime in. :) 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:
> * Kevin Fenzi:
> 
> > Yes. They were mistakenly running the normal kernel (so they had ~3GB
> > memory available). I moved them back to the lpae kernel (so they see
> > 40GB memory), but this causes 
> >
> > https://bugzilla.redhat.com/show_bug.cgi?id=1920183
> >
> > basically OOM kills kojid, which restarts kojid, which restarts the
> > build, which kills kojid, etc... 
> 
> Why aren't the builders running 64-bit kernels, like we do for x86-64?

This is not a configuration that I think we support in any way. 

At least rpm rejects trying to install a aarch64 kernel on a 32bit
userspace. 

> As far as I understand it, there is both kernel and CPU support for that
> (on CPUs that support 32-bit at all).

If it can be made to work that would be great, but my understanding is
that it does not.

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Sat, Aug 14, 2021 at 12:57:40PM -0400, Frank Ch. Eigler wrote:
> Kevin Fenzi  writes:
> 
> > It makes me wonder if we should consider letting 32bit arm go...
> > (insert pitchforks and torches). 
> 
> ... or go back to an F32 kernel?

At this point it would have a gillion CVE's... but also I am not sure it
would work due to the change from direct boot to uefi booting. 

Might be worth trying tho... 

kevin


signature.asc
Description: PGP signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Florian Weimer]

* Kevin Fenzi:

> Yes. They were mistakenly running the normal kernel (so they had ~3GB
> memory available). I moved them back to the lpae kernel (so they see
> 40GB memory), but this causes 
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1920183
>
> basically OOM kills kojid, which restarts kojid, which restarts the
> build, which kills kojid, etc... 

Why aren't the builders running 64-bit kernels, like we do for x86-64?
As far as I understand it, there is both kernel and CPU support for that
(on CPUs that support 32-bit at all).

Thanks,
Florian
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Fabio Valentini]

On Sun, Aug 15, 2021 at 11:44 AM Miro Hrončok  wrote:
>
> On 14. 08. 21 18:19, Kevin Fenzi wrote:
> > It makes me wonder if we should consider letting 32bit arm go...
> > (insert pitchforks and torches).
>
> Let's propose it and see what people say? As a package maintainer, I would
> certainly appreciate this.
>
> I can draft a change proposal next week.

I think there's some variables we need to consider here, for example:

- Which 32-bit ARM hardware does Fedora currently support, and how old
are those boards? Can we drop support for those with, let's say,
Fedora 36?
- What's the impact on different editions, like on the IoT Edition? My
guess is that it's the one that's most impacted by dropping 32-bit ARM
support (though I may be wrong here).

If those two are no problem, then I'd probably support dropping
armv7hl from Fedora as well.
It's probably easier to do than dropping 32-bit x86, because there's
no multilib support on aarch64 in Fedora.

Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Miro Hrončok]

On 14. 08. 21 18:19, Kevin Fenzi wrote:

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).


Let's propose it and see what people say? As a package maintainer, I would 
certainly appreciate this.


I can draft a change proposal next week.

--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Miro Hrončok]

On 15. 08. 21 4:13, Orion Poplawski wrote:
Or perhaps at least a statement that support for it is "best effort" only to 
make it more acceptable to ExcludeArch it on some packages.


Due to the nature of our dependency chain, this would be just slower version of 
not including it. E.g. we have trouble building Python and if we ExcludeArch 
it, the distro is not working any more.


--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Vitaly Zaitsev via devel]

On 14/08/2021 20:29, Jeff Law wrote:
Letting 32bit arm go would have my support.  I suspect it's less and 
less interesting as a platform every day and it causes nothing but 
headaches.


+1 for dropping armv7.

--
Sincerely,
  Vitaly Zaitsev (vit...@easycoding.org)
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Orion Poplawski]

On 8/14/21 12:29 PM, Jeff Law wrote:



On 8/14/2021 10:19 AM, Kevin Fenzi wrote:

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
Have there been any recent changes to the arm (32bit) builders?  It 
seems

like I'm having much more issues there with builds likely running out of
memory or similar.

Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).

Anyhow, if anyone has any ideas, let me know.
Letting 32bit arm go would have my support.  I suspect it's less and 
less interesting as a platform every day and it causes nothing but 
headaches.


Jeff


I would certainly be in favor of it.  Many of the large C++ based 
packages I maintain take forever to build on it (and are now failing 
apparently due to various platform issues).


Or perhaps at least a statement that support for it is "best effort" 
only to make it more acceptable to ExcludeArch it on some packages.


--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems  720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Jeff Law]

On 8/14/2021 10:19 AM, Kevin Fenzi wrote:

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:

Have there been any recent changes to the arm (32bit) builders?  It seems
like I'm having much more issues there with builds likely running out of
memory or similar.

Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders.
I suspect the number of people using lpae on 32bit arm is... low.
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches).

Anyhow, if anyone has any ideas, let me know.
Letting 32bit arm go would have my support.  I suspect it's less and 
less interesting as a platform every day and it causes nothing but 
headaches.


Jeff
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Frank Ch. Eigler]

Kevin Fenzi  writes:

> It makes me wonder if we should consider letting 32bit arm go...
> (insert pitchforks and torches). 

... or go back to an F32 kernel?

- FChE
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Kevin Fenzi]

On Fri, Aug 13, 2021 at 09:34:11PM -0600, Orion Poplawski wrote:
> Have there been any recent changes to the arm (32bit) builders?  It seems
> like I'm having much more issues there with builds likely running out of
> memory or similar.

Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes 

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc... 

I've tried all kinds of things here, but haven't been able to find any
way to make it work. Arm folks can't duplicate it on non koji builders. 
I suspect the number of people using lpae on 32bit arm is... low. 
We could just go back to non lpae, but that breaks building some other
packages (llvm fails to build for example).

It makes me wonder if we should consider letting 32bit arm go...
(insert pitchforks and torches). 

Anyhow, if anyone has any ideas, let me know. 

kevin
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Any recent changes to the arm builders?

[Orion Poplawski]

Have there been any recent changes to the arm (32bit) builders?  It 
seems like I'm having much more issues there with builds likely running 
out of memory or similar.


--
Orion Poplawski
he/him/his - surely the least important thing about me
Manager of NWRA Technical Systems  720-772-5637
NWRA, Boulder/CoRA Office FAX: 303-415-9702
3380 Mitchell Lane   or...@nwra.com
Boulder, CO 80301 https://www.nwra.com/



smime.p7s
Description: S/MIME Cryptographic Signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure