Re: F40 Change Proposal: Optimized Binaries for the AMD64 Architecture (System-Wide)

[Jeremy Linton]

Hi,

On 12/28/23 10:12, Aoife Moloney wrote:

Wiki -> 
https://fedoraproject.org/wiki/Changes/Optimized_Binaries_for_the_AMD64_Architecture

This is a proposed Change for Fedora Linux.
This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


== Summary ==

Additional paths will be inserted into the search path used for
executables on systems which have a compatible CPU.
Those additional paths will mirror the AMD64 "microarchitecture
levels" supported by the glibc-hwcaps mechanism: `x86-64-v2`,
`x86-64-v3`, `x86_64-v4`.
Systemd will be modified to insert the additional directories into the
`$PATH` environment variable (affecting all programs on the system)
and the equivalent internal mechanism in `systemd` (affecting what
executables are used by services).
Individual packages can provide optimized libraries via the


(trimming)


Glibc-hwcaps together with the new feature in systemd provide a
generic mechanism. It will be up to individual packages to actually
provide code which makes use of it. Individual package maintainers are
encouraged to benchmark their packages after recompilation, and
provide the optimized variants if useful. (I.e. the code in question
is measureably faster '''and''' the program is ran often enough for
this to make a difference.)

The Change Owners will implement the packaging changes for a few
packages while developing the general mechanism and will submit those
as pull requests. Other maintainers are asked to do the same for their
packages.

Optimized variants of programs and libraries MAY be packaged in a
separate subpackage. The general packaging rules should be applied,
i.e. a separate package or packages SHOULD be created if it is files
are large enough.

Available benchmark results [2,4] are narrow and not very convincing.
We should plan an evaluation of results after one release.  If it
turns out that the real gains are too small, we can scrap the effort.
On the other hand, we should also consider other architectures. For
example, microarchitecture levels `z{14,15}` for `s390x` or
`power{9,10}` for `ppc64le`. Other architectures are not included in
this Change Proposal to reduce its scope.


While I realize this is intended as an amd64 specific change, I'm 
interested in helping to assure the infrastructure is in place for 
similar testing of aarch64 packages, since we have similar problems.


There remains a desire to support arm v8.0 machines (ex: the rpi4) into 
the foreseeable future, but significant uplifts exist for some 
packages/workloads with newer arch revisions. In most cases those 
features are currently being enabled at runtime but have perf hits for 
operations that traditionally are just part of the instruction stream. 
Outline-atomics, which leverage the v8.2 atomics are one such example, 
but also SVE (vectors, used for crypto, regex, etc), and more recently 
the v8.8 CPY (memcpy) should all ideally be inline.


So, while some other organizations have moved their platform/OSs to a 
v8.2 baseline, it would be nice to have another fedora tool that can be 
utilized to remain perf competitive.






== Feedback ==


== Benefit to Fedora ==
The developers who are interested in this kind of optimization work
can perform it within Fedora, without having to build separate
repositories. The users who have the appropriate hardware will gain
performance benefits. Faster code is also more energy-efficient. The
change will be automatic and transparent to users.

Note that other distributions use higher microarchitecture levels. For
example RHEL 9 uses x86-64-v2 as the baseline, RHEL 10 will use
x86-64-v3, and other distros provide optimized variants (OpenSUSE,
Arch Linux). We implement the same change in Fedora in a way that is
scoped more narrowly, but should provide the same performance and
energy benefits.

== Scope ==
* Proposal owners:
** Extend systemd to set the executable search path using the same
criteria as the dynamic linker.
** Implement packaging changes for at least one package with a library
and at least one package with executables and submit this as pull
requests.
** Provide a pull request for the Packaging Guidelines to describe the
changes listed in Description above.

* Other developers:
** Do benchmarking and implement packaging changes for other packages
if beneficial.

* Release engineering: [https://pagure.io/releng/issue/11864 #11864]

* Policies and guidelines: TBD.

* Trademark approval: N/A (not needed for this Change)

* Alignment with Community Initiatives:


== Upgrade/compatibility impact ==
No impact.


== How To Test ==

* Use `/usr/bin/ld.so --help` to check which hwcaps are supported by the system.
* Install one or more packages which provide optimized code.
* Restart the system or re-login to reinitialize `$PATH`.
* Check that appropriate 

Re: F40 Change Proposal: Unified Kernel Support Phase Two (System-Wide)

[Jeremy Linton]

Hi,

On 12/18/23 06:41, Gerd Hoffmann wrote:

On Fri, Dec 15, 2023 at 02:03:27PM -0600, Jeremy Linton wrote:

Hi,


 Phase 2 goals 

* Add support for booting UKIs directly.
** Boot path is shim.efi -> UKI, without any boot loader (grub,
sd-boot) involved.


This is IMHO a mistake, the systemd-boot and UKI paths are the perfect time
to break with shim and require some form of actual fedora/whatever secure
boot key enrollment on the machine.


This is not going to fly.  There are too many cases where you simply
can't enroll fedora keys, so booting on machines with the MS 3rd party
certificate enrolled IMHO must continue to work.


I agree solving this for every possible machine combination is an 
intractable problem at the moment. But, the UKI use case, as I 
understand it, is a handful of hyperscalers and machines. Those 
organizations either participate in this community or we have 
engineering contacts at who can assist with cleaning it up.


And this isn't unheard of, poke around in a few machine's key database 
and you will find other distro's keys.





I agree that depending on MS signing exclusively is a problem.  I'd love
to see fedora signing as an option.  Given that EFI binaries can have
multiple signatures we could have shim.efi signed by both microsoft and
fedora.  Which would allow to enroll the fedora keys instead of the
microsoft keys (in case your platform offers that option) and everything
works as it did before, except that the machine would only accept
fedora-signed EFI binaries.

Problem #1 is we don't have that in fedora today.  Which btw is
different from rhel/centos, where we actually have a second,
distro-signed shim binary.  Not as useful as one binary with two
signatures, but better than nothing.


I'm talking about removing shim from the boot flow. The UKI would be 
signed with the fedora key same as would be done with shim in the boot 
path. The fedora public key is itself enrolled in the UEFI key db 
alongside the assortment of existing db entries, and the boot path would 
be UEFI->UKI. Alternatively, better instructions for putting specific 
machines into UEFI setup mode can be provided, and something like the 
key enroller being used for fedora/libvirt/qemu today is used to replace 
the key infrastructure (PK/KEK/etc) on a given machine. The argument 
against this has always been that it breaks multiboot, but that is not 
applicable here.



Frankly, none of this should be an issue for any machine that conforms 
to MS's requirements, as there is already a windows/everyone else boot 
switch to enable the keys needed to boot linux/shim vs the keys needed 
to boot modern windows versions.





Problem #2 is we don't have a signed system-boot binary.  Switching over
to use systemd-boot when this has changed should be easy.  The UKIs are
already placed in $ESP/EFI/Linux, according to the boot loader spec,
where systemd-boot would look for them.  So the kernel-install workflow
would need only minor changes.


I'm not sure that is strictly needed. Your example boot flow shim->UKI 
depends on the BDS as the boot selector. If that boot flow works for the 
end user, there isn't a need the systemd-boot loader itself. It does 
solve various problems, like boot selection and command line editing, 
and a few other things, but isn't otherwise necessary. Of course it too, 
would be signed with the fedora keys rather than the MS ones, and maybe 
it should be built without the shim + LoadImage/StartImage hacks.





Problem #3 is that apparently everything touching fedora secure boot
signing takes ages to make any progress.  One ticket I've looked at
recently (IIRC the one to enable secure boot signing for aarch64) was
roughly five years (!) old.  So I'm not going to make my change
proposals depend on any progress there.  But I'll happily file a
Phase #3 proposal once the problems outlined above are solved.


Right, but little of it has been strictly technical. Other distro's have 
signed their aarch64 shim/boot path. That isn't to say there haven't 
been plenty of technical things needing attention, but those have been 
in the, "this should be cleaned up" category rather than "it doesn't 
work" category.


But, your not really asking for more or less of the fedora infra with or 
without shim in the path. In both cases the UKI is signed with a fedora 
key. The difference is where the public key(s) gets enrolled.






whole UKI concept works at all. Put another way, there isn't really an
answer to a generic boots everywhere UKI at the movement unless one is
willing to create GB+ UKIs with every boot critical driver in existence,


Well, CoreOS actually does that.  They don't use UKIs specifically, but
they ship a generic initrd created on fedora infrastructure instead of
generating an host-specific initrd on the installed machine (with only
the drivers needed on the host included).

Last time I looked it was ~80 MB in size.


Well on x86, and a fair numb

Re: F40 Change Proposal: Unified Kernel Support Phase Two (System-Wide)

[Jeremy Linton]

Hi,

On 12/5/23 14:38, Aoife Moloney wrote:

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Improve support for unified kernels in Fedora.

== Owner ==
* Name: [[User:kraxel| Gerd Hoffmann]]
* Email: kra...@redhat.com

* Name: [[User:vittyvk| Vitaly Kuznetsov]]
* Email: vkuzn...@redhat.com


== Detailed Description ==
See [[ Changes/Unified_Kernel_Support_Phase_1 ]] for overview and Phase 1 goals.

 Phase 2 goals 

* Add support for booting UKIs directly.
** Boot path is shim.efi -> UKI, without any boot loader (grub,
sd-boot) involved.



This is IMHO a mistake, the systemd-boot and UKI paths are the perfect 
time to break with shim and require some form of actual fedora/whatever 
secure boot key enrollment on the machine. Shim's fundamentally 
backdooring the UEFI security infrastructure, and frankly some of what 
is being done is pretty sketchy and its somewhat amazing it hasn't 
broken by vendors cleaning up their UEFI implementations*. Furthermore, 
the dependency on MS signing shim is also strongly in the pragmatic but 
not idea category as well.


Some of the stronger reasons for using shim (MOK's and 3rd party binary 
modules, ex nvidia) really shouldn't be considered a problem for UKI 
based machines as the hardware profiles need to be restricted enough 
that the whole UKI concept works at all. Put another way, there isn't 
really an answer to a generic boots everywhere UKI at the movement 
unless one is willing to create GB+ UKIs with every boot critical driver 
in existence, at which point its probably worth revisiting the entire 
initramfs boot method. For example, the current method of: load a 
compressed filesystem, decompress it, and then pick out the needed 
pieces, switch root then free the initramfs is very inferior to a 
"sealed volume" method that can mount and read the fs directly without 
all the overhead of loading/decompressing/freeing a huge blob of unused 
data. Furthermore, UKI are largely just a stopgap to solve the lack of a 
manifest like system that can validate the executable and shared 
libraries in the initramfs itself. Nevermind the piles and piles of 
configuration options that end up in the initrd for every obscure boot 
method (ex: where will the iscsi authentication information be placed, 
surely not the the kernel command line)




* I would expect that the UEFI hardening to continue to the point where 
shim's antics are no longer allowed now that people are continuing to 
look at the weaknesses in the current vendors UEFI boot paths.



Thanks,






** The UEFI boot configuration will get an entry for each kernel installed.
** Newly installed kernels are configured to be booted once (via BootNext).
** Successful boot of the system will make the kernel update permanent
(update BootOrder).
* Enable UKIs for aarch64.
** Should be just flipping the switch, dependencies such as kernel
zboot support are merged.
* Add a UEFI-only cloud image variant which uses UKIs.
** Also suitable for being used in confidential VMs.
** Cover both x86_64 and aarch64.

 Related bugs 

* shim: remove dependency on grub2-efi-x64
([https://bugzilla.redhat.com/show_bug.cgi?id=2240989 buzilla
2240989])
* shim: handling of multiple lines in BOOT.CSV is inconsistent
([https://issues.redhat.com/browse/RHEL-10704 jira RHEL-10704],
[https://github.com/rhboot/shim/issues/554 github 554])
* anaconda: add support for
[https://www.freedesktop.org/wiki/Specifications/DiscoverablePartitionsSpec/
discoverable partitions]
([https://bugzilla.redhat.com/show_bug.cgi?id=2160074 bugzilla
2160074], [https://bugzilla.redhat.com/show_bug.cgi?id=2178043
bugzilla 2178043])
* dracut: do not create yet another initramfs for UKIs
([https://github.com/dracutdevs/dracut/pull/2521 github PR 2521])
* kernel: enable UKIs on aarch64
([https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2818 MR
2818])

== Feedback ==


== Benefit to Fedora ==
* Better secure boot support: the UKI initrd is covered by the signature.
* Better support for tpm measurements and confidential computing.
** measurements are more useful if we know what hashes to expect for the initrd.
** measurements are more useful without grub.efi in the boot path
(which measures each grub.cfg line processed).
* More robust boot process
** generating the initrd on the installed system is fragile

== Scope ==
* Proposal owners:
** updates for virt-firmware and uki-direct packages.
** enable UKIs on aarch64
([https://gitlab.com/cki-project/kernel-ark/-/merge_requests/2818 MR
2818]).
** prepare kickstart ([https://pagure.io/fedora-kickstarts.git Fedora
kickstarts]) changes for generating UKI enabled images.

* Other developers:
** installer/anaconda: implement discoverable partition support.
** bootloader/shim: fix bugs.
** Fedora 

Re: F40 Change Proposal: Unified Kernel Support Phase Two (System-Wide)

[Jeremy Linton]

Hi,

On 12/6/23 11:26, Vitaly Kuznetsov wrote:

Gerd Hoffmann  writes:


   Hi,


Does that mean that the Linux EFI boot code knows how to call back to
shim to get the certificates instead of reading the firmware directly?


No.  The linux efi stub doesn't need that.

shim.efi does:

   (a) Set efi variables, where the linux kernel can read the
   certificates from.  This works the same way for both traditional
   kernels and UKIs.

   (b) provide an efi protocol for bootloaders, which can be called by grub
   and systemd-boot to verify the signature for binaries they load
   (typically the linux kernel, but could also be fwupd.efi).


Note, the protocol is also used by systemd-stub (the base for UKIs) to
verify cmdline addons, see

commit 05c9f9c2517c54b98d55f08f8afa67c79be861e8
Author: Luca Boccassi 
Date:   Fri May 12 00:55:58 2023 +0100

 stub: allow loading and verifying cmdline addons

this AFAIU means that we also need shim in the boot chain if we want to
support these addons.


That is true, buts its also false. The LoadImage protocol is more than 
capable of doing exactly what that patch does (AFAIK), and using 
strictly the UEFI protocols for validation. Of course if you want to 
backdoor the process then you need to add shim into it, which is part of 
what that patch is doing.




--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: zlib-ng as a compat replacement for zlib

[Jeremy Linton]

Hi,

On 8/6/23 08:33, John Reiser wrote:

On 8/6/23 02:00, Peter Robinson wrote:

We tried to pull some of the optimisations in some time ago to the
Fedora package and they caused some issues with compatibility.


Please provide *any* documentation!  Such as: the dates the work was 
performed,

the participants, the nature of the issues, the "other side" of the problem
cases (the other packages, the use cases, etc.)


Waves, some of this was my fault.

example bug: https://bugzilla.redhat.com/show_bug.cgi?id=1582555
look at the zlib rpm history you will see things like:

https://src.fedoraproject.org/rpms/zlib/c/71a74f9c8684dd99c0e0657f53affb90a3ca3219?branch=rawhide

there were a few others that were ignored, or we reverted part of the 
original set of 5 or 6 patches. The original patches were aarch64 + NEON 
optimizations, but there were a number of issues around unittests in 
various packages that zipped something then validated the results 
against a known crc/hash/etc which then failed because the hash changed, 
the size changed, padding issues, the optimized code touched valid parts 
of the buffer and tripped buffer poisoning logic, etc.


Turns out zlib is old school byte oriented and any slight behavioral 
change can result in compatibility issues. The first obvious 
optimization is to increase the fetched word/matching sizes, which 
maintain binary compatibility with the zlib format/decompressor but 
results in buffer len/compressed size deltas.


Of course some of these were potentially the fault of the patches, but 
you have to decide between perf or compatibility when writing these, and 
if the goal is faster, then the compatibility gets sacrificed.


Bugzilla is taking its time retrieving some of the BZs that were closed 
without fixes. So you will have to search for them yourself.


In the end most of the patches were dropped the uplift wasn't worth the 
effort of maintaining them downstream, when the effort can be better 
spent getting a 10X uplift using a more modern compression 
implementation (ex zstd/lz4/lzo/etc) that isn't written with so many 
byte oriented assumptions.






___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/

List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Jeremy Linton]

Hi,

Thanks for looking at this, and sorry about the delay I was on PTO for a 
few days.


On 6/28/23 09:15, Richard W.M. Jones wrote:

On Thu, Jun 22, 2023 at 12:24:04PM -0500, Jeremy Linton wrote:

But, IMHO the largest change is moving the boot kernel/initrd to the
ESP, rather than the use of systemd-boot itself. Given the
filesystem layout remains the same outside of those two items and
the BLS entries, all the common tools (dracut/etc) seem to be able
to deal with it, and random other packages not manipulating
kernel/initrd images are behavior should not be affected anymore
than putting reFind/whatever in the boot path.


This change is likely to affect supermin, libguestfs, guestfs-tools &
virt-v2v, some BZs would be appreciated.


I have tested some of these, and opened:

supermin: Appears to have problem with the PE + compressed kernel image 
(although likely more a qemu issue) otherwise basic functionality works?


https://bugzilla.redhat.com/show_bug.cgi?id=137

libguestfs, largely a  placeholder bug:

https://bugzilla.redhat.com/show_bug.cgi?id=139


gestfs-tools, again a placeholder:

https://bugzilla.redhat.com/show_bug.cgi?id=140


and finally virt-v2v is confused by systemd-boot based images and cannot 
detect the bootloader.


https://bugzilla.redhat.com/show_bug.cgi?id=485


Thanks again,

jlinton
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

OT: Re: F40 Change: Privacy-preserving Telemetry for Fedora Workstation (System-Wide)

[Jeremy Linton]

On 7/10/23 13:16, Demi Marie Obenour wrote:

On 7/10/23 02:30, Vitaly Zaitsev via devel wrote:

On 10/07/2023 02:49, Demi Marie Obenour wrote:

QtWebEngine (used by Falkon) was a
month or more behind upstream Chromium last I checked.


Qt5QtWebEngine is an extremely vulnerable thing. It still uses Chromium
87.0[1].

Current Chromium version: 105.0.

[1]: https://wiki.qt.io/QtWebEngine/ChromiumVersions


In that case it should be removed from the distribution.  Can KDE
mail clients be built without QtWebEngine?  This would disable
HTML email support, but plain text mail might still work.


The problem isn't QtWebEngine, the latest Qt 6.X is using 108 according 
to the link above.


The problem seems to be that not everything has moved to the 6.x branch yet.

https://iskdeusingqt6.org/





More generally, WebKit is the only major browser engine with
upstream support for being embedded, so it is the only embedded
browser engine that is supportable security-wise.  Unfortunately,
it is also the least secure of the major browser engines on Linux
last I checked, and in particular is far behind Chromium.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F40 Change: Privacy-preserving Telemetry for Fedora Workstation (System-Wide)

[Jeremy Linton]

Hi,


On 7/6/23 11:10, Aoife Moloney wrote:

Important process note: we are experimenting with using Fedora

(trimming stuff because this proposal is huge)



We intend to deploy the Endless OS metrics system.
[https://blogs.gnome.org/wjjt/2023/07/05/endless-oss-privacy-preserving-metrics-system/
This blog post] contains a description of how the system works. We do
not plan to deploy the eos-phone-home component in Fedora.


So, the following is just _my_ opinion, don't read more than that into it:


Having finally had a chance to look at the list of collected metrics i'm 
a bit worried about just how much information is being/can be gathered 
by the project, as well as the frequency it is being gathered.


Personally, I think it would benefit fedora if questions such as "is 
anyone actually using this hardware/driver/package" could be answered. 
OTOH, the metrics presented above go far beyond that. I'm not sure why 
its necessary to know how many times, or how long a particular 
application is being used.





=== How will data collection be approved? ===

The proposal owners feel it is essential to ensure the Fedora
community has ultimate oversight over metrics collection. Community
control is required to maintain user trust. If this change proposal is
approved, then we'll need new policies and procedures to ensure
community oversight over metrics collection and ensure Fedora users
can be confident that our metrics collection does not violate their
privacy.


So, I would suggest that the intended metrics are included as part of 
this proposal as well as the interval, and that it wouldn't be changed 
without further community approval. Doing this would go a long way to 
convincing me, and likely others, that its not worth the effort to 
manually rip the entire subsystem out of fedora at the first chance on 
my machines.


If there is to be a "process" for changing them, then I think that needs 
to be documented here rather than hand waving it away too.




We can say "we would never collect personally-identifiable data" and
write software that really doesn't collect any such data, but this
alone will never be enough to ensure user confidence. We will need a
metrics collection policy that describes what sort of data may be
collected by Fedora (anonymous, non-invasive), and what sort of data
may not be collected. Such a policy does not exist currently. We will
also want to ensure the Fedora community has ultimate control over
which particular metrics are collected. One option is that each metric
to be collected should be separately approved by FESCo. Collection of
particular metrics in a particular data format is ultimately an
engineering decision, and therefore FESCo seems like an appropriate
approval point. Because FESCo members are elected regularly by the
Fedora community, this also provides the community with ultimate
control over metrics collection via the election process. But other
oversight and approval structures would work too.

=== What data might we collect? ===

We are not proposing to collect any of these particular metrics just
yet, because a process for Fedora community approval of metrics to be
collected does not yet exist. That said, in the interests of maximum
transparency, we wish to give you an idea of what sorts of metrics we
might propose to collect in the future.

One of the main goals of metrics collection is to analyze whether Red
Hat is achieving its goal to make Fedora Workstation the premier
developer platform for cloud software development. Accordingly, we
want to know things like which IDEs are most popular among our users,
and which runtimes are used to create containers using Toolbx.



IMHO, the data shouldn't be collected more frequently than every 6 
months or so, which allows each collection to be presented to the user, 
rather than having it just uploading the data in the background. Nor 
should it be tracking _user_ actions, which I would differentiate from 
machine state (bios machine type, RAM, installed packages, application 
crashes, failed suspend/resume, kinds of things).



But given course grained tracking, why isn't it part of server/IoT/etc 
as well, other than the current focus on gnome? Surely knowing that only 
one user is running $APPLICATION on a server is useful too.




Metrics can also be used to inform user interface design decisions.
For example, we want to collect the clickthrough rate of the
recommended software banners in GNOME Software to assess which banners
are actually useful to users. We also want to know how frequently
panels in gnome-control-center are visited to determine which panels
could be consolidated or removed, because there are other settings we
want to add, but our usability research indicates that the current
high quantity of settings panels already makes it difficult for users
to find commonly-used settings.


(trimming)


=== User control ===

A new metrics collection setting will be added to the privacy page in

Re: F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Jeremy Linton]

Hi,

On 6/22/23 12:28, Zbigniew Jędrzejewski-Szmek wrote:

https://fedoraproject.org/wiki/Changes/cleanup_systemd_install



== Detailed Description ==
As a first pass, the 'inst.sdboot' option already in anaconda should
work. As it stands, that replaces grub+shim with the systemd-boot
loader, and moves the kernel + initrd to the EFI system partition
(ESP).


In general, I think it's great to see this happening. Thank you!


It doesn't attempt to create unified kernel images, so the
existing `dnf update`, `kdumpctl`, and `make install` in a kernel
source directory should all work.


Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
invokes kernel-install which will create a UKI, if the configuration
specifies that. This sentence implies that 'make install' has some
issue with UKIs, but that is not true.


As clarification, I added a "." on the wiki version of the proposal so 
that its hopefully clearer that we aren't excluding UKIs from working 
with the usual tools.


Thanks,



/usr/bin/installkernel does a lot of stuff, most of it probably wrong.
We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
and then 'make install' would work without grubby.


The vast majority of this work has
been done, leaving only two action items, removing grubby from core,
and merging a shimming package (sdubby) into the fedora repos.


As I wrote before in the bug, what is really grubby and sdubby needed for?
Most of what grubby does is arcane archaic stuff that we don't need.
With grubby/sdubby we have an additional layer that adds complexity
and is very inflexible. Why can't we have Anaconda write the Boot Loader
Specification files directly or invoke kernel-install?

Zbyszek
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Jeremy Linton]

Hi,

On 6/23/23 05:27, Gerd Hoffmann wrote:

== Detailed Description ==
As a first pass, the 'inst.sdboot' option already in anaconda should
work. As it stands, that replaces grub+shim with the systemd-boot
loader, and moves the kernel + initrd to the EFI system partition
(ESP). It doesn't attempt to create unified kernel images, so the
existing `dnf update`, `kdumpctl`, and `make install` in a kernel
source directory should all work.


What is the plan for existing UKIs, for example kernel-uki-virt.rpm?

I'd expect they should work fine without extra work (i.e. try a
kickstart file with '-kernel' + '-kernel-core' + 'kernel-uki-virt' in
the file list).


Since no one else said anything, I will comment that I've not tested the 
uki package on top of systemd-boot + normal kernel flow. But the 
knowledge of where to put the kernels (/boot or ESP) is autodetected by 
the kernel-install script sorta automatically, and when grub isn't in 
the picture its happy to put them on the ESP alongside the BLS entries.


Presumably as you note below kernel-install should be able to do this 
for UKI kernels as well, removing the need to move them. Although the 
BLS type is set to "type1" and puts the kernel + initrd in 
/boot/efi/`cat /etc/machine-id`/`uname -r`/. So, setting the 
KERNEL_INSTALL_LAYOUT=uki should override the entries.srel and get them 
in the right place.


So, in theory it should work, but probably needs tweaking here/there.

Something else to test is what happens if both UKIs and traditional 
kernel + initrd exist on the ESP as the docs seem to want one or the 
other tied to "type1"/"type2"





[ Side node: Right now the kernel-uki-virt %postinstall just does a copy
   to /boot/efi/EFI/Linux where systemd-boot should find them just fine.
   After systemd v254 release we which brings some kernel-install
   improvements for UKIs we should be able to switch over to use
   kernel-install instead ].

take care,
   Gerd
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Jeremy Linton]

Hi,

On 6/22/23 12:28, Zbigniew Jędrzejewski-Szmek wrote:

https://fedoraproject.org/wiki/Changes/cleanup_systemd_install



== Detailed Description ==
As a first pass, the 'inst.sdboot' option already in anaconda should
work. As it stands, that replaces grub+shim with the systemd-boot
loader, and moves the kernel + initrd to the EFI system partition
(ESP).


In general, I think it's great to see this happening. Thank you!


Well thanks for looking at it :)




It doesn't attempt to create unified kernel images, so the
existing `dnf update`, `kdumpctl`, and `make install` in a kernel
source directory should all work.


Nitpick: 'make install' invokes /usr/bin/installkernel which (sometimes?)
invokes kernel-install which will create a UKI, if the configuration
specifies that. This sentence implies that 'make install' has some
issue with UKIs, but that is not true.

/usr/bin/installkernel does a lot of stuff, most of it probably wrong.
We could 'ln -sf --relative /usr/bin/kernel-install /usr/sbin/installkernel'
and then 'make install' would work without grubby.


Right, which is one of the 1/2 dozen things in the sdubby package, and 
with my limited testing appears to work fine.





The vast majority of this work has
been done, leaving only two action items, removing grubby from core,
and merging a shimming package (sdubby) into the fedora repos.


As I wrote before in the bug, what is really grubby and sdubby needed for?
Most of what grubby does is arcane archaic stuff that we don't need.
With grubby/sdubby we have an additional layer that adds complexity
and is very inflexible. Why can't we have Anaconda write the Boot Loader
Specification files directly or invoke kernel-install?


The critical missing piece of grubby has been the BLS entry 
manipulation, usually adding or stripping kernel boot parameters. Which 
at least in fedora, grubby's shell "API" is the defacto way for other 
programs (ex: kdumpctl) to perform this. That is what is being provided 
in the sdubby package, outside of symlinks and packaging stuff. So, your 
right there is a _LOT_ of seemingly redundant stuff in grubby if the 
system fits into the somewhat narrow set of criteria already required 
for systemd-boot. AKA basic UEFI secure boot path without 3rd party 
kernel modules/etc.


Maybe a better name would have been bls-entry-tools, since its largely 
just shimming between bootctl and everything else with sed. It might 
make sense from a fedora perspective if something like bootctl did this 
by default, but at least when it comes to output formats/etc its a bit 
ugly, and I don't really think the grubby "API" as such is being used by 
anyone outside of the fedora derived distros.


So, it makes sense if that functionality exists, that anaconda would use 
it, rather than attempting an install only version. Hence why this isn't 
in anaconda.


Thanks again,
jlinton
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F39 Change Proposal: Clean Systemd-boot installs (Self-Contained_

[Jeremy Linton]

Hi,

On 6/22/23 11:21, Daniel P. Berrangé wrote:

On Thu, Jun 22, 2023 at 04:59:38PM +0100, Aoife Moloney wrote:

https://fedoraproject.org/wiki/Changes/cleanup_systemd_install

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.

== Summary ==
Fedora default installs with a shim + grub bootloader on EFI
platforms, yet has been shipping systemd-boot in various forms for a
number of releases. There are a few howto's which describe how to
replace grub with systemd-boot with varying levels of functionality.
This should be easier with a formalized default method that can be
built upon. This proposal aims to complete the work started with
anaconda (inst.sdboot), kickstart (bootloader --sdboot) such that the
"everything" media can install a grub free machine.


snip


Beyond that there are various enhancements which can be made to remove
the /boot partition (leaving the EFI at /boot/efi), enrolling fedora
keys if the secure boot mode is "Setup", adding options to enable
shim+systemd-boot, assuring that there is a systemd-boot-signed
package, etc.


This is the $million question to - is there any proposal and/or agreement
by relevant maintainers, to start signing systemd-boot with Fedora SecureBoot
certs yet ? Without that, sdboot looks destined to remain a niche use case.


I think there remain a number of open questions, WRT how systemd-boot 
will settle down. In this case, AFAIK there is an open fedora infra work 
item for signing it, per zbyszek, who I've put on CC in case he misses 
this discussion. Of course due to the fact that not every platform has 
UEFI, and there are features in grub which aren't/shouldn't be 
duplicated in systemd-boot means that it will never be capable of 100% 
replacing grub on every platform.





If that's part of this proposal, then it feels more like a system-wide
change, than self-contained.


Well given it only affects a couple packages, and is optional, I flipped 
a coin and said its not really system wide, but given its in the boot 
path I do see the argument the other way as well.


But, IMHO the largest change is moving the boot kernel/initrd to the 
ESP, rather than the use of systemd-boot itself. Given the filesystem 
layout remains the same outside of those two items and the BLS entries, 
all the common tools (dracut/etc) seem to be able to deal with it, and 
random other packages not manipulating kernel/initrd images are behavior 
should not be affected anymore than putting reFind/whatever in the boot 
path.



Thanks,

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: U-Boot for x86 BIOS systems

[Jeremy Linton]

Hi,

On 5/22/23 06:01, Neal Gompa wrote:

On Mon, May 22, 2023 at 6:57 AM Gerd Hoffmann  wrote:


   Hi,


But could we use U-Boot to fill in this gap so these systems still
work? We'd then treat x86 like ARM (if no UEFI, use U-Boot UEFI).


How does u-boot handle EFI variables in that case?



Unless the variable store is disabled, I believe it's written to a
file? That seems to be how it works on ARM.


Usually with u-boot the runtime service is simply unavailable, which 
tends to break all kinds of things in subtle ways.


More recently though there have been patches merged to support runtime 
services in the uboot/uefi shim by trapping to lower level firmware 
which has always had runtime components available (unlike uboot), ex 
PSCI. So, if you have runtime variable storage with uboot its most 
likely being handled by Trusted Firmware A 
(https://www.trustedfirmware.org/).


That said TFA has all the same problems as EDK2 on some of these 
platforms WRT variable storage because its not generally possible to 
share something like an emmc/etc device between the Linux kernel and the 
firmware. Meaning the firmware needs to have a dedicated device, used to 
store the variables. Usually some SPI flash, but not always. I don't 
think anyone has managed to get a workable runtime service that writes 
the variable store to a file on a shared FS outside of maybe some tech 
demos. The problem is that the low level firmware need to understand and 
be able to read/write something like the ESP + USB/SD/MMC and then 
release it while keeping a volatile copy around long enough for a OS 
specific service to then take over persistence of the data. Its brittle 
and OS dependent as every single OS in existence then needs to have the 
service ported and active. The closest solutions I'm aware of are things 
like the older rpi3 EDK2 ports which wrote the variable store back to 
the firmware image itself on the ESP. But this doesn't really provide 
non-volatile runtime storage either because power loss/whatever causes 
updates done while the OS is running to be lost. It also technically 
violates the relevant specs as well since it fails to be "tamperproof" .


So "hardware bug", but once you fix all those bugs, it becomes 
straightforward just to toss TFA + EDK2 on it and run a full blown UEFI 
implementation providing all the infrastructure to make things like 
secure/measured boot, firmware updates, persistent variables, TRNGs, 
RTCs, etc all "just work". And then once all that works, it becomes 
possible to consider ACPI in the same bucket and avoid a bunch of kernel 
drivers for clock, power, battery, thermal, etc management.







How does u-boot do storage access in that case?  Go call BIOS INT13h?
Or use its own device drivers?

Last time I checked there have been a few gaps, no virtio-scsi driver
for example.



I don't know. I imagine with U-Boot using a similar build process to
the kernel that it does have its own equivalent of drivers for
platform configuration.




___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: Making systemd-boot option available for installation?

[Jeremy Linton]

Hi,

On 6/5/23 03:24, Zbigniew Jędrzejewski-Szmek wrote:

On Fri, Jun 02, 2023 at 05:25:22PM -0700, Luya Tshimbalanga wrote:

Hello team,

I would like to bring back the topic related to the selection of bootloader
notably either GRUB2 and systemd-boot. With the recent adoption on UKI
kernel, it would be great to get systemd-boot ready for at least Fedora 39
which is useful for devices like laptops. Currently, some methods allow to
install systemd-boot with extra step to keep supporting secure boot while
preserving GRUB2 [1]. What is the missing step to enable secure boot for
systemd-boot without at least keeping GRUB 2?


Hi Luya,

my goal is to have systemd-boot built as a ready-to-install Fedora package
with a Fedora signature for SecureBoot. The signature would use a different
certificate than grub2, and would not be trusted by our shim build. (This
way, we don't have to touch the complicated issue of making shim trust sd-boot.)
Users would be able to self-enroll those sd-boot singing keys on their machines,
getting reasonable protection from SecureBoot and being able to build
useful policies for tpm-encrypted secrets.


I tend to agree this is the right way to use systemd-boot although users 
depending on 3rd party modules will be again locked out of the secure 
boot path without the shim + MOKs. That support exists in systemd-boot 
today but it might be a good idea to provide a switch to disable the 
shim support in sdboot if the plan is to have a clean secure boot setup 
without MOKs. AKA systemd-boot-signed and a systemd-boot-signed-for-shim 
package.





Unfortunately, this requires releng to adjust the infrastructure to do the
signing, and this is not progressing at all [1].

Also, there has been work to add support for sd-boot to Anaconda [2,3].
There has been more progress there, but what we have is not a complete solution.


It is with the comps change below and the acceptance of the sdbubby 
package which is shimming things like kdumpctl to the systemd interface. 
Granted its not complete given the secure boot path hasn't been solved, 
but it does allow for testing a machine that can be updated, and 
generally works transparently. Once a signed package is available it 
should be fairly simple to assure the secure boot keys that need 
enrolling are positioned on the ESP for systemd-boot to enroll during 
the next reboot.




[1] https://pagure.io/releng/issue/10765
[2] https://bugzilla.redhat.com/show_bug.cgi?id=2106706
[3] https://github.com/rhinstaller/anaconda/pull/4368

In general, I think it'd be nice to make the process of installing sd-boot much
much simpler than it is currently. 'bootctl install' takes care of installation
process, if the system already has the expected layout. So the installation 
procedure
for Fedora should be just 'dnf install …' of a single package. But this doesn't
currently work because of a few issues:

1. the /boot partition is formatted with ext4


I don't think this is really a problem with the current anaconda + 
systemd-boot-unsigned + sdubby package. If the user default installs 
with inst.sdboot they get a separate /boot partition, and it continues 
to hold parts of the kernel debug/etc packages that aren't needed for 
boot. The kernel, initrd and loader entries have all been moved to the 
/boot/efi ESP partition under the assumption that systemd-boot systems 
will have larger ESPs and the /boot partition can be removed in its 
entirety. This works quite well today if the user uses the inst.sdboot 
option and simply deletes the /boot partition with the partition tools 
in anaconda.




2. partitions don't have parttype uuids conforming to Discoverable Partitions 
Spec [4]
(or has this been fixed? I need to check.)


I'm not sure this is needed either given the kernel + initrd is on the ESP.


3. grub2 and shim carry files directly in their rpm payload, hardcoding paths 
and
causing any changes to layout to conflict with what rpm thinks about the 
file system.
(This part was discussed on fedora-devel recently too.)


The anaconda inst.sdboot option explicitly doesn't install grub, grubby, 
grub efi tools, nor shim. Although as noted in the review comments the 
sdubby package "pins" the install location that bootctl/etc are aware 
of. Which IMHO isn't a problem given the above.


https://bugzilla.redhat.com/show_bug.cgi?id=2134972

And as I mention in that ticket I'm not sure I view it as a problem 
either since fedora has an existing filesystem layout, and something has 
to make the choice about where the files are located and that is usually 
the responsibility of rpm/etc on fedora systems since it also aids in 
auditing the files, dealing with updates, etc. I'm personally not super 
happy using `bootctl install` to move the EFI bootloader executable to 
the ESP, since I think its totally unnecessarily now that systemd-boot 
its its own package, but that is what anaconda is doing at the moment.


Nor am I entirely sure the default location 

Re: SPECfiles - conditionals with EOLed Fedora releases - any value in keeping them ?

[Jeremy Linton]

On 1/19/23 04:52, Michal Schorm wrote:

Hello,
While playing around with Sourcegraph, which indexed all Fedora
package repositories, I was able to craft a query listing all '%if'
conditionals referencing Fedora releases that reached EOL.

https://sourcegraph.com/search?q=context:global+repo:%5Esrc%5C.fedoraproject%5C.org/rpms+file:.spec+%28%25if.*%25%7B%5C%3Ffedora%7D.*%5B%3C%3D%3E%5D.*%29%28%5B12%5D%5B0-9%5D%7C%5B3%5D%5B012345%5D.*%29+count:all=regexp=yes=0=group


Similarly as F36/32-bit arm reaches EOL,

https://sourcegraph.com/search?q=context:global+repo:%5Esrc%5C.fedoraproject%5C.org/rpms+file:.spec+%28%25.*armv%5B567%5D.*%29+count:all=regexp=yes=0=group

Which finds spec's with arm[567] conditionals, and reports there are 230 
of them.


It might be a bit early to nag about them but I would assume most of 
them will start dropping off RSN.




I don't believe such conditions have any value and I think we can
remove them right away.
I think the removal shouldn't affect neither Fedora nor derived
operating systems.

If removed, they will be preserved in the git history anyway, for
anyone seeking historical code.

In some cases the conditionals hold patches that could be removed with them:

https://sourcegraph.com/search?q=context:global+repo:%5Esrc%5C.fedoraproject%5C.org/rpms+file:.spec+%28%25if.*%25%7B%5C%3Ffedora%7D.*%5B%3C%5D.*%29%28%5B12%5D%5B0-9%5D%7C%5B3%5D%5B0123456%5D.*%5Cn%2B%29%28.*%5Cn%3F%29%28%25patch.*%29+count:all=regexp=yes=0=group

--

Do you agree it would be safe to remove such conditionals and the code
they hold ?

Do you agree that removing obsolete code such as this brings value to
the package codebase ?

Would you see a value in e.g. some kind of a robot reminding
maintainers of such obsolete code? (e.g. new RPMinspect or ZUUL CI
check)

Michal

--

Michal Schorm
Software Engineer
Core Services - Databases Team
Red Hat

--
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam, report it: 
https://pagure.io/fedora-infrastructure/new_issue

Re: F37 proposal: Add -fno-omit-frame-pointer to default compilation flags (System-Wide Change proposal)

[Jeremy Linton]

Hi,


On 6/16/22 15:53, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/fno-omit-frame-pointer

This document represents a proposed Change. As part of the Changes
process, proposals are publicly announced in order to receive
community feedback. This proposal will only be implemented if approved
by the Fedora Engineering Steering Committee.


Given how this just went in the FESCo meeting, I might toss out an 
alternative I've not seen anyone else suggest.


Why not turn this on just for rawhide and leave it off for the main 
distro releases?


That is sorta what happens with the kernel already (extra debug 
options). Its not perfect but rawhide is largely intended to be the dev 
target anyway, so this solves both the "how do I do detailed 
testing/profiling" as well as maintaining peak performance for everyone 
else.





== Summary ==

Fedora will add -fno-omit-frame-pointer to the default C/C++
compilation flags, which will improve the effectiveness of profiling
and debugging tools.

== Owner ==
* Name: [[User:daandemeyer| Daan De Meyer]], [[User:Dcavalca| Davide
Cavalca]], [[ Andrii Nakryiko]]
* Email: daandeme...@fb.com, dcava...@fb.com, andr...@fb.com


== Detailed Description ==

Credits to Mirek Klimos, whose internal note on stacktrace unwinding
formed the basis for this change proposal (myre...@gmail.com).

Any performance or efficiency work relies on accurate profiling data.
Sampling profilers probe the target program's call stack at regular
intervals and store the stack traces. If we collect enough of them, we
can closely approximate the real cost of a library or function with
minimal runtime overhead.

Stack trace capture what’s running on a thread. It should start with
clone - if the thread was created via clone syscall - or with _start -
if it’s the main thread of the process. The last function in the stack
trace is code that CPU is currently executing. If a stack starts with
[unknown] or any other symbol, it means it's not complete.

=== Unwinding ===

How does the profiler get the list of function names? There are two parts of it:

# Unwinding the stack - getting a list of virtual addresses pointing
to the executable code
# Symbolization - translating virtual addresses into human-readable
information, like function name, inlined functions at the address, or
file name and line number.

Unwinding is what we're interested in for the purpose of this
proposal. The important things are:

* Data on stack is split into frames, each frame belonging to one function.
* Right before each function call, the return address is put on the
stack. This is the instruction address in the caller to which we will
eventually return — and that's what we care about.
* One register, called the "frame pointer" or "base pointer" register
(RBP), is traditionally used to point to the beginning of the current
frame. Every function should back up RBP onto the stack and set it
properly at the very beginning.

The “frame pointer” part is achieved by adding push %rbp, mov
%rsp,%rbp to the beginning of every function and by adding pop %rbp
before returning. Using this knowledge, stack unwinding boils down to
traversing a linked list:

https://i.imgur.com/P6pFdPD.png

=== Where’s the catch? ===

The frame pointer register is not necessary to run a compiled binary.
It makes it easy to unwind the stack, and some debugging tools rely on
frame pointers, but the compiler knows how much data it put on the
stack, so it can generate code that doesn't need the RBP. Not using
the frame pointer register can make a program more efficient:

* We don’t need to back up the value of the register onto the stack,
which saves 3 instructions per function.
* We can treat the RBP as a general-purpose register and use it for
something else.

Whether the compiler sets frame pointer or not is controlled by the
-fomit-frame-pointer flag and the default is "omit", meaning we can’t
use this method of stack unwinding by default.

To make it possible to rely on the frame pointer being available,
we'll add -fno-omit-frame-pointer to the default C/C++ compilation
flags. This will instruct the compiler to make sure the frame pointer
is always available. This will in turn allow profiling tools to
provide accurate performance data which can drive performance
improvements in core libraries and executables.

== Feedback ==

=== Potential performance impact ===

* Meta builds all its libraries and executables with
-fno-omit-frame-pointer by default. Internal benchmarks did not show
significant impact on performance when omitting the frame pointer for
two of our most performance intensive applications.
* Firefox recently landed a change to preserve the frame pointer in
all jitted code
(https://bugzilla.mozilla.org/show_bug.cgi?id=1426134). No significant
decrease in performance was observed.
* Kernel 4.8 frame pointer benchmarks by Suse showed 5%-10%
regressions in some benchmarks
(https://lore.kernel.org/all/20170602104048.jkkzssljsompj...@suse.de/T/#u)

Re: F37 Change: Deprecate Legacy BIOS (System-Wide Change proposal)

[Jeremy Linton]

Hi,

On 5/2/22 22:53, Chris Murphy wrote:

On Mon, May 2, 2022 at 5:29 PM Jeremy Linton  wrote:


On 4/6/22 12:57, Neal Gompa wrote:
(trimming)

* NVIDIA graphics
* Broadcom wireless

The former case is excessively common, and the latter case is fairly
common with HP and Dell machines as well as some smaller OEMs. I
literally helped someone this past week with both[1][2][3]. The
Workstation WG has been tracking both issues for years now[4][5]. This
situation is *worse* now because we have Fedora Linux preloaded on
computers, and OEMs basically have to disable Secure Boot to make
things "work". How's that for improving security?


I too have been a bit surprised at some of the difficulties of
hibernate/secure boot on recent fedora releases. It seems people are
entirely unaware that ACPI/S3 standby is gone from most consumer
laptops, and the modern standby replacement implementations tend to work
very poorly WRT conserving battery with the lid closed in Linux.


It's a kernel problem. I'm not sure to what degree upstream is aware
of it. But there's not a lot we can do about it except file bugs and
ask for improvement.




So, on a recent fedora machine, it took me more than 4 hours to get a
hibernation file on btrfs plus LUKS encrypted partition working. The
documentation for that wasn't to be found anywhere on the fedora/RH
sites and required compiling a tool to do the block offset calculations
and manually adding the resume_offset options to grub/etc. All while
avoiding the mass of incorrect information found on the internet. And of
course it also requires disabling swap on zram (which was nonsense on
the machine anyway, given the disks are faster than it can
compress/decompress pages).


I don't think it requires disabling swap on zram per se - from what
I've been told the hibernation code knows it can't use it for the
hibernation image, not least of which is it's not big enough for a
contiguous write of the image. The issue might be that so much needs
to be swapped out, to free ~50% RAM, which is used to create the
hibernation image in memory before it's written out. We need a clear
reproducer with logs and get it posted to the Linux memory management
mailing list to see what's going wrong. Since zram is threaded, it's
pretty unlikely drive writes are faster than memory writes with
compression. LZO+RLE is computationally pretty cheap.


DMA is computationally free. and at >3GB/sec on modern hw with 
sufficient queue depth or contiguous. And the RLE improvements only help 
when the page is basically empty. AKA its a great chrome benchmark tool, 
less so for real workloads.







And of course the lockdown patches in the kernel still aren't smart
enough to be able to detect that the swapfile is actually encrypted, so
it also requires disabling secure boot (this IMHO is frankly
unacceptable, that one can't have both options enabled at the same time).


Encryption isn't enough to ensure the image is valid. It needs to be
signed. But in any case this is also upstream effort required,
including discovering the offset via a standard API for all file
systems.


Yes, I'm aware much of this is a kernel problem, but the point being 
that in the meantime, most random machines people are buying at retail 
won't last more than a day or so without being plugged in using fedora, 
vs many days with windows because its hibernating with secure boot 
turned on.









So, this is really less about BIOS/EFI and more about some pretty basic
functionality being broken in the distro.




___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F37 Change: Deprecate Legacy BIOS (System-Wide Change proposal)

[Jeremy Linton]

On 4/6/22 12:57, Neal Gompa wrote:
(trimming)

* NVIDIA graphics
* Broadcom wireless

The former case is excessively common, and the latter case is fairly
common with HP and Dell machines as well as some smaller OEMs. I
literally helped someone this past week with both[1][2][3]. The
Workstation WG has been tracking both issues for years now[4][5]. This
situation is *worse* now because we have Fedora Linux preloaded on
computers, and OEMs basically have to disable Secure Boot to make
things "work". How's that for improving security?


I too have been a bit surprised at some of the difficulties of 
hibernate/secure boot on recent fedora releases. It seems people are 
entirely unaware that ACPI/S3 standby is gone from most consumer 
laptops, and the modern standby replacement implementations tend to work 
very poorly WRT conserving battery with the lid closed in Linux.


Leaving hibernate as the only workable solution if you want to just 
close the laptop lid, and come back the next day without having the 
machine at 0% battery. This is the expected/default windows behavior 
too. After ~5% IIRC battery loss in modern standby mode, it hibernates. 
(look up windows adaptive hibernate).


So, on a recent fedora machine, it took me more than 4 hours to get a 
hibernation file on btrfs plus LUKS encrypted partition working. The 
documentation for that wasn't to be found anywhere on the fedora/RH 
sites and required compiling a tool to do the block offset calculations 
and manually adding the resume_offset options to grub/etc. All while 
avoiding the mass of incorrect information found on the internet. And of 
course it also requires disabling swap on zram (which was nonsense on 
the machine anyway, given the disks are faster than it can 
compress/decompress pages).


And of course the lockdown patches in the kernel still aren't smart 
enough to be able to detect that the swapfile is actually encrypted, so 
it also requires disabling secure boot (this IMHO is frankly 
unacceptable, that one can't have both options enabled at the same time).


So, this is really less about BIOS/EFI and more about some pretty basic 
functionality being broken in the distro.






On the cloud side, it's been very difficult to articulate any benefits
for supporting UEFI when the majority of the consumers of Fedora Cloud
don't have any pressing need to do it and things like hibernation and
snapshotting are non-functional. Last year, I changed Fedora Cloud to
hybrid boot[6] so that our image artifacts support both boot modes.
While GCP requires UEFI and Azure prefers it, AWS has very basic
support for UEFI and using UEFI causes you to lose some features that
exist only in BIOS mode. One of those is leveraging hibernation in the
cloud for spot instances[7]. Moving past the Big Three(tm), the actual
cloud providers that matter from a Fedora context are the smaller
outfits that principally serve Linux users. These are companies like
DigitalOcean, Linode (Akamai), Hetzner, VexxHost, and others who
graciously do offer Fedora Linux in their platforms. All of their
virtualization platforms are BIOS only right now, and getting them to
switch requires them to uplift their platforms to support UEFI in the
first place. And again, when UEFI means things like VM snapshots and
cloud hibernation don't work, it's not very compelling.

You'd think that given how important this is for the Cloud that it
would have mattered for RHEL, but nope. These problems are not new.
They've existed since we supported UEFI Secure Boot, and given how
people have responded saying these issues are irrelevant to this
Change, it shows how out of sync with reality this Change is.





Frankly, I'm extremely frustrated and exhausted over the situation.

[1]: https://twitter.com/Det_Conan_Kudo/status/1508968025785049088
[2]: https://twitter.com/Det_Conan_Kudo/status/1508984123339202560
[3]: https://twitter.com/Det_Conan_Kudo/status/1511755879687012354
[4]: https://pagure.io/fedora-workstation/issue/155
[5]: https://pagure.io/fedora-workstation/issue/116
[6]: https://fedoraproject.org/wiki/Changes/FedoraCloudHybridBoot
[7]: https://lwn.net/Articles/821158/


--
真実はいつも一つ！/ Always, there's only one truth!
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: 

Re: Any recent changes to the arm builders?

[Jeremy Linton]

Hi,

On 8/17/21 2:06 AM, Florian Weimer wrote:

* Jeremy Linton:


That said, there as you mention various rpm/package build/etc problems
caused by `uname -m` returning armv8.


Is this something that can be changed with setarch?  It works on other
architectures (at least on x86 and POWER).


Beyond the defaults? I don't know how to pull that off. You get three 
differing unames on a armv8 machine that supports aarch64 and aarch32.


32-bit kernel armv7l
64-bit kernel 64-bit process, aarch64
64-bit kernel 32-bit process (or via setarch), armv8l

Trying to force the 64-bit kernel to armv7l via compat/etc just results 
in the armv8 moniker. This is probably fixable/etc, or there is a way I 
don't know about off hand.


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Any recent changes to the arm builders?

[Jeremy Linton]

Hi,

On 8/16/21 12:55 AM, Florian Weimer wrote:

* Kevin Fenzi:


On Sun, Aug 15, 2021 at 01:51:16PM +0200, Florian Weimer wrote:

* Kevin Fenzi:


Yes. They were mistakenly running the normal kernel (so they had ~3GB
memory available). I moved them back to the lpae kernel (so they see
40GB memory), but this causes

https://bugzilla.redhat.com/show_bug.cgi?id=1920183

basically OOM kills kojid, which restarts kojid, which restarts the
build, which kills kojid, etc...


Why aren't the builders running 64-bit kernels, like we do for x86-64?


This is not a configuration that I think we support in any way.


Who is “we”?

I expect that 64-bit kernel bugs will get more attention upstream.


At least rpm rejects trying to install a aarch64 kernel on a 32bit
userspace.


The host (including kojid) should probably be 64-bit, and only the
chroot 32-bit.  If that doesn't work, it's an RPM bug/missing feature.


OS wise, 32-bit containers/etc work fine on 64-bit fedora kernels. This 
also solves the problem that server grade HW with 32-bit guest (EL1+) 
support is becoming rarer (basically only older A72 based platforms now) 
and would provide a path forward on more recent Gravaton2, Ampere, etc 
platforms.



That said, there as you mention various rpm/package build/etc problems 
caused by `uname -m` returning armv8.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Drop the the "Allow SSH root login with password" option from the installer GUI (Self-Contained Change proposal)

[Jeremy Linton]

Hi,

On 5/17/21 2:26 PM, Martin Kolman wrote:

On Sat, 2021-05-15 at 17:53 +0200, Ralf Corsepius wrote:

On 5/14/21 2:50 PM, Martin Kolman wrote:

On Thu, 2021-05-13 at 20:09 +0200, Peter Boy wrote:



We discussed that in the Fedora Server Edition Working Group and
opted to leave it as is for the Server installation iso. A lot of
servers are running in a protected environment. And there are
situations when you need urgent access but do not sit at your
desktop
and don’t have the key available. So let the server admin decide
what
is best in a given installation context. In most cases it is the
current default (disallow password login)

Do those server deployments not have any users accounts other than
root
? Creating a non-root user account, possibly with admin rights (all
possible from within Anaconda) would seem like a safer option for
accasional/emergency password based access to such machines over
SSH.


I don't see, how this would any safer than directly using "root".

As far as I understand the original change in upstream OpenSSH it's
about only having to remotely guess a password to gain access to the
root account.

In comparison to remotely attack a user account you need to guess both
the user name *and* password, making the potential search space quite a
bit larger (provided the user name is reasonably unique).


So presumably, its a problem for which a single additional bit of 
password entropy provides more security.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: F35 Change: Drop the the "Allow SSH root login with password" option from the installer GUI (Self-Contained Change proposal)

[Jeremy Linton]

Hi,

On 5/14/21 1:05 AM, Juha Tuomala wrote:



On Thursday, 13 May 2021 18:50:33 EEST PGNet Dev wrote:

On 5/13/21 10:48 AM, Juha Tuomala wrote:

Virtual machine installation is hopefully a special use case and majority
of installations are bare metal end users.


hardly.

here,


Sure. But this is devel list. Are developers themselves the target audience?
:) Hopefully not. Is it defined somewhere?

I would certainly enjoy the polished user interface that normal users require.


Yes, it would be helpful know to know the userbase better, but I would 
hazard a guess that the percentage of non IT related people _installing_ 
fedora is a tiny fraction of the userbase. I don't think that is unique 
to linux, not many mac/windows users have seen the osx/windows installer

either.

I would suggest then the point of the installer (vs just a random disk 
image, or pre-installed machine) is to give the user choices about the 
systemm behavior, be that the partitioning, DE, system services, etc. 
Sure having a streamlined "just do it" mode is helpful, but its a 
shortcomming of the installer if the first thing I have to do with a 
newly installed machine is reverse a lot of the defaults it set. Sadly I 
find myself doing this more and more with fedora, as i'm not given the 
choice to not to use zram, or avoid starting iscsi, I have to manually 
disable those things. So, while zram and iscsi have their place, its not 
in my environment.





  

for that, a simple password option is more than sufficient.
again, why not simply 'leave it be'.


To make it clear, I agree. Unix/Linux has always been about options and
flexibility. And hence having option to pull the root's existing public key
somewhere easier is just good progress.


Tuju


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fedora 34 Change: Enable btrfs transparent zstd compression by default (System-Wide Change proposal)

[Jeremy Linton]

Hi,

On 2/14/21 2:20 PM, Chris Murphy wrote:

On Sat, Feb 13, 2021 at 9:45 PM Jeremy Linton  wrote:


Hi,

On 2/11/21 11:05 PM, Chris Murphy wrote:

On Thu, Feb 11, 2021 at 9:58 AM Jeremy Linton  wrote:


Hi,

On 1/1/21 8:59 PM, Chris Murphy wrote:



Anyway, compress=zstd:1 is a good default. Everyone benefits, and I'm
not even sure someone with a very fast NVMe drive will notice a slow
down because the compression/decompression is threaded.


I disagree that everyone benefits. Any read latency sensitive workload
will be slower due to the application latency being both the drive
latency plus the decompression latency. And as the kernel benchmarks
indicate very few systems are going to get anywhere near the performance
of even baseline NVMe drives when its comes to throughput.


It's possible some workloads on NVMe might have faster reads or writes
without compression.

https://github.com/facebook/zstd

btrfs compress=zstd:1 translates into zstd -1 right now; there are
some ideas to remap btrfs zstd:1 to one of the newer zstd --fast
options, but it's just an idea. And in any case the default for btrfs
and zstd will remain as 3 and -3 respectively, which is what
'compress=zstd' maps to, making it identical to 'compress=zstd:3'.

I have a laptop with NVMe and haven't come across such a workload so
far, but this is obviously not a scientific sample. I think you'd need
a process that's producing read/write rates that the storage can meet,
but that the compression algorithm limits. Btrfs is threaded, as is
the compression.

What's typical, is no change in performance and sometimes a small
small increase in performance. It essentially trades some CPU cycles
in exchange for less IO. That includes less time reading and writing,
but also less latency, meaning the gain on rotational media is
greater.


Worse, if the workload is very parallel, and at max CPU already
the compression overhead will only make that situation worse as well. (I
suspect you could test this just by building some packages that have
good parallelism during the build).


This is compiling the kernel on a 4/8-core CPU (circa 2011) using make
-j8, the kernel running is 5.11-rc7.

no compression

real55m32.769s
user369m32.823s
sys 35m59.948s

--

compress=zstd:1

real53m44.543s
user368m17.614s
sys 36m2.505s

That's a one time test, and it's a ~3% improvement. *shrug* We don't
really care too much these days about 1-3% differences when doing
encryption, so I think this is probably in that ballpark, even if it
turns out another compile is 3% slower. This is not a significantly
read or write centric workload, it's mostly CPU. So this 3% difference
may not even be related to the compression.


Did you drop caches/etc between runs?


Yes. And also did the test with uncompressed source files when
compiling without the compress mount option. And compressed source
files when compiling with the compress mount option. While it's
possible to mix those around (there's four combinations), I kept them
the same since those are the most common.




Because I git cloned mainline,
copied the fedora kernel config from /boot and on a fairly recent laptop
(12 threads) with a software encrypted NVMe. Dropped caches and did a
time make against a compressed directory and an uncompressed one with
both a semi constrained (4G) setup and 32G ram setup (compressed
swapping disabled, because the machine has an encrypted swap for
hibernation and crashdumps).

compressed:
real22m40.129s
user221m9.816s
sys 23m37.038s

uncompressed:
real21m53.366s
user221m56.714s
sys 23m39.988s

uncompressed 4G ram:
real28m48.964s
user288m47.569s
sys 30m43.957s

compressed 4G
real29m54.061s
user281m7.120s
sys 29m50.613s



While the feature page doesn't claim it always increases performance,
it also doesn't say it can reduce performance. In the CPU intensive
workloads, it stands to reason there's going to be some competition.
The above results strongly suggest that's what's going on, even if I
couldn't reproduce it. But performance gain/loss isn't the only factor
for consideration.



and that is not an IO constrained workload its generally cpu
constrained, and since the caches are warm due to the software
encryption the decompress times should be much faster than machines that
aren't cache stashing.


I don't know, so I can't confirm or deny any of that.



The machine above, can actually peg all 6 cores until it hits thermal
limits simply doing cp's with btrfs/zstd compression, all the while
losing about 800MB/sec of IO bandwidth over the raw disk. Turning an IO
bound problem into a CPU bound one isn't ideal.


It's a set of tradeoffs. And there isn't a governor that can assess
when an IO bound bottleneck becomes a CPU bound one.


Compressed disks only work in the situation where the CPUs can
compress/decompress faster than the disk, or the workload is managing to
significantly reduce IO because the working set

Re: Fedora 35 Change proposal: POWER 4k page size (System-Wide Change proposal)

[Jeremy Linton]

Hi,

On 2/13/21 10:41 PM, Tom Seewald wrote:

  > The GPUs also have firmware blobs
Could you provide some links to mailing list posts or bug reports where AMD 
developers confirm that their GPU firmware requires 4k pages? I think having 
some definitive sources will make this situation more clear.

So far the only amdgpu bug report I could find that relates to 64k pages[1] is 
a regression, as the reporter states the driver works with the 5.4 kernel. If 
someone with a power9 machine is willing to bisect the issue I think that would 
greatly increase the odds of this bug being resolved.


I can confirm that amdgpu worked with 64k pages on arm in the past with 
various GPUs. I haven't been testing it regularly though, but we will 
want it for rhel/centos which use 64k pages on aarch64 too.





[1] https://gitlab.freedesktop.org/drm/amd/-/issues/1446
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure


___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fedora 34 Change: Enable btrfs transparent zstd compression by default (System-Wide Change proposal)

[Jeremy Linton]

Hi,

On 2/11/21 11:05 PM, Chris Murphy wrote:

On Thu, Feb 11, 2021 at 9:58 AM Jeremy Linton  wrote:


Hi,

On 1/1/21 8:59 PM, Chris Murphy wrote:



Anyway, compress=zstd:1 is a good default. Everyone benefits, and I'm
not even sure someone with a very fast NVMe drive will notice a slow
down because the compression/decompression is threaded.


I disagree that everyone benefits. Any read latency sensitive workload
will be slower due to the application latency being both the drive
latency plus the decompression latency. And as the kernel benchmarks
indicate very few systems are going to get anywhere near the performance
of even baseline NVMe drives when its comes to throughput.


It's possible some workloads on NVMe might have faster reads or writes
without compression.

https://github.com/facebook/zstd

btrfs compress=zstd:1 translates into zstd -1 right now; there are
some ideas to remap btrfs zstd:1 to one of the newer zstd --fast
options, but it's just an idea. And in any case the default for btrfs
and zstd will remain as 3 and -3 respectively, which is what
'compress=zstd' maps to, making it identical to 'compress=zstd:3'.

I have a laptop with NVMe and haven't come across such a workload so
far, but this is obviously not a scientific sample. I think you'd need
a process that's producing read/write rates that the storage can meet,
but that the compression algorithm limits. Btrfs is threaded, as is
the compression.

What's typical, is no change in performance and sometimes a small
small increase in performance. It essentially trades some CPU cycles
in exchange for less IO. That includes less time reading and writing,
but also less latency, meaning the gain on rotational media is
greater.


Worse, if the workload is very parallel, and at max CPU already
the compression overhead will only make that situation worse as well. (I
suspect you could test this just by building some packages that have
good parallelism during the build).


This is compiling the kernel on a 4/8-core CPU (circa 2011) using make
-j8, the kernel running is 5.11-rc7.

no compression

real55m32.769s
user369m32.823s
sys 35m59.948s

--

compress=zstd:1

real53m44.543s
user368m17.614s
sys 36m2.505s

That's a one time test, and it's a ~3% improvement. *shrug* We don't
really care too much these days about 1-3% differences when doing
encryption, so I think this is probably in that ballpark, even if it
turns out another compile is 3% slower. This is not a significantly
read or write centric workload, it's mostly CPU. So this 3% difference
may not even be related to the compression.


Did you drop caches/etc between runs? Because I git cloned mainline, 
copied the fedora kernel config from /boot and on a fairly recent laptop 
(12 threads) with a software encrypted NVMe. Dropped caches and did a 
time make against a compressed directory and an uncompressed one with 
both a semi constrained (4G) setup and 32G ram setup (compressed 
swapping disabled, because the machine has an encrypted swap for 
hibernation and crashdumps).


compressed:
real22m40.129s
user221m9.816s
sys 23m37.038s

uncompressed:
real21m53.366s
user221m56.714s
sys 23m39.988s

uncompressed 4G ram:
real28m48.964s
user288m47.569s
sys 30m43.957s

compressed 4G
real29m54.061s
user281m7.120s
sys 29m50.613s

and that is not an IO constrained workload its generally cpu 
constrained, and since the caches are warm due to the software 
encryption the decompress times should be much faster than machines that 
aren't cache stashing.


The machine above, can actually peg all 6 cores until it hits thermal 
limits simply doing cp's with btrfs/zstd compression, all the while 
losing about 800MB/sec of IO bandwidth over the raw disk. Turning an IO 
bound problem into a CPU bound one isn't ideal.


Compressed disks only work in the situation where the CPUs can 
compress/decompress faster than the disk, or the workload is managing to 
significantly reduce IO because the working set is streaming rather than 
random. Any workload which has a random read component to it and is 
tending closer to page sized read/writes is going to get hurt, and god 
help if its a RMW cycle. Similarly for parallelized compression, which 
is only scalable if the IO sizes are large enough that its worth the IPI 
overhead of bringing additional cores online and the resulting chunks 
are still large enough to be dealt with individually.








Plus, the write amplification comment isn't even universal as there
continue to be controllers where the flash translation layer is
compressing the data.


At least consumer SSDs tend to just do concurrent write dedup. File
system compression isn't limited to Btrfs, there's also F2FS
contributed by Samsung which implements compression these days as
well, although they commit to it at mkfs time, where as on Btrfs it's
a mount option. Mix and match compressed extents is routine on Btrfs
anyway, so there's

Re: Ars claims: Fedora 32 is sluggish

[Jeremy Linton]

Hi,

On 2/5/21 3:03 AM, Roberto Ragusa wrote:

On 2/4/21 9:52 PM, Alexander Ploumistos wrote:


considerable lag. In the last 4 or so years I remember issues with
tracker, gnome-shell, mutter/clutter and friends on specific GPUs,
default or popular shell extensions and dbus services. A recent bug


If tracker is enabled the performance drop after booting is huge.

rpm -e tracker-miners



I regularly have problems with runaway processes on fedora. Frequently 
the  first indication of a problem are fans on a laptop running when the 
machine is idle. Just yesterday, clean install of F33, enabled Plasma, 
and korgac sat there for a few minutes at 100% until I killed it and 
told it never to start again. There isn't anything like a couple runaway 
processes to make the whole machine "sluggish".


Fedora suffers from having a lot of things starting by default that are 
only used by a fraction of the user base. If it were smarter about 
starting/installing things provided by systemd/gnome/kde/etc I suspect 
that not only would it utilize less resources, but there security 
benefits of simply not having much of this running all the time would be 
clearer too. To pick on a couple. iscsi and avahi, are very important 
for a subset of the users, but frankly i suspect they are trivial 
fraction of the overall userbase. But they show up in security audits 
(lynis) simply because systemd-analyze itself reports them as insecure.


Runaway processes are also a bit of an abrt failure in that there isn't 
a good automated way to report them. A package flag to the effect "this 
package doesn't contain anything which should consume 100% cpu for > 10 
seconds" could be set on a wide range of these services to at least 
notify the upstream developers that there might be something wrong. Of 
course this would require yet another always on monitoring daemon.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fedora 34 Change: Enable btrfs transparent zstd compression by default (System-Wide Change proposal)

[Jeremy Linton]

Hi,

On 1/1/21 8:59 PM, Chris Murphy wrote:

On Fri, Jan 1, 2021 at 11:31 AM Artem Tim  wrote:


It's faster. Here is some benchmark with different zstd compression ratios 
https://lkml.org/lkml/2019/1/28/1930. Could be outdated a little bit though.

But for HDD it makes sense to increase it probably. And IIRC Chris wrote about 
such plans.


There are ideas but it's difficult because the kernel doesn't expose
the information we really need to make an automatic determination.
sysfs commonly misreports rotational devices as being non-rotational
and vice versa. Since this is based on the device self-reporting, it's
not great.

I use zstd:1 for SSD/NVMe. And zstd:3 (which is the same as not
specifying a level) for HDD/USB sticks/eMMC/SD Card. For the more
archive style of backup, I use zstd:7. But these can all be mixed and
matched, Btrfs doesn't care. You can even mix and match algorithms.

Anyway, compress=zstd:1 is a good default. Everyone benefits, and I'm
not even sure someone with a very fast NVMe drive will notice a slow
down because the compression/decompression is threaded.


I disagree that everyone benefits. Any read latency sensitive workload 
will be slower due to the application latency being both the drive 
latency plus the decompression latency. And as the kernel benchmarks 
indicate very few systems are going to get anywhere near the performance 
of even baseline NVMe drives when its comes to throughput. With PCIe 
Gen4 controllers the burst speeds are even higher (>3GB/sec read & 
write). Worse, if the workload is very parallel, and at max CPU already 
the compression overhead will only make that situation worse as well. (I 
suspect you could test this just by building some packages that have 
good parallelism during the build).


So, your penalizing a large majority of machines built in the past 
couple years.


Plus, the write amplification comment isn't even universal as there 
continue to be controllers where the flash translation layer is 
compressing the data.


OTOH, it makes a lot more sense on a lot of these arm/sbc boards 
utilizing MMC because the disks are so slow. Maybe if something like 
this were made the default the machine should run a quick CPU 
compress/decompress vs IO speed test and only enable compression if the 
compress/decompress speed is at least the IO rate.





I expect if we get the "fast" levels (the negative value levels) new
to zstd in the kernel, that Btrfs will likely remap its level 1 to one
of the negative levels, and keep level 3 set to zstd 3 (the default).
So we might actually see it get even faster at the cost of some
compression ratio. Given this possibility, I think level 1 is the best
choice as a default for Fedora.




___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: Fwd: Re: CppunitTest_sw_htmlexport failing due to zlib variation?

[Jeremy Linton]

Hi,

On 8/28/20 1:38 AM, Stephan Bergmann wrote:

All,

The below question came up in the context of a LibreOffice unit test, 
where LibreOffice writes out a PNG image (involving zlib for 
compression) and the test checked the exact sequence of bytes, which 
failed on aarch64 when using Fedora's zlib.  (Though the resulting 
images look rather identical.  Full thread starting at 
 
"CppunitTest_sw_htmlexport failing due to zlib variation?")


Given the Fedora zlib aarch64 optimization patches quoted below:  Does 
anybody know whether it is indeed the case that the above scenario 
(client code using zlib to generate a PNG image) can legitimately 
generate varying output depending on zlib implementation details?  Or 
would that rather indicate a bug somewhere?


Yes, the short answer is that it is possible to create valid zlib 
archives that vary depending on the exact zlib implementation or 
compression setting. This is even part of the library's compression 
level (Z_BEST_SPEED vs Z_BEST_COMPRESSION) selection.


If one is trying to test a compression function the best way to test it 
is to turn around and decompress the output and compare it with the 
original rather than checking that the output from a compressor hasn't 
changed.


The zlib patches in fedora are an attempt to gain a bit of performance 
on an algorithm that compared with more modern algorithm/implementations 
is frightfully slow without any significant upside on the compression 
ratio. A lot of the byte oriented code runs poorly on modern cores. 
Looking at the fedora patches, there appears to be a s390 patch 
offloading the whole thing to hardware. This isn't just a arm/s390 
problem either as IIRC there were x86 perf fixes posted as well that 
haven't been merged either.






Thanks,
Stephan


 Forwarded Message 
Subject: Re: CppunitTest_sw_htmlexport failing due to zlib variation?
Date: Wed, 26 Aug 2020 08:37:15 +0200
From: Stephan Bergmann 
To: libreoff...@lists.freedesktop.org

On 25/08/2020 11:07, Stephan Bergmann wrote:
At least when building recent master on recent Fedora rawhide aarch64 
with (among others) --with-system-zlib, CppunitTest_sw_htmlexport 
fails with

[...]
The base64-encoded payload apparently is a PNG image.  And from what 
little I know about PNG, it looks plausible to me that there can be 
different (compressed) PNG content that decompress to identical raw 
data, and that the LibreOffice code would be allowed to generate 
differing (compressed) PNG content for the above data:image/png URL 
payload.


What supports the theory that "the LibreOffice code would be allowed to 
generate differing (compressed) PNG content" is the Fedora zlib commit 
 
"aarch64 optimizations", adding lots of code that presumably is only 
relevant for aarch64 and presumably changes details of the compression 
algorithm.



___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 32 aarch64 build failures on copr

[Jeremy Linton]

On 7/28/20 4:39 PM, Jeff Law wrote:

On Tue, 2020-07-28 at 15:29 -0500, Michael Catanzaro wrote:

On Tue, Jul 28, 2020 at 2:01 pm, Jeff Law  wrote:

If this is a new failure (say in the last week), it could be an out
of memory
scenario.  Try disabling LTO.  The standard way to do that is

%define _lto_cflags %{nil}

In your %build stanza in the spec file.

Heff


I agree, it's almost certainly OOM because it says "fatal error:
Killed." I've never seen that happen for any reason other than OOM.

I've seen it happen for a variety of reasons.  Please test with LTO disabled and
let me know if that helps.


As a FYI: I had similar problems not long ago building a tensorflow. 
Reducing the build parallelism in that case helped reduce the memory 
footprint sufficiently that the build completed.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Upcoming Fedora 33 Change proposal deadlines

[Jeremy Linton]

Hi,


On 7/17/20 9:07 AM, Ben Cotton wrote:

This is the final reminder of the Fedora 33 Change proposal deadlines:

  * 2020-07-21: Proposal deadline for Self-Contained Changes

The mass rebuild is scheduled to begin next week as well.

For the full schedule, see
https://fedorapeople.org/groups/schedule/f-33/f-33-key-tasks.html




So, the PAC proposal should have this defect 
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=94891 integrated before its 
enabled.


That defect is in 10.2 and has been backported to 10.1/9, but I don't 
see it in the version currently in rawhide. Is there a plan to roll gcc 
forward before or as part of the mass rebuild?



Thanks,
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Aarch64 Pointer Authentication & Branch Target Enablement

[Jeremy Linton]

Hi,

On 5/19/20 2:21 PM, Igor Raits wrote:

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

On Mon, 2020-05-18 at 15:36 -0400, Ben Cotton wrote:

https://fedoraproject.org/wiki/Changes/Aarch64_PointerAuthentication

== Summary ==
Arm Pointer Authentication (PAC) is a method of hardening code from
Return Oriented Programming (ROP) attacks. It uses a tag in a pointer
to sign and verify pointers. Branch Target Identification (BTI) is
another code hardening method, where the branch/jump target is
identified with a special landing pad instruction. Outside of some
system support in glibc+kernel, packages gain the additional
hardening
by compiling with the -mbranch-protection= flag available in recent
versions of GCC. In particular -mbranch-protection=standard enables
both BTI and PAC, with backwards compatible to armv8.0 code sequences
that activate on v8.3 (PAC) & v8.5 (BTI) enabled Arm machines.


Is there some noticeable performance drop or anything like that?


Potentially, please see my longer response in the other email.





== Owner ==
* Name: [[User:jlinton| Jeremy Linton]] & ARM SIG
* Email: jeremy.lin...@arm.com


== Benefit to Fedora ==

PAC & BTI are code hardening features, they should serve to make
fedora more resistant to a couple further classes of runtime attacks.
By enabling this early, fedora is once again proven to be at the
leading edge of security and linux development. If everything works
as
planned, this change will be invisible to the end user, except in
cases where the applications will trap behaviour that appears to be
caused by exploit attempts.

== Scope ==
* Proposal owners:
Work with individual package maintainers in the case of build
failures
or runtime exceptions. In the latter case there are two
possibilities.
First on v8.0 hardware, which is currently the most common, the
additional instruction sequences are treated as NOP's and should be
completely ignored by the hardware. It may be possible on v8.3/8.5
hardware that PAC or BTI may need additional tweaks for hand written
assembly which interacts with PAC/BTI enabled code.


It would be nice if you would specify which exact flags and where you
will add them, so that it would be easy to see which people we need to
get on board for this change.


Its the gcc flag "-mbranch-protection=standard" I mentioned in the 
summary above. The assumption is that we place it in the arch specific 
opt stanza in /usr/lib/rpm/redhat/rpmrc, or we extend 
/usr/lib/rpm/redhat/macros hardening section to allow an arch specific 
hardening.


I've rebuilt a minimal install by creating a custom mock template, but I 
don't think that is the right official answer.



Thanks,
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fedora 33 System-Wide Change proposal: Aarch64 Pointer Authentication & Branch Target Enablement

[Jeremy Linton]

Hi,

On 5/19/20 1:38 PM, Przemek Klosowski via devel wrote:

On 5/18/20 3:36 PM, Ben Cotton wrote:

Arm Pointer Authentication (PAC) is a method of hardening code from
Return Oriented Programming (ROP) attacks. It uses a tag in a pointer
to sign and verify pointers. Branch Target Identification (BTI) is
another code hardening method, where the branch/jump target is
identified with a special landing pad instruction.


Is there a performance impact? do those landing pad instructions take an 
execution pipeline slots?


Potentially, depends on the microarch. In general on existing machines 
they decode as HIT/NOPs and get tossed, and since most of the ARM cores 
aren't decode limited it won't have an effect. On cores where its 
actually active its a lot harder to generalize and compare performance 
at the moment. In theory if a particular machine has a bad 
implementation we can just disable it for that given machine to return 
the behavior to just HIT/NOP.




We're planning to use AUTIASP+RET, not RETAA, right?
I believe that is the case, retaa requires a 8.3 machine and we need 
compatibility with 8.0 so we need to stick to the pac portion which is 
in the HINT space and translates to NOPs on < v8.3 hardware.


Note:

https://github.com/gcc-mirror/gcc/blob/master/gcc/config/aarch64/aarch64.c#L8208


Thanks,
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Previous awesome background images

[Jeremy Linton]

Hi,

On 4/17/20 4:43 PM, Miro Hrončok wrote:

On 17. 04. 20 16:07, Kamil Paral wrote:
I'm disappointed with default wallpapers in the latest releases. I 
wonder if we could go back to more artistic images from previous 
releases? Here are some of my favorite ones:

https://fedoraproject.org/wiki/Wallpapers#Fedora_29
https://fedoraproject.org/wiki/Wallpapers#Fedora_27
https://fedoraproject.org/wiki/Wallpapers#Fedora_21
https://fedoraproject.org/wiki/Wallpapers#Fedora_16
https://fedoraproject.org/wiki/Wallpapers#Fedora_15
https://fedoraproject.org/wiki/Wallpapers#Fedora_11
https://fedoraproject.org/wiki/Wallpapers#Fedora_7

Especially the one in Fedora 15 (GNOME edition) and 16 was 
outstanding. Can we do more of those, please?


I love both the design and concept of the F25-F28 wallpapers.

"As of the past 3 releases, we choose a sequential letter of the 
alphabet and come up with a list of scientists / mathematicians / 
technologists to serve as an inspiration for the desktop background’s 
visual concept"


https://blog.linuxgrrl.com/2018/03/06/fedora-28s-desktop-background-design/

I am sad we haven't followed the pattern. (However I don't know the 
reasoning for stopping that.)


(I've changed the subject line, because I don't want to participate in 
what was in there.)




(agree)

IMHO, A desktop background serves a different purpose than a login/lock 
screen. A desktop background shouldn't be distracting, high contrast, or 
busy. Making beautiful art with those restrictions is quite difficult. 
The fedora backgrounds tend to grow on me over time, but I also tend to 
immediately just set my desktop background to straight black, with a 
dark theme (breeze dark variation) and reserve the fedora art for the 
screen lock.


The login/lock screen, should be looking for maximum impact. Its after 
all what people see as they walk by a fedora machine. In that regard 
something like the F7 image really wins.


So maybe this is a case of trying to answer to conflicting requirements. 
Maximum art, high contrast login screen, more soothing muted desktop 
backgrounds.



Anyway, i'm going to spread some love to another distro in this thread 
too. The Opensuse 12.3 theme/background was one of the few times I've 
left the default background on for any length of time. They seemed to 
have pulled off just the right amount of mostly non distracting 
background, yet somehow managed to still achieve that minimalist beauty 
everyone seems to be in search of these days. Its all gray tones, with a 
single green vine/logo. It fit with the theme in a way that made me 
think, now this is what a linux desktop should look and behave like.


https://news-cdn.softpedia.com/images/news2/openSUSE-12-3-Is-Approaching-End-of-Life-Fast-466700-2.jpg
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org

Re: Fwd: Re: Fedora27: NFS v4 terrible write performance, is async working

[Jeremy Linton]

On 01/31/2018 09:49 AM, J. Bruce Fields wrote:

On Tue, Jan 30, 2018 at 01:52:49PM -0600, Jeremy Linton wrote:

Have you tried this with a '-o nfsvers=3' during mount? Did that help?

I noticed a large decrease in my kernel build times across NFS/lan a while
back after a machine/kernel/10g upgrade. After playing with mount/export
options filesystem tuning/etc, I got to this point of timing a bunch of
these operations vs the older machine, at which point I discovered that
simply backing down to NFSv3 solved the problem.

AKA a nfsv3 server on a 10 year old 4 disk xfs RAID5 on 1Gb ethernet, was
slower than a modern machine with a 8 disk xfs RAID5 on 10Gb on nfsv4. The
effect was enough to change a kernel build from ~45 minutes down to less
than 5.


Did you mean "faster than"?


Yes, sorry about that.



Definitely worth trying, though I wouldn't expect it to make that big a
difference in the untarring-a-kernel-tree case--I think the only RPC
avoided in the v3 case would be the CLOSE, and it should be one of the
faster ones.

In the kernel compile case there's probably also a lot of re-opening and
re-reading files too?  NFSv4 is chattier there too.  Read delegations
should help compensate, but we need to improve the heuristics that
decide when they're given out.


The main kernel include files get repeatedly hammered, despite them in 
theory being in cache, IIRC. So yes, if the concurrent (re)open path is 
even slightly slower its going to hurt a lot.




All that aside I can't think what would explain that big a difference
(45 minutes vs. 5).  It might be interesting to figure out what
happened.


I had already spent more than my time allotted looking in the wrong 
direction at the filesystem/RAID (did turn off intellipark though) by 
the time I discovered the nfsv3/v4 perf delta. Its been sitting way down 
on the "things to look into" list for a long time now. I'm still using 
it as a NFS server so at some point I can take another look if the 
problem persists.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org

Re: Fwd: Re: Fedora27: NFS v4 terrible write performance, is async working

[Jeremy Linton]

Hi,

On 01/30/2018 01:03 PM, Terry Barnaby wrote:


Being a daredevil, I have used the NFS async option for 27 years 
without an issue on multiple systems :)


I have just mounted my ext4 disk with the same options you were using 
and the same NFS export options and the speed here looks the same as I 
had previously. As I can't wait 2+ hours so I'm just looking at 
ksysguard and it is showing a network rate of about 10 KBytes/s and 
the directory on the server is growing in size very very slowly.


This is using the current Fedora27 kernel 4.14.14-300.fc27.x86_64.

I will have a look at using wireshark to see if this shows anything.


This is a snippet from a wireshark trace of the NFS when untaring the 
linux kernel 4.14.15 sources into an NFSv4.2 mounted directory with 
"sync" option on my NFS server. The whole untar would take > 2 hours vs 
13 seconds direct to the disk. This is about 850 MBytes of 60k files. 
The following is a single, small file write.


No. Time   Source Destination   Protocol Length Info
    1880 11.928600315   192.168.202.2 192.168.202.1 NFS 380
V4 Call (Reply In 1881) OPEN DH: 0xac0502f2/sysfs-c2port
    1881 11.950329198   192.168.202.1 192.168.202.2 NFS 408
V4 Reply (Call In 1880) OPEN StateID: 0xaa72
    1882 11.950446430   192.168.202.2 192.168.202.1 NFS 304
V4 Call (Reply In 1883) SETATTR FH: 0x825014ee
    1883 11.972608880   192.168.202.1 192.168.202.2 NFS 336
V4 Reply (Call In 1882) SETATTR
    1884 11.972754709   192.168.202.2 192.168.202.1 TCP 
1516   785 → 2049 [ACK] Seq=465561 Ack=183381 Win=8990 Len=1448 
TSval=1663691771 TSecr=3103357902 [TCP segment of a reassembled PDU]
    1885 11.972763078   192.168.202.2 192.168.202.1 TCP 
1516   785 → 2049 [ACK] Seq=467009 Ack=183381 Win=8990 Len=1448 
TSval=1663691771 TSecr=3103357902 [TCP segment of a reassembled PDU]
    1886 11.972979437   192.168.202.2 192.168.202.1 NFS 332
V4 Call (Reply In 1888) WRITE StateID: 0xafdf Offset: 0 Len: 2931
    1887 11.973074490   192.168.202.1 192.168.202.2 TCP 
68 2049 → 785 [ACK] Seq=183381 Ack=468721 Win=24557 Len=0 
TSval=3103357902 TSecr=1663691771
    1888 12.017153631   192.168.202.1 192.168.202.2 NFS 248
V4 Reply (Call In 1886) WRITE
    1889 12.017338766   192.168.202.2 192.168.202.1 NFS 260
V4 Call (Reply In 1890) GETATTR FH: 0x825014ee
    1890 12.017834411   192.168.202.1 192.168.202.2 NFS 312
V4 Reply (Call In 1889) GETATTR
    1891 12.017961690   192.168.202.2 192.168.202.1 NFS 328
V4 Call (Reply In 1892) SETATTR FH: 0x825014ee
    1892 12.039456634   192.168.202.1 192.168.202.2 NFS 336
V4 Reply (Call In 1891) SETATTR
    1893 12.039536705   192.168.202.2 192.168.202.1 NFS 284
V4 Call (Reply In 1894) CLOSE StateID: 0xaa72
    1894 12.039979528   192.168.202.1 192.168.202.2 NFS 248
V4 Reply (Call In 1893) CLOSE
    1895 12.040077180   192.168.202.2 192.168.202.1 NFS 392
V4 Call (Reply In 1896) OPEN DH: 0xac0502f2/sysfs-cfq-target-latency
    1896 12.061903798   192.168.202.1 192.168.202.2 NFS 408
V4 Reply (Call In 1895) OPEN StateID: 0xaa72


It looks like this takes about 100ms to write this small file. With the 
approx 60k files in the archive this would take about 6000 secs, so is 
in the 2 hours ballpark or the untar that I am seeing.


Looks like OPEN 21ms, SETATTR 22ms, WRITE 44ms, second SETATTR 21ms a 
lot of time ...


The following is for an "async" mount:

No. Time   Source Destination   Protocol Length Info
   37393 7.630012608    192.168.202.2 192.168.202.1 NFS 396
V4 Call (Reply In 37394) OPEN DH: 0x1f828ac9/vidioc-dbg-g-chip-info.rst
   37394 7.630488451    192.168.202.1 192.168.202.2 NFS 408
V4 Reply (Call In 37393) OPEN StateID: 0xaa72
   37395 7.630525117    192.168.202.2 192.168.202.1 NFS 304
V4 Call (Reply In 37396) SETATTR FH: 0x0f65c554
   37396 7.630980560    192.168.202.1 192.168.202.2 NFS 336
V4 Reply (Call In 37395) SETATTR
   37397 7.631035171    192.168.202.2 192.168.202.1 TCP 
1516   785 → 2049 [ACK] Seq=13054241 Ack=3620329 Win=8990 Len=1448 
TSval=1664595527 TSecr=3104261711 [TCP segment of a reassembled PDU]
   37398 7.631038994    192.168.202.2 192.168.202.1 TCP 
1516   785 → 2049 [ACK] Seq=13055689 Ack=3620329 Win=8990 Len=1448 
TSval=1664595527 TSecr=3104261711 [TCP segment of a reassembled PDU]
   37399 7.631042228    192.168.202.2 192.168.202.1 TCP 
1516   785 → 2049 [ACK] Seq=13057137 Ack=3620329 Win=8990 Len=1448 
TSval=1664595527 TSecr=3104261711 [TCP segment of a reassembled PDU]
   37400 7.631195554    192.168.202.2 192.168.202.1 NFS 448
V4 Call (Reply In 37402) WRITE StateID: 0xafdf Offset: 0 Len: 4493
   37401 7.631277423    192.168.202.1 192.168.202.2 TCP 
68 2049 → 785 [ACK] Seq=3620329 

Re: Stuck maxima builds on aarch64

[Jeremy Linton]

Hi,

On 03/21/2017 07:33 AM, Jerry James wrote:

On Tue, Feb 28, 2017 at 2:05 PM, Jerry James  wrote:

I'm really not sure what to check next.  If an aarch64 box for
packagers will be available in the not too distant future, I will try
to debug this.  Thanks for the information, Kevin.


This issue is now blocking a large update of sagemath and many of its
dependencies.  Sagemath invokes maxima during its build.  That
currently fails because the maxima in both F26 and Rawhide was built
with the pre-mass-rebuild version of ecl.  If this issue is not
resolved, we will ship a nonfunctional sagemath with F26.

I either need access to an aarch64 box so I can try to figure out the
problem, or I need somebody with access to the aarch64 koji builders
to figure it out.  I would file a bug, except I don't know which
component to file it against.  Probably rpm, but I don't know that for
sure.


I started a local build of this and it appears to be hanging like this:

make  check-TESTS
make[2]: Entering directory '/root/maxima/maxima-5.39.0/tests'
make[3]: Entering directory '/root/maxima/maxima-5.39.0/tests'
sed <"test.sh.in" >"gcl-test.sh"  \
  -e 's#!LOCAL_MAXIMA!#/bin/sh ../maxima-local#'   \
  -e 's#!LISPNAME!#gcl#' \
  -e 's#!OUTPUT_LOG!#../tests/gcl.log#'
chmod a+x "gcl-test.sh"
FAIL: gcl-test.sh
sed <"test.sh.in" >"ecl-test.sh"  \
  -e 's#!LOCAL_MAXIMA!#/bin/sh ../maxima-local#'   \
  -e 's#!LISPNAME!#ecl#' \
  -e 's#!OUTPUT_LOG!#../tests/ecl.log#'
chmod a+x "ecl-test.sh"


Breaking into maxima it seems to be stuck in:
(gdb) bt

#0  0xccdf2b3c in LC37testsuite (narg=0) at binary-ecl/mload.c:2800
#1  0xccdf2d8c in LC38__g239 (narg=0) at binary-ecl/mload.c:2967
#2  0xb30532fc in L1do_time () from /lib64/libecl.so.16.1
#3  0xccdf2060 in L39run_testsuite (narg=) at 
binary-ecl/mload.c:2692

#4  0xb3108f48 in cl_apply () from /lib64/libecl.so.16.1
#5  0xccdead24 in L44_run_testsuite (narg=) at 
binary-ecl/mload.c:3269

#6  0xb3108f48 in cl_apply () from /lib64/libecl.so.16.1
#7  0xccc5c42c in L13meval1 (v1form=) at 
binary-ecl/mlisp.c:1146
#8  0xccc41b1c in L10meval (v1form=0xed4c6991) at 
binary-ecl/mlisp.c:631
#9  0xccdfc7d4 in L1meval_ (v1expr=) at 
binary-ecl/suprv1.c:30
#10 0xccde4ec8 in L7toplevel_macsyma_eval (v1x=) 
at binary-ecl/macsys.c:179
#11 0xccde85d4 in L9continue (narg=) at 
binary-ecl/macsys.c:369
#12 0xccde75b8 in L24macsyma_top_level (narg=) at 
binary-ecl/macsys.c:1154
#13 0xcd3044c4 in L39common_lisp_user__run () at 
binary-ecl/init-cl.c:1355

#14 0xb310b840 in ecl_interpret () from /lib64/libecl.so.16.1
#15 0xb310f82c in eval_form () from /lib64/libecl.so.16.1
#16 0xb310f8e0 in execute_each_form () from /lib64/libecl.so.16.1
#17 0xb310ec04 in compile_form () from /lib64/libecl.so.16.1
#18 0xb310f4f8 in compile_with_load_time_forms () from 
/lib64/libecl.so.16.1

#19 0xb310f7d8 in eval_form () from /lib64/libecl.so.16.1
#20 0xb31130d0 in si_eval_with_env () from /lib64/libecl.so.16.1
#21 0xb30f1120 in si_safe_eval () from /lib64/libecl.so.16.1
#22 0xccbb9130 in main (argc=, argv=out>) at /tmp/eclinitVioCS0.c:1843


So, I'm not sure how helpful that is. I will spend a little more time 
looking at it.



In the mean time, linaro has instances for loan ( 
https://www.linaro.org/leg/servercluster/ ) That said, an aarch64 target 
in fedora/qemu on x86 isn't a great solution, but on a high end machine 
is actually fairly reasonable when compared with some of the low end 
SOCs (particularly with regard to IO).  So its possible to install 
fedora/aarch64 and do basic builds. I wouldn't try building kernels on a 
regular basis but it is doable.

___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org