This has been keeping me up for the last week or so.. so I finally took the
time this morning to try and come up with an 'alternative plan'
# Papyrus Project
The following is a very short attempt to meld Gordon Messmer's general idea
of faster delivery with existing tools and ideas that Fedora has. It is
not a full plan but more of a "concept of a plan".
:toc:
## Why Papyrus?
Papyrus is one of the first writing materials known to exist. It is both
fragile and yet durable at the same time. It is also an font avatar for
madness in some readers. As such I figure a tool which is upstream for
rawhide which might be useful but also prone to fits should be named it.
## Moving things faster without breaking things
The first things we need to do is work out any rules we need to
follow. The first rule is "we can't break rawhide." The next rule is "we
need to use tools already built as much as possible". Third rule is that
"The bill comes due. Always" so have an idea of what the costs would
be. There are probably others but we need to start with the 'simple'
ones first.
## Don't break rawhide
In order to not break rawhide we need to work out where rawhide is and
how it is generated. In general, it is built from packages which have
their 'spec' files stored in source.fedoraproject.org and source code
stored in a lookaside cache using koji as the recorder of all builds
which will go into releases. Rawhide is meant to be an 'always ready to
branch' release, which I consider different from an rolling release
since the goal isn't to be always usable but always ready to 'branch'
into a more solid release. This means breakages are supposed to be short
lived and planned by packagers on how the fixes are to happen.
One problem that has been seen is that this means a lot of packages can
get delayed as various integrations occur. Another problem is that
people may not know that upstream has pushed multiple updates unless
they are dialed into that upstream. There are a lot of upstreams
however which takes too much time from a limited volunteer budget. The
helpers on this in the past have been anitya/thenewhotness which can be
set up to alert the developer. Tie this into packit and you can get a
scratch build which will tell you if you need to do additional work. The
problem of course is that there are still a lot of work which needs to
be done as many software projects like to pull in tons of additional
dependencies every couple of hours.
Additional tooling would need to be written:
1. take upstream releases at various hubs like pypi, rustoleum(no idea
what it is called), etc., and then work out what dependencies are
needed and what versions they are. Determine what 'cooldown' periods
are active for the various dependcies (to try and avoid upstream
attacks on code), and then download if ok.
2. See if the spec file and code has already been written for the
various dependencies.
1. If there is a spec file, then make a branch and update with
the new version of the code.
2. If not, use the various toolkits that have been written to
write new spec files as needed.
3. Do scratch builds to see if they build.
4. If they don't.. add to report on what is currently broken to
build X.
5. If not check in the spec file to a papyrus-source system, and
the source into papyrus-lookaside.
3. Send a report to the interested parties about what happened and what
didn't. Person either checks in/fixes the broken specs and pushes
them into the system for a retry or this build is listed 'orphaned'.
4. rawhide maintainers of these items can be alerted of this work if
they want but they are **NOT** responsible unless they sign up for
this. When they want to update in rawhide, they can check out the
code from this and work on solidifying it for 'Always Ready Rawhide'.
5. Other automated tools to be written would be to check for active CVEs
or other problems against anything that is in papyrus and then mark
the appropriate spec files in their branches as insecure with said
CVE or other problems. Source code in the look-aside would also be
tagged as such (this changes how look-aside looks and will need
work).
6. Future work could also improve any automated spec file writers so
that templates and other things could be used for areas which are
more complicated than rust2rpm or other tools can do.
The goal of this is to automate various work which could be taking up
time that people don't have anymore. The work is stored in ways which
could be useful for other parts of the buildsystem but not directly
affecting rawhide. It is also useful for rawhide in that you can see
what stuff will not work in rawhide way before you actually want to put
it there.
## Workflow
My 20k light-year view of a workflow is something like this:
1. Packager gets added to the papyrus group
2. Packager adds in the first level of packages they want 'watched' and
built.
3. Automatic tools build the first round of package spec files
4. Review tools check and find problems.
5. Test builds are attempted.
6. Packager updates base spec files on any failures that the auto-tools
did not fix.
7. Cycle of search for new versions, attempt to build with packit, push
changes to papyrus-git, etc starts.
8. When packages are needed by multiple items, each leaf packager is
added to the child packages as a comaintainer. If a leaf package is
orphaned, then they are removed.
### Rawhide
The main benefit for rawhide would be to allow for 'faster' builds to show
where things are falling down or where things could be better automated to help
people. This allows for a code path where a lot of scans we don't do in Fedora
could be done regularly:
1. CVE marking
2. License reviews
3. General package reviews to keep spec files up2date with current packaging
standards.
4. [I think we all have others that we have wanted but trying to get it into
the existing Fedora build system without causing a multi month outage is very
low.]
### Flatpaks
This is where it falls more into what Gordon and others have wanted for a long
time.. a faster alternative to distributions where they can put whatever
explosive packages into bubblewrap and mostly protect the user who also wants
possibly dangerous code.
I expect this plan has holes the size of planets, but like I said, I am
looking at this from a long way.
## The bill comes due. Always
This is the harder part of most proposed projects. There is no such
thing as a free lunch, and the bill always comes due if you don't pay it
in advance. The above project requires multiple tools which need to be written
still, it needs a project team to tie the tools together, and it needs a
much larger set of storage and builders in order to happen.
I would figure it would take 3-4 developers fulltime about a year to
work on making a new forgejo, backing store, spec-system writer, tie ins
to copr for building, an updated package permission model, and a running
budget to continue going that does not impact Fedora's existing small
budget. There would be other project constraints and timelines which
would need to be factored in so we are talking about adding someone to
be a IT, Project Manager, Team Lead, and Marketing (because someone has to
figure out how to make sure it doesn't land like a Marvel Sequel no one
asked for.)
So I am looking at about 7 or 8 full time positions which will have some
unknown operational expense cost. Then there is the storage, build, and
other services which we can limit down to aarch64 and x86_64 only (no
i686, s390x, or powerpc). A rough cost in the US would be 5.2 million
per year with a minimum 1 year timeline before minimum integration. We
could hopefully cut some of the costs by working with Amazon and
Microsoft partnerships to use as much cloud resources as possible. The
reason to "sell" it to them is that they would gain the most from this
possible faster container workflow.
## End thoughts
I don't know if anyone would want to invest in this to get it done. It
still requires a lot of work to move this to 12% of a plan. However, it
is what the brain said at 2am was a first stab of a plan. [THis could also be
99% of what konflux is supposed to do so a lot of this work may be already
done.]
On Mon, Sep 28, 2026, at 12:17, Kevin Fenzi wrote:
> On Sat, Sep 26, 2026 at 01:43:55PM -0700, Gordon Messmer wrote:
>> On 2026-09-26 12:38 AM, Björn Persson wrote:
>> > So that particular program is ahead of some other programs that are
>> > holding the distribution back. That seems to confirm that programs lag
>> > and use old libraries.
>>
>>
>> Yes, right now Fedora Rawhide lags behind and uses old libraries because
>> sometimes updating libraries is disruptive to Rawhide.
>>
>> Decoupling some applications from Rawhide would remove some of the friction
>> that is currently causing Rawhide to lag.
>
> Thats one reason, but there are many more, including, but not limited
> to:
>
> * maintainers just haven't gotten to that update yet
> * It's deliberately not being updated because there's an issue with it
> and thats being worked out.
> * The upstream update has no changes that matter for the fedora package.
> (windows bugfixes, changes in some feature thats not enabled).
> * The update is minor and maintainers don't think it's worth updating.
> * Maintainers are waiting for a larger update thats landing soon, so
> skipping minor ones until that happens
>
> repology used to track the 'behind' versions FWIW.
> Sadly that site is gone, but:
>
> https://web.archive.org/web/20260914132657/https://repology.org/
>
> But anyhow, I think if we want rawhide to be 'more up to date', we
> should look more at helping maintainers rather than just saying 'oh we
> are 100% up to date, but all these branches aren't ones we use'.
>
> kevin
> --
> _______________________________________________
> devel mailing list -- [email protected]
> To unsubscribe send an email to [email protected]
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/[email protected]
> Do not reply to spam, report it:
> https://forge.fedoraproject.org/infra/tickets/issues/new
--
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard battle. --
Ian MacClaren
--
_______________________________________________
devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/[email protected]
Do not reply to spam, report it:
https://forge.fedoraproject.org/infra/tickets/issues/new
