Re: on rpms

[Stephen Smoogen]

On Mon, 30 May 2022 at 21:26, Neal Gompa  wrote:

> On Mon, May 30, 2022 at 11:20 PM Miroslav Suchý  wrote:
> >
> > Dne 23. 05. 22 v 20:57 Kevin Fenzi napsal(a):
> > > However, now a days we have a number of new apps that are deployed in
> > > openshift and aren't using rpms, but pip or s2i or other things.
> >
> > Regarding pip applications - we have pyp2rpm and pyp2spec which can
> convert to rpm easily. And we have
> >
> >   https://copr.fedorainfracloud.org/coprs/g/copr/PyPI/
> >
> > with about 70k packages.
> >
>
> The bigger problem is that those applications are *not* able to easily
> be deployed outside of Fedora infrastructure. One consequence of
> OpenShift based deployments is that it's become almost too easy to
> assume nobody else would ever want to run that code. Noggin and Bodhi
> both have had a number of code changes in the past couple of years
> that have made it increasingly difficult to use outside of Fedora
> deployments in their default code state. I've given up on Bodhi, but
> I'm still trying to get Noggin into good shape.
>
> I've been able to evaluate those issues by packaging them as RPMs,
> because RPM packaging forces a total decoupling of development,
> deployment, and configuration. None of that is true with our container
> based deployments. They're not discoverable, and if you can find them,
> they're not independently useful.
>
> Because of this, it becomes hard for community growth around these
> projects.
>
>
I think we should be clearer about if community growth is expected. 15 to
10 years ago, we tried very hard to grow community support into helping to
build things but found that the requirements of what was wanted in the
applications always took more time and energy than volunteers had. Over the
last 10 years, we have had less and less time to do this community growth
work.

 Fedora Infrastructure's primary mission is to provide a working build
system and produce N thousand deliverables every day (be this individually
built RPMS, containers, spins, isos, raw images, etc etc). We are to do
this while also maintaining the core community features which allow for
mail, badges, and outsourced tools to work as best as possible. The amount
of resources to actually make it work where we could have people do the
marketing and community support needed to make generic apps or even build
an infrastructure community is about 2x what we have people for IF we only
had to keep the current applications going. However as soon as you hire
someone in this space, you find that 3-4 new projects which were waiting to
be resourced become high priority. And the timeframes to delivery are
shortened as it is no longer acceptable to have a tool in development for N
years before initial deployment like Bodhi and some others had. Instead you
usually go from prototype to full production in 1 to 1.5 release cycles (6
to 9 months).

In order to meet this, we usually have to bake all the 'business rules' of
the time into the application. The idea being that when the rules change,
the developer will just likely rewrite it from scratch to meet those.
[Since one of the lessons learned from when we did 'long form development'
was that a lot of code was rewritten every time a new framework which
helped solve something better was found.]

Finally, I think trying to decouple deployment, development, and
configuration is going against the tide. "Infrastructure as Code" sounds
great but it generally means that those 3 things are tied together deeply
in every application and tool written these days. Like the tide, this is a
cyclical pattern in the industry and it will roll out and leave a pile of
bad example flotsam that few will remember the next time the tide comes in.


-- 
Stephen Smoogen, Red Hat Automotive
Let us be kind to one another, for most of us are fighting a hard battle.
-- Ian MacClaren
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Announcing sqlalchemy-helpers

[Aurelien Bompard]

Hey folks!

A few months ago I started a library to share some boilerplate code in our
applications when it comes to SQLAlchemy.

Remember the thread about Flask and SQLAlchemy
?
It ended with us admitting that Flask-SQLAlchemy is pretty cool, but does
not cover our non-Flask needs, can introduce issues with scripts that do
multithreading manually, and ties us into Flask more than what we may feel
comfortable with.
But the problem remains the same: by doing SQLAlchemy integration manually
over the years, we've copied code from one project to the next, improving
it when we were working on it but not backporting those improvements to
older projects, and having rather sparse (I should say "distributed" ;-) )
knowledge of what the SQLAlchemy & Alembic best practices are (everybody
here knows about constraint naming conventions? Foreign key enforcement on
SQLite?)

In the spirit of avoiding code duplication and sharing maintenance, I've
started the sqlalchemy-helpers library
.
The idea is to standardize on best practices with SQLAlchemy & Alembic
usage without depending on Flask, but with an optional Flask integration.
The README has a list of features and usage examples, so I'll let you go
over there and skim read it.

Basically what I've done is go over all our Flask apps using SQLAlchemy,
look at what features they implemented, and bring them into
sqlalchemy-helpers. As a result if you decide to move to it you will only
gain features.
The only notable exception is the pagination system that Flask-SQLAlchemy
implements and that at least one project has copied. If it proves useful,
we can always bring it to sqlalchemy-helpers, but let's start small for now.
It targets SQLAlchemy 1.3+ & 2.0, Flask 2.0+ and Python 3.6+.

I've used it personally for a few months already and I'd say it works quite
well. I'm very interested in feedback. My next step is to improve user
documentation because right now it's only the README and the APIdocs.
I hope we can join forces and maintain this sort of boilerplate code in a
central place.

Cheers!
Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: on rpms

[Aurelien Bompard]

> The bigger problem is that those applications are *not* able to easily
> be deployed outside of Fedora infrastructure. One consequence of
> OpenShift based deployments is that it's become almost too easy to
> assume nobody else would ever want to run that code.

Because of this, it becomes hard for community growth around these
> projects.
>

I think that's a fair point.


> I'm still trying to get Noggin into good shape.
>

I'm interested in which parts of the Noggin source code make it hard to be
deployed outside Fedora. In my opinion those are bugs, because we do want
others to be able to deploy it. Do you have some pointers? I'll easily
admit we may have gone the easier route by assuming what our infra looks
like in a few places, so I'm happy to fix that.


> I've been able to evaluate those issues by packaging them as RPMs,
> because RPM packaging forces a total decoupling of development,
> deployment, and configuration. None of that is true with our container
> based deployments. They're not discoverable, and if you can find them,
> they're not independently useful.
>

Hmm, I'm not sure I agree. Containers can be decoupled from the deployment
and configuration, can they not? That's how people use all those generic
containers on DockerHub, no?
It's probably extra effort to make our containers runnable in different
infra, but I'm pretty sure there's also some work involved with making our
RPMs buildable/runnable in other distros, no?
So I'm not convinced RPMs are inherently better than containers at that.

Aurélien
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure

Re: on rpms

[Pierre-Yves Chibon]

On Tue, May 24, 2022 at 06:24:10PM +0200, Aurelien Bompard wrote:
> > Something like:
> >>
> >> Applications in Fedora Infrastructure may be deployed via non rpm
> >> methods (as long as they obey licensing guidelines (
> >> https://fedoraproject.org/wiki/Infrastructure_Licensing )). For those
> >> applications, creating and maintaining an rpm is optional.
> >>
> >>
> > How about:
> >
> > Applications in Fedora Infrastructure need to be deployed in an auditable
> > and repeatable way. These methods need to allow someone to determine which
> > software was installed, when it was installed, and what it was meant to be
> > done (example: rpms or podman build scripts for containers). The goal is to
> > be kind to our future selves at 2 am who need to figure out why a critical
> > application is broken and how to rebuild and redeploy as needed.
> >
> 
> Yeah that seems sensible (although I'm not sure of the wording of "what it
> was meant to be done", but I think I get it).
> This would satisfy apps built with s2i as long as they are pinning their
> dependencies with something like poetry or pipenv. We are currently
> standardizing on poetry but any would do as long as deps are pinned).
> 
> For s2i based apps, I see two ways of ensuring repeatability, one being
> stricter but more transparent than the other:
> 1. have the buildconfig track a production branch upstream, and rely on the
> build log to know which exact commit was built
> 2. have the buildconfig specify the commit hash, and change the buildconfig
> each time we want to deploy a new prod version

A third option which we've used in a few of our apps is to have a specific
branch (or branches) for our deployment. That branch can then have commits
dedicated to our deployment, such as support for s2i which aren't needed in the
'main' branch.
We could do things like the version pining in that branch as well, making it
easy/easier for people to package the application while still helping us ensure
the reproducibility we want in openshift.


Pierre
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct: 
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: 
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it: 
https://pagure.io/fedora-infrastructure