> This is now done and I think everything is working.
>
Congrats!
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct:
So, something broke, I forgot that the bodhi user also publishes to the
org.fedoraproject.{env}.pungi.
I fixed that now but there were quite a few messages rejected during my
night. It may be necessary to restart the compose.
Aurélien
Le lun. 10 juil. 2023 à 17:43, Aurelien Bompard
a écrit
Done. The following users are not protected by ACLs (which means they can
send to any topics):
- notifs-web and notifs-backend, because we'll remove the old FMN soonish
- alt-src: I couldn't contact the owner (Siteshwar?). Related to CentOS
Stream. I tried to contact Brian Stinston.
- coreos:
> I watched the recording today. Thanks for starting all the way at the
> beginning with the easyfix page. It was interesting to see your dev
> environment with VS Code at the beginning and OpenShift GitHub
> automation at the end, plus the tiny-stage concept. I learned a few
> things!
>
Hey folks!
This Friday at 13:00 UTC I'll be steaming on Twitch[1] about the
development of Fedora infrastructure apps. I'll start on a clean env,
checkout one of our apps, setup a dev env, fix a small bug, test it, and
create a PR.
[1] https://twitch.tv/ohwellien
I haven't decided which app
> I was going to say that one thing you need to 'add' is announcing this plan
> of changes to devel and users mailing lists and the equivalent discourse at
> least 3 times.
Very true, thanks for the suggestion, I would not have communicated
enough. Sadly, people don't like surprises.
> I guess
Oh yeah one more thing:
> - How do you see the transition to the new system? We were thinking:
> - move the current FMN to a different URL, such as
> notifications-old.fp.o. It will still be processing messages and
> sending notifications
> - run the new system in notifications.fp.o (in place
Hey folks!
I have a few questions about the final deployment of the FMN replacement:
- There's been a request to handle a user being disabled in IPA, which
should trigger their rules being disabled (FMN#826). We can do that
but we have questions about re-enablement: should the rules be
> We should drop that from dns. [...]
> Anyhow, the ssh access SOP should be updated with all this info.
I looked for the SOP and found this:
https://docs.fedoraproject.org/en-US/infra/sysadmin_guide/sshaccess/
It still mentions bastion-iad01. Am I on the wrong docs? It looks like
the right
Hey folks!
To help me search through the FMN logs during development I've written
a small script that parses and stores the logs in a SQLite database on
log01 (that I remove afterwards :-).
While doing that I noticed that MDAPI produces quite a bit of logs.
Here is the number of log lines
> I'd like to setup log forwarding on our production cluster to log all
> application level logs to log01.
+1 to that, it would be very useful to the folks developing apps as
well, as we all know that no bug ever shows up when we deploy
something to production.
Thanks!
Aurélien
> I'm missing any kind of release guide and I'm missing how to run tests
> in contribution guide (I only found how to setup dev env with vagrant
> and setup quick test instance in README).
Thanks Michal, I've added that to my TODO list.
A.
___
Hey folks!
I would like to ask you about Ipsilon and its documentation. I have
made the last significant commits to it over the past few years
(mostly during the AAA project development), and I might be one of the
few people who *kinda* know how it works. At least some parts.
That's not a very
> Well, we can actually do persistent storage in the ocp4 cluster. ;)
Oh, that's interesting! Are we using it already in one of our
ansible-deployed apps?
> I'm not sure how slow/fast it might be, but it is there...
I think it's fine, Redis will use memory first and snapshot to disk
e,
but we haven't stored a lot of data in there yet.
Le lun. 28 nov. 2022 à 01:17, Kevin Fenzi a écrit :
>
> On Thu, Nov 24, 2022 at 10:56:57AM +0100, Aurelien Bompard wrote:
> > Hey folks!
> >
> > The new version of FMN will run in OpenShift and will use Redis as a
> &
Hey folks!
The new version of FMN will run in OpenShift and will use Redis as a
cache backends (we chose it over memcached because it can do native
"is-this-string-in-this-set" operations).
I can deploy redis inside my openshift project easily enough , but I
was wondering if it would be
> > 1. Sync the prod DB to staging.
>
> I think it might work, but not sure we have anyplace off hand with
> enough disk space. We might. I can look more if this is the way we want
> to go.
Well if we don't have the disk space on staging then let's do something else.
> > 2. Having a second
Hey folks!
There's been a report of queries long enough to cause a timeout in
datagrepper:
https://github.com/fedora-infra/datagrepper/issues/467
I don't think those queries should take so much time, and I'd like to debug
this performance issue, possibly try a couple new indexes on the tables,
> Hey, folks. Just a note on the FMN replacement plan - as part of that
> involves making sure important things have fedora-messaging message
> schemas, I thought I'd link to a thing I wrote a while back which may
> be handy:
>
> https://pagure.io/fedora-qa/python-ci_messages
Thanks Adam, I'll
Hey folks!
I have begun setting topic authorizations on our message bus: apps will no
longer be able to send messages to any topics, only to those they are
explicitly allowed to. I'll need your help to make sure I'm not forgetting
topics that your app wants to send to.
In RabbitMQ these
> However, fas is still there, so when we take down the cluster, badges
> will break. Ideally we would fix that before we take down the old
> cluster, but I don't want to leave it running there too long.
>
I'll check if there's an easyfix for badges' reliance on FAS. It may not be
that much work.
> I've moved a bunch more projects the last few days.
>
I've realized with datagrepper that we need to move apps that share a
virtualhost at the same time. Otherwise the SSLProxyCACertificateFile value
in the HTTP proxy will conflict and things will fail.
Luckily datagrepper only conflicted with
REMOVE: fas-changes.yml
> ( I think this was just needed for a short time for the account system
> migration, please correct me if it's got some better use)
>
Correct, it can be dropped.
> REMOVE: ipsilon.yml
> ( we moved this to vm's because we couldn't run pam_sssd in openshift.
> Has
Hey folks!
A few months ago I started a library to share some boilerplate code in our
applications when it comes to SQLAlchemy.
Remember the thread about Flask and SQLAlchemy
> The bigger problem is that those applications are *not* able to easily
> be deployed outside of Fedora infrastructure. One consequence of
> OpenShift based deployments is that it's become almost too easy to
> assume nobody else would ever want to run that code.
Because of this, it becomes hard
>
> I'm going to deploy the recent changes to production soonish (probably
> tomorrow early morning UTC).
>
And it's done. There were a couple hiccups because of course I did not
record everything I did on staging to make it work, but it's now working
fine. Enjoy the new OTP field! :-)
Aurélien
> As a package maintainer... I LOATHE pinning. ;(
>
Let me rephrase that and please tell me if I'm correctly representing your
thoughts.
You loathe somebody else deciding which dependencies you must use.
That's fair, it's a distro packager's hell.
However in this case I think it's pretty
Hey folks!
I have recently been given the powers to make Ipsilon releases, so I'm
going to deploy the recent changes to production soonish (probably tomorrow
early morning UTC). We've been working with a snapshot so it's not as big
an update as you'd think when looking at the date of the last
> Something like:
>>
>> Applications in Fedora Infrastructure may be deployed via non rpm
>> methods (as long as they obey licensing guidelines (
>> https://fedoraproject.org/wiki/Infrastructure_Licensing )). For those
>> applications, creating and maintaining an rpm is optional.
>>
>>
> How
Hey everyone!
Bodhi 6.0 will be published in a few days, and deployed to production a
couple weeks after the Fedora release. It has backwards-incompatible
changes, here's what you need to know.
== Authentication ==
Bodhi gained support for OpenID Connect (OIDC) authentication, like most of
Hey Ondrej!
On Wed, Mar 16, 2022 at 12:50 AM Ondrej Nosek wrote:
> I don't have expertise in Irish holidays (I know there is one on this
Thursday), so I don't know how much time I have.
This was my attempt at a joke: I was suggesting the worst possible moment,
when everybody is on holiday.
> well, the cron job that does daily bodhi updates pushes (when we are not
> in freeze) calls 'bodhi-push --username releng'.
>
Would this be affected? I am not sure how it authenticates currently. :(
>
Nope that's not the same bodhi client, the bodhi client I'm asking about is
just "bodhi". This
Hey folks!
We are preparing for the deployment of the next major release of Bodhi
(planned for a Friday evening on an Irish bank holiday during freeze in the
Thanksgiving extended weekend), and the authentication has changed, which
means automated calls of the bodhi client ("bodhi" command line)
> But yeah, making it impossible to use the bodhi cli without opening a web
> browser for authentication would be bad for my use cases / my projects -
> particularly fedora-update-feedback. If I need to open a web browser for
> authentication, I can just use it to submit bodhi feedback as well,
Hey folks!
A long email to give you some context, please bear with me :-)
A while back, Bodhi's integration tests stopped working on the "pip"
release (basically the latest python packages from PyPI). Since the
integration tests were flaky at that time, they were disabled on the
"pip" release.
> Do you think it's a good idea to do this on Friday?
Well, I did not say Friday *evening*, so, this is fine :-D
Yeah Monday is better, I realized it after sending the email :-)
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> I am back now so we can do this whenever suits you
Cool! What about this Friday morning? Too short notice? It should take
an hour or two, less if we're lucky.
WDYT?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To
Hey folks!
I am happy to report that the datanommer data migration is finally
complete! \o/
Now we can move on to migrating the apps themselves. I had initially
written this plan:
https://github.com/fedora-infra/datanommer/wiki/Migration-plan
We'll need to set a downtime window, I'd say of a
Hey folks!
I published version 3.0.0 of Fedora Messaging this morning. It is a major
release, and the backwards-incompatible changes are:
- Queues created by the CLI are now non-durable, auto-deleted and
exclusive, as server-named queues are.
- It is no longer necessary to declare a queue in the
> Sorry for taking so long to reply. I'm afraid I don't check this mailing
> list as often as I should. :)
>
Totally fine, thanks for the reply!
When I thought about that use case, I supposed it would be OK to
> instantiate the app and start the app context from within the script, as it
> would
>
> - we end up with many slightly different integrations, written by
>> different people or even the same person at different points in time.
>> Ironically, our attempt at avoiding tech debt has caused us more tech
>> debt.
>>
>
> One approach could be to build your data-models as a dedicated
Thanks for your input!
1. We're using a clustered database (CockroachDB, for those who care)
> that uses optimistic concurrency, so automatic transaction retries are
> a must, and we need control over how those retries are done.
>
Interesting, we don't use that, but then again we've recently
Hey folks!
I'd like to open a can of worms: SQLAlchemy integration in Flask. It's a
long read but I hope you'll like it. I try not to be on the ranty side.
First, some context: we have been plagued by tech debt for a very long
time, maybe more than other development projects because web tech has
Hey!
I'm trying to write an application that is cloud native, that needs to
> be able to interract with the FAS for Fedora Account System User ID,
>
If you want to auth your users against FAS, the best way to go is OIDC
(OpenID Connect)
> also for Fedora Badges. I am wiritng this using
Hey folks!
I just released and deployed Noggin 1.4.0. Here are the release notes:
== Features ==
* Improve the display of group communication channels (IRC or Matrix)
(#309).
* Add the email address in the user’s profile (#568).
* Display the SSH public keys on the user’s profile (#676).
*
Hey everyone!
I just released and deployed FASJSON 1.3.0, it only contains a new feature
and a bugfix.
* Add some more user fields: github_username, gitlab_username, website, and
pronouns (#213).
* Respect the user's privacy setting on the search endpoint (#257).
This last item fixes an
> So, how about this: Just disable notifications there in #fedora-apps for
> now.
>
> If someone wants them (or a subset back), they can propose re-adding it
> there or in another channel?
>
Works for me! Thanks!
___
infrastructure mailing list --
> > At the moment I get notified of my own
> > actions which is extra annoying.
>
> Yeah. But if you silent the notification channel, do those notices do
> any good?
>
I could decide to look at it when I'm waiting for something to finish, or
re-enable temporarily the notifications if I'm waiting
Yeah I see what you mean. I don't think IRC notifications are useless, but
if they are in a different channel I can set this channel to be silent even
on messages with my nickname. At the moment I get notified of my own
actions which is extra annoying.
The use case you described still generates
> > - either "dev", "devel", or "develop"
>
> Had a quick look, and there are over 50 already at "develop" as the
> main branch. -- most of the others are 'main' or 'master' -- so it
> looks like 'develop' is a bit of a standard already.
>
Alright. I think I've setup most of the projects that
Hey folks!
We currently have messages posted on the #fedora-apps and
#fedora-infrastructure IRC channels when there's a ticket change or a
pull-request change. I don't know about the infrastructure channel, but it
makes it difficult to have a development conversation in #fedora-apps, the
Hey folks!
I think most of the repos just went with GitHub default, which recently
> changed from master to main.
> In Anitya and the-new-hotness I have:
> - master
> - staging
> - production
> The staging and production corresponds to deployment in OpenShift. This is
> why I named them like
> * Do we want to get noggin to be able to verify nicks first?
>
> How will the verification works?
>
We don't know yet. I was thinking of having and IRC bot that would get an
HTTP request from Noggin to verify a user, and would send a link with a JWT
token as a private message that the user
Hey folks!
I have released and deployed Noggin 1.2.0 to production a few minutes ago.
Here are the release notes:
Features
- Display the version in the page footer (#592).
- Allow sponsors to resign from their position in the group (#599).
- Disallow login and register with mixed-case usernames
Hey folks!
I have released and deployed FASJSON 1.1.0 to production a few minutes ago.
It's a small release, as you can see. I've also rebased the Openshift image
on F34 (it was on F32).
*Features:*
- Field mask support: request more or less object attributes with a HTTP
header (#144
Hey folks!
I have released and deployed Noggin 1.1.0 to production a few minutes
ago. Here are the release notes:
Features
- Add a verification step when enrolling a new OTP token (#422).
- The GPG key ID fields now refuse key IDs shorter than 16 characters,
and allow up to 40 characters (the
> > Just one note: I'm not sure how the token generation works in noggin, but
> > usually you get a few seconds to use the old code when the new one is
> > generated, but I just got invalid code when the new one was generated during
> > typing the old one.
>
> I guess this is a question for IPA
> Once it's merged and deployed, the tokens will only be enabled once
> users have proven that their app works, so it should cut down on those
> "I'm locked out" requests.
OK, it's merged and deployed on staging. If you folks want to test it
out, it's at
https://accounts.stg.fedoraproject.org/
> So, we have at least a half-dozen of these pending now. ;(
I have implemented a verification step for OTP tokens, it's currently
under review:
https://github.com/fedora-infra/noggin/pull/584
Once it's merged and deployed, the tokens will only be enabled once
users have proven that their app
> So technically you can have something like:
> - create OTP token and mark it disabled
> - show OTP token configuration details to a user
> - ask user for this token validation: enter a password and a value
> - enable token
> - verify token
> - if verification fails, disable the token again
Some
> > * Could we require someone enter their password + token before accepting
> > the token? ie, they try and enroll, ipa adds it, they have to verify, if
> > they can't, it's removed?
>
> This is _very_ common in other implementations.
Yeah, but there is no API in IPA to do that (we did consider
Hey folks!
The AAA team would like to test a re-import of the accounts in staging. We
have learnt of a way to speed up the import significantly (20 times) and
we'd like to test it.
For that we'll need to remove all existing accounts and start from scratch.
It means that if you're currently
> But yeah, I think if the fas sync is going to take a bit, perhaps we
> should disable the new account creation for now.
I've added the feature to disable registration yesterday, once it's
reviewed and merged I'll push it to the staging instance and disable
the registration. Thanks for pointing
> Sure, but if you could clean them up afterward that would be good.
Will do, thanks.
> +1 for me, though I'm not sure I follow the advantage of them over say
> fedocal,
> elections or the wiki.
I could check the features I'm testing independently, such as group
membership, agreement signing,
Hey folks,
To test authentication with the new AAA system I'd like to deploy a couple
very basic apps that do nothing but auth in staging's openshift. It
shouldn't touch any configuration besides the reverse proxies and the new
project in openshift. And it's staging only.
Is it OK?
Thanks.
Thanks for the update!
> Account system / noggin:
IPA is deployed, Noggin is deployed, FASJSON (the REST API) is
deployed, Ipsilon is deployed.
Yesterday we manage to have the elections app authenticate a random
user (that would be me) through Ipsilon (OIDC) as before, except
Ipsilon is now
> I'm happy to announce that We have approved several folks into the
>> sysadmin-main group:
>>
>> mobrien - Mark O'Brien
>> abompard - Aurelien Bompard
>>
>> This is the core group of trusted folks that high level access to most
>> everything in fedo
> It doesn't? What about https://github.com/freeipa/freeipa-container ?
>
> My understanding is that it is an experimental implementation
> currently. FreeIPA does not necessarily work very well broken up into
> containers right now.
>
Yes, and running FreeIPA in a container requires the
> > I am not sure what to do.. I do not know how hard it would be to pull
>> basset out of the system and I do not have the time to update/fix/improve
>> Patrick's code on this. So I figured it would be good to get some feedback
>> on this.
>>
>
So, I guess the new AAA system doesn't have to
> We have been having the cluster fall over for still unknown reasons,
> but this patch should at least help prevent them
>
I wish I understood what's actually going on, but +1 on those changes to
see if they help.
If they do we may consider reverting to the default when we upgrade to the
newer
> I hit some permissions problems with the playbook that I can't figure
> out.
>
I found why, apparently when tags (rabbitmq tags, not ansible tags) aren't
specified with the rabbitmq_user ansible module, it clears them while I
thought it would leave them alone.
I've fixed it, it should work now.
Hey folks,
I thought I'd make a summary of where I'm at. Here are the issues I found
and what I did about it:
- We ran into an Ansible issue that the PR
https://github.com/ansible/ansible/pull/50381 fixes. I've asked pingou to
patch batcave since it's basically a one-liner that will keep working
> Patch looks good and you have a plan of action. +1
>
Thanks. I've pushed the Ansible change and moved the build from
epel7-infra-stg to epel7-infra, but now I need someone in sysadmin-main to
update the RPM and run the playbook on autosign01, since I don't have the
permissions for that.
I'm on
Hey folks,
Last Monday, before the freeze, we updated Robosignatory in prod with a few
new features, some of which could not be tested in staging as thoroughly as
we wanted to. As a result, the version currently in prod has issues with
the CoreOS artifacts. We've worked on that and our tests in
> Do you have a ansible patch here?
>
>
Yes, sorry, this is it.
diff --git a/roles/rabbit/queue/tasks/main.yml b/roles/rabbit/queue/tasks/main.yml
index 7259984f6..68ced3015 100644
--- a/roles/rabbit/queue/tasks/main.yml
+++ b/roles/rabbit/queue/tasks/main.yml
@@ -66,7 +66,7 @@
Hey folks,
The fedora-messaging consumers are currently subscribed to the amq.topic
exchange where they get all messages sent over AMQP. However, the bridges
that forward messages from fedmsg publish to the zmq.topic exchange,
therefore consumers need to subscribe to that one too to benefit from
Alright, I now have quite a few checks for the RabbitMQ servers. Those
checks also give out interesting metrics like queue sizes, connections,
message throughput, etc.
Do we have something in place to use and display those metrics?
I'd like to look at what our common usage values and trends are
>
> What should I do? Create a SELinux module to allow that connection? Do we
> have a policy regarding that sort of module creation?
>
I see that the Copr role has a policy module in Ansible (both source and
binary), copies the binary to the destination and loads it with "semodule
-i". Can I do
>
> I've made a few changes to Jeremy's proposal, because I wanted to make use
> of the configuration file that the NRPE plugin already deploys.
> Attached is my proposed change to the Ansible repo.
>
> If that works I'll add more checks later on.
>
>
OK I deployed that config but now SELinux is
>
> I'd like to try to implement this, and possibly add app-specific
> monitoring of queues afterwards.
>
I've made a few changes to Jeremy's proposal, because I wanted to make use
of the configuration file that the NRPE plugin already deploys.
Attached is my proposed change to the Ansible repo.
Le jeu. 16 mai 2019 à 16:52, Jeremy Cline a écrit :
> Commit eae92f73e95 installed the nagios scripts[0] that are packaged for
> epel7-infra on the RabbitMQ hosts. This is an attempt to use them with
> nagios. I don't know anything about nagios though, so I have no idea if
> this is even close
> My fear here is that someone will manually create something and we have
> to redeploy for some reason. They will be broken untll they manually
> remember to do what they did again. :(
>
It's not manual at all actually, the queues that should be declared are in
the app's configuration file,
> Can I get +1s for the patch to the "dns" repo underneath?
> This should make "rabbitmq.fp.o" resolve to proxy101/proxy110 internally.
+1 ! :-)
A.
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email
Hey folks,
I'm migrating bugzilla2fedmsg to Fedora Messaging, and I thought it'd
be a good opportunity to migrate it to OpenShift also.
It only requires a connection to the STOMP brokers[0] on port 61612.
Is this available from inside OpenShift?
[0]
> my overall feeling is that the
> risk of DoS should be one of the factor we take into account to make
> the decision but we should also consider how easy is it to use, how
> easy is it to maintain, how much effort is it to setup.
I agree, and since both burdens (daily maintenance and dealing
I'm assuming you're considering the solution where we have a single
broker and we make it publicly accessible (option 1).
> how easy would it be to turn off the possibility for external
> publisher to flood the broker ?
External clients won't publish anything, they'll be read-only (with a
few
Hey y'all,
Fedora Messaging, the replacement for fedmsg, is using AMQP and thus a
message broker. The current clusters we have deployed in staging and
prod are only accessible from inside our infrastructure.
There are two needs for an externally accessible broker:
- the CentOS folks, who are
Hey Leigh!
- Project you are actively working regularly on
>
- Fedora Messaging
- Bodhi
> - Link to the Landing Page / Tracker / Source / Docs / anything relevant
> really that might help me get a handle on the project
>
Fedora Messaging:
- https://fedora-messaging.readthedocs.io/en/stable/
-
Hey folks!
Pingou would like to announce the availability of mailing-lists on
lists.pagure.io with the 5.0 release. The following patch should add
the new domain to our mailing list server.
Affected services are the mailman server and the proxies.
Can I get a couple +1s?
A.
commit
> I like the CI test idea, a little bit like when we tests that the code base
> is pep8 compliant or the test coverage in above 90%. There are a couple of
> python packages that could be useful to help with that [0] [1].
>
> [0] https://github.com/dhatim/python-license-check
> [1]
> Could this caching proxy just use EmpyDir (ie, only for the life of that
> pod) and just refresh when it restarts? If it really needs disk, might
> be better to do on a vm at this point.
Since it's just caching, I guess that would be sufficient, unless we
cycle the pod frequently. It would be
> everything should be there for
> this with one exception: We really want to have some check in place for
> s2i so that it checks license, so we don't accidentally push out
> something thats not under a open source license. This doesn't need to be
> a blocker, but it would be great to get in
> I've been playing around with openshift staging for the last few weeks
> and enabling some cool features. :)
Cool! I seem to remember that having persistent storage in our
Openshift instance was a difficult thing. I'm considering Openshift to
setup a PyPI caching proxy for us, and that will
>> https://github.com/noirbizarre/flask-restplus
Actually, I haven't tried that one. It seems pretty good (from the
docs), has anybody tried it?
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to
> Within limits. It should be a version thats supported and gets at least
> security updates. Hopefully the one(s) in Fedora follow this.
Yeah it's 1.11 now which is LTS, since it'll be the last version to
support Python 2
> There are a few flask rest frameworks, but I have not much idea how
> I'm a little worried about Django. True, we have to maintain a version
> for mailman3, but it's rhel7/python3. Is this new app going to use that?
Actually, HyperKitty and Postorius are using Django on Python 2.7. The
Django version is 1.8 and it's pretty old now.
I would recommend against
> It's nice to give the flexibility to clients by exposing both. I
> haven't seen a problem with topic matching in my experience so far.
While I like the idea of adding flexibility, it'll probably also be
harder on the debugging and maintenance side of things. We will keep
the ZeroMQ gateway for
Hey folks!
Jeremy and I have been working on a proposal to migrate fedmsg from our
current brokerless architecture to a broker-based architecture.
The overview and reasons for the migration are described on this page:
> > The "pdc-lite" options are attractive, across the board.
>
I know Django and Django-REST-Framework, and I've made a small contribution
to PDC a few months ago, so I may be of use if that's the path we choose.
Aurélien
___
infrastructure mailing
> Is there a list of changes in this new version?
>
Not exactly, there's a lot of fixes but very few new features (and nothing
very obvious for the user anyway).
> Has staging been updated ok?
>
Yes, it's been a couple weeks now, it works fine.
Thanks for the +1 folks!
A.
1 - 100 of 161 matches
Mail list logo
