[PATCH 2/4] Support running tests from any directory

--- test/setup.sh | 5 ++--- test/t1100-git-auth.t | 2 +- test/t1200-git-serve.t | 2 +- test/t1300-git-update.t | 2 +- test/t2100-mkpkglists.t | 2 +- test/t2200-tuvotereminder.t | 2 +- test/t2300-pkgmaint.t | 2 +- test/t2400-aurblup.t| 2 +-

[PATCH 1/4] TESTING: Step for installing the Python package

--- TESTING | 13 +++-- 1 file changed, 11 insertions(+), 2 deletions(-) diff --git a/TESTING b/TESTING index 4a1e6f4c..7476bc97 100644 --- a/TESTING +++ b/TESTING @@ -22,7 +22,16 @@ INSTALL. Be sure to change backend to sqlite and name to the file location of your created test

[PATCH 4/4] Write test/README.md to help working with tests

--- test/README.md | 38 ++ 1 file changed, 38 insertions(+) create mode 100644 test/README.md diff --git a/test/README.md b/test/README.md new file mode 100644 index ..c48001d8 --- /dev/null +++ b/test/README.md @@ -0,0 +1,38 @@ +Running tests

[PATCH 3/4] test/Makefile: Run tests with prove when available

--- test/Makefile | 7 +++ 1 file changed, 7 insertions(+) diff --git a/test/Makefile b/test/Makefile index d310c8f5..758befa3 100644 --- a/test/Makefile +++ b/test/Makefile @@ -1,6 +1,13 @@ T = $(sort $(wildcard *.t)) +PROVE := $(shell command -v prove 2> /dev/null) + +ifdef PROVE

Re: [PATCH] Set up Alembic for database migrations

Lukas Fleischer [2020-02-26 14:00:51 +0100] > We should also think about phasing out the text files in the upgrading/ > subdirectory. Maybe we should simply keep them for one or two releases > and remove the directory entirely later? If anybody ever still needs > them after they have been removed,

[PATCH] rendercomment: respectful linkification of git commits

Turn the git-commits markdown processor into an inline processor, which is smart enough not to convert Git hashes contained in code blocks or links. --- aurweb/scripts/rendercomment.py | 36 ++--- 1 file changed, 20 insertions(+), 16 deletions(-) diff --git

[PATCH] rendercomment: safer auto-linkification of URLs

Fixes a few edge cases: - URLs within code blocks used to get redundant <> added, breaking bash code snippets like `curl https://...` into `curl `. - Links written with markdown’s syntax also used to get an extra pair of brackets. --- aurweb/scripts/rendercomment.py | 19

[PATCH 1/3] rendercomment: safer Flyspray task linkification

When an FS#123 is part of a code block, it must not be converted into a link. FS#123 may also appear inside an URL, in which case regular linkifaction of URLs must take precedence. --- aurweb/scripts/rendercomment.py | 21 ++--- test/t2600-rendercomment.sh | 26

[PATCH 3/3] rendercomment: use python-markdown's new registration API

First, this gets rid of the deprecation warnings Python displayed. Second, this fixes the case where a link contained a pair of underscores, which used to be interpreted as an emphasis because the linkify processor ran after the emphasis processor. --- aurweb/scripts/rendercomment.py | 10

[PATCH 2/3] rendercomment: test headings lowering

--- test/t2600-rendercomment.sh | 26 ++ 1 file changed, 26 insertions(+) diff --git a/test/t2600-rendercomment.sh b/test/t2600-rendercomment.sh index 1da422d..1ba560a 100755 --- a/test/t2600-rendercomment.sh +++ b/test/t2600-rendercomment.sh @@ -129,4 +129,30 @@

Automated database migrations with Alembic

Hello everyone, After discussing with Lukas about automation and database migrations, he suggested I write a proposal. Here it is. By the way, I’m a new contributor, and nothing has yet been decided on who would implement this proposal if we do decide to follow it. Feedback and contributions

TAP conventions for aurweb’s test suites

Hello everyone, The aurweb test suites are are currently run by make, looking for all files matching `t-*.sh`. See `test\Makefile`. Each test program is standalone and follows the TAP conventions. More about TAP at https://en.wikipedia.org/wiki/Test_Anything_Protocol While make is better

Re: TAP conventions for aurweb’s test suites

Lukas Fleischer [2020-02-22 09:58:15 +0100] > The current tests mostly execute commands and compare their outputs, and > I don't see much benefit in rewriting them in Python. That being said, I > think we could benefit from additional unit tests. Exactly. Using .t instead of .sh is about opening

Re: [PATCH] migrate the database schema to SQLAlchemy

Lukas Fleischer [2020-02-22 22:43:31 +0100] > > +5) Create a new MySQL database and a user and import the aurweb SQL schema: > > + > > +$ python -m aurweb.initdb > > + > > I noticed that this step is slightly confusing: the command must be > executed from the aurweb/ subdirectory, not the

[PATCH] Set up Alembic for database migrations

--- INSTALL | 4 +- TESTING | 3 +- alembic.ini | 86 +++ aurweb/initdb.py | 9 aurweb/schema.py | 8 migrations/README | 48 ++ migrations/env.py

[PATCH 2/2] Disable Alembic support on test databases

--- aurweb/initdb.py | 12 +--- test/setup.sh| 2 +- 2 files changed, 10 insertions(+), 4 deletions(-) diff --git a/aurweb/initdb.py b/aurweb/initdb.py index c02fb961..91777f7e 100644 --- a/aurweb/initdb.py +++ b/aurweb/initdb.py @@ -36,14 +36,17 @@ def run(args): # Ensure

[PATCH] Change the extension of TAP test suites to .t

This is the common convention for TAP, and makes harnesses like prove automatically detect them. Plus, test suites don’t have to be shell scripts anymore. --- test/Makefile| 2 +- test/{t1100-git-auth.sh => t1100-git-auth.t} | 0

Re: [PATCH] Set up Alembic for database migrations

Frédéric Mangano-Tarumi [2020-02-22 22:31:26 +0100] > diff --git a/aurweb/initdb.py b/aurweb/initdb.py > index e3e96503..c02fb961 100644 > --- a/aurweb/initdb.py > +++ b/aurweb/initdb.py > @@ -31,10 +33,17 @@ def feed_initial_data(conn): > > > def run(args): > +

Re: Automated database migrations with Alembic

Lukas Fleischer [2020-02-07 11:11:35 +0100] > Following that, I guess we could also port the existing Python code to > use SQLAlchemy? There currently seem to be ~100 database queries that we > would need to work on. Absolutely. Maybe we should wait until we have a few successful deployments with

[PATCH] Proof-of-concept of a dual PHP–Python stack

Here’s a demonstration of a Python web stack running next to the current PHP stack, such that it’s invisible to the client. This approach aims at providing a way to migrate the PHP code base to Python, one endpoint after another, with as little glue as possible to have the two backends

Functional HTTP testing

Hello everyone, With the current plans of porting the PHP codebase to Python, we should consider setting up a suite of functional tests to squash most bugs introduced by the rewrite. I suggest the following testing plan, to be applied on an endpoint-by-endpoint basis: 1. Write functional HTTP

Re: [PATCH] First HTTP functional test of the RPC interface

Lukas Fleischer [2020-04-15 17:06:15 -0400] > The new tests are going to require mock data in the database, too, > right? Do you intend to populate the database separately for each test? > Note that many tests require a common basic set of data, such as user > accounts and packages. If we want

Re: [PATCH] First HTTP functional test of the RPC interface

Lukas Fleischer [2020-04-16 16:58:51 -0400] > On Wed, 15 Apr 2020 at 17:54:45, Frédéric Mangano-Tarumi wrote: > > When I run test/t2600-rendercomment.t and measure the time, it spends > > almost 8 seconds on setup.sh, then 3 seconds to actually run the tests. > > That\u20

Re: [PATCH] First HTTP functional test of the RPC interface

Filipe Laíns [2020-04-16 22:58:47 +0100] > On Sun, 2020-04-12 at 18:55 +0200, Frédéric Mangano-Tarumi wrote: > > +assert parts.path in ("", "/") > > Why a tuple? In Python, tuples are essentially immutable lists, and I think it’s a good practice to use

[PATCH] First HTTP functional test of the RPC interface

Though barely useful in its current state, it shows how to integrate pytest with SQLAlchemy and Werkzeug, providing a test framework for the potential future Flask port, while actually testing the current PHP implementation. --- aurweb/test/__init__.py | 0 aurweb/test/conftest.py | 51

Re: [PATCH] First HTTP functional test of the RPC interface

Lukas Fleischer [2020-04-14 09:36:11 -0400] > On Tue, 14 Apr 2020 at 09:26:44, Frédéric Mangano-Tarumi wrote: > > No way! aur_location refers to the development setup address. Its > > default value in our sample configuration is http://localhost:8080, > > which is fine. Note

Re: [PATCH] First HTTP functional test of the RPC interface

Hi Lukas, Lukas Fleischer [2020-04-14 08:54:03 -0400] > On Sun, 12 Apr 2020 at 12:55:46, Frédéric Mangano-Tarumi wrote: > > +base_uri = aurweb.config.get("options", "aur_location") > > +proxy = WsgiHttpProxy(base_uri) > > +return werkze

[PATCH] Create aurweb.spawn for spawing the test server

This program makes it easier for developers to spawn the PHP server since it fetches automatically what it needs from the configuration file, rather than having the user explicitly pass arguments to the php executable. When the setup gets more complicated as we introduce Python, aurweb.spawn will

Re: [PATCH] First HTTP functional test of the RPC interface

Lukas Fleischer [2020-04-15 08:08:08 -0400] > Our test suite already uses a separate configuration which is > auto-generated in test/setup.sh, so it is already self-contained in that > sense; we only need to update aur_location there. I’m hesitant about using setup.sh because it does more than we

Re: [PATCH] Proof-of-concept of a dual PHP–Python stack

Filipe Laíns [2020-03-15 13:50:39 +] > Not really, we just put it in maintenance mode -- no more features just > bugfixes. That won’t benefit the end user, but if we manage to port the code fast enough I guess it’s a compromise. There’s still the second point though: stability. > We can

Re: [PATCH] Proof-of-concept of a dual PHP–Python stack

Lukas Fleischer [2020-03-15 08:25:31 -0400] > Being overloaded is a relative term. Yes, the AUR servers are often > under heavy load, with millions of requests every day. To get more concrete: are the AUR servers sometimes at 100% CPU capacity, or do they hardly ever reach the point of

Re: [PATCH] Proof-of-concept of a dual PHP–Python stack

Lukas Fleischer [2020-03-11 19:44:37 -0400] > Thanks! I like the approach. I wonder what the performance impact of > always querying the Python backend first would be, though, especially at > the beginning when most requests are expected to yield a 404. One way or the other, I don’t think it’s

TAP testing in Python

In order to write tests in Python, we first need to make one important choice: the test framework. Our main constraint is the output format, which should be TAP to run along with the existing scripts. My favorite candidate is pycotap, which holds in a 157-line no-dependency module that we could

[PATCH] Map BIGINT to INTEGER for SQLite

--- aurweb/schema.py | 11 +++ 1 file changed, 11 insertions(+) diff --git a/aurweb/schema.py b/aurweb/schema.py index fde6512f..6792cf1d 100644 --- a/aurweb/schema.py +++ b/aurweb/schema.py @@ -17,6 +17,17 @@ def compile_tinyint_sqlite(type_, compiler, **kw): return 'INTEGER'

Re: TAP testing in Python

I’ve never used unittest, but from my experience pytest has all the qualities you could wish for. Lukas Fleischer [2020-03-22 17:35:23 -0400] > How hard would it be to make pytest-tap compatible with the usual TAP > test invocation? I didn’t realize it at first, but since pytest-tap outputs TAP

Re: [PATCH] First HTTP functional test of the RPC interface

Lukas Fleischer [2020-05-23 12:35:46 -0400] > On Sun, 12 Apr 2020 at 12:55:46, Frédéric Mangano-Tarumi wrote: > > --- > > aurweb/test/__init__.py | 0 > > aurweb/test/conftest.py | 51 + > > aurweb/test/test_rpc.py | 21 +++

Re: [PATCH] aurweb.spawn: Integrate FastAPI and nginx

Lukas Fleischer [2020-05-31 10:16:43 -0400] > > diff --git a/conf/config.defaults b/conf/config.defaults > > index 86fe765c..ed495168 100644 > > --- a/conf/config.defaults > > +++ b/conf/config.defaults > > @@ -41,9 +41,16 @@ cache = none > > cache_pkginfo_ttl = 86400 > > memcache_servers =

[PATCH v2 2/2] Introduce conf/config.dev for development

conf/config.dev’s purpose is to provide a lighter configuration template for developers, and split development-specific options off the default configuration file. --- TESTING | 8 +++- conf/config.defaults | 10 -- conf/config.dev | 36

[PATCH v2 1/2] aurweb.spawn: Integrate FastAPI and nginx

aurweb.spawn used to launch only PHP’s built-in server. Now it spawns a dummy FastAPI application too. Since both stacks spawn their own HTTP server, aurweb.spawn also spawns nginx as a reverse proxy to mount them under the same base URL, defined by aur_location in the configuration. ---

Re: [PATCH v2 2/2] Introduce conf/config.dev for development

Lukas Fleischer [2020-06-01 21:05:57 -0400] > On Mon, 01 Jun 2020 at 12:50:23, Frédéric Mangano-Tarumi wrote: > > diff --git a/conf/config.dev b/conf/config.dev > > new file mode 100644 > > index ..e9c2112e > > --- /dev/null > > +++ b/

[PATCH] Fix typos in CONTRIBUTING.md

--- CONTRIBUTING.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index a37d980a..7b9ff466 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -2,9 +2,9 @@ Patches should be sent to the [aur-dev@archlinux.org][1] mailing list.

[PATCH] Update the aurweb Git URLs to GitLab

--- INSTALL | 2 +- README.md | 4 ++-- TESTING | 2 +- doc/i18n.txt| 2 +- web/lib/version.inc.php | 1 + web/template/footer.php | 2 +- 6 files changed, 7 insertions(+), 6 deletions(-) diff --git a/INSTALL b/INSTALL index

Re: [PATCH] Exclude suspended Users from being notified

Hi Kevin, Kevin Morris [2020-07-03 18:29:16 -0700] > diff --git a/aurweb/scripts/notify.py b/aurweb/scripts/notify.py > index 5b18a476..223ed61f 100755 > --- a/aurweb/scripts/notify.py > +++ b/aurweb/scripts/notify.py > @@ -171,7 +171,8 @@ class CommentNotification(Notification): >

[PATCH 1/4] SSO: Explain the rationale behind prompt=login

We might reconsider it in the future. --- aurweb/routers/sso.py | 7 +++ 1 file changed, 7 insertions(+) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index d0802c34..e1ec7efe 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -28,6 +28,13 @@ oauth.register(

[PATCH 2/4] SSO: Add an SSO option in the login page

We’ll probably change the whole login page in the future, but this makes development easier. --- web/html/login.php | 3 +++ 1 file changed, 3 insertions(+) diff --git a/web/html/login.php b/web/html/login.php index 01454414..3a146f60 100644 --- a/web/html/login.php +++ b/web/html/login.php @@

[PATCH 3/4] Implement SSO logout

--- aurweb/routers/sso.py | 18 ++ web/html/logout.php | 14 +- 2 files changed, 31 insertions(+), 1 deletion(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index e1ec7efe..a8d4b141 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@

[PATCH 4/4] Save id_token for the SSO logout

As far as I can see, Keycloak ignores it entirely. I can login in as SSO user A, then disconnect from the SSO directly and reconnect as user B, but when I disconnect user A from AUR, Keycloak disconnects B even though AUR passed it an ID token for A. --- aurweb/routers/sso.py | 19

Re: [PATCH 3/4] Crude OpenID Connect client using Authlib

Lukas Fleischer [2020-06-06 08:42:05 -0400] > > +; Passphrase FastAPI uses to sign client-side sessions. > > +session_secret = > > \u6975\u79d8\u3001\u8a33\u3059\u306a\uff01\u3042\u3001\u9045\u904e\u304e\u305f\u3002 > > Nit: Since this is not really a secret, can we just use a plain text > value

[PATCH 3/3] Open AUR sessions from SSO

Only the core functionality is implemented here. See the TODOs. --- aurweb/routers/sso.py | 51 +-- 1 file changed, 49 insertions(+), 2 deletions(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index b16edffb..d0802c34 100644 ---

[PATCH 1/3] Add SSO account ID in table Users

This column holds a user ID issed by the single sign-on provider. For Keycloak, it is an UUID. For more flexibility, we will be using a standardly-sized VARCHAR field. --- aurweb/schema.py | 1 + ...6e1cd_add_sso_account_id_in_table_users.py | 30 +++

[PATCH 2/3] Integrate SQLAlchemy into FastAPI

--- aurweb/db.py | 30 ++ 1 file changed, 30 insertions(+) diff --git a/aurweb/db.py b/aurweb/db.py index 1ccd9a07..02aeba38 100644 --- a/aurweb/db.py +++ b/aurweb/db.py @@ -10,6 +10,8 @@ except ImportError: import aurweb.config +engine = None # See get_engine +

[PATCH] aurweb.l10n: Translate without side effects

The install method in Python’s gettext API aliases the translator’s gettext method to an application-global _(). We don’t use that anywhere, and it’s clear from aurweb’s Translator interface that we want to translate a piece of text without affecting any global namespace. --- aurweb/l10n.py | 3

[PATCH 1/4] Remove the FastAPI /hello test route

--- aurweb/asgi.py | 5 - aurweb/spawn.py | 3 --- 2 files changed, 8 deletions(-) diff --git a/aurweb/asgi.py b/aurweb/asgi.py index 5f30471a..9bb71ecc 100644 --- a/aurweb/asgi.py +++ b/aurweb/asgi.py @@ -1,8 +1,3 @@ from fastapi import FastAPI app = FastAPI() - - -@app.get("/hello/")

[PATCH 4/4] Guide to setting up Keycloak for the SSO

--- conf/config.dev | 2 +- doc/SSO | 38 ++ 2 files changed, 39 insertions(+), 1 deletion(-) create mode 100644 doc/SSO diff --git a/conf/config.dev b/conf/config.dev index 27e981f8..fb0cc99a 100644 --- a/conf/config.dev +++ b/conf/config.dev @@

[PATCH 3/4] Crude OpenID Connect client using Authlib

Developers can go to /sso/login to get redirected to the SSO. On successful login, the ID token is displayed. --- .gitlab-ci.yml | 3 ++- TESTING| 3 ++- aurweb/asgi.py | 13 + aurweb/routers/__init__.py | 5 + aurweb/routers/sso.py

[PATCH 2/4] aurweb.spawn: Fix isort errors

--- aurweb/spawn.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/aurweb/spawn.py b/aurweb/spawn.py index 7fe59e65..e86f29fe 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@ -8,8 +8,8 @@ configuration anyway. """ -import atexit import argparse +import atexit

Re: [PATCH v2 2/2] Introduce conf/config.dev for development

Lukas Fleischer [2020-06-02 17:40:23 -0400] > On Tue, 02 Jun 2020 at 08:41:28, Frédéric Mangano-Tarumi wrote: > > Developers need to set AUR_CONFIG to spawn aurweb. If we introduced it > > earlier like \u201cDefine the path to you configuration: > > export AUR_CONFIG=&q

[PATCH v3 2/2] Introduce conf/config.dev for development

conf/config.dev’s purpose is to provide a lighter configuration template for developers, and split development-specific options off the default configuration file. --- TESTING | 11 ++- conf/config.defaults | 10 -- conf/config.dev | 32

[PATCH 1/3] Update last login information on SSO login

--- aurweb/routers/sso.py | 8 +++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index 7b9c67c8..817adadb 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -63,7 +63,13 @@ def open_session(request, conn, user_id):

[PATCH 2/3] HTML error pages for FastAPI

--- aurweb/asgi.py | 16 +++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/aurweb/asgi.py b/aurweb/asgi.py index 60c7ade7..9293ed77 100644 --- a/aurweb/asgi.py +++ b/aurweb/asgi.py @@ -1,4 +1,7 @@ -from fastapi import FastAPI +import http + +from fastapi import

[PATCH 3/3] Guard OAuth exceptions to provide better messages

--- aurweb/routers/sso.py | 16 +--- 1 file changed, 13 insertions(+), 3 deletions(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index 817adadb..2e4fbacc 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -5,7 +5,7 @@ from urllib.parse import urlencode

[PATCH] Remove the per-user session limit

This feature was originally introduced by f961ffd9c7f2d3d51d3e3b060990a4fef9e56c1b as a fix for FS#12898 . As of today, it is broken because of the `q.SessionID IS NULL` condition in the WHERE clause, which can’t be true because SessionID is not nullable. As

[PATCH] Redirect to referer after SSO login

Introduce a `redirect` query argument to SSO login endpoints so that users are redirected to the page they were originally on when they clicked the Login link. --- aurweb/routers/sso.py | 23 +-- web/html/login.php| 18 -- 2 files changed, 29 insertions(+),

[PATCH 2/2] Stop redirecting stderr with proc_open

Error outputs were piped to a temporary buffer that wasn’t read by anyone, making debugging hard because errors were completely silenced. By not explicitly redirecting stderr on proc_open, the subprocess inherits its parent stderr. --- web/lib/acctfuncs.inc.php| 2 --

[PATCH 1/2] aurweb.spawn: Support stdout redirections to non-tty

Only ttys have a terminal size. If we can’t obtain it, we’ll just use 80 as a sane default. --- aurweb/spawn.py | 6 +- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/aurweb/spawn.py b/aurweb/spawn.py index 5da8587e..46d534d9 100644 --- a/aurweb/spawn.py +++ b/aurweb/spawn.py @@

[PATCH 3/3] Build a translation facility for FastAPI

--- aurweb/l10n.py| 22 ++ aurweb/routers/sso.py | 20 +++- 2 files changed, 33 insertions(+), 9 deletions(-) diff --git a/aurweb/l10n.py b/aurweb/l10n.py index 51b56abb..a476ecd8 100644 --- a/aurweb/l10n.py +++ b/aurweb/l10n.py @@ -16,3 +16,25 @@

[PATCH 2/3] SSO: Port account suspension

--- aurweb/routers/sso.py | 11 ++- 1 file changed, 10 insertions(+), 1 deletion(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index efd4462c..3e3b743d 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -41,11 +41,20 @@ async def login(request: Request):

[PATCH 1/3] SSO: Port IP ban checking

--- aurweb/routers/sso.py | 19 +-- 1 file changed, 17 insertions(+), 2 deletions(-) diff --git a/aurweb/routers/sso.py b/aurweb/routers/sso.py index 04ecdca6..efd4462c 100644 --- a/aurweb/routers/sso.py +++ b/aurweb/routers/sso.py @@ -14,7 +14,7 @@ from starlette.requests import