Re: Functional HTTP testing

[Lukas Fleischer]

On Sat, 14 Mar 2020 at 10:58:54, Frédéric Mangano-Tarumi wrote:
> With the current plans of porting the PHP codebase to Python, we should
> consider setting up a suite of functional tests to squash most bugs
> introduced by the rewrite.
> [...]
> HTTP testing being a kind of language-agnostic blackbox testing, the
> tests would also resist any future rewrite or major refactoring without
> requiring maintenance.
> [...]
> I think it\u2019s safe to start with Requests, and consider Mechanize later
> if need be, or even Selenium. Mechanize and Selenium kind of overlap
> with each other, so we should pick only one of the two, but Requests is
> well-known and should never be unwelcome.

Thanks for all the work. I agree that functional tests are important and
your suggestions sound good to me.

We should also roughly define what we expect from a rewrite. We probably
don't want to require all HTML to be exactly the same on a byte level.
Do we want the DOM representation to remain exactly the same or are we
fine with some small changes? We don't need to be super strict with this
but having some guidelines that are only broken in special cases would
be helpful.

> If it were only me, I\u2019d pick Perl for its shell-like features and regex
> support that make it a great tool for writing tests in my opinion.
> However, not that many developers are familiar with Perl nowadays.

I'd prefer Python, even if only for consistency with the main code base.
I don't feel strongly about it but introducing another programming
language has its downsides.

Functional HTTP testing

[Frédéric Mangano-Tarumi]

Hello everyone,

With the current plans of porting the PHP codebase to Python, we should
consider setting up a suite of functional tests to squash most bugs
introduced by the rewrite.

I suggest the following testing plan, to be applied on an
endpoint-by-endpoint basis:

1. Write functional HTTP tests for the existing software.
2. Port the code to Python.
3. Run the tests.

This differs from the more appealing plan to test only the new code in
Python using tools provided by the framework we’d use. I think testing
the PHP code first will help find surprising edge cases we could
accidentally change but that other parts of the system depend on.

HTTP testing being a kind of language-agnostic blackbox testing, the
tests would also resist any future rewrite or major refactoring without
requiring maintenance.

I’d now like to introduce a few solutions to implement that.

Requests


Using Python’s Requests package with an HTML and a JSON parser should
cover most cases. We would typically run POST requests to change the
state then parse the HTML to validate the updated data.

Any other programming language would work too. Perl offers better
integration with our current test architecture, and its support for
regex is quite practical for matching results.

Mechanize
-

Perl has a cool WWW::Mechanize package that provides a higher-level
interface to HTTP testing, offering basic support for forms and link
detection. We would simulate use cases like “Click the ‘login’ link,
submit the form, etc.”, ensuring the website is usable by a human user.
More at https://metacpan.org/pod/WWW::Mechanize#SYNOPSIS

Someone made a port of WWW::Mechanize for Python, called
python-mechanize.

Selenium


Selenium a software suite for testing web applications among other
things. Its WebDriver component controls a full-fledged browser like
Firefox, supporting in particular JavaScript. You can’t get more
powerful than that. See https://www.selenium.dev/documentation/en/

I think it’s overkill because I believe AUR should never require
JavaScript, as it should be usable without a GUI. I myself occasionally
access it on a text browser. It’s also simpler for bots, including
search engine’s.

Conclusion
--

I think it’s safe to start with Requests, and consider Mechanize later
if need be, or even Selenium. Mechanize and Selenium kind of overlap
with each other, so we should pick only one of the two, but Requests is
well-known and should never be unwelcome.

If it were only me, I’d pick Perl for its shell-like features and regex
support that make it a great tool for writing tests in my opinion.
However, not that many developers are familiar with Perl nowadays.