On Tue, Dec 24, 2024 at 8:22 PM Ben Ramsey <b...@benramsey.com> wrote:
> On Dec 16, 2024, at 07:21, Jakub Zelenka <bu...@php.net> wrote: > > > Hi, > > I have been looking into how to test some cases where integration tests > are very difficult or even impossible to create for. Those are often found > in networking related and system specific code code (network.c, streams, > FPM and more). I was recently fixing one such bug and decided to give a try > which resulted in this PR: https://github.com/php/php-src/pull/16987 . > > There was a suggestion of RFC but that might be a bit too much as it's > just an internal change / addition. But certainly some overview on > internals should be done so writing this instead. > > I decided to use cmocka in that PR because I had some experience with > that. It's quite small and still very powerful and allow vast mocking > options. It's a bit manual but it gives a bigger control over the mock. It > relies on --wrap linking option that replaces original functions with > wraps. This is however available only on Linux (or maybe some other Unix > variants) but doesn't work on MacOS or Windows. The developers that want to > use it on those platforms would need to use some Linux Virtualisation > option (e.g. Docker). It also requires static library which is supported by > embed SAPI that can be compiled statically. That limits number of > extensions to use but the main use cases don't really have deps so it > should be fine. > > I did also some research into the other mocking libraries in C. There is a > Unity with CMock, FFF and some C++ libs like GUnit, Criterion and > Trompeloeil that I looked into. I quickly discarded GUnit and Trompeloeil > as they relay on C++ virtual methods and require wrapping C code to C++ > which is very inconvenient. FFF seems too simple and maybe quite inflexible > for our needs as well. Criterion also optionally uses wrap so I didn't see > much advantages compare to cmocka. So it left Unity with CMock that allows > generating custom mocks using a Ruby script. That seemed initially quite > nice but after spending around two hours with trying to make it works for > PHP codebase, I just gave up. It gets quite messy for complex scenarios and > I just didn't figure out how to nicely mock libc functions without any > modification to php-src. > > In terms of CI. It has got its own build which is very simple and it tests > just specific parts so we could just limit it to run only for changed files > which might be quite convenient. > > So the proposed PR is probably the only reasonable unit testing that I can > come up with. I think it should be completely optional initially for people > to use - more like an experiment. If it becomes used, then good of course. > And if it becomes pain, we can just get rid of it. Has anyone got any > objections to get this merged? If not I plan to merge it early in January. > > Cheers > > Jakub > > > > FWIW, as someone still very new to C, I found Criterion quite easy to use, > and I was able to quickly grasp its concepts and start using it right away. > I can’t say the same for other C testing libraries I tried. > I just checked Criterion a bit more and it actually does not have built-in mocking. It could be used with --wrap but it doesn't have any expectations like cmocka. There's Mimick [1] from the same author which is indendent but its docs still say that it's experimental so probably not a good idea to use an experimental tool as its API can change. So all in all it's not probably an option for us. [1] https://github.com/Snaipe/Mimick Cheers Jakub
