The issue is probably not in you algorithm but in memory management. You are using out-of-place routines like `x = (x*x + 1) mod n`, that means an extra allocation at each loop. F(7) is 2^128 + 1 so it does take some space and for F(7) it seems like the GC cannot reclaim that memory. And you have a dozens of out-of-place operand per-loop.
Instead you should refactor your program to allocate the minimum possible by using the in-place operators that would re-use one of your bigint buffer instead of allocating a new one. That will make it uglier obviously. With the new Nim semantics of destructors, especially the `sink` parameters, we could recreate a wrapper that would detect when a parameter is actually not used later and its buffer could be reused without allocating. I leave such an exercise to the reader ;).
