• [email protected] [2025-10-20 22:46]:
On Oct 20, 2025, at 1:11 PM, Kirill Miazine <[email protected]> wrote:
2025-10-20T21:45:53+0200 [pyng.master.PyngMaster#info] Unable to
format event {'check': <pyng.PING o
bject at 0x294df13f3510>, 'log_logger': <Logger
'pyng.master.PyngMaster'>, 'log_level': <LogLevel=in
fo>, 'log_namespace': 'pyng.master.PyngMaster', 'log_source':
<pyng.master.PyngMaster object at 0x29
4df34c8090>, 'log_format': 'Ready to run {check.type} check
"{check.desc}" [{check.id <http://check.id/>}]', 'log_time'
: 1760989553.4371765}: maximum recursion depth exceeded
My guess is that what's happening here is that because logging
consumes several stack frames for the emitting, the formatting, etc,
you're just getting unlucky and happening to blow your stack here as
it's the biggest increase in the repeated function. The message here
is telling you that it can't format the message, but given that
PyngCheck does not seem to have a dynamic __getattr__ or any @property
descriptors, I don't think that that's where you're seeing the recursion.
The actual problem that I think you're seeing is just too large a
number of Deferreds synchronously running their callbacks (which you
can see via the addCallbacks on the top of the stack synchronously
resulting in a _runCallbacks call. You're creating a Deferred for each
go round the loop, without actually returning back up to the event
loop at any point, eventually building up a huge stack of
Deferred-calling-Deferred-calling-Deferred. IIRC we have some
mitigations for this to reduce the amount of recursion that Deferred
does but it is not avoidable in the general case.
I suspect that the thing that you need here is just some backoff and
jitter on some part of the retry here, rather than synchronously
attempting to do things? I don't have time to fully absorb this logic
but I can definitely see some mutual synchronous recursion between
_get_check and _run_check, which is, I think, where you are running
into problems (specifically with the implicit callback addition caused
by returning a Deferred from _run_check into _get_check's addCallback).
Even if your stack is not meaningfully infinite, it's easy to get up
around the stack limit if you just have a huge pile of Deferreds
synchronously chain on to one another like this. But even a
deferLater(0) will break up the compounding callback stack and allow
work to proceed.
thanks a lot for the ideas!
initially I was using reactor.callLater(0-5, foo) quite a lot, and then
I was recommended to just call foo instead, and so I did. so I will
callLater() foo now again and see if it helps. initially it looks promising
-g
_______________________________________________
Twisted mailing list -- [email protected]
To unsubscribe send an email to [email protected]
https://mail.python.org/mailman3/lists/twisted.python.org/
Message archived at
https://mail.python.org/archives/list/[email protected]/message/I74D2AODU6FCDM6IABCWPG6CK762UXSS/
Code of Conduct: https://twisted.org/conduct
