dstandish commented on PR #27758: URL: https://github.com/apache/airflow/pull/27758#issuecomment-1403935741
TLDR: added some parameters that can be set on task handler to control setup behavior. we have a lot of variation in how our handlers work. some can support triggers more-or-less natively, i.e. without wrapping in wrapper handler and attaching to queuelistener (namely, stackdriver), some can't (all the rest). some could be updated to be native with some changes (ES when in stdout mode, cloudwatch) and others probably not (the rest of them, since they are file-based). Backstory if you're interested: cloudwatch task handler it uses a cloudwatch logging handler from a library called watchtower which is a small project maintained i think by one. internally this library writes to a queue and launches a thread to consume it, batch the messages , and upload to cloudwatch periodically. so the problem is, as it is now, it will create a thread for each trigger. the handler (in watchtower) has a flag to disable the queue/thread logic, but unfortunately this also disables batching, so it uploads for every log message instead of batching. i think that the thing to do is, for cloudwatch just disable the threading in watchtower and let it upload once per message. since we pipe through queue to a listener thread anyway it won't cause blocking, albeit worse performance since no batching. if someone has time to improve the handler, so that it could not thread but still batch (or just be completely native), then that will improve things. and in the meantime, it should work, and in any case, the whole thing is disableable if the current behavior proves problematic. but it got me thinking that we should have a way to allow for a “native” handler. and then the plot thickens. then i discovered stackdriver task handler. it doesn’t inherit from FileTaskHandler which is why i never saw it before. and then i took a look and i discovered that actually, this handler was almost already “native”. it already launches a thread and pipes messages through a queue. and, it already adds necessary “labels” to each record before queuing it for upload. the only thing i needed to do was make it so that when in trigger context it used the TI attached to LogRecord to create the labels (instead of using static labels generated in set_context time) So, I added attributes to allow handlers to disable wrapping with handler and disable queuelistener while still being "supported" for trigger, and updated stackdriver to work with triggers. So now we end up with these params settable on each handler: * trigger_should_wrap (wrap with triggerhandler) * trigger_should_queue (create listener with handler and write message to it through queue) * trigger_send_end_marker (whether we should send an END signal e.g. to close a handler, necessary e.g. for file-based handlers with 1-per-trigger) * trigger_supported (if not trigger_should_wrap and not trigger_should_queue then this should be true so that it’s still used as trigger handler, essentially it’s for case “trigger is native”) Then of course the two main setup behaviors, queuing and wrapping, are disableable by global logging config. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
