dstandish commented on PR #27758:
URL: https://github.com/apache/airflow/pull/27758#issuecomment-1403935741

   TLDR:
   
   added some parameters that can be set on task handler to control setup 
behavior.  we have a lot of variation in how our handlers work.  some can 
support triggers more-or-less natively, i.e. without wrapping in wrapper 
handler and attaching to queuelistener (namely, stackdriver), some can't (all 
the rest).  some could be updated to be native with some changes (ES when in 
stdout mode, cloudwatch) and others probably not (the rest of them, since they 
are file-based).
   
   Backstory if you're interested:
   
   cloudwatch task handler it uses a cloudwatch logging handler from a library 
called watchtower which is a small project maintained i think by one.
   internally this library writes to a queue and launches a thread to consume 
it, batch the messages , and upload to cloudwatch periodically.  so the problem 
is, as it is now, it will create a thread for each trigger.
   the handler (in watchtower) has a flag to disable the queue/thread logic, 
but unfortunately this also disables batching, so it uploads for every log 
message instead of batching.
   i think that the thing to do is, for cloudwatch just disable the threading 
in watchtower and let it upload once per message.  since we pipe through queue 
to a listener thread anyway it won't cause blocking, albeit worse performance 
since no batching.  if someone has time to improve the handler, so that it 
could not thread but still batch (or just be completely native), then that will 
improve things.  and in the meantime, it should work, and in any case, the 
whole thing is disableable if the current behavior proves problematic.
   
   but it got me thinking that we should have a way to allow for a “native” 
handler.
   
   and then the plot thickens.
   
   then i discovered stackdriver task handler.  it doesn’t inherit from 
FileTaskHandler which is why i never saw it before.  and then i took a look and 
i discovered that actually, this handler was almost already “native”.  it 
already launches a thread and pipes messages through a queue.  and, it already 
adds necessary “labels” to each record before queuing it for upload.  the only 
thing i needed to do was make it so that when in trigger context it used the TI 
attached to LogRecord to create the labels (instead of using static labels 
generated in set_context time)
   
   So, I added attributes to allow handlers to disable wrapping with handler 
and disable queuelistener while still being "supported" for trigger, and 
updated stackdriver to work with triggers.
   
   So now we end up with these params settable on each handler:
   * trigger_should_wrap (wrap with triggerhandler)
   * trigger_should_queue (create listener with handler and write message to it 
through queue)
   * trigger_send_end_marker (whether we should send an END signal e.g. to 
close a handler, necessary e.g. for  file-based handlers with 1-per-trigger)
   * trigger_supported (if not trigger_should_wrap and not trigger_should_queue 
then this should be true so that it’s still used as trigger handler, 
essentially it’s for case “trigger is native”)
   
   Then of course the two main setup behaviors, queuing and wrapping, are 
disableable by global logging config.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to