Alex (cc'd) brought this up to me about this a while ago too, and I agreed with him. It is definitely something we should do, I remember there were some things that were a bit tricky about removing the intermediate process and would be a bit of work to fix (something about the tasks needing to heartbeat the parent process maybe?).
TLDR: No blockers from me, just might be a bit of work to implement. On Sat, Aug 4, 2018 at 9:15 AM Bolke de Bruin <bdbr...@gmail.com> wrote: > Hi Max, Dan et al, > > Currently, when a scheduled task runs this happens in three steps: > > 1. Worker > 2. LocalTaskJob > 3. Raw task instance > > It uses (by default) 5 (!) different processes: > > 1. Worker > 2. Bash + Airflow > 3. Bash + Airflow > > I think we can merge worker and LocalTaskJob as the latter seems exist > only to track a particular task. This can be done within the worker without > side effects. Next to thatI think we can limit the amount of (airflow) > processes to 2 if we remove the bash dependency. I don’t see any reason to > depend on bash. > > Can you guys shed some light on what the thoughts were around those > choices? Am I missing anything on why they should exist? > > Cheers > Bolke > > Verstuurd vanaf mijn iPad
