potiuk commented on PR #37931: URL: https://github.com/apache/airflow/pull/37931#issuecomment-1981389611
> Also: > > * Should we put the internal API component in each tenant box? They are really "on the edge" so it's difficult to make them both in and out the tenant box - but I will try to figure out something :) > * I am a bit confused by the links between the DAG files and triggerers and workers. Do these link exist? I thought the only component using DAG files is DAG processor and then the other components just use parsed DAGs from DB. But I might be wrong The picture is correct. 1) Workers Workers cannot execute the DAGs from DB because they have to execute the code (With the imports and all the other things that are in the DAG folder) - the DB contains two things there: a) Serialized for of DAG - json (that UI uses to display the DAG and Scheduler to schedule tasks from) b) Code of the DAG file where the DAG comes from - but this one does not contain any imported libraries, utils etc. - it's literally source code of the DAG file the DAG came from. So not nearly enough to get workers to run tasks. You **really** need to have access to DAG folder and be able to execute any code there from the worker. 2) The Triggerers Triggerers have to execute Triggers. And you can define Triggers not only in your operators but also you can define custom Triggers in your DAG code and Trigger shoud be able to execute them in the event loop. And for that Triggerer similarly to worker should have access to DAG folder and be able to parse the complete code (including imports of any shared code(, -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
