koustreak commented on PR #43408: URL: https://github.com/apache/airflow/pull/43408#issuecomment-2440093226
Hi , @potiuk , First of all , many thanks for taking a look at my PR . lets say you have setup ```common``` and ```dag_team_a``` in the multirepo setup , and as per wait parameter both will be synced independently , So lets say for common repo the value of wait is 60 and for dag_team_a the value of wait is 80 , and they will be synced accordingly , so if ```common.util.function``` introduced a mandatory parameter and the user forgot to change the function call accordingly in dag_team_a , then it will throw an error , the user will change the function call in dag_team_a and wait for 80 seconds . The scenario you referred is somewhat depends on end-user error , lets say you combine both dag_team_a and common in a single repo , still if you forgot to mention the mandatory parameter in your function call , you still gonna get the error . the helper function **airflow.dags_poststart** makes sure once the scheduler / triggerer / worker / webserver .. container is started , it creates a symlink in /opt/airflow/dags/bags/<container name> which points to /opt/airflow/dags/repo/container_name ( this is where the git-sync changes will be synced every time . Also the dag_folder points to /opt/airflow/dags/bags , ( if dags.path is mentioned in values.yaml ) Yes , i agree to your point that the cross-repo sync won't happen , if such scenario with mandatory parameter occur the it will throw and error in dag processing , then the user will change the code in dag_team_a and after 80 seconds it will be synced again and we can run the dag . A dag sync hook or dependence can be created , to make sure both repo sync at same time . I agree to your point there will be a problem with sync staleness , but if we keep the wait parameter value is same for both repo then it will somewhat do the job will less lag . -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
