[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-536679117 @clacke sorry I was on vacation and was not able to reply (should have let you know).Let me know if you have a PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-531281411 So a few things to clarify: Is your parent DAG triggered by backfill or scheduler? Via airflow scheduler, the `DagRun` of a subdag is created via a `SubDagOperator` [here](https://github.com/apache/airflow/pull/5498/files#diff-45749879e4753a355c5bdb5203584698R134) and scheduler only creates `DagRun` for non-subdags [here](https://github.com/apache/airflow/pull/5498/files#diff-c35269bcfbbe386e269ffa7487e86192R1210). So a `DagRun` of a subdag should only be created when the corresponding `SubDagOperator` runs. Can you share your DAG and your setup so that I can take a closer look? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-530896915 I think it is expected that the sub-DAG does not execute when it is paused. When we pause/un-pause the parent DAG, its sub-DAGs would be paused/un-paused accordingly. sub-DAG would only execute after its dependencies are met. It should not be free-floating like you described. Can you check if we have same issue on latest master? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-521825390 We can probably include this PR as part of our next release but I am not sure how the release timeline looks like. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-518532599 Shall we merge it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-516649293 @ashb just resolved the conflicts This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-516552402 @ashb just resolved the conflicts This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-516260138 I am fine merging it but released in 1.10.5 or later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-516080646 I am fine with releasing it in later versions. No rush. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-515325179 Shall we merge it? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-514710063 @Fokko just did a rebase to latest master and the build went through This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-514335580 It passed on my personal travis: https://travis-ci.org/milton0825/airflow/builds/562446362 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-511025184 People were advised not to use `SubDagOperator` because of the performance issues and the bugs. I hope this PR and the following PRs can improve the user experience of `SubDagOperator` and hopefully people can start leveraging the benefits of subdags. @Fokko do you know which committer has experience around subdags? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-510582823 I can break the PR into several smaller PRs if that makes it easier to review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-510358550 PTAL @Fokko @ashb @kaxil This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [airflow] milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill
milton0825 commented on issue #5498: [AIRFLOW-4509] SubDagOperator using scheduler instead of backfill URL: https://github.com/apache/airflow/pull/5498#issuecomment-509447354 PTAL @ashb @feng-tao @potiuk This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
