jaketf edited a comment on issue #6210: [AIRFLOW-5567] [Do not Merge] prototype BaseAsyncOperator URL: https://github.com/apache/airflow/pull/6210#issuecomment-537192679 @nuclearpinguin I think those concerns are quite valid. > How should I decide whether I should create an async operator or operator+sensor? IMO if your operator and sensor refer to the same resource (job, query, etc.) you'd almost always want to use async operator. This has to do w/ retries if your sensor fails and you want to retry the submit, then it should be async operator (this wouldn't be possible w/ operator+sensor w/o a whole re-run of the DAG). I'd love to hear differing opinions here. @dstandish mentioned ([slack](https://apache-airflow.slack.com/archives/CCY359SCV/p1569608756022600?thread_ts=1569540521.006000&cid=CCY359SCV)) if polling / retrival is lower priority than kicking the job off this could be a reason to have separate operator + sensor. > Should I try to create an operator that could be used in sync or async mode by passing an `async=True` flag? I was just thinking about this today as well. Similarly: "for our existing operators how do we expose async versions". Are these new operators? I think you could develop a new operator that extends `BaseAsyncOperator` you would define a submit_request, process_request, and poke. The you'd override `execute` with something that checks ```python3 if self.async: super().execute(context) # use the `BaseAsyncOperator`'s execute defined in this PR. else: #Note this is what most operators I'm familiar with look like. self.submit_request(context) while not self.poke(): time.sleep(self.poke_interval) self.process_results(context) ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
