FWIW, I'm liking the Breeze idea. I hadn't thought of it at first, but it
seems to make the most sense.
- ferruzzi
From: Beck, Vincent
Sent: Wednesday, June 7, 2023 7:21 AM
To: dev@airflow.apache.org
Subject: RE: [EXTERNAL]Deferrable operators in system
Same here, I like the breeze option
On 2023-06-07, 3:32 AM, "Pankaj Koti" mailto:pankaj.k...@astronomer.io.inva>LID> wrote:
CAUTION: This email originated from outside of the organization. Do not click
links or open attachments unless you can confirm the sender and know the
content is safe.
+1 to both ideas.
I have a stronger +1 to include the flag with breeze since triggerer
is an Airflow component and breeze commands having support for
it would be nice.
Regards,
Pankaj Koti
*Senior Software Engineer, *OSS Engineering Team.
Location: Pune, India
Timezone: Indian Standard Time
Don't think fixture would break that. It would just be test code not in the
dag. It would just ensure that the triggerer is running before the tests
that use the triggerer need it. But doing it in breeze makes more sense for
sure. Although I suppose a combination approach could be considered EG,
Thanks for pointing out solutions I hadn't considered.
> we could simply have a pytest fixture that will do the job (based on what
> Daniel proposed).
> It will not support running tests via Airflow UI or regular python execution
> of the test files,
> but maybe it's a good idea that does not
There are two options I think:
1) rely on Breeze as execution environment for System Tests and add a flag
to start triggerer with system tests
2) add a pytest fixture that will do it automatically when system tests are
run via pytests (that could be done via tests/system/conftest.py)
I believe
I just took a look and it turns out that DebugExecutor works fine with
triggerer you just need to have one running.
You could run one in a subprocess. I experimented with refactoring the
subprocess hook for this purpose (so you can start the subprocess
asynchronously) and then ran this dag with
My preference would be Option 2 as that is closest to prod. Our approach is
to spin up Airflow , install the providers that currently have deferrable
functionality, and run system tests on those example DAGs on a daily basis
. We are also running them on multiple executors (Celery, Kubernetes
Hello,
As we implement deferrable capabilities for operators, we're going to want to
update the sample dags/system tests.
The problem we encounter at the moment is that system tests are run with pytest
using the DebugExecutor, and it only runs the "worker part" of the operators,
without
