Any discoveries from this thread should probably be added to
https://airflow.apache.org/concepts.html#core-ideas
FWIW, I'm not sure if it's still this way on the master branch, but at one
point a couple useful features (e.g. was it accessing task logs from the UI
and maybe running from the UI?)
Hi Brian,
Really appreciate your quick reply. Just to be clear, I did not intend to
run them in particular order. as a matter of fact, these are expensive db
queries that I cant afford to run in parallel.
I think I have setup the tasks correctly to use pool but may be missing the
priority_weight
I should probably mention that our implementation will be running multiple
executors/schedulers/workers for the sake of resilience and recovery. We
are more concerned about task state correctness than performance if that
makes sense, i.e. the same task should not run twice by accident etc.
On
To be clear, you’re hoping that setting the slots to 1 will cause the tasks
across district dags to run in order based on the assumption that they’ll queue
up and then execute off the pool?
I don’t think it will quite work that way - there’s no guarantee the scheduler
will execute your tasks
Hi
We are using Airflow to build up a DWH from many source systems but I have a
couple of edge cases that I’m not quite sure how to proceed with.
Each table from a source system is supplied as a CSV. With each set of files
delivered (daily) we receive a manifest/description file containing a
Hi All,
Greetings. I am writing unit tests for some Elasticsearch related changes and
want to use this library: https://github.com/vrcmarcos/elasticmock for mocking.
Unfortunately it is not working with current version of elasticsearch and
elasticsearch_dsl libs so I had to copy/paste the code
Hi All,
Fokko and I are picking up the release management role for the 1.10 release.
The aim of this release is to make sure we are, from the perspective of a
release, graduation ready. So license compliancy is paramount and proper
publishing of security vulnerabilities. It should also be our
