nuclearpinguin commented on issue #6750: [WIP] [AIRFLOW-6329] Add AirflowExecutor and native worker URL: https://github.com/apache/airflow/pull/6750#issuecomment-570776666 > > What I would really love is to change the "type" of task message. Instead of sending `["airflow", "tasks", "run", ...]` it would be nice to send `{'dag_id': dag_id, 'task_id': task_id, ... }` > > As an aside, while considering options for an asyncio executor, there could be various ways that the task "type" (data/spec/payload - you name it) needs to be parsed by some kind of asyncio coroutine that can spawn and manage 1000's of coroutines on an event loop, such that each of those task coroutines can `await` any awaitable to yield control back to the event loop. In a way, this idea might be similar to whatever is managing the messages in this PR. I'd like to understand any changes to task designs. I've kinda bumped into this while trying to do a small POC on the dask executor, i.e. #6984 @dazza-codes what do you think about approach used by @dimberman in https://github.com/apache/airflow/pull/7027? > I like the idea of a persistent message queue for tasks: > > * an option for using Kafka might help for users who need massive scale with fault tolerance > * redis with persistence options could be useful too Yes, we hope that Kafka and Google PubSub will get some love from Kombu/Celery community (unofficial transports already exists).
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
