dazza-codes edited a comment on issue #6750: [WIP] [AIRFLOW-6329] Add 
AirflowExecutor and native worker
URL: https://github.com/apache/airflow/pull/6750#issuecomment-570740844
 
 
   > What I would really love is to change the "type" of task message. Instead 
of sending `["airflow", "tasks", "run", ...]` it would be nice to send 
`{'dag_id': dag_id, 'task_id': task_id, ... }`
   
   As an aside, while considering options for an asyncio executor, there could 
be various ways that the task "type" (data/spec/payload - you name it) needs to 
be parsed by some kind of asyncio coroutine that can spawn and manage 1000's of 
coroutines on an event loop, such that each of those task coroutines can 
`await` any awaitable to yield control back to the event loop.  In a way, this 
idea might be similar to whatever is managing the messages in this PR.  I'd 
like to understand any changes to task designs.  I've kinda bumped into this 
while trying to do a small POC on the dask executor, i.e. #6984 
   
   I like the idea of a persistent message queue for tasks:
   - an option for using Kafka might help for users who need massive scale with 
fault tolerance
   - redis with persistence options could be useful too

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
[email protected]


With regards,
Apache Git Services

Reply via email to