potiuk commented on issue #39717: URL: https://github.com/apache/airflow/issues/39717#issuecomment-2265548922
OK. I think we are getting closer to the root cause thanks to that stacktrace. No picking is involved here. What happens there: 1) Mini-scheduler is enabled 2) During that mini-scheduling - partial_subset() of the dag is prepared (including deepcopy of the task + all the downstream tasks it has. 3) deepcopying one of the tasks fails - because the tasks apparently uses CFFI to interact with a C code and the implementation of whatever you have there does not allow for deepcopying the task I think a good solution would be @ashb @ephraimbuddy - following what you've done in https://github.com/apache/airflow/pull/27506 - to just skip mini-scheduler when something like that happens. In this case that would mean exiting mini-scheduler without raising any error if anything wrong happens during partial_subset. WDYT @ashb @ephraimbuddy ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
