stephenonethree commented on issue #8541: URL: https://github.com/apache/airflow/issues/8541#issuecomment-840813246
Ok, I think I might have actually found the cause. Unlike other people who have reported this error message, in my case I traced the error to a very specific configuration: I had a single SubDag with direct upstream dependencies on 2 other SubDags. Attempting to clear the status of that SubDag (the downstream one) caused this error. I was able to stop this error by reworking the dependencies so that no SubDag directly depended on more than 1 other SubDag. It looks like, in my case, this had nothing to do with directly passing a non-pickleable parameter to a Dag or Task. Sorry for all the spam, but I hope this latest update helps anybody who wants to look into this (or finds this while Googling). -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
