potiuk commented on issue #25964: URL: https://github.com/apache/airflow/issues/25964#issuecomment-1229082778
No, I think I think this is pretty expected stil. I do not think it's ever defined what will be in the exception - it depends on the circumstances. Sometimes it is string, sometims exception, and there is no guarantee about printing stacktrace mostly because callbacks can be executed in different context: 1) sometimes they are executed in Worker 2) sometimes they are executed in DagFileProcessor (and serialized to be send over the database sometimes) 3) sometimes they are executed in scheduler 4) sometiumes they are executed in mini-scheduler There is lot more complexity in Airlfow 2 than 1 involved in running callbacks. Serialization is involved and I think the decision was made (and semi-documented in the types possible to return) that "exception" might be more than just excpetion. Side effect of changes in Airlfow 2 are that some of the things from 1 are not preserved. And to be perfectly honest relaying on having a stack-trace and its content by the logic of your DAG is super-brittle. What happens if the code change ? How do you keep track of it? How does it work when you change different versions? how do you react on different provider versions that might change every month and basically have the stack trace change every month? I thinkt it is unreasonable to rely on it, so lack of it is not a bug, rally and should not hinder any "reasonable" logic you might implement there. The fact that is not "perfect" when even "perfect" solution is unusable realisticall makes me very reluctant on even attempting to make it consistent - because that might make other users like you atempt to rely on it - but this is a futile effort. If you want to implement any logic based on the exceoption - tou will do much better if relay on str() coercion of anytning you get in the 'exception' field and check if it contain what you expect it to contain (exception or string message that you throw when you fail). this is far better strategy. So I do not argue your "inconsistency" statement does not hold. It does. But it's more of a feature than a but , and I think it is highly unlikely it will turn into something you will be able to rely one. I will convert it into a discussion rather than just closing - maybe others will have other opinion, or maybe you will have other arguments - why relying on stacktrace being present is important and useful. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
