uranusjr commented on issue #28199: URL: https://github.com/apache/airflow/issues/28199#issuecomment-1977756610
The context is that when we render the templates (which are saved to RTIF later), the rendering process also implicitly resolves XCom, causing big values to be loaded (and thus saved to RTIF). We do want the values to be loaded for execution, so one possible solution would be to render the templates twice if a custom XCom backend is detected (by checking if `orm_deserialize_value` is overriden), once normally for execution, once without loading the large values for storage. I think handling this in `serialize_template_field` might be more difficult; when we reach there, all templates have been rendered and we may miss a lot of context on whether we want a value to be abbrevated or not. But thinking of this again, maybe that is fine…? Custom XCom is just one special case (where the problem is more significant). It could be argued we should reduce _all_ large values anyway regardless of where they are from, and handling that in `serialize_template_field` would be reasonable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
