GitHub user potiuk added a comment to the discussion: Best Way to Track Step Execution Like Airflow Tasks in a Python Data Workflow
You can't now. This has been discussed that in the future we might want to serve this better (and we already started to discuss add Arrow ADBC provider. So far we hsmeave not seen anyone **wanting** this feature - usually Airflow is orchestrating external systems and data processing itself by Airflow task has not been the main focus. Some of the workflows requireing multiple tasks sharing memory were done with - for example - Ray (Astronomer has Ray provider https://github.com/astronomer/astro-provider-ray). But yes, seeing a "need" is a signal that yeah, maybe we should implement something like that. GitHub link: https://github.com/apache/airflow/discussions/58088#discussioncomment-14914454 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
