GitHub user potiuk added a comment to the discussion: Compatibility issues with polars: Any calculation of polars out of a task will block calculation in the task
Also another interesting thing https://docs.pola.rs/user-guide/misc/multiprocessing/ -> you might want to look at starting your Polars context with spawn. Airlfow uses heavily multiprocessing and forks and it seems that Polars has specific guidelines and instructions that you should follow on this and some guidelines suggesting to use spawn when forking is involved.. This is a polars doc to follow, not Airflow. GitHub link: https://github.com/apache/airflow/discussions/45756#discussioncomment-11870106 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
