potiuk commented on issue #33688:
URL: https://github.com/apache/airflow/issues/33688#issuecomment-1695059348
@jaetma and @mpolatcan
So can you reiterate and explain again what were the differences between the
environments you have? It's clear that you seem to hit the problem when you
upgrade Airflow to 2.7.0 but it seems this is not the only thing you are
upgrading. @mpolatcan you mentioned that you also upgreaded Python version and
you mentioned piclkle library but this is a bit vague. Can you plese help us to
narrow it down and maybe even do some experimenting to help us?
1. Can you please extract out the difference with your enviroments. Ideally
in a short summary:
A. Airflow 2.6.3, Python version any other differences <- does not have
problems
B. Airflow 2.7.0, Python version any other differences <- have problem
2. @mpolatcan especially, you mentioned some suspicion about Python version
3.8 and pickling library. Can you please elaborate on this and maybe (if that
is possible) - attempt to do the same upgrades you did for Airlfow 2.7 but
WITHOUT upgrading Airflow (staying with 2.6.3) and see if you observe the same
memory growth and stability issues? Is this possible? If not cany you explain
what requiremets etc. are preventing it ? That woudl help us enormously in an
attempt to track down the root cause of the problem. Also @jaetma - maybe you
have a possibility to do similar excercise.
Also - are you using pickling?
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#donot-pickle
- airflow's "donot-pickle" is set to "True" by default. Similarly
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#enable-xcom-pickling
is set to "False"- XCom pickling is disabled by default. You mentioned
pickling library, but (except Python Virtualenv/External Python operator) we
are not really using pickling for anything in Airflow. We do use serialization,
but without Pickling so I'd be surprised to see pickling library has an effect.
But maybe you have those parameter's changed? Can you also (if you get to the
point that you have an installation that exhibits the problem) try to change
those values (if your current configuration uses pickling).
I am trying to narrow down the problem - we know already that in some
cases, Airflow 2.7.0 upgrade might trigger some problems. But we also know that
it is not in all environments, only some. So we need to narrow down and track
what is really causing it - is it just Airflow, or Python version or some
dependencies.
I'd relaly love if we can do this remote experiments with you to see if we
can track the root cause of the problem.
And Many thanks for the reports so far, this is already helpful to see that
there is something we need to track down.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
