This is an automated email from the ASF dual-hosted git repository.
jedcunningham pushed a commit to branch v2-10-test
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/v2-10-test by this push:
new da79e4dbab7 [v2-10-test] Update XCom docs around containers/helm
(#44570) (#44573)
da79e4dbab7 is described below
commit da79e4dbab77c2099ff634315c642ca6b5531de1
Author: github-actions[bot]
<41898282+github-actions[bot]@users.noreply.github.com>
AuthorDate: Mon Dec 2 09:42:40 2024 -0700
[v2-10-test] Update XCom docs around containers/helm (#44570) (#44573)
This removes the whole section about helm, as it really isn't directly
related to the XCom concept at all. I also simplified the section about
containers as well - this one is a bit more practical, so I've left it.
(cherry picked from commit 3747c91afdcd0470ca29e911c589b334b357b778)
Co-authored-by: Jed Cunningham
<[email protected]>
---
docs/apache-airflow/core-concepts/xcoms.rst | 27 +++------------------------
1 file changed, 3 insertions(+), 24 deletions(-)
diff --git a/docs/apache-airflow/core-concepts/xcoms.rst
b/docs/apache-airflow/core-concepts/xcoms.rst
index b6bd160d89d..fad9420cea6 100644
--- a/docs/apache-airflow/core-concepts/xcoms.rst
+++ b/docs/apache-airflow/core-concepts/xcoms.rst
@@ -98,36 +98,15 @@ There is also an ``orm_deserialize_value`` method that is
called whenever the XC
You can also override the ``clear`` method and use it when clearing results
for given DAGs and tasks. This allows the custom XCom backend to process the
data lifecycle easier.
-Working with Custom XCom Backends in Containers
------------------------------------------------
+Verifying Custom XCom Backend usage in Containers
+-------------------------------------------------
Depending on where Airflow is deployed i.e., local, Docker, K8s, etc. it can
be useful to be assured that a custom XCom backend is actually being
initialized. For example, the complexity of the container environment can make
it more difficult to determine if your backend is being loaded correctly during
container deployment. Luckily the following guidance can be used to assist you
in building confidence in your custom XCom implementation.
-Firstly, if you can exec into a terminal in the container then you should be
able to do:
+If you can exec into a terminal in an Airflow container, you can then print
out the actual XCom class that is being used:
.. code-block:: python
from airflow.models.xcom import XCom
print(XCom.__name__)
-
-which will print the actual class that is being used.
-
-You can also examine Airflow's configuration:
-
-.. code-block:: python
-
- from airflow.settings import conf
-
- conf.get("core", "xcom_backend")
-
-Working with Custom Backends in K8s via Helm
---------------------------------------------
-
-Running custom XCom backends in K8s will introduce even more complexity to
your Airflow deployment. Put simply, sometimes things go wrong which can be
difficult to debug.
-
-For example, if you define a custom XCom backend in the Chart ``values.yaml``
(via the ``xcom_backend`` configuration) and Airflow fails to load the class,
the entire Chart deployment will fail with each pod container attempting to
restart time and time again.
-
-When deploying in K8s your custom XCom backend needs to be reside in a
``config`` directory otherwise it cannot be located during Chart deployment.
-
-An observed problem is that it is very difficult to acquire logs from the
container because there is a very small window of availability where the trace
can be obtained. The only way you can determine the root cause is if you are
fortunate enough to query and acquire the container logs at the right time.
This in turn prevents the entire Helm chart from deploying successfully.
