This is an automated email from the ASF dual-hosted git repository.
potiuk pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/airflow.git
The following commit(s) were added to refs/heads/main by this push:
new f4af596beee Xcoms object storage doc clean (#47924)
f4af596beee is described below
commit f4af596beeed2a29fc4e878479f81b90ea8b140c
Author: Amol Saini <[email protected]>
AuthorDate: Thu Mar 20 14:40:49 2025 +0530
Xcoms object storage doc clean (#47924)
* doc: fix contribution doc ports (#47176)
* doc: clean up for xcoms docs
---
docs/apache-airflow/core-concepts/xcoms.rst | 24 +-----------------------
1 file changed, 1 insertion(+), 23 deletions(-)
diff --git a/docs/apache-airflow/core-concepts/xcoms.rst
b/docs/apache-airflow/core-concepts/xcoms.rst
index d49d9b0e591..ab9813826b6 100644
--- a/docs/apache-airflow/core-concepts/xcoms.rst
+++ b/docs/apache-airflow/core-concepts/xcoms.rst
@@ -83,29 +83,7 @@ An example of pushing multiple XComs and pulling them
individually:
Object Storage XCom Backend
---------------------------
-The default XCom backend is the :class:`~airflow.models.xcom.BaseXCom` class,
which stores XComs in the Airflow database. This is fine for small values, but
can be problematic for large values, or for large numbers of XComs.
-
-To enable storing XComs in an object store, you can set the ``xcom_backend``
configuration option to
``airflow.providers.common.io.xcom.backend.XComObjectStorageBackend``.
-You will also need to set ``xcom_objectstorage_path`` to the desired location.
The connection id is obtained from the user part of the url that you will
provide, e.g. ``xcom_objectstorage_path = s3://conn_id@mybucket/key``.
Furthermore, ``xcom_objectstorage_threshold`` is required
-to be something larger than -1. Any object smaller than the threshold in bytes
will be stored in the database and anything larger will be be
-put in object storage. This will allow a hybrid setup. If an xcom is stored on
object storage a reference will be
-saved in the database. Finally, you can set ``xcom_objectstorage_compression``
to fsspec supported compression methods like ``zip`` or ``snappy`` to
-compress the data before storing it in object storage.
-
-So for example the following configuration will store anything above 1MB in S3
and will compress it using gzip::
-
- [core]
- xcom_backend =
airflow.providers.common.io.xcom.backend.XComObjectStorageBackend
-
- [common.io]
- xcom_objectstorage_path = s3://conn_id@mybucket/key
- xcom_objectstorage_threshold = 1048576
- xcom_objectstorage_compression = gzip
-
-
-.. note::
-
- Compression requires the support for it is installed in your python
environment. For example, to use ``snappy`` compression, you need to install
``python-snappy``. Zip, gzip and bz2 work out of the box.
+The default XCom backend, BaseXCom, stores XComs in the Airflow database,
which works well for small values but can cause issues with large values or a
high volume of XComs. To overcome this limitation, object storage is
recommended for efficiently handling larger data. For a detailed overview,
refer to the :doc:`documentation
<apache-airflow-providers-common-io:xcom_backend>`.
Custom XCom Backends
