jedcunningham commented on PR #46854:
URL: https://github.com/apache/airflow/pull/46854#issuecomment-2673448887
I gave this a spin with the following DAG, and it would sometimes still
retrieve just part of the xcom file.
```python
from airflow import DAG
from airflow.operators.python import PythonOperator
from airflow.providers.cncf.kubernetes.operators.pod import
KubernetesPodOperator
def consume_xcom(**context):
ti = context['ti']
data = ti.xcom_pull(task_ids='produce')
if data is None:
raise ValueError("No data was received from XCom.")
if len(data) != (105 * 1024):
raise ValueError(f"Data has unexpected length: {len(data)}")
print(f"Successfully retrieved xcom value with length: {len(data)}")
with DAG(dag_id='xcom_large_dag', schedule=None):
produce = KubernetesPodOperator(
task_id="produce",
name="produce",
namespace="default",
image="python:3.12-slim",
cmds=["python", "-c"],
arguments=[
"import json; data = 'A' * (105 * 1024); "
"json.dump(data, open('/airflow/xcom/return.json', 'w'))"
],
get_logs=True,
do_xcom_push=True,
)
consume = PythonOperator(
task_id='consume',
python_callable=consume_xcom,
provide_context=True
)
produce >> consume
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
