[
https://issues.apache.org/jira/browse/AIRFLOW-1898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16651693#comment-16651693
]
jack commented on AIRFLOW-1898:
-------------------------------
Regardless if there should be a limit or not.... There shouldn't be a silent
fail. If the XCOM was not saved as expected it should raise an exception to the
user.
This is a reliability issue.
> Large XComs are not supported and fail silently
> -----------------------------------------------
>
> Key: AIRFLOW-1898
> URL: https://issues.apache.org/jira/browse/AIRFLOW-1898
> Project: Apache Airflow
> Issue Type: Bug
> Components: xcom
> Affects Versions: 1.8.0
> Environment: MySQL
> Reporter: Len Frodgers
> Priority: Major
>
> I am using Airlfow backed by MySQL and having problems with large XComs (> 64
> KB). Xcom uses PickleType which is backed by BLOB on MySQL.
> Unfortunately, MySQL by default truncates anything longer than BLOB (64 KB)
> when saving, so when unpickling such XComs, they are corrupt and an EOFError
> is raised.
> Two things we need:
> 1) Validation when saving the XCom that it is not too big
> 2) Use MEDIUMBLOB as the underlying data type for the XCOM column on MySQL so
> large XComs can be stored (supports up to 12 MB i think)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
