GitHub user Nataneljpwd added a comment to the discussion: Proposal: Enable the possibility to store in memory video/images/raw-bytes through common.io xcom backend
Hello, as I said, I think that it is a good Idea, yet we need to take into acount that it can only work for when we store the data on s3, as otherwise, we are required to change the schema of the `xcom` table, and I think that storing raw bytes only without the option for json in the table will add to the current overhead of pulling an xcom, as most xcoms use json schemas, and if we change it to be binary only, we will need to know what the underlying data type actually is, if it is a serialized json or a video/audio file. I also think that in no case we should save binary data (blobs) in the metadata database itself, as it is not recomended by any of the database vendors, in addition to there bein a hard limit for the binary data size (in pgsql around 1gb without plugins, in maria/mysql up to 4gb), and so I think the implementation should target the object storage xcom backend instead GitHub link: https://github.com/apache/airflow/discussions/61843#discussioncomment-15794768 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
