sighingnow commented on pull request #17405: URL: https://github.com/apache/airflow/pull/17405#issuecomment-929962066
Hi @MatrixManAtYrService, I would like to share our experiences of implementing the airflow XCom backend in vineyard (see also: https://github.com/v6d-io/v6d/blob/main/python/vineyard/contrib/airflow/xcom/backend.py) Background: vineyard a in-memory storage that that share complex composable data objects with clients in a zero-copy fashion using memory sharing. We implements a XCom backend based on vineyard. Each object in vineyard has own unique identifier (`ObjectID`), then + for serialization, we first put the data into vineyard, then store the returned object id into the db of airflow + for deserailization, we implemented as first retrieve the object id, then getting object from vineyard + for deletion, we first query the object id for required `(dag, task, execution_date)` then deleting it from vineyard. Hope that could be helpful for you. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
