jscheffl commented on PR #36029: URL: https://github.com/apache/airflow/pull/36029#issuecomment-1990524324
> > I feel too much overhead for a niche feature. > > I dont think this ia a niche feature. This is the heart and core of what orchestrator needs to have. I suggested before to mark this feature as experimintal thus allowing us to move fast, get it reeleased and hear feedback from users while preserving the option to change the feature details as needed until it's stable. "niche" in context we came to version 2.9.0 now and so far most of the demand is covered in contrast to adding a persistent blob to every task instance which we need million times in the DB - which is a high performance impact. Even it is experimental we are extending the DB model at the core where we need to carefully think about every byte we spend because it will be multiplied a million times in each instance. Even if experimental (which I think is rather about the interface) I doubt that once established we will rollback the DB scheme changes. Even the example supplied does not require persistence of any information and the use cases I have in mind also do not need a blob/JSON persisted. On top how is the persisted data filled initially when "class" is attached to the task? At parse time? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
