villebro commented on PR #36368:
URL: https://github.com/apache/superset/pull/36368#issuecomment-3850028379
@michael-s-molina and @mistercrunch: I have done extensive work to minimize
overhead as much as possible: heavy optimization of current distributed locking
feature, use atomic SQL wherever possible (only use locks when explicitly
needed), throttle updates originating from task etc. Please review the updated
description for all the changes introduced.
@mistercrunch: regarding your comment on providing a gradual path for
migrating legacy tasks to GTF: I've now added a feature flag, so the feature
will not be made generally available until it's gone through extensive
hardening. For the task migration, I'm proposing creating new GTF-enabled tasks
that will be used when the FF is flipped, but continue using the old tasks for
envs where GTF isn't enabled. I did some testing to see if supporting direct
Celery scheduling with the new task decorator would be possible, but it added a
lot of dangerous complexity to the already complex internals of the feature, so
ultimately I decided against it. But if you feel strongly about being able to
run legacy tasks in parallel with GTF, e.g. with a progressive rollout where
you could define a percentage of thumbnails to be scheduled on GTF vs pure
Celery, I'm happy to bake that in. However, I suspect it will just add
confusion ("why am I only seeing some tasks on the list view?", "why are some ta
sks failing while others are not?", "why is this thumbnail sometimes
deduplicated and other times not?" etc). So I would recommend just hardening
the hell out of this feature in the next few months, and then doing a few
canary releases to make sure things are running smoothly before making it GA
for everyone.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
