lhotari commented on PR #23094: URL: https://github.com/apache/pulsar/pull/23094#issuecomment-2258720029
> The unbounded concurrent calls is only part of the problem. I agree. > IMO, the set call should only be touching the "changed/affected" namespaces and unloading those, if at all. It should be very straight forward to compare the list of namespaces matching before and after the change and only unload the delta (newly added and now-removed namespaces) so that their placement can be updated as per the policy. I think that this part is now addressed as part of #23100 > Currently, it can unload 100s of namespaces in one go, practically making the cluster down as it struggles to cope up with so many placement calls. When this feature was added in #8976 , there was a flag to control it, but the async work done in #15527 forgot about that and broke the contract, then it was later removed in #22449 after being called "deprecated".. This feels like a weird loophole to ignore backward compatibility and bring in breaking changes easily :) That's unfortunate. > I believe that the flag was useful, but not at a broker level config. It should be part of the command itself. I am thinking a three way behavior that users may want based on their usecases: > > 1. unload nothing as part of the set policy call > 2. unload every matching namespace as part of the policy set call > 3. unload only the changed namespaces (newly added + removed) Makes sense. > My goal is to tackle this, rather than tackling the general problem of unbounded concurrency, which really is a much wider problem not only limited to this particular "logical" flaw/regression. +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
