grssam commented on PR #23094: URL: https://github.com/apache/pulsar/pull/23094#issuecomment-2258573858
> > 2. When there are too many namespaces in a cluster and adding a single namespace to the ns isolation policy data leads to too many bundle unload calls. This can exhaust broker resources (open connections). This should be controlled by a flag if bundle unload can be done post setPolicy call > > Good catch! This problem is related to the migration from synchronous calls to asynchronous calls in the Pulsar code base (/cc @mattisonchao). I have discussed the problem with @mattisonchao and there's some description [in this comment](https://github.com/apache/pulsar/pull/22541#issuecomment-2071568113). The main problem is that with asynchronous calls there aren't concurrency limits in place. One of the possible solutions would be to use https://github.com/spotify/completable-futures/blob/master/src/main/java/com/spotify/futures/ConcurrencyReducer.java to limit the concurrency of asynchronous calls. I wonder if we would need to also fix this problem although it seems like a useful solution to prevent unloading when policies are modified to reduce load on the system. The unbounded concurrent calls is only part of the problem. IMO, the set call should only be touching the "changed/affected" namespaces and unloading those, if at all. It should be very straight forward to compare the list of namespaces matching before and after the change and only unload the delta (newly added and now-removed namespaces) so that their placement can be updated as per the policy. Currently, it can unload 100s of namespaces in one go, practically making the cluster down as it struggles to cope up with so many placement calls. When this feature was added in https://github.com/apache/pulsar/pull/8976 , there was a flag to control it, but the async work done in https://github.com/apache/pulsar/pull/15527 forgot about that and broke the contract, then it was later removed in https://github.com/apache/pulsar/pull/22449 after being called "deprecated".. This feels like a weird loophole to ignore backward compatibility and bring in breaking changes easily :) I believe that the flag was useful, but not at a broker level config. It should be part of the command itself. I am thinking a three way behavior that users may want based on their usecases: 1. unload nothing as part of the set policy call 2. unload every matching namespace as part of the policy set call 3. unload only the changed namespaces (newly added + removed) My goal is to tackle this, rather than tackling the general problem of unbounded concurrency, which really is a much wider problem not only limited to this particular "logical" flaw/regression. What are everyone else's thoughts? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
