Hi Enrico,
I agree with Jiuming, we do not need a flag to control this
behavior. We will call `maybeOffloadInBackground` method in the
managedledger init stage. This method only checks the offload policies
to decide if we need to trigger offload. If the offload conditions are
met, use the scheduled executor to trigger offload. The whole check
process has no blocking calls and won't impact the managedledger init.
https://github.com/apache/pulsar/blob/eee3694f00e269eef0f75d791521d0d35d8ff411/managed-ledger/src/main/java/org/apache/bookkeeper/mledger/impl/ManagedLedgerImpl.java#L2475-L2489
Thanks,
Hang
太上玄元道君 <dao...@apache.org> 于2024年5月6日周一 00:31写道：
>
> hi Enrico,
>
> I still think it doesn't matter
>
> 1. Offload a topic or not, configured by
> `managedLedgerOffloadAutoTriggerSizeThresholdBytes`/`managedLedgerOffloadThresholdInSeconds`,
> even though we may trigger a bunch of stuff, it's expected behavior
>
> 2. After triggered to offload a topic, the offload task will be added to an
> Executor, currently, we have 2 threads to execute offload with sync mode,
> so, even though there are thousands of topics, only 2 tasks can be executed
> at the same time, other tasks wait.
>
> 3. If you are worrying about system overwhelmed, there is a PR to introduce
> offload throttling(https://github.com/apache/pulsar/pull/22385),
> it is for the purpose of protect the broker, it has been pending in review
> for a loooong time, I'm very appreciate it that if you can help review
>
> 4. There are toooo many flags in pulsar(~500), I think we should be
> cautious when introducing new FLAGs
>
> Thanks,
> Tao Jiuming
>
> Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日13:44写道：
>
> > daojun,
> >
> > Il giorno sab 4 mag 2024 alle ore 22:06 太上玄元道君 <dao...@apache.org> ha
> > scritto:
> >
> > > it doesn't matter, we don't need to wait until offload finished when
> > > loading topic, since offloading may take a long time.
> > > it just trigger offloading, and we don't need to handle the result.
> > >
> >
> > We don't have to wait for the results, but we still trigger a bunch of
> > stuff and if the broker has thousand of topics, there may be an herd effect
> > as this is a behavior change, I think it is safest to have a flag to
> > disable this new behavior,
> > otherwise upgrading an existing big cluster may lead to unexpected results,
> > like overwhelmed systems
> >
> >
> > Enrico
> >
> >
> >
> > >
> > > Enrico Olivelli <eolive...@gmail.com>于2024年5月5日 周日02:41写道：
> > >
> > > > Overall I think it is a good idea.
> > > >
> > > > What about adding a flag to enable this behavior?
> > > >
> > > > In case there are problems with the offloader the topic won't be able
> > to
> > > > load.
> > > >
> > > > Enrico
> > > >
> > > > Il Sab 4 Mag 2024, 17:12 Hang Chen <chenh...@apache.org> ha scritto:
> > > >
> > > > > Hi guys,
> > > > > I pushed a new proposal to improve the tiered storage offload
> > > > > efficiency. Please help take a look, thanks.
> > > > >
> > > > > PIP: https://github.com/apache/pulsar/pull/22650
> > > > >
> > > > > Thanks,
> > > > > Hang
> > > > >
> > > >
> > >
> >
