sandynz opened a new issue, #19457:
URL: https://github.com/apache/shardingsphere/issues/19457
Part of #19421.
Currently, all jobs progress persistence is in
RuleAlteredJobSchedulerCenter, it's scheduled by ScheduledExecutorService with
fixed delay. The persistence frequency is every 60 seconds before and 10
seconds now.
```
private static final ScheduledExecutorService JOB_PERSIST_EXECUTOR =
Executors.newSingleThreadScheduledExecutor(ExecutorThreadFactoryBuilder.build("scaling-job-persist-%d"));
static {
JOB_PERSIST_EXECUTOR.scheduleWithFixedDelay(new
PersistJobContextRunnable(), 10, 10, TimeUnit.SECONDS);
}
```
It's not effient enough:
- All jobs progresses will be persisted at the same time.
- Scheduled by fixed delay, no matter there's new progress update or not,
and it might cost much seconds to persist latest progress.
Possible improvements:
- Separate all jobs progresses persistence, every job use dedicated
persistence, make sure progress could be persisted ASAP for job that has new
progress, else no persistence for idle job;
- Replace fixed delay, persist when there's job progress update ASAP; but we
could not persist immediately once there's job progress update, since job
progress might be updated very frequently, just persist to registry center
might hurt performance; so it need a balance on persistence frequence, a
suitable delay might be 2 seconds, it is configurable for every type of job if
possible;
Possible changes:
- It might need observer pattern to capture new progress update, make sure
underlying dumper and importer doesn't have job dedicated logic for common
usage;
- Every type of job could implement dedicated logic of job progress
marsharling and unmarsharling;
- Remove JOB_PERSIST_EXECUTOR and PersistJobContextRunnable in
RuleAlteredJobSchedulerCenter;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
