sandynz opened a new issue, #2526:
URL: https://github.com/apache/shardingsphere-elasticjob/issues/2526
## Bug Report
### Which version of ElasticJob did you use?
The issue was reproduced with the released ElasticJob 3.0.4 and 3.0.5
lifecycle artifacts.
A verified reproduction environment included:
- ElasticJob 3.0.4
- Apache Curator 5.7.0
- ZooKeeper client and server 3.9.5
- JDK 17
### Expected behavior
Lifecycle APIs are management-facing APIs used to inspect and manage jobs.
After a job configuration update has been committed successfully to ZooKeeper,
the following queries should return the latest committed
configuration:
- `JobConfigurationAPI.getJobConfiguration()`
- `JobStatisticsAPI.getJobBriefInfo()`
Their results should be consistent with
`ZookeeperRegistryCenter.getDirectly()` and `zkCli.sh`.
### Actual behavior
When a running job has a `CuratorCache` registered for its job path,
lifecycle APIs may temporarily return an older job configuration.
During the observed cache refresh window:
- `ZookeeperRegistryCenter.get()` returns the old cached value.
- `ZookeeperRegistryCenter.getDirectly()` returns the new value.
- `zkCli.sh` returns the new value.
- `JobConfigurationAPI.getJobConfiguration()` returns the old value.
- `JobStatisticsAPI.getJobBriefInfo()` may be built from the old value.
The cached value eventually converges to the latest ZooKeeper value, but
management queries can expose stale job information before that happens.
### Reason analyze (If you can)
ElasticJob registers a `CuratorCache` for each running job.
`ZookeeperRegistryCenter.get()` prefers data from a matching cache and
only falls back to `getDirectly()` when the cache does not contain the
requested node.
A successful `ZookeeperRegistryCenter.update()` commits the new value to
ZooKeeper, but the local `CuratorCache` is refreshed asynchronously through
ZooKeeper watch events. Therefore, a short interval can exist in
which:
```text
cached value = old configuration
direct ZooKeeper value = new configuration
```
The lifecycle implementations currently read job configurations through
the cached method:
```java
regCenter.get(jobNodePath.getConfigNodePath());
```
The affected reads are in:
- `JobConfigurationAPIImpl.getJobConfiguration()`
- `JobStatisticsAPIImpl.getJobBriefInfo()`
This exposes the registry-center cache refresh window through
management-facing lifecycle APIs.
The scheduling kernel benefits from cached reads and should remain
unchanged. The consistency boundary can instead be corrected by using
authoritative reads only when lifecycle APIs retrieve job configurations.
### Steps to reproduce the behavior.
1. Start a ZooKeeper server.
2. Create two `ZookeeperRegistryCenter` instances connected to the same
namespace.
3. Persist an initial job configuration at `/job/config`.
4. Register a cache for `/job` on the observer registry center.
5. Wait until the observer cache contains the initial configuration.
6. Update `/job/config` through the writer registry center.
7. Immediately compare the observer's cached and direct reads.
8. When the cache refresh window is observed, invoke the lifecycle API
using the observer registry center.
> Two registry-center instances are used to separate the writer and
observer roles and to make the cache-refresh window easier to capture. Multiple
instances are not a prerequisite for the underlying race. A single
> instance may also observe the window because a successful ZooKeeper
update does not synchronously refresh its CuratorCache.
The relevant sequence is:
```java
ZookeeperRegistryCenter observer = new
ZookeeperRegistryCenter(configuration);
ZookeeperRegistryCenter writer = new
ZookeeperRegistryCenter(configuration);
observer.init();
writer.init();
writer.persist("/job/config", oldJobConfiguration);
observer.addCacheData("/job");
awaitCachedValue(observer, "/job/config", oldJobConfiguration);
writer.update("/job/config", newJobConfiguration);
String cached = observer.get("/job/config");
String direct = observer.getDirectly("/job/config");
if (!Objects.equals(cached, direct)) {
JobConfigurationPOJO actual =
new
JobConfigurationAPIImpl(observer).getJobConfiguration("job");
// Before the fix, actual is created from the old cached configuration,
// while direct and zkCli.sh already return the new configuration.
}
```
The race can be made deterministic enough for testing by repeatedly
updating the node until the following condition is captured:
```text
observer.get("/job/config") != observer.getDirectly("/job/config")
```
An observed run produced the following sequence:
```text
cached=value-0
direct=value-1
writerDirect=value-1
cached.version=0
direct.version=1
CONVERGED cached=value-1
```
The newer `version`, `mzxid`, and `mtime` from the direct read confirm
that ZooKeeper has committed the update and that the old value comes from the
local cache.
### Example codes for reproduce this issue (such as a github link).
A lifecycle-level regression can mock the registry center and verify that
job configuration queries use authoritative reads:
```java
when(regCenter.getDirectly("/test_job/config")).thenReturn(jobConfigurationYaml);
JobConfigurationPOJO actual =
new
JobConfigurationAPIImpl(regCenter).getJobConfiguration("test_job");
verify(regCenter).getDirectly("/test_job/config");
verify(regCenter, never()).get("/test_job/config");
```
The same expectation should cover the configuration read performed by:
```java
new JobStatisticsAPIImpl(regCenter).getJobBriefInfo("test_job");
```
A minimal fix is to replace cached configuration reads with
`getDirectly()` in these two lifecycle methods.
The fix should not remove `ZookeeperRegistryCenter` caches or change
cached reads used by the scheduling kernel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail:
[email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
