GorlosHao opened a new issue, #68875:
URL: https://github.com/apache/doris/issues/68875
### Search before asking
I searched the public Doris and doris-thirdparty issues/PRs for
`LatencyPercentile`, `Feeder.java:981`, and the feeder array-index exception. I
did not find a previous matching report.
### Version
- Apache Doris 4.1.3-rc02, as reported by the runtime.
- Java 17.0.2 on the affected master FE.
- Installed jar: `je-18.3.14-doris-SNAPSHOT.jar`; JE exception messages
report `18.3.12`.
- Existing FEs run in Docker; the joining FE runs natively on Linux.
### What happened
Starting a new FE with an empty JE replication environment and a helper
repeatedly causes the existing master FE to fail during initial catch-up. In my
environment this reproduces almost every time.
The master throws an uncaught exception while processing a replication
heartbeat. JE invalidates its environment, journal operations fail, and the FE
service is interrupted until restart/recovery. This is not merely a failed
start on the joining FE.
```text
com.sleepycat.je.EnvironmentFailureException:
Uncaught exception in feeder thread
UNCAUGHT_EXCEPTION: Uncaught Exception in internal thread, unable to
continue.
Environment is invalid and must be closed.
Caused by: java.lang.ArrayIndexOutOfBoundsException:
Index -2038877704 out of bounds for length 1001
at
java.base/java.util.concurrent.atomic.AtomicLongArray.incrementAndGet(AtomicLongArray.java:234)
at
com.sleepycat.je.utilint.LatencyPercentile.add(LatencyPercentile.java:288)
at
com.sleepycat.je.rep.impl.node.Feeder$InputThread.processHeartbeatResponse(Feeder.java:981)
at
com.sleepycat.je.rep.impl.node.Feeder$InputThread.runResponseLoop(Feeder.java:856)
at com.sleepycat.je.rep.impl.node.Feeder$InputThread.run(Feeder.java:755)
```
A previous reproduction produced the same stack with index `-1871525518`.
### Reproduction and evidence
1. Have an existing cluster with substantial historical JE logs and
relatively low current journal activity.
2. Register a new FE and start it with an empty JE replication environment,
using `start_fe.sh --helper <existing-fe>:9010 --daemon`.
3. Observe the master feeder during initial catch-up.
Selected joining-FE JE logs (UTC, October 10, 2026):
```text
08:09:03.441 Replica range: first=-1 last=-1 sync=-1 txnEnd=-1
08:09:03.450 start stream at VLSN: 1
08:09:03.462 Replica initialization completed. Replica VLSN: -1
Heartbeat master commit VLSN: 710,294
Replica VLSN delta: 710,295
```
The master exception was recorded around 08:09:06. The joining FE subsequent
shutdown dump showed:
```text
vlsnRange=first=1 last=113,190 sync=113,190 txnEnd=113,190
lastReplayedTxn= VLSN: 113,190
masterTxnEndTime=Tue Sep 15 08:59:33 UTC 2026
```
Catch-up was replaying historical records within seconds of startup. The
September timestamp is the historical transaction timestamp, not evidence of a
host clock mismatch.
### Root cause
In `Feeder.InputThread.processHeartbeatResponse()`, when the replica has not
reached the heartbeat commit VLSN, JE estimates time lag using:
```java
(long) (60000.0 * ((double) vlsnLag / vlsnRatePerMinute))
```
A large initial VLSN gap divided by a low current master generation rate can
produce a very large estimated delay. This is not measured network transfer
duration.
`LatencyPercentile.add(long latencyMillis)` narrows before clamping:
```java
Math.min((int) latencyMillis, maxTrackedLatencyMillis)
```
Large positive values can narrow to negative histogram indices or incorrect
lower buckets. For example, `2256089592L` narrows to `-2038877704`. The exact
delay input and generation rate at the failing heartbeat were not logged; this
example does not establish the exact runtime input.
### Proposed fix
Clamp in the long domain before narrowing:
```java
(int) Math.min(latencyMillis, maxTrackedLatencyMillis)
```
Patch: https://github.com/apache/doris-thirdparty/pull/419
StarRocks previously addressed the same failure:
https://github.com/StarRocks/starrocks-bdb-je/pull/12
Local isolated JUnit 4.12 tests using the upstream class and supporting
classes: 4 tests, 2 failures before the patch, all 4 passed after it. The full
Maven build and a live cluster test with the patched jar have not been run.
Please consider the thirdparty fix, updating the Doris JE
dependency/artifact, and backporting to affected supported releases.
### Expected behavior
A newly joining FE should be able to catch up without a latency monitoring
statistic invalidating the master JE environment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
