Doris-Breakwater commented on issue #68875: URL: https://github.com/apache/doris/issues/68875#issuecomment-6095818815
Breakwater-GitHub-Analysis-Slot: slot_dcd8fca4fba4 This warrants prioritization as an FE availability bug in JE replication statistics: the supplied stack identifies a master-side histogram failure, and the narrowing defect is independently reproducible. The issue is currently open with no labels. Suggested triage area: FE / BDB JE / replication; the exact affected release range still needs artifact verification. **Verified findings.** The local public `4.1.3-rc02` tag (`7126cf65d96ebc43fce0906f51e92c1a2ccf24a6`) declares `org.apache.doris:je:18.3.14-doris-SNAPSHOT`. See the [release dependency declaration](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/pom.xml#L272). The locally cached jar's bytecode executes `l2i` before `Math.min(int,int)`. On Java 17.0.18, calling its actual `LatencyPercentile.add()` with synthetic inputs reproduces both reported indices: | Input milliseconds | Current implementation, maximum 1000 | Proposed long-domain clamp | | --- | --- | --- | | `2256089592L` | Exception, index `-2038877704` | Bucket 1000 | | `2423441778L` | Exception, index `-1871525518` | Bucket 1000 | | `4294967296L` | Incorrect percentile 0 | Bucket 1000 | | `4294967338L` | Incorrect percentile 42 | Bucket 1000 | These are synthetic reproductions, not recovered runtime inputs. Ordinary values retain their expected buckets; negative inputs are already ignored. Consequently, PR #419's clamp before narrowing preserves the existing input policy and correctly sends oversized nonnegative values to the maximum bucket. The [histogram implementation and contract](https://github.com/apache/doris-thirdparty/blob/70324a2452e4ff717c872e431b5b385c82a54f84/src/main/java/com/sleepycat/je/utilint/LatencyPercentile.java#L272-L291) support this change. The [feeder heartbeat path](https://github.com/apache/doris-thirdparty/blob/70324a2452e4ff717c872e431b5b385c82a54f84/src/main/java/com/sleepycat/je/rep/impl/node/Feeder.java#L901-L983) calculates `delay = lastResponseTime - statCommitTimestamp` and passes it to the percentile statistic. When behind, it subtracts the VLSN/rate estimate from the commit timestamp; a nonpositive generation rate skips this calculation. The [uncaught-exception handler](https://github.com/apache/doris-thirdparty/blob/70324a2452e4ff717c872e431b5b385c82a54f84/src/main/java/com/sleepycat/je/rep/impl/node/Feeder.java#L1578-L1605) promotes the failure to `UNCAUGHT_EXCEPTION`, invalidating the environment. Doris journals use that environment, and [database-name lookup can exit after repeated environment failures](https://github.com/apache/doris/blob/7126cf65d96ebc43fce0906f51e92c1a2ccf24a6/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java#L430-L453). This explains why a monitoring sta tistic can interrupt the existing master. **Qualification of the trigger.** Large lag with a low positive generation rate is plausible, but is not yet proven to be the sole source of the oversized sample. The [feeder output path](https://github.com/apache/doris-thirdparty/blob/70324a2452e4ff717c872e431b5b385c82a54f84/src/main/java/com/sleepycat/je/rep/impl/node/Feeder.java#L1333-L1442) records transaction timestamps from streamed records. Historical timestamps can themselves make the delay large, including when the replica has reached the feeder's recorded commit VLSN. The supplied September timestamp is approximately 2.157 billion milliseconds before the failure, already above `Integer.MAX_VALUE`; it is not necessarily the timestamp used by the failing heartbeat. The heartbeat's advertised master VLSN likewise need not equal the feeder's recorded statistics VLSN. Neither slow networking nor host clock mismatch is established by these logs. The cached jar also reports JE `18.3.12`, consistent with [the fork's embedded version constant](https://github.com/apache/doris-thirdparty/blob/70324a2452e4ff717c872e431b5b385c82a54f84/src/main/java/com/sleepycat/je/JEVersion.java#L33-L34). That string alone does not prove the wrong jar was loaded. **Information needed to close the remaining gaps:** - Exact Doris build commit/package provenance for all FEs, loaded JE jar location and SHA-256, and any duplicate JE jars on the classpath. A `SNAPSHOT` filename does not identify immutable contents. - Sanitized master and joining-FE `je.info*`, `fe.log`, and `fe.out` covering startup through the first exception and subsequent journal failure; FE roles/topology and relevant replication configuration. - For a controlled reproduction, capture `lastHeartbeatCommitVLSN`, `replicaTxnVLSNSeq`, `lastHeartbeatCommitTimestamp`, `lastResponseTime`, `vlsnRatePerMinute`, the selected calculation branch, and the final `delay` before percentile insertion. This distinguishes historical timestamp age from the rate-based estimate. - The four-test source and command, plus results from the patched artifact in a live catch-up reproduction. Current evidence does not establish successful cluster recovery with the patch. **Recommended maintainer actions:** 1. Review [doris-thirdparty PR #419](https://github.com/apache/doris-thirdparty/pull/419), currently open and unmerged, and commit regression tests with it. Cover `Integer.MAX_VALUE + 1L`, both sample values above, `2^32`, `2^32 + 42`, `Long.MAX_VALUE`, normal bucket boundaries, ignored negatives, and maximum zero. Assert both histogram/count behavior and percentile results. The PR currently contains only the implementation change; its reported tests are not committed. 2. [StarRocks PR #12](https://github.com/StarRocks/starrocks-bdb-je/pull/12) confirms the same failure pattern, but its post-conversion lower clamp can record wrapped positive samples in incorrect buckets. PR #419's long-domain upper clamp addresses both the exception and that statistic corruption. 3. Build and publish a traceable patched JE artifact, update Doris to consume it, and test empty/lagging FE catch-up against old logs with low journal activity. Verify master environment validity, continued journal writes, catch-up completion, and ordinary replication. Backport according to supported releases whose shipped jars contain the vulnerable bytecode; do not infer safety from a newer version number alone. 4. Until validated, avoid repeated joining-FE retries against an affected production master. Preserve metadata and logs, and confirm healthy electable replicas/quorum before recovery. Clearing the joining FE's metadata recreates the reported empty-environment trigger and is not a demonstrated fix. This investigation verified source, cached-jar behavior, and boundary arithmetic. It did not modify repository code, build the patched JE project, or run a live cluster test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
