MaMengzhen opened a new issue, #70655:
URL: https://github.com/apache/doris/issues/70655
## Version / environment
- Version reported by all FEs: `doris-4.0.5-rc01-59de8c4c524`
- Java 17; Linux
- Topology: 3 FOLLOWER FEs (including the Master) and 1 OBSERVER FE.
- The affected node is a non-Master FOLLOWER, referred to below as FE-A.
## What happened
FE-A exited at approximately 2026-11-14 01:36:35 (UTC+08:00). Its last
successful heartbeat was 01:36:26. Afterward, SHOW FRONTENDS reported
Join=true, Alive=false, and `java.net.ConnectException: Connection refused`. No
DorisFE process or listeners on ports 8030/9010/9020/9030 were found on FE-A.
Immediately before shutdown, the replayer logged a failure to deserialize a
journal entry through LoadJob.read():
```text
2026-11-14 01:36:35,700 ERROR (replayer|134)
[BDBJournalCursor.next():130] fail to read journal entity key=19295330, will
exit
com.google.gson.JsonSyntaxException:
com.google.gson.stream.MalformedJsonException:
Expected ':' at line 1 column 660 path $.:
at com.google.gson.internal.Streams.parse(Streams.java:61)
at
org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:306)
at com.google.gson.TypeAdapter$1.read(TypeAdapter.java:204)
at com.google.gson.Gson.fromJson(Gson.java:1227)
at com.google.gson.Gson.fromJson(Gson.java:1137)
at com.google.gson.Gson.fromJson(Gson.java:1047)
at com.google.gson.Gson.fromJson(Gson.java:982)
at org.apache.doris.load.loadv2.LoadJob.read(LoadJob.java:828)
at
org.apache.doris.journal.JournalEntity.readFields(JournalEntity.java:559)
at
org.apache.doris.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:127)
at org.apache.doris.catalog.Env.replayJournal(Env.java:3088)
at org.apache.doris.catalog.Env$4.runOneCycle(Env.java:2872)
at org.apache.doris.common.util.Daemon.run(Daemon.java:119)
```
At 01:36:35,701 it logged `Received shutdown signal, starting graceful
shutdown...`, followed by shutdown of the HTTP server. The fatal replay error
precedes shutdown; there is no evidence establishing an external kill as the
cause.
## Other observations
- The other FEs were alive and reported ReplayedJournalId values beyond
19295330 in a later SHOW FRONTENDS snapshot. This alone does not establish
whether they replayed this exact entry or loaded a later image.
- Searches for the exact journal ID in the healthy FEs' retained logs did
not find it. Their logs for the incident date are present.
- The displayed healthy-node extracts were limited by `head -n 100` and
mainly cover earlier progress around journal IDs 19294606–19294651. They do not
prove success or failure at 19295330.
- FE-A and healthy nodes also logged recurring exceptions in the statistics
daemon:
```text
daemon thread got exception. name: Follower Column Sender
java.lang.NullPointerException: Cannot invoke
"org.apache.doris.catalog.Column.getType()"
because the return value of
"org.apache.doris.catalog.TableIf.getColumn(String)" is null
at
org.apache.doris.statistics.FollowerColumnSender.getNeedAnalyzeColumns(FollowerColumnSender.java:133)
at
org.apache.doris.statistics.FollowerColumnSender.send(FollowerColumnSender.java:77)
at
org.apache.doris.statistics.FollowerColumnSender.runAfterCatalogReady(FollowerColumnSender.java:65)
```
Replay continued after those daemon exceptions. Their relationship to the
fatal JSON error is unknown.
- A kernel journal query for the incident interval returned no entries. This
is not sufficient to rule out system/storage problems.
- The actual raw bytes/opcode of entry 19295330 have not been extracted. Its
LoadJob ID/label are unknown; neighboring successfully replayed job IDs should
not be attributed to it.
- JAR checksums across nodes, disk diagnostics, and repeatability after
restart have not yet been verified.
## Expected behavior
The Follower should successfully deserialize and replay valid committed
metadata. If an entry cannot be decoded, diagnostic information and a
documented safe recovery procedure would help identify the underlying problem.
## Reproduction
Observed on a running cluster. No minimal reproduction or confirmed
triggering SQL is available yet. The failure occurred in internal metadata
replay; we have not established that any user-supplied JSON input was malformed.
## Questions for maintainers
1. Is there a known LoadJob journal serialization/deserialization issue
affecting this build, with a corresponding fix/PR?
2. What version-matched tool/procedure can safely export the opcode and raw
payload of entry 19295330 from an offline copy of FE-A's metadata, even if
normal deserialization fails?
3. How can we safely compare this entry with a healthy FE's copy if it is
still retained?
4. If the error repeats at the same entry, what is the recommended recovery
procedure for this single non-Master Follower while preserving the healthy
quorum? Can recovery from a newer healthy image avoid replaying the problematic
entry, and what checks are required?
5. Could the FollowerColumnSender exception be related, or should it be
investigated separately?
We can provide additional sanitized logs, package checksums, and diagnostic
output. Internal IP addresses, business identifiers, credentials, and
authentication headers have been omitted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]