MaMengzhen opened a new issue, #70655:
URL: https://github.com/apache/doris/issues/70655

   ## Version / environment
   
   - Version reported by all FEs: `doris-4.0.5-rc01-59de8c4c524`
   - Java 17; Linux
   - Topology: 3 FOLLOWER FEs (including the Master) and 1 OBSERVER FE.
   - The affected node is a non-Master FOLLOWER, referred to below as FE-A.
   
   ## What happened
   
   FE-A exited at approximately 2026-11-14 01:36:35 (UTC+08:00). Its last 
successful heartbeat was 01:36:26. Afterward, SHOW FRONTENDS reported 
Join=true, Alive=false, and `java.net.ConnectException: Connection refused`. No 
DorisFE process or listeners on ports 8030/9010/9020/9030 were found on FE-A.
   
   Immediately before shutdown, the replayer logged a failure to deserialize a 
journal entry through LoadJob.read():
   
   ```text
   2026-11-14 01:36:35,700 ERROR (replayer|134)
   [BDBJournalCursor.next():130] fail to read journal entity key=19295330, will 
exit
   com.google.gson.JsonSyntaxException: 
com.google.gson.stream.MalformedJsonException:
   Expected ':' at line 1 column 660 path $.:
       at com.google.gson.internal.Streams.parse(Streams.java:61)
       at 
org.apache.doris.persist.gson.RuntimeTypeAdapterFactory$1.read(RuntimeTypeAdapterFactory.java:306)
       at com.google.gson.TypeAdapter$1.read(TypeAdapter.java:204)
       at com.google.gson.Gson.fromJson(Gson.java:1227)
       at com.google.gson.Gson.fromJson(Gson.java:1137)
       at com.google.gson.Gson.fromJson(Gson.java:1047)
       at com.google.gson.Gson.fromJson(Gson.java:982)
       at org.apache.doris.load.loadv2.LoadJob.read(LoadJob.java:828)
       at 
org.apache.doris.journal.JournalEntity.readFields(JournalEntity.java:559)
       at 
org.apache.doris.journal.bdbje.BDBJournalCursor.next(BDBJournalCursor.java:127)
       at org.apache.doris.catalog.Env.replayJournal(Env.java:3088)
       at org.apache.doris.catalog.Env$4.runOneCycle(Env.java:2872)
       at org.apache.doris.common.util.Daemon.run(Daemon.java:119)
   ```
   
   At 01:36:35,701 it logged `Received shutdown signal, starting graceful 
shutdown...`, followed by shutdown of the HTTP server. The fatal replay error 
precedes shutdown; there is no evidence establishing an external kill as the 
cause.
   
   ## Other observations
   
   - The other FEs were alive and reported ReplayedJournalId values beyond 
19295330 in a later SHOW FRONTENDS snapshot. This alone does not establish 
whether they replayed this exact entry or loaded a later image.
   - Searches for the exact journal ID in the healthy FEs' retained logs did 
not find it. Their logs for the incident date are present.
   - The displayed healthy-node extracts were limited by `head -n 100` and 
mainly cover earlier progress around journal IDs 19294606–19294651. They do not 
prove success or failure at 19295330.
   - FE-A and healthy nodes also logged recurring exceptions in the statistics 
daemon:
   
   ```text
   daemon thread got exception. name: Follower Column Sender
   java.lang.NullPointerException: Cannot invoke 
"org.apache.doris.catalog.Column.getType()"
   because the return value of 
"org.apache.doris.catalog.TableIf.getColumn(String)" is null
       at 
org.apache.doris.statistics.FollowerColumnSender.getNeedAnalyzeColumns(FollowerColumnSender.java:133)
       at 
org.apache.doris.statistics.FollowerColumnSender.send(FollowerColumnSender.java:77)
       at 
org.apache.doris.statistics.FollowerColumnSender.runAfterCatalogReady(FollowerColumnSender.java:65)
   ```
   
     Replay continued after those daemon exceptions. Their relationship to the 
fatal JSON error is unknown.
   - A kernel journal query for the incident interval returned no entries. This 
is not sufficient to rule out system/storage problems.
   - The actual raw bytes/opcode of entry 19295330 have not been extracted. Its 
LoadJob ID/label are unknown; neighboring successfully replayed job IDs should 
not be attributed to it.
   - JAR checksums across nodes, disk diagnostics, and repeatability after 
restart have not yet been verified.
   
   ## Expected behavior
   
   The Follower should successfully deserialize and replay valid committed 
metadata. If an entry cannot be decoded, diagnostic information and a 
documented safe recovery procedure would help identify the underlying problem.
   
   ## Reproduction
   
   Observed on a running cluster. No minimal reproduction or confirmed 
triggering SQL is available yet. The failure occurred in internal metadata 
replay; we have not established that any user-supplied JSON input was malformed.
   
   ## Questions for maintainers
   
   1. Is there a known LoadJob journal serialization/deserialization issue 
affecting this build, with a corresponding fix/PR?
   2. What version-matched tool/procedure can safely export the opcode and raw 
payload of entry 19295330 from an offline copy of FE-A's metadata, even if 
normal deserialization fails?
   3. How can we safely compare this entry with a healthy FE's copy if it is 
still retained?
   4. If the error repeats at the same entry, what is the recommended recovery 
procedure for this single non-Master Follower while preserving the healthy 
quorum? Can recovery from a newer healthy image avoid replaying the problematic 
entry, and what checks are required?
   5. Could the FollowerColumnSender exception be related, or should it be 
investigated separately?
   
   We can provide additional sanitized logs, package checksums, and diagnostic 
output. Internal IP addresses, business identifiers, credentials, and 
authentication headers have been omitted.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to