Doris-Breakwater commented on issue #70655:
URL: https://github.com/apache/doris/issues/70655#issuecomment-6937166405

   Breakwater-GitHub-Analysis-Slot: slot_586cbd41118b
   
   Initial triage: this is an FE metadata decoding failure with a confirmed 
process-exit path. The evidence does **not yet establish why the JSON presented 
to Gson was malformed**, or identify a matching LoadJob serialization fix. I 
checked the exact reported commit, `59de8c4c524008e8ab2e43b79312f716a3a423a8`, 
rather than assuming a release/tag mapping. The issue currently has no labels; 
FE persistence/journal replay is the primary area for maintainer routing, with 
the statistics exception tracked separately.
   
   **What the matching code establishes**
   
   - `BDBJournalCursor.next()` retrieves a record with `READ_COMMITTED`, then 
calls `JournalEntity.readFields()`. Its deserialization exception handler logs 
this exact fatal message and calls `System.exit(-1)`. `DorisFE` registers the 
shutdown hook that prints “Received shutdown signal”; that message also follows 
an internal `System.exit`, so it does not establish an external kill. See 
[cursor lines 
120–131](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBJournalCursor.java#L120-L131)
 and [shutdown 
hook](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/DorisFE.java#L160-L169).
   - At the reported `JournalEntity.java:559`, the dispatch case is 
`OP_CREATE_LOAD_JOB` (230). This is a code-based inference from the stack, 
pending confirmation from the actual opcode bytes. It does not identify the job 
ID, label, or load subtype. The record layout is a two-byte opcode followed, 
for this operation, by a four-byte UTF-8 byte length and the JSON bytes. 
`LoadJob.write()` uses `GsonUtils.GSON.toJson(this)` and `Text.writeString()`; 
`read()` reverses that path. This is internal metadata JSON, with no evidence 
that malformed user input caused it. See 
[dispatch](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/JournalEntity.java#L558-L561),
 [LoadJob 
read/write](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/load/loadv2/LoadJob.java#L827-L829),
 and [Text framing](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b
 
79312f716a3a423a8/fe/fe-common/src/main/java/org/apache/doris/common/io/Text.java#L404-L427).
   - The exception occurs at `Streams.parse(in)`, before subtype selection and 
object reconstruction. A missing subtype registration is therefore not 
supported by this trace. Column 660 is a parser position in the decoded text, 
not necessarily byte offset 660 in the BDB value. See [adapter read 
path](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/persist/gson/RuntimeTypeAdapterFactory.java#L303-L330).
   - Successful per-entry replay is logged at DEBUG in this build. Absence of 
`19295330` from normal healthy-node logs is inconclusive; a later replay ID 
alone also does not distinguish replay from loading a newer image.
   
   **Exporting the failing record without successful deserialization**
   
   The matching build already contains a useful tool: `BDBTool.getValueByKey()` 
catches the decode exception and subsequently prints `key`, `op code`, `num 
bytes`, and escaped `bytes`, even when `value` is null. It opens the JE 
environment/database read-only. See [tool 
implementation](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBTool.java#L144-L184).
   
   1. Preserve FE-A's full metadata directory and incident logs before 
restarting or rebuilding it. Run diagnostics against a disposable, consistent 
offline copy, using the exact FE package and JE/Gson libraries. For a healthy 
FE, obtain a consistent backup/snapshot through the deployment's supported 
procedure; do not stop either remaining voting FE merely to get a comparison 
copy.
   2. Use an isolated copied FE installation. Set its `DORIS_HOME`, `PID_DIR`, 
logging paths, and effective `meta_dir` to the diagnostic locations, including 
any `fe_custom.conf` overrides. With that package's Java 17 options and `lib/*` 
classpath, the Java main-class/argument forms are:
   
      ```text
      org.apache.doris.DorisFE --bdb --listdb
      org.apache.doris.DorisFE --bdb --db <segment-db-name> --from 19295330 
--to 19295330
      ```
   
      These are JVM main-class arguments, not commands for `start_fe.sh`: that 
version's shell wrapper does not expose the `--bdb` options. List the numeric 
segment databases and choose the greatest segment start no greater than 
`19295330`; do not assume the segment name equals the journal ID. Preserve 
stdout, stderr, and diagnostic logs. The tool supports `--metaversion` if 
decoding requires the recorded metadata version; do not substitute the product 
version number.
   3. Keep the original export private: LoadJob metadata can contain 
identifiers and source properties. For a lossless comparison, have a maintainer 
export the raw JE `DatabaseEntry` value slice (respecting offset/size) as 
hex/base64 and hash those original bytes before sanitization. The lookup key 
uses JE `TupleBinding<Long>`, not a decimal-text key. The tool's mixed 
printable/escaped `bytes:` output is useful for inspection but is not an 
unambiguous binary interchange format.
   
   One diagnostic limitation matters: this `BDBTool` explicitly disables JE log 
checksum reads. Successful extraction therefore does **not** establish storage 
integrity. If it cannot open the offline copy, retain that error for a 
version-matched JE diagnostic procedure rather than repairing the evidence copy.
   
   **How the comparison would narrow the cause**
   
   - Different values for the same journal ID in confirmed copies of the same 
cluster would direct investigation toward record divergence and the 
storage/replication path; it would not by itself prove a disk fault.
   - Identical values that fail the matching decoder on both copies would 
direct investigation toward the persisted representation, framing, and writer 
path.
   - Identical values that decode successfully offline while FE-A fails would 
require comparison of actual loaded libraries, JVM/configuration, and 
repeatability before attributing a cause.
   - `NOT FOUND` on a healthy copy can reflect retention/compaction and does 
not prove that node replayed or rejected the entry.
   
   Please provide: the opcode, value length and original-byte hashes from both 
copies if retained; a sanitized structural excerpt around the failing JSON 
position; SHA-256 checksums for `doris-fe.jar`, Gson and JE JARs across all 
FEs; actual JVM versions/options and effective metadata read settings, 
especially `metadata_text_read_max_batch_bytes`; FE-A's full 
exception/startup/shutdown window; healthy-node image-load and replay-range 
logs covering this entry; and available JE, filesystem, device, and 
service-manager diagnostics. Please align timestamps across sources: the 
incident time is `2026-11-13 17:36:35 UTC`. Once the record identifies the load 
subtype/job, its creation SQL and preceding DDL/load sequence would help 
construct a targeted reproduction. A query execution profile is not the first 
diagnostic needed for this metadata parser failure.
   
   **Recovery of this single non-Master Follower**
   
   Keep the two healthy voting FEs running; the OBSERVER does not supply the 
missing vote. After preserving evidence, the preferred recovery direction is a 
controlled rebuild/rejoin of FE-A from authoritative healthy metadata, with the 
exact membership procedure confirmed for this deployment. Do not treat 
force-skipping `19295330` as a routine fix: skipping a job-creation record can 
leave later metadata operations without their prerequisite state. Cluster-wide 
metadata recovery mode is also inappropriate as a default response to one 
failed follower; the [4.x metadata operations 
guide](https://doris.apache.org/docs/4.x/admin-manual/trouble-shooting/metadata-operation/)
 documents its risks.
   
   A valid same-cluster image with journal sequence **N >= 19295330** can avoid 
decoding this record: `Env.loadImage()` sets the replay position to N, and 
replay starts at N+1. This is conditional recovery, not proof that the 
underlying cause has been fixed. Verify the completed image's integrity and 
package compatibility, continuous availability of journals after N, correct 
FE-A identity/role and cluster membership, and absence of skip/recovery 
overrides. Preserve FE-A's identity files appropriately; do not copy another 
node's `ROLE`/node identity wholesale. After rejoin, confirm `Alive=true`, 
advancing/caught-up replay position, and successful metadata operations. See 
[image 
loading](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/catalog/Env.java#L2192-L2203)
 and [replay starting 
position](https://github.com/apache/doris/blob/59de8c4c524008e8ab2e43b79312f716a3a423a8/fe/fe-core/src/main/java/org/apache/doris/ca
 talog/Env.java#L3068-L3078).
   
   Simply adding `--helper` to an existing installation does not guarantee a 
newer image: when both local `ROLE` and `VERSION` exist, this build follows the 
local identity path and ignores helpers. Rebuilding/reseeding must therefore 
follow the deliberate node recovery procedure rather than assuming a restart 
downloads an image.
   
   **The statistics exception has a separate matching fix**
   
   The reported commit dereferences `table.getColumn(column.colName)` without 
checking for null. A queued column can become stale after a column is dropped 
while its table still exists. Public [PR 
#63882](https://github.com/apache/doris/pull/63882) adds the null check, and 
[PR #63947](https://github.com/apache/doris/pull/63947) is its merged 4.0 
backport. The reported commit lacks that change; release inclusion should be 
checked against the intended upgrade package. `Daemon.run()` catches and logs 
this exception and continues subsequent cycles. The inspected statistics path 
processes queued columns and sends a statistics RPC; it does not serialize 
LoadJob journal records. Treat this as a separate known defect unless 
additional evidence links the two failures. Applying that fix is not an 
established remedy for the fatal JSON error.
   
   No source changes were made, and no production metadata was available for 
reproduction. The exit mechanism and diagnostic/recovery conditions above are 
verified from the matching source; the origin of the malformed JSON remains 
unresolved pending the record-level evidence.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to