github-actions[bot] commented on code in PR #69642:
URL: https://github.com/apache/doris/pull/69642#discussion_r4384680497
##########
fe/fe-core/src/main/java/org/apache/doris/journal/bdbje/BDBEnvironment.java:
##########
@@ -228,6 +236,47 @@ public void setup(File envHome, String selfNodeName,
String selfNodeHostPort,
}
}
}
+ if (replicatedEnvironment == null || epochDB == null) {
+ // e.g. still got InsufficientLogException at the last retry
+ LOG.error("failed to open replicated environment after {} retries.
will exit.", RETRY_TIME);
+ exitHook.accept(-1);
+ }
+ }
+
+ /*
+ * Handle InsufficientLogException thrown by an opened environment: exit
the process WITHOUT NetworkRestore.
+ *
+ * Once the opened environment is invalidated, every thread which touches
it (replayer, http/rpc threads, ...)
+ * gets the same exception. Running NetworkRestore in this process is
unsafe:
+ * - NetworkRestore deletes and renames log files in the environment
directory, concurrent restores by
+ * different threads corrupt the directory;
+ * - the invalidated environment is still open, other threads hold its
handles and its bdbje daemon threads
+ * may still touch the log files;
+ * - the in-memory meta may have replayed logs which have been rolled
back, it must be rebuilt anyway.
+ *
+ * After restart, opening the environment throws InsufficientLogException
again because the cause still
+ * exists, and setup() performs the restore in a clean state. If no
exception is thrown after restart, the
+ * node can sync up with the master normally and no restore is needed.
+ */
+ public static void exitForNetworkRestore(InsufficientLogException
insufficientLogEx, String reason) {
+ synchronized (NETWORK_RESTORE_LOCK) {
+ LOG.error("bdb environment needs network restore, thread: {},
reason: {}. "
+ + "will exit, and restore when opening bdb environment
after restart.",
+ Thread.currentThread().getName(), reason,
insufficientLogEx);
+ exitHook.accept(-1);
Review Comment:
[P2] Avoid joining the RPC thread that triggers fatal recovery
With `thrift_server_type=SIMPLE`, a `fetchSchemaTableData` request for
`FRONTENDS` runs on `ThriftServer.serverThread` and reaches this call through
`FrontendsProcNode.getFrontendsInfo` -> `getMaxJournalId` ->
`getDatabaseNames`. [TSimpleServer processes requests inline on its serving
thread](https://github.com/apache/thrift/blob/v0.24.0/lib/java/src/main/java/org/apache/thrift/server/TSimpleServer.java).
The default hook calls `System.exit`, which waits for shutdown hooks, but
`DorisFE`'s hook calls `OssThriftServerStarter.stop` -> `ThriftServer.join` ->
unbounded `serverThread.join()`. The hook therefore waits for the same RPC
thread that is waiting for the hook: the FE never exits, so its supervisor
cannot restart it to perform the startup restore. If the replayer wins the
monitor first, this RPC can instead remain blocked on `NETWORK_RESTORE_LOCK`
with the same result. Make fatal shutdown independent of joining these blocked
callers, and cover this with a subprocess test using
the real shutdown hook; the injected exit hooks do not exercise it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
