fresh-borzoni commented on code in PR #3649:
URL: https://github.com/apache/fluss/pull/3649#discussion_r3595622612
##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorLeaderElection.java:
##########
@@ -186,18 +140,138 @@ public void notLeader() {
public void close() {
LOG.info("Closing LeaderLatch for server {}.", serverId);
- if (leaderLatch != null) {
+ if (closing.compareAndSet(false, true)) {
try {
leaderLatch.close();
} catch (Exception e) {
LOG.error("Failed to close LeaderLatch for server {}.",
serverId, e);
}
+
+ // Events submitted after closing starts are ignored by their
executor-side check.
+ // Since the executor is single-threaded, this task runs after all
leadership work
+ // already queued before close and completes only after leader
cleanup has finished.
+ leaderCallbackExecutor.execute(
+ () -> {
+ try {
+ boolean cleanupRequired = state == State.LEADER;
+ state = State.CLOSED;
+ if (cleanupRequired) {
+ cleanupLeaderServices(null);
+ }
+ } finally {
+ closeFuture.complete(null);
+ }
+ });
}
- leaderCallbackExecutor.shutdownNow();
+ closeFuture.join();
+ leaderCallbackExecutor.shutdown();
+ awaitCallbackExecutorTermination();
}
public boolean isLeader() {
- return this.isLeader.get();
+ return !closing.get() && state == State.LEADER;
+ }
+
+ private void submitLeadershipEvent(boolean leader, Runnable
initLeaderServices) {
+ if (closing.get()) {
+ return;
+ }
+ try {
+ leaderCallbackExecutor.execute(
+ () -> {
+ if (closing.get() || state == State.CLOSED) {
+ return;
+ }
+ if (leader) {
+ becomeLeader(initLeaderServices);
+ } else {
+ becomeStandby();
+ }
+ });
+ } catch (RejectedExecutionException e) {
+ if (!closing.get()) {
+ throw e;
+ }
+ }
+ }
+
+ private void becomeLeader(Runnable initLeaderServices) {
+ if (closing.get() || state == State.LEADER || state == State.CLOSED) {
+ return;
+ }
+
+ LOG.info("Coordinator server {} has become the leader.", serverId);
+ Throwable initializationFailure = null;
+ try {
+ initLeaderServices.run();
+ } catch (CoordinatorEpochFencedException e) {
+ LOG.warn(
+ "Coordinator server {} has been fenced and not become
leader successfully.",
+ serverId);
+ initializationFailure = e;
+ } catch (Exception e) {
+ LOG.error("Failed to initialize leader services for server {}",
serverId, e);
+ initializationFailure = e;
+ }
+
+ if (initializationFailure == null && !closing.get()) {
+ state = State.LEADER;
+ } else {
+ cleanupLeaderServices(initializationFailure);
Review Comment:
If init throws we clean up and go STANDBY but still hold the latch, so
nothing re-elects and we're left with a leader that isn't serving.
Looks pre-existing, maybe a follow-up, but could we at least add a test for
the cleanup path here? wdyt?
##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorLeaderElection.java:
##########
@@ -186,18 +140,138 @@ public void notLeader() {
public void close() {
LOG.info("Closing LeaderLatch for server {}.", serverId);
- if (leaderLatch != null) {
+ if (closing.compareAndSet(false, true)) {
try {
leaderLatch.close();
} catch (Exception e) {
LOG.error("Failed to close LeaderLatch for server {}.",
serverId, e);
}
+
+ // Events submitted after closing starts are ignored by their
executor-side check.
+ // Since the executor is single-threaded, this task runs after all
leadership work
+ // already queued before close and completes only after leader
cleanup has finished.
+ leaderCallbackExecutor.execute(
+ () -> {
+ try {
+ boolean cleanupRequired = state == State.LEADER;
+ state = State.CLOSED;
+ if (cleanupRequired) {
+ cleanupLeaderServices(null);
+ }
+ } finally {
+ closeFuture.complete(null);
+ }
+ });
}
- leaderCallbackExecutor.shutdownNow();
+ closeFuture.join();
Review Comment:
Think my earlier "can't hang" only covered init - cleanup takes lock and
shuts down the event processor / channel manager, none of which are
curator-bounded, so close() can now block the shutdown thread until it's
SIGKILL'd.
The callback threads are daemon and old close() didn't wait at all, so maybe
just bound the wait (timeout on join) and move on? shutdownNow() won't help if
it's stuck on lock anyway. wdyt?
##########
fluss-server/src/main/java/org/apache/fluss/server/coordinator/CoordinatorLeaderElection.java:
##########
@@ -186,18 +140,138 @@ public void notLeader() {
public void close() {
LOG.info("Closing LeaderLatch for server {}.", serverId);
- if (leaderLatch != null) {
+ if (closing.compareAndSet(false, true)) {
try {
leaderLatch.close();
} catch (Exception e) {
LOG.error("Failed to close LeaderLatch for server {}.",
serverId, e);
}
+
+ // Events submitted after closing starts are ignored by their
executor-side check.
+ // Since the executor is single-threaded, this task runs after all
leadership work
+ // already queued before close and completes only after leader
cleanup has finished.
+ leaderCallbackExecutor.execute(
+ () -> {
+ try {
+ boolean cleanupRequired = state == State.LEADER;
+ state = State.CLOSED;
+ if (cleanupRequired) {
+ cleanupLeaderServices(null);
+ }
+ } finally {
+ closeFuture.complete(null);
+ }
+ });
}
- leaderCallbackExecutor.shutdownNow();
+ closeFuture.join();
+ leaderCallbackExecutor.shutdown();
+ awaitCallbackExecutorTermination();
}
public boolean isLeader() {
- return this.isLeader.get();
+ return !closing.get() && state == State.LEADER;
+ }
+
+ private void submitLeadershipEvent(boolean leader, Runnable
initLeaderServices) {
Review Comment:
nit: initLeaderServices is passed to the standby branch too but only leader
uses it — drop it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
