DaanHoogland commented on code in PR #14677:
URL: https://github.com/apache/cloudstack/pull/14677#discussion_r4642622469
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -177,6 +190,14 @@ public static void setRetryInterval(Integer retryInterval)
{
RabbitMQEventBus.retryInterval = retryInterval;
}
+ public static void setBufferSize(Integer bufferSize) {
+ RabbitMQEventBus.bufferSize = bufferSize;
+ }
Review Comment:
```suggestion
/**
* maximum size to which the event buffer may grow
*/
public static void setBufferSize(Integer bufferSize) {
maxEventBufferSize = bufferSize;
}
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
private static Integer retryInterval;
+ // max number of events kept in memory while the AMQP server is
unreachable, 0 disables buffering
+ private static Integer bufferSize = 1000;
+
+ // time in milliseconds after which a buffered event is dropped if it
could not be sent
+ private static Integer bufferExpiry = 300000;
+
+ // events (routing key, description, time of buffering) waiting to be
sent, oldest first
+ private static final ArrayDeque<Ternary<String, String, Long>> s_buffer =
new ArrayDeque<>();
Review Comment:
```suggestion
/**
* events, stored as {routing key, description, time of buffering},
waiting to be sent, oldest first
*/
private static final ArrayDeque<Ternary<String, String, Long>>
eventBuffer = new ArrayDeque<>();
```
needs renaming below as well!
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
private static Integer retryInterval;
+ // max number of events kept in memory while the AMQP server is
unreachable, 0 disables buffering
+ private static Integer bufferSize = 1000;
Review Comment:
```suggestion
private static Integer maxEventBufferSize = 1000;
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
private static Integer retryInterval;
+ // max number of events kept in memory while the AMQP server is
unreachable, 0 disables buffering
+ private static Integer bufferSize = 1000;
+
+ // time in milliseconds after which a buffered event is dropped if it
could not be sent
+ private static Integer bufferExpiry = 300000;
Review Comment:
```suggestion
private static Integer eventBufferExpiryTime = 300000;
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
private static Integer retryInterval;
+ // max number of events kept in memory while the AMQP server is
unreachable, 0 disables buffering
+ private static Integer bufferSize = 1000;
+
+ // time in milliseconds after which a buffered event is dropped if it
could not be sent
+ private static Integer bufferExpiry = 300000;
+
+ // events (routing key, description, time of buffering) waiting to be
sent, oldest first
+ private static final ArrayDeque<Ternary<String, String, Long>> s_buffer =
new ArrayDeque<>();
+
+ // events are buffered only after the AMQP server was reachable at least
once
+ private static volatile boolean s_connectedOnce = false;
+
// hashmap to book keep the registered subscribers
private static ConcurrentHashMap<String, Ternary<String, Channel,
EventSubscriber>> s_subscribers;
// connection to AMQP server,
- private static Connection s_connection = null;
+ private static volatile Connection s_connection = null;
Review Comment:
now we're touching it let's rename to `connection`!?
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -273,20 +294,106 @@ public void publish(Event event) throws
EventBusException {
String routingKey = createRoutingKey(event);
String eventDescription = event.getDescription();
+ // keep the order: nothing is sent directly while older events are
waiting or the connection is down
+ if (bufferEvent(routingKey, eventDescription, false)) {
+ return;
+ }
+
try {
- Connection connection = getConnection();
- Channel channel = createChannel(connection);
- createExchange(channel, amqpExchangeName);
- publishEventToExchange(channel, amqpExchangeName, routingKey,
eventDescription);
- channel.close();
+ sendEvent(getConnection(), routingKey, eventDescription);
} catch (AlreadyClosedException e) {
closeConnection();
+ if (bufferEvent(routingKey, eventDescription, true)) {
+ return;
+ }
throw new EventBusException("Failed to publish event to message
broker as connection to AMQP broker in lost");
+ } catch (IOException e) {
+ if (bufferEvent(routingKey, eventDescription, true)) {
+ return;
+ }
+ throw new EventBusException("Failed to publish event to message
broker due to " + e.getMessage());
} catch (Exception e) {
throw new EventBusException("Failed to publish event to message
broker due to " + e.getMessage());
}
}
+ private void sendEvent(Connection connection, String routingKey, String
eventDescription) throws Exception {
+ Channel channel = createChannel(connection);
+ createExchange(channel, amqpExchangeName);
+ publishEventToExchange(channel, amqpExchangeName, routingKey,
eventDescription);
+ channel.close();
+ }
+
+ /** adds the event to the buffer, dropping the oldest one if the buffer is
full
+ *
+ * @param sendFailed false if the event was not tried yet and has to be
buffered only to keep the order
+ * @return true if the event was buffered, false if the caller has to deal
with it
+ */
+ private boolean bufferEvent(String routingKey, String eventDescription,
boolean sendFailed) {
+ if (!s_connectedOnce || bufferSize <= 0) {
+ return false;
+ }
+ synchronized (s_buffer) {
+ if (!sendFailed && s_buffer.isEmpty() && s_connection != null) {
+ return false;
+ }
+ if (s_buffer.isEmpty()) {
+ logger.warn("Unable to send events to AMQP server, buffering
them until they can be sent");
+ }
+ if (s_buffer.size() >= bufferSize) {
+ logger.warn("Event buffer is full, dropping the oldest event "
+ s_buffer.poll().first());
+ }
+ s_buffer.add(new Ternary<>(routingKey, eventDescription,
System.currentTimeMillis()));
+ }
+ return true;
+ }
+
+ // drops the expired events and, if connected, sends the buffered ones in
the order they were added
Review Comment:
top of method comment:
```suggestion
/**
* drops the expired events and, if connected, sends the buffered ones
in the order they were added
*/
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException
shutdownSignalException) {
}
abortConnection(); // disconnected to AMQP server, so abort
the connection and channels
+ // the ReconnectionTask started with the event bus takes care
of the re-connect
Review Comment:
```suggestion
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -273,20 +294,106 @@ public void publish(Event event) throws
EventBusException {
String routingKey = createRoutingKey(event);
String eventDescription = event.getDescription();
+ // keep the order: nothing is sent directly while older events are
waiting or the connection is down
+ if (bufferEvent(routingKey, eventDescription, false)) {
+ return;
+ }
+
try {
- Connection connection = getConnection();
- Channel channel = createChannel(connection);
- createExchange(channel, amqpExchangeName);
- publishEventToExchange(channel, amqpExchangeName, routingKey,
eventDescription);
- channel.close();
+ sendEvent(getConnection(), routingKey, eventDescription);
} catch (AlreadyClosedException e) {
closeConnection();
+ if (bufferEvent(routingKey, eventDescription, true)) {
+ return;
+ }
throw new EventBusException("Failed to publish event to message
broker as connection to AMQP broker in lost");
+ } catch (IOException e) {
+ if (bufferEvent(routingKey, eventDescription, true)) {
+ return;
+ }
+ throw new EventBusException("Failed to publish event to message
broker due to " + e.getMessage());
} catch (Exception e) {
throw new EventBusException("Failed to publish event to message
broker due to " + e.getMessage());
}
}
+ private void sendEvent(Connection connection, String routingKey, String
eventDescription) throws Exception {
+ Channel channel = createChannel(connection);
+ createExchange(channel, amqpExchangeName);
+ publishEventToExchange(channel, amqpExchangeName, routingKey,
eventDescription);
+ channel.close();
+ }
+
+ /** adds the event to the buffer, dropping the oldest one if the buffer is
full
+ *
+ * @param sendFailed false if the event was not tried yet and has to be
buffered only to keep the order
+ * @return true if the event was buffered, false if the caller has to deal
with it
+ */
+ private boolean bufferEvent(String routingKey, String eventDescription,
boolean sendFailed) {
+ if (!s_connectedOnce || bufferSize <= 0) {
+ return false;
+ }
+ synchronized (s_buffer) {
+ if (!sendFailed && s_buffer.isEmpty() && s_connection != null) {
+ return false;
+ }
+ if (s_buffer.isEmpty()) {
+ logger.warn("Unable to send events to AMQP server, buffering
them until they can be sent");
+ }
+ if (s_buffer.size() >= bufferSize) {
+ logger.warn("Event buffer is full, dropping the oldest event "
+ s_buffer.poll().first());
+ }
+ s_buffer.add(new Ternary<>(routingKey, eventDescription,
System.currentTimeMillis()));
+ }
+ return true;
+ }
+
+ // drops the expired events and, if connected, sends the buffered ones in
the order they were added
+ private void flushBuffer() {
+ int sent = 0;
+ while (true) {
+ Ternary<String, String, Long> bufferedEvent;
+ synchronized (s_buffer) {
+ bufferedEvent = s_buffer.peek();
+ }
+ if (bufferedEvent == null) {
+ break;
+ }
+
+ boolean expired = System.currentTimeMillis() -
bufferedEvent.third() > bufferExpiry;
+ Connection connection = s_connection;
+ if (!expired && connection == null) {
+ break;
+ }
+
+ if (expired) {
+ logger.warn("Dropping expired event " + bufferedEvent.first()
+ " as it could not be sent to AMQP server in time");
+ } else {
+ try {
+ sendEvent(connection, bufferedEvent.first(),
bufferedEvent.second());
+ sent++;
+ } catch (AlreadyClosedException e) {
+ closeConnection();
+ break;
+ } catch (IOException e) {
+ break; // retry with the next run
+ } catch (Exception e) {
+ logger.warn("Dropping buffered event " +
bufferedEvent.first() + " due to " + e.getMessage());
+ }
+ }
+
+ // the event stays in the buffer while being sent, so that newer
events queue up behind it
+ synchronized (s_buffer) {
+ if (s_buffer.peek() == bufferedEvent) {
+ s_buffer.poll();
+ }
Review Comment:
can you extract this to a separate method? sounds trivial but it makes for
better reading
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException
shutdownSignalException) {
}
abortConnection(); // disconnected to AMQP server, so abort
the connection and channels
+ // the ReconnectionTask started with the event bus takes care
of the re-connect
logger.warn("Connection has been shutdown by AMQP server.
Attempting to reconnect.");
-
- // initiate re-connect process
- ReconnectionTask reconnect = new ReconnectionTask();
- executorService.submit(reconnect);
}
}
}
- // retry logic to connect back to AMQP server after loss of connection
+ // runs for the lifetime of the event bus: connects back to AMQP server
after any loss of connection
+ // and sends the events buffered in the meantime
Review Comment:
```suggestion
/*
* runs for the lifetime of the event bus: connects back to AMQP server
after any loss of connection
* and sends the events buffered in the meantime
*/
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -85,11 +86,23 @@ public static void setUseSsl(String useSsl) {
private static Integer retryInterval;
+ // max number of events kept in memory while the AMQP server is
unreachable, 0 disables buffering
+ private static Integer bufferSize = 1000;
+
+ // time in milliseconds after which a buffered event is dropped if it
could not be sent
+ private static Integer bufferExpiry = 300000;
+
+ // events (routing key, description, time of buffering) waiting to be
sent, oldest first
+ private static final ArrayDeque<Ternary<String, String, Long>> s_buffer =
new ArrayDeque<>();
+
+ // events are buffered only after the AMQP server was reachable at least
once
+ private static volatile boolean s_connectedOnce = false;
Review Comment:
this comment confuses more than explaining (for me) why is it private static
(only one connection possible is a lack of feature I think but that would be a
reason).
Please drop the comment or make it truely prozaic.
Also the naming convention of 's_' is obsolete and no reason to keep it up.
It seems this is nnever set back to false so I'd make it something like
```suggestion
/**
* indicates wether a connection has ever been made, which is acted on
by checking that events are buffered only after the AMQP server was reachable
at least once
*/
private static volatile boolean hasBeenConnectedOnce = false;
```
needs renaming elsewhere as well.
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -177,6 +190,14 @@ public static void setRetryInterval(Integer retryInterval)
{
RabbitMQEventBus.retryInterval = retryInterval;
}
+ public static void setBufferSize(Integer bufferSize) {
+ RabbitMQEventBus.bufferSize = bufferSize;
+ }
+
+ public static void setBufferExpiry(Integer bufferExpiry) {
+ RabbitMQEventBus.bufferExpiry = bufferExpiry;
+ }
Review Comment:
```suggestion
/**
* expriy time to set for events that cannot be send
*
*/
public static void setBufferExpiry(Integer bufferExpiry) {
eventBufferExpiryTime = bufferExpiry;
}
```
##########
plugins/event-bus/rabbitmq/src/main/java/org/apache/cloudstack/mom/rabbitmq/RabbitMQEventBus.java:
##########
@@ -540,35 +649,38 @@ public void shutdownCompleted(ShutdownSignalException
shutdownSignalException) {
}
abortConnection(); // disconnected to AMQP server, so abort
the connection and channels
+ // the ReconnectionTask started with the event bus takes care
of the re-connect
logger.warn("Connection has been shutdown by AMQP server.
Attempting to reconnect.");
-
- // initiate re-connect process
- ReconnectionTask reconnect = new ReconnectionTask();
- executorService.submit(reconnect);
}
}
}
- // retry logic to connect back to AMQP server after loss of connection
+ // runs for the lifetime of the event bus: connects back to AMQP server
after any loss of connection
+ // and sends the events buffered in the meantime
private class ReconnectionTask extends ManagedContextRunnable {
- boolean connected = false;
Connection connection = null;
@Override
protected void runInContext() {
- while (!connected) {
+ while (true) {
Review Comment:
not all of this is on you, @durdin85 , but this is a forever loop with
nested try blocks and an anonymous class inside. Could you at least put the new
part (the sync block in a separate method? If you have the energy, feel free to
simplify it further.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]