aaron-ai commented on code in PR #252: URL: https://github.com/apache/rocketmq-site/pull/252#discussion_r976565505
########## i18n/en/docusaurus-plugin-content-docs/version-5.0/04-功能行为/02delaymessage.md: ########## @@ -0,0 +1,154 @@ +# Delay Message + +Delay messages are messages with advanced features in Apache RocketMQ. This topic describes the scenarios, working mechanism, limits, usage examples, and usage notes of delay messages and delayed messages. + +:::note + +Scheduled message and delay message are essentially the same. Both of them deliver messages to consumers at a fixed time according to the timing time set by the message. Therefore, delay messages are used in the following sections. + +::: + +## Scenarios + +Accurate and reliable time-based event triggers are required in scenarios such as distributed timed scheduling and task timeout processing. Apache RocketMQ provides delay messages to help you simplify the development of timed scheduling tasks and implement high-performance, scalable, and reliable timed triggering. + +**Scenario 1: Distributed timed scheduling** + + +A distributed timed scheduling scenario involves tasks that require various time granularity levels, for example, a task to execute file cleanup at 5 o'clock every day or a task to trigger push messages every 2 minutes. Traditional dataset-based timed scheduling solutions are complex and inefficient in distributed scenarios. In comparison, delay messages in Apache RocketMQ allow you to encapsulate multiple types of time triggers. + +**Scenario 2: Task timeout processing** + + +A typical scenario that involves task timeout processing is e-commerce payment, where an unpaid order is canceled after it remains unpaid for a specific time period instead of being canceled immediately. In this case, you can use delay messages in Apache RocketMQ to check and trigger timeout tasks. + +Task timeout processing based on delay messages provides the following benefits: + +* Various time granularity levels and simplified development: Scheduled messaging in Apache RocketMQ does not have the limit of fixed time increments. You can trigger tasks at any time granularity level and without deduplication. + +* High performance and scalability: delay messages in Apache RocketMQ offer high concurrency and scalability. This outperforms traditional database scanning methods, which are complex to implement and can cause performance bottlenecks due to frequent API calls for scanning. + +## Working mechanism + +**Definition of delay messages** + +delay messages are messages with advanced features in Apache RocketMQ. delay messages allow consumers to consume messages that are sent to the server only after a specified period of time or at a specified time. You can use delay messages to implement delayed scheduling and triggering in distributed scenarios. + +**Time setting rules** + +* The scheduled or delayed time for delay messages in Apache RocketMQ is represented as a timestamp, not a time period. + +* The scheduled time is in the format of a millisecond-level Unix timestamp. You must convert the scheduled time of message delivery to a millisecond-level Unix timestamp. You can use the [Unix timestamp converter](https://www.unixtimestamp.com/) to convert a time to a millisecond-level Unix timestamp. + +* The scheduled time must be within the allowed time range. If the scheduled time exceeds the range, the scheduled time does not take effect and the messages are immediately delivered from the server side. + +* By default, the maximum time range for delay messages is 24 hours. You cannot change the default value. For more information, see[Parameter limits](../01-基础介绍/03limits.md). + +* The scheduled time must be later than the current time. If the scheduled time is set to a time earlier than the current time, the scheduled time does not take effect and the messages are immediately delivered from the server side. + + +**The following section provides two time setting examples:** + +* delay messages: If the current time is 2022-06-09 17:30:00 and you want to deliver messages at 2022-06-09 19:20:00, the millisecond-level Unix timestamp of the scheduled time is 1654773600000. + +* Delayed messages: If the current time is 2022-06-09 17:30:00 and you want to deliver messages after 1 hour, the message delivery time is 2022-06-09 18:30:00 and the millisecond-level Unix timestamp is 1654770600000. + +**Lifecycle of a scheduled message** + + + +* Initialized: The message is built and initialized by the producer and is ready to be sent to the server. + +* Timing: The message is sent to the server side, where the message is stored in a time-based storage system until the specified delivery time. An index is not immediately created for the message. + +* Ready: At the specified time, the message is written into a regular storage engine, where the message is visible for consumers and waits for consumption by consumers. + + + +* Inflight: The message is obtained by the consumer and processed based on the local business logic of the consumer. + + In this process, the broker waits for the consumer to complete the consumption and submit the consumption result. If no response is received from the consumer in a certain period of time, Apache RocketMQ retries the message. For more information, see [Consumption retry](../04-功能行为/10consumerretrypolicy.md). + + +* Acked: The consumer completes consumption and submits the consumption result to the broker. The broker marks whether the current message is successfully consumed. + + By default, Apache RocketMQ retains all messages. When the consumption result is submitted, the message data is logically marked as consumed instead of being deleted immediately. Therefore, the consumer can backtrack the message for re-consumption before it is deleted due to the expiration of the retention period or insufficient storage space. + + +* Deleted: When the retention period of the message expires or the storage space is insufficient, Apache RocketMQ deletes the earliest saved message from the physical file in a rolling manner. For more information, see [Message storage and cleanup](../04-功能行为/11messagestorepolicy.md). + +## Usage limits + +**Message type consistency** + +delay messages can be sent only to topics whose MessageType is Delay. + +**Time granularity** + +The time granularity for delay messages in Apache RocketMQ is down to milliseconds. The default granularity value is 1000 ms. + +The status of delay messages in Apache RocketMQ can be persistently stored. If the messaging system experiences a failure and is restarted, messages are still delivered based on the specified delivery time. However, if the storage system experiences an exception or is restarted, latency may occur in delivering delay messages. + + +## Example + +Unlike normal messages, delay messages must have a delivery timestamp specified for them. + +The following code provides Java examples of delivery and consumption of delay messages: + +```java + // Send delay messages. + MessageBuilder messageBuilder = null; + // Specify a millisecond-level Unix timestamp. In this example, the specified timestamp indicates that the message will be delivered in 10 minutes from the current time. + Long deliverTimeStamp = System.currentTimeMillis() + 10L * 60 * 1000; + Message message = messageBuilder.setTopic("topic") + // Specify the message index key. The system uses the key to locate the message. + .setKeys("messageKey") + // Specify the message tag. The consumer can use the tag to filter messages. + .setTag("messageTag") + .setDeliveryTimestamp(deliverTimeStamp) + // Configure the message body. + .setBody("messageBody".getBytes()) + .build(); + try { + // Send the messages. Focus on the result of message sending and exceptions such as failures. + SendReceipt sendReceipt = producer.send(message); + System.out.println(sendReceipt.getMessageId()); + } catch (ClientException e) { + e.printStackTrace(); + } + // Consumption example 1: If a scheduled message is consumed by a push consumer, the consumer needs to process the message only in the message listener. + MessageListener messageListener = new MessageListener() { +@Override +public ConsumeResult consume(MessageView messageView) { + System.out.println(messageView.getDeliveryTimestamp()); + // Return the status based on the consumption result. + return ConsumeResult.SUCCESS; + } + }; + // Consumption example 2: If a scheduled message is consumed by a simple consumer, the consumer must obtain the message for consumption and submit the consumption result. + List<MessageView> messageViewList = null; + try { + messageViewList = simpleConsumer.receive(10, Duration.ofSeconds(30)); + messageViewList.forEach(messageView -> { + System.out.println(messageView); + // After consumption is complete, the consumer must invoke ACK to submit the consumption result. + try { + simpleConsumer.ack(messageView); + } catch (ClientException e) { + e.printStackTrace(); + } + }); + } catch (ClientException e) { + // If the pull fails due to system traffic throttling or other reasons, you must re-initiate the request to obtain the message. + e.printStackTrace(); + } Review Comment: Fix code indent here. ########## i18n/en/docusaurus-plugin-content-docs/version-5.0/04-功能行为/10consumerretrypolicy.md: ########## @@ -0,0 +1,212 @@ +# Consumption retry + +If a message fails to be consumed, Apache RocketMQ redelivers the message based on a consumption retry policy. This helps remove some faults. This topic describes the working mechanism, version compatibility, and usage notes of the consumption retry feature. + +## Scenarios + +The consumption retry feature of Apache RocketMQ ensures consumption integrity that may be affected by the failure of a business processing logic. This feature is a protective measure against business logic failures. It cannot be used to control the business process. + +The feature is suitable for use in the following scenarios: + +* The business fails because of the message content. For example, the transaction status is not returned and the business is expected to be restored within a specific period of time. + +* The cause of consumption failure does not affect business continuity. The failure has a small possibility of occurring and subsequent messages are very likely to be delivered and consumed as expected. In these cases, you can use the retry mechanism to redeliver the message to avoid blocking the process. + + +Do not use the feature in the following scenarios: + +* Consumption failure is used as a condition to divert message flows in the processing logic. The processing logic assumes that many messages will fail to be consumed. + +* Consumption failure is used to limit the rate of message processing. Rate limiting should be used to temporarily stack excessive messages in the queue for later processing instead of making the messages enter the retry link. + + + +## Purpose + + +A common issue of message middleware in asynchronous decoupling is how to ensure the integrity of the entire call link if the downstream service fails to process messages. As a financial-grade reliable message middleware service, Apache RocketMQ uses a well-designed message acknowledgement and retry mechanism to ensure that every message is processed according to business expectations. + +Understanding the message acknowledgement and retry mechanism of Apache RocketMQ helps solve the following issues: + +* How to ensure that every message is processed: You can ensure that every message is processed based on their consumer logic and business statuses are consistent. + +* How to ensure that the status of messages that are being processed are correct when an exception occurs: You can ensure the correct message status when an exception, such as power failure, occurs. + + + +## Policy overview + +When the consumption retry feature is enabled, the Apache RocketMQ broker resends a message when the message fails to be consumed. If the message fails to be consumed even after a specified number of retries, the broker sends the message to the dead-letter queue. + +**Trigger conditions** + +* A message fails to be consumed. In this case, the consumer returns a failure status or the system throws an exception. + +* A timeout error occurs or a message stays in a push consumer queue for an excessive period of time. + + +**Behaviors** + +* Retry process state machine: controls the state and the change logic of messages in the retry process. + +* Retry interval: the time that elapses from when a consumption failure or timeout occurs to when the message is retried. + +* Maximum retries: the maximum number of times that a message can be retried for consumption. + +**Policy differences** + +Message retry policies use different retry mechanisms and configuration methods based on the consumer type. The following table describes the differences between the policies. + + +| Consumer type | Retry process state machine | Retry interval | Maximum number of retries | +|----------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------| +| PushConsumer | * Ready * Inflight * WaitingRetry * Commit * DLQ | Specified in the metadata when a consumer group is created. * Unordered messages: incremental * Ordered messages: fixed | Specified in the metadata when a consumer group is created. | +| SimpleConsumer | * Ready * Inflight * Commit * DLQ | Specified in the InvisibleDuration parameter in the API. | Specified in the metadata when a consumer group is created. | + +For more information about retry policies, see [Retry policy for push consumers](#section-qqo-bil-rc6) and [Retry policy for simple consumers](#section-my2-2au-7gl). + +## Retry policy for PushConsumer + + +**Retry process state machine** + +When a push consumer consumes a message, the message can be in one of the following states: + +* Ready The message is waiting to be consumed on the Apache RocketMQ broker. + + +* Inflight The message has been obtained and is being consumed by the consumer. However, the consumption result has not been returned. + + +* WaitingRetry This state is exclusive to push consumers. The message fails to be consumed or a timeout error occurs when the broker waits for the consumer to return the consumption status. In these cases, the consumption retry logic is triggered. If the maximum number of retries is not reached, the message goes back to the Ready state after the retry interval elapses. Messages that are in the Ready state can be consumed again. You can increase the interval between retries to prevent frequent retries. + + +* Commit The message is consumed. After the consumer returns a success response, the state machine can be terminated. + + +* DLQ A preventive measure for the consumption logic. If the message fails to be consumed even after the maximum number of retries is reached, the message is no longer retried and is sent to the dead-letter queue. You can consume messages in a dead-letter queue to restore your business. + +When a message is retried, its state changes from Ready to Inflight and then to WaitingRetry. The interval between two consumptions is the sum of the actual time spent on consumption and the retry interval. The maximum consumption interval is specified by a system parameter on the broker and cannot be exceeded.  + +**Maximum number of retries** + +The maximum number of retries for a push consumer is specified in the metadata when the consumer group is created. For more information, see [Consumer groups](../03-领域模型/07consumergroup.md). + +For example, if the maximum number of retries is three, the message can be delivered four times: one original attempt and three retries. + +**Retry interval** + +* Unordered messages (messages that are not ordered messages): incremental. The following table describes the details. + + | Retry number | Interval | Retry number | Interval | + |--------------|------------|--------------|------------| + | 1 | 10 seconds | 9 | 7 minutes | + | 2 | 30 seconds | 10 | 8 minutes | + | 3 | 1 minute | 11 | 9 minutes | + | 4 | 2 minutes | 12 | 10 minutes | + | 5 | 3 minutes | 13 | 20 minutes | + | 6 | 4 minutes | 14 | 30 minutes | + | 7 | 5 minutes | 15 | 1 hour | + | 8 | 6 minutes | 16 | 2 hours | + + +:::info +If the number of retries exceeds 16, the interval of each subsequent retry is 2 hours. +::: + +* Ordered messages: fixed. For more information, see[Parameter limits](../01-基础介绍/03limits.md). + + +**Example** + +For push consumers, a message retry is triggered only by the status code of consumption failure. Unexpected exceptions are also captured by the SDK. + +```java +SimpleConsumer simpleConsumer = null; + // Consumption example: Consume normal messages as a push consumer and trigger a message retry by using a consumption failure. + MessageListener messageListener = new MessageListener() { + @Override + public ConsumeResult consume(MessageView messageView) { + System.out.println(messageView); + // Retry the message until the maximum number of retries is reached. + return ConsumeResult.FAILURE; + } + }; + +``` + +## Retry policy for SimpleConsumer + + +**Retry process state machine** + +When a simple consumer consumes a message, the message can be in one of the following states: + +* Ready The message is waiting to be consumed on the Apache RocketMQ broker. + +* Inflight The message has been obtained and is being consumed by the consumer. However, the consumption result has not been returned. + +* Commit The message is consumed. After the consumer returns a success response, the state machine can be terminated. + +* DLQ A preventive measure for the consumption logic. If the message fails to be consumed even after the maximum number of retries is reached, the message is no longer retried and is sent to the dead-letter queue. You can consume messages in a dead-letter queue to restore your business. + + +The retry interval is fixed and pre-allocated. It is configured in the InvisibleDuration parameter by the consumer when the consumer calls the API. The parameter specifies the maximum processing duration of the message. When a message is retried, the value of the parameter is reused. You do not need to configure the interval for the subsequent retries. + + +Because the InvisibleDuration value is pre-allocated, it may not meet your business requirements. You can change it in the code that is used to call the API. + +For example, if you set the InvisibleDuration value to 20 ms and a message cannot be processed within the duration, you can change the value to a larger value to avoid triggering the retry mechanism. + +Before you can change the InvisibleDuration value, the following conditions must be met: + +* A timeout error has not occurred on the current message. + +* A consumption status of the current message is not returned. + +As shown in the following figure, the change takes effect immediately, that is, the InvisibleDuration value is recalculated from the point in time when the API is called. + + +**Maximum number of retries** + +The maximum number of retries for a simple consumer is specified in the metadata when the consumer group is created. For more information, see [Consumer groups](../03-领域模型/07consumergroup.md). + +**Message retry interval** + +Message retry interval = InvisibleDuration value − Actual duration of message processing + +The consumption retry interval is therefore controlled by the InvisibleDuration value. For example, if the InvisibleDuration value is 30 ms and a consumption failure is returned 10 ms after the processing starts, the time to the next retry is 20 ms, which means that the retry interval is 20 ms. If no consumption result is returned within 30 ms, a timeout error occurs and a retry is triggered. Then, the retry interval is 0 ms. + +**Examples** + +Simple consumers need only to wait for a message to be retried. + +```java + // Consumption example: Consume normal messages as a simple consumer. If you want a message to be retried, do not process the message. Wait for it to time out, and the broker retries it automatically. + List<MessageView> messageViewList = null; + try { Review Comment: ditto -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
