Hi, I still don't think that it's the job of the Transient channel to do this throttling for you. Have you experimented with your proposed change? Does it really help? What stops you from running out of memory by just growing the number of messages that are in flight indefinitely?
B/ On 15 May 2014 at 10:52:26, 王东 Dong Wang ([email protected]) wrote: Sorry for that. It's just redeliveries, a lot of redeliveries, we didn't really encounter redelivery failures. Thank you for the suggestions regarding throttling. We surely haven't done that yet. I just want to share with you and the akka community that when our load is high, we can achieve a QPS of 10000 (of course this is very application specific) but when the load is 5x higher, the QPS dropped significantly to 200. And to me it was caused by a lot of redeliveries. Maybe the redelivery mechanism can be enhanced a little bit by implementing my little idea :) On May 15, 2014, at 4:10 PM, Björn Antonsson <[email protected]> wrote: Hi Daniel, You still haven't explained if there are redelivery failures or just redeliveries, and if the destination is local or remote. If you see redeliveries or failures while the the destination is alive and healthy, then you are flooding your destination. That is a flow control issue that the channel can't solve for you in a generic way. You need to add some throttling in the form of high/low watermarks, work pulling and or fan-out. The PersistenChannel will limit the number of messages in flight for you. It might work for bursts of traffic, but if you continuously send more messages than the destination can handle you will eventually overflow. B/ On 15 May 2014 at 07:05:20, 王东 Dong Wang ([email protected]) wrote: On May 14, 2014, at 2:32 PM, Björn Antonsson <[email protected]> wrote: Hi Daniel, I'm not sure where your logic would be used. Are you using transient Channels and a Processor or are you using PersistentChannels? If you are using a PersistentChannel then you could limit the maximum number of outstanding messages via the pendingConfirmationsMax setting. We are using transient channels with a processor. A redeliverInterval setting of 10 seconds seems fine, but what is your redeliverMax setting. Are you flooding the channel and your code would be a back off in the sender? You only need to resend from the sender if you get a redelivery failure message from the channel. We use the default value of redeliverMax setting. We saw what we saw during a load testing, which means we don't back off senders at all. Is the channel destination remote or local? Do you see any remoting issues? B/ On 13 May 2014 at 17:23:25, Dong Wang ([email protected]) wrote: I se the interval to 10 seconds. Does my suggestion make any sense to you? 在 2014年5月13日，下午7:11，Björn Antonsson <[email protected]> 写道： Hi Daniel, Are you talking about getting RedeliveryFailure messages when your load is high? The Channel should handle redelivery for you. If you have a lot of redeliveries happening, then maybe your redeliver interval is set too tight, or you are simply overloading your system. B/ On 12 May 2014 at 13:56:49, dong ([email protected]) wrote: I'm seeing in my application a lot of channels message redelivery when the overall load of the system is high. Is it a good idea to re-deliver channel message this way: record the latest confirmed time [ of any message] re-deliver the message only iff (current_timestamp - latest_confirmed_timestamp) > timeout1 && current_timestamp - the_message's_delivery_timestamp > timeout_2) I may missed many details, but you get the idea. -Daniel -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/g47kdVSOqMw/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/g47kdVSOqMw/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to a topic in the Google Groups "Akka User List" group. To unsubscribe from this topic, visit https://groups.google.com/d/topic/akka-user/g47kdVSOqMw/unsubscribe. To unsubscribe from this group and all its topics, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout. -- Björn Antonsson Typesafe – Reactive Apps on the JVM twitter: @bantonsson JOIN US. REGISTER TODAY! Scala Days June 16th-18th, Berlin -- >>>>>>>>>> Read the docs: http://akka.io/docs/ >>>>>>>>>> Check the FAQ: >>>>>>>>>> http://doc.akka.io/docs/akka/current/additional/faq.html >>>>>>>>>> Search the archives: https://groups.google.com/group/akka-user --- You received this message because you are subscribed to the Google Groups "Akka User List" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. Visit this group at http://groups.google.com/group/akka-user. For more options, visit https://groups.google.com/d/optout.
