The root cause of this issue is as follows. As soon as a new worker node joins, and its HTTP/S ports become available, the LB tries to send messages to that node. However, that node need a few more seconds to "warm up", hence a few initial requests fail. The failover mechanism kicks in and tries to resends the message to another instance. However, an error occurs when it tries to retrieve the original request envelope, and ends up sending an empty message to the worker node, which shows up as a malformed XML error when the StAX parser tries to parse the message.
One measure I have implemented is allowing some warm up time for newly added worker nodes which solved this problem for the moment. However, the failover logic is still broken and needs to be fixed. On Sun, Jun 12, 2011 at 9:21 PM, Sanjiva Weerawarana <[email protected]>wrote: > Not very likely dude :) .. its been in HEAVY use in eBay for a long time > now. There has to be some other combination of events that triggers this. > > Sanjiva. > > On Sun, Jun 12, 2011 at 4:25 PM, Sameera Jayasoma <[email protected]>wrote: > >> Binary relay impl might be failing to handle messages in high concurrency. >> >> Thanks, >> Sameera >> >> >> On Fri, Jun 10, 2011 at 3:34 PM, Afkham Azeez <[email protected]> wrote: >> >>> Folks, >>> Please note $subject. This happens rarely. We are using binary relay in >>> the LB. >>> >>> -- >>> *Afkham Azeez* >>> Director of Architecture; WSO2, Inc.; http://wso2.com >>> Member; Apache Software Foundation; http://www.apache.org/ >>> * <http://www.apache.org/>** >>> email: **[email protected]* <[email protected]>* cell: +94 77 3320919 >>> blog: **http://blog.afkham.org* <http://blog.afkham.org>* >>> twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> >>> * >>> linked-in: **http://lk.linkedin.com/in/afkhamazeez* >>> * >>> * >>> *Lean . Enterprise . Middleware* >>> >>> >> >> >> -- >> Sameera Jayasoma >> Technical Lead and Product Manager, WSO2 Carbon >> >> WSO2, Inc. (http://wso2.com) >> email: [email protected] >> blog: http://tech.jayasoma.org >> >> Lean . Enterprise . Middleware >> >> _______________________________________________ >> Carbon-dev mailing list >> [email protected] >> http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev >> >> > > > -- > Sanjiva Weerawarana, Ph.D. > Founder, Chairman & CEO; WSO2, Inc.; http://wso2.com/ > email: [email protected]; phone: +94 11 763 9614; cell: +94 77 787 6880 | +1 > 650 265 8311 > blog: http://sanjiva.weerawarana.org/ > > Lean . Enterprise . Middleware > -- *Afkham Azeez* Director of Architecture; WSO2, Inc.; http://wso2.com Member; Apache Software Foundation; http://www.apache.org/ * <http://www.apache.org/>** email: **[email protected]* <[email protected]>* cell: +94 77 3320919 blog: **http://blog.afkham.org* <http://blog.afkham.org>* twitter: **http://twitter.com/afkham_azeez*<http://twitter.com/afkham_azeez> * linked-in: **http://lk.linkedin.com/in/afkhamazeez* * * *Lean . Enterprise . Middleware*
_______________________________________________ Carbon-dev mailing list [email protected] http://mail.wso2.org/cgi-bin/mailman/listinfo/carbon-dev
