Re: Websockets now considered stable
Hi Joe, thanks for the pointer, had totally forgotten about that demo app. I've removed the call from the demo, the wsServer logic has been removed completely since it's now Redis that is responsible for routing messages through the pub / sub handling. I've also updated the readme with the fact that Redis is now a required dependency. On Thu, Jan 14, 2016 at 6:17 PM, Joe Bogner <joebog...@gmail.com> wrote: > Hi Henrik, > > Thanks for sharing. I get the following when running the ws-demo: > > ./pil pl-web/ws-demo/main.l -go > ... > !? (wsServer) > wsServer -- Undefined > > I can't find the definition of wsServer anywhere. Is it missing from the > repo? > > Thanks, > Joe > > On Mon, Jan 4, 2016 at 4:27 PM, Henrik Sarvell <hsarv...@gmail.com> wrote: >> >> Update: >> >> The socketserver is now completely reliant on Redis, using Redis' pub >> / sub functionality: http://redis.io/topics/pubsub >> >> The reason for this is that I was using the websocket server to handle >> all websockets functionality for the site I'm being paid to work on >> and it started running into problems as the site grew, the first issue >> was an easy fix after Alex pointed me to it, increasing the amount of >> file descriptors in src64/sys/x86-64.linux.defs.l, my line #115 now >> looks like this: (equ FD_SET 1024) # 1024 bit >> >> After re-compiling I could easily handle more than 500 clients and all >> was well for a while. >> >> Unfortunately the site is growing so fast that just some month(s) >> later the parent / root process started intermittently running at 100% >> CPU utilization and the service stopped working for perhaps 10-20 >> minutes before resolving on its own. At this point peak usage involved >> 2000 clients being connected at the same time. >> >> Alex suspects that the issue has got to do with how the internal logic >> handles new processes being created when there are already a lot of >> them present. In a normal HTTP server scenario this probably never >> happens, imagine that every request takes on average 1 second to >> perform before the socket closes, you would then need about 2000 >> requests per second in order to trigger the CPU problem, you'll run >> into many other issues long before that happens in a non-trivial >> scenario (trust me I've tested). >> >> In the end we switched over to a node.js based solution that also >> relies on Redis' pub / sub functionality (that's where I got the idea >> from to make the PL based solution also use it). >> >> I have tried to replicate the real world situation load wise and >> number of clients wise but not been able to trigger the CPU issue >> (this also seems to imply that Alex's suspicion is not completely on >> target), it's impossible for me to replicate the real world situation >> since I can't commandeer hundreds of machines all over the world to >> connect to my test server. What I did manage to trigger though was >> fairly high CPU usage in the child processes though, a situation that >> also involved loss of service. After the switch to using pub / sub I >> haven't been able to trigger it, so that's a win at least. >> >> Now for the real improvement, actually making HTTP requests to publish >> something becomes redundant when publishing from server to client >> since it's just a matter of issuing a publish call directly to Redis >> instead. That lowers the amount of process creation by more than 90% >> in my use case. >> >> Even though I can't be 100% sure as it currently stands I believe that >> if I had implemented the websocket server using Redis' pub / sub to >> begin with the CPU issue would probably never have happened and there >> would've been no need to switch over to node.js. >> >> That being said, this type of service / application is better suited >> for threads since the cost in RAM etc is lower. >> >> Final note, my decision to use one socket per feature was poor, it >> allowed me a simpler architecture but had I opted for one socket with >> "routing" logic implemented in the browser instead I could have >> lowered the amount of simultaneous sockets up to 8 times. Peak usage >> would then have been 2000 / 8 = 250 processes. Not only that, it turns >> out that IE (yes, even version 11 / edge) only allows 6 simultaneous >> sockets (including in iframes) per page. We've therefore been forced >> to turn off for instance the tournament functionality for IE users. >> >> >> >> On Fri, Jun 26, 2015 at 9:30 PM, Henrik Sarvell <hsarv...@gmail.com> &g
Re: Websockets now considered stable
Hi Henrik, Thanks for sharing. I get the following when running the ws-demo: ./pil pl-web/ws-demo/main.l -go .. !? (wsServer) wsServer -- Undefined I can't find the definition of wsServer anywhere. Is it missing from the repo? Thanks, Joe On Mon, Jan 4, 2016 at 4:27 PM, Henrik Sarvell <hsarv...@gmail.com> wrote: > Update: > > The socketserver is now completely reliant on Redis, using Redis' pub > / sub functionality: http://redis.io/topics/pubsub > > The reason for this is that I was using the websocket server to handle > all websockets functionality for the site I'm being paid to work on > and it started running into problems as the site grew, the first issue > was an easy fix after Alex pointed me to it, increasing the amount of > file descriptors in src64/sys/x86-64.linux.defs.l, my line #115 now > looks like this: (equ FD_SET 1024) # 1024 bit > > After re-compiling I could easily handle more than 500 clients and all > was well for a while. > > Unfortunately the site is growing so fast that just some month(s) > later the parent / root process started intermittently running at 100% > CPU utilization and the service stopped working for perhaps 10-20 > minutes before resolving on its own. At this point peak usage involved > 2000 clients being connected at the same time. > > Alex suspects that the issue has got to do with how the internal logic > handles new processes being created when there are already a lot of > them present. In a normal HTTP server scenario this probably never > happens, imagine that every request takes on average 1 second to > perform before the socket closes, you would then need about 2000 > requests per second in order to trigger the CPU problem, you'll run > into many other issues long before that happens in a non-trivial > scenario (trust me I've tested). > > In the end we switched over to a node.js based solution that also > relies on Redis' pub / sub functionality (that's where I got the idea > from to make the PL based solution also use it). > > I have tried to replicate the real world situation load wise and > number of clients wise but not been able to trigger the CPU issue > (this also seems to imply that Alex's suspicion is not completely on > target), it's impossible for me to replicate the real world situation > since I can't commandeer hundreds of machines all over the world to > connect to my test server. What I did manage to trigger though was > fairly high CPU usage in the child processes though, a situation that > also involved loss of service. After the switch to using pub / sub I > haven't been able to trigger it, so that's a win at least. > > Now for the real improvement, actually making HTTP requests to publish > something becomes redundant when publishing from server to client > since it's just a matter of issuing a publish call directly to Redis > instead. That lowers the amount of process creation by more than 90% > in my use case. > > Even though I can't be 100% sure as it currently stands I believe that > if I had implemented the websocket server using Redis' pub / sub to > begin with the CPU issue would probably never have happened and there > would've been no need to switch over to node.js. > > That being said, this type of service / application is better suited > for threads since the cost in RAM etc is lower. > > Final note, my decision to use one socket per feature was poor, it > allowed me a simpler architecture but had I opted for one socket with > "routing" logic implemented in the browser instead I could have > lowered the amount of simultaneous sockets up to 8 times. Peak usage > would then have been 2000 / 8 = 250 processes. Not only that, it turns > out that IE (yes, even version 11 / edge) only allows 6 simultaneous > sockets (including in iframes) per page. We've therefore been forced > to turn off for instance the tournament functionality for IE users. > > > > On Fri, Jun 26, 2015 at 9:30 PM, Henrik Sarvell <hsarv...@gmail.com> > wrote: > > Hi all, after over a month without any of the prior issues I now > > consider the websockets part of pl-web stable: > > https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU > > usage and zombie processes. > > > > With Alex's help the main web server is now more stable (he made me > > throw away a few throws in favour of a few byes). The throws were > > causing the zombies. > > > > I was also including dbg.l (it was causing hung processes at 100% > > CPU), it's basically been deprecated or something, I'll leave it up to > > him to elaborate. It's just something I've been including by habit > > since years ago when at some point I needed to include it to do some > > kind o
Re: Websockets now considered stable
Update: The socketserver is now completely reliant on Redis, using Redis' pub / sub functionality: http://redis.io/topics/pubsub The reason for this is that I was using the websocket server to handle all websockets functionality for the site I'm being paid to work on and it started running into problems as the site grew, the first issue was an easy fix after Alex pointed me to it, increasing the amount of file descriptors in src64/sys/x86-64.linux.defs.l, my line #115 now looks like this: (equ FD_SET 1024) # 1024 bit After re-compiling I could easily handle more than 500 clients and all was well for a while. Unfortunately the site is growing so fast that just some month(s) later the parent / root process started intermittently running at 100% CPU utilization and the service stopped working for perhaps 10-20 minutes before resolving on its own. At this point peak usage involved 2000 clients being connected at the same time. Alex suspects that the issue has got to do with how the internal logic handles new processes being created when there are already a lot of them present. In a normal HTTP server scenario this probably never happens, imagine that every request takes on average 1 second to perform before the socket closes, you would then need about 2000 requests per second in order to trigger the CPU problem, you'll run into many other issues long before that happens in a non-trivial scenario (trust me I've tested). In the end we switched over to a node.js based solution that also relies on Redis' pub / sub functionality (that's where I got the idea from to make the PL based solution also use it). I have tried to replicate the real world situation load wise and number of clients wise but not been able to trigger the CPU issue (this also seems to imply that Alex's suspicion is not completely on target), it's impossible for me to replicate the real world situation since I can't commandeer hundreds of machines all over the world to connect to my test server. What I did manage to trigger though was fairly high CPU usage in the child processes though, a situation that also involved loss of service. After the switch to using pub / sub I haven't been able to trigger it, so that's a win at least. Now for the real improvement, actually making HTTP requests to publish something becomes redundant when publishing from server to client since it's just a matter of issuing a publish call directly to Redis instead. That lowers the amount of process creation by more than 90% in my use case. Even though I can't be 100% sure as it currently stands I believe that if I had implemented the websocket server using Redis' pub / sub to begin with the CPU issue would probably never have happened and there would've been no need to switch over to node.js. That being said, this type of service / application is better suited for threads since the cost in RAM etc is lower. Final note, my decision to use one socket per feature was poor, it allowed me a simpler architecture but had I opted for one socket with "routing" logic implemented in the browser instead I could have lowered the amount of simultaneous sockets up to 8 times. Peak usage would then have been 2000 / 8 = 250 processes. Not only that, it turns out that IE (yes, even version 11 / edge) only allows 6 simultaneous sockets (including in iframes) per page. We've therefore been forced to turn off for instance the tournament functionality for IE users. On Fri, Jun 26, 2015 at 9:30 PM, Henrik Sarvell <hsarv...@gmail.com> wrote: > Hi all, after over a month without any of the prior issues I now > consider the websockets part of pl-web stable: > https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU > usage and zombie processes. > > With Alex's help the main web server is now more stable (he made me > throw away a few throws in favour of a few byes). The throws were > causing the zombies. > > I was also including dbg.l (it was causing hung processes at 100% > CPU), it's basically been deprecated or something, I'll leave it up to > him to elaborate. It's just something I've been including by habit > since years ago when at some point I needed to include it to do some > kind of debugging. > > Anyway atm the WS router is regularly routing up to 40 messages per > second to upwards 300-500 clients which means that roughly 20,000 > messages are being pushed out per second during peak hours. > > The PL processes show up with 0 CPU and 0 RAM usage when I run top, > sometimes 1% CPU :) They hardly register even i aggregate, the server > would be running 99% idle if it was only running the WS server. > > To work around the inter-process limit of 4096 byte long messages the > router now supports storing the messages in Redis (raw disk is also > supported if Redis is not available), this is also in effect in > production and is working flawlessly since months. >
Re: Websockets now considered stable
Hi Rick, seems like a fix would be a check there: if sessions dir doesn't exist (and Redis isn't used to store the session) create it and move on instead of breaking down in tears. Hi Henrik! Yes, I agree. BTW, thanks. I forgot to thank you before for sharing this! -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets now considered stable
Hi Rick, seems like a fix would be a check there: if sessions dir doesn't exist (and Redis isn't used to store the session) create it and move on instead of breaking down in tears. On Sun, Jun 28, 2015 at 10:47 PM, Rick Hanson cryptor...@gmail.com wrote: I downloaded pl-web and ext and ran the demo-app. When I went to the login page, I got this Open error in the console: # excmd redefined # exlst redefined # exlst redefined !? (out Sf (print (list (list sid *Sid /home/rick/projects/pl-web-demo/./sessions/6d61fa61b9cc1d8fd878f4b534703473 -- Open error: No such file or directory ? But after I quit, issued a: $ mkdir sessions and re-started the server, I never got the error again -- everything worked as expected. (I got the login creds from main.l.) -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets now considered stable
What gives?! This stuff is broken!!! $ git clone https://bitbucket.org/hsarvell/pl-web Cloning into 'pl-web'... fatal: repository 'https://bitbucket.org/hsarvell/pl-web/' not found Just yanking your chain. I know this is a mercurial repo. :) Thanks, man. Looks good. I'll study the code when I get time in the next few days. On Fri, Jun 26, 2015 at 3:30 PM, Henrik Sarvell hsarv...@gmail.com wrote: Hi all, after over a month without any of the prior issues I now consider the websockets part of pl-web stable: https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU usage and zombie processes. With Alex's help the main web server is now more stable (he made me throw away a few throws in favour of a few byes). The throws were causing the zombies. I was also including dbg.l (it was causing hung processes at 100% CPU), it's basically been deprecated or something, I'll leave it up to him to elaborate. It's just something I've been including by habit since years ago when at some point I needed to include it to do some kind of debugging. Anyway atm the WS router is regularly routing up to 40 messages per second to upwards 300-500 clients which means that roughly 20,000 messages are being pushed out per second during peak hours. The PL processes show up with 0 CPU and 0 RAM usage when I run top, sometimes 1% CPU :) They hardly register even i aggregate, the server would be running 99% idle if it was only running the WS server. To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis (raw disk is also supported if Redis is not available), this is also in effect in production and is working flawlessly since months. This is how I start the WS server in production: (load pl-web/pl-web.l) (setq *Mobj (new '(+Redis) pl-ws-)) (undef 'app) (setq *WsAuth '((notifications ((send (put your password/key here)) (de app () (splitPath) (wsApp) (bye)) (de go () (wsServer) (server 9090) ) -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets now considered stable
Hi Henrik, hi Andreas, Question: To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis 1) Where comes this limit from? POSIX IPC? PicoLisp IPC ? 1) As far as I remember from a discussion with Alex it's a hard limit (OS related). I think this was about the constant PIPE_BUF /usr/include/linux/limits.h #define PIPE_BUF4096 /* # bytes in atomic write to a pipe */ Used to be just 512 bytes on older Unixes ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets now considered stable
On Fri, Jun 26, 2015 at 09:30:58PM +0200, Henrik Sarvell wrote: I was also including dbg.l (it was causing hung processes at 100% CPU), it's basically been deprecated or something, I'll leave it up to him to elaborate. IIRC, the problem was not so much including dbg.l, but starting the application without stdio redirection to some log file. As a result, errors or other messages in the background server caused broken pipe exceptions. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
RE: Websockets now considered stable
Hi Henrik Awesome! That's really cool, thank you for your effort and for sharing the code :-) 20k message with nearly zero server load sounds very impressive. Question: To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis 1) Where comes this limit from? POSIX IPC? PicoLisp IPC ? 2) I couldn't find the redis part in the code, maybe you can give me a hint where to look? Thanks, your work on websockets will definitely help me in in the future :-) - beneroth - Original Message - From: Henrik Sarvell [mailto:hsarv...@gmail.com] To: picolisp@software-lab.de Sent: Fri, 26 Jun 2015 21:30:58 +0200 Subject: Hi all, after over a month without any of the prior issues I now consider the websockets part of pl-web stable: https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU usage and zombie processes. With Alex's help the main web server is now more stable (he made me throw away a few throws in favour of a few byes). The throws were causing the zombies. I was also including dbg.l (it was causing hung processes at 100% CPU), it's basically been deprecated or something, I'll leave it up to him to elaborate. It's just something I've been including by habit since years ago when at some point I needed to include it to do some kind of debugging. Anyway atm the WS router is regularly routing up to 40 messages per second to upwards 300-500 clients which means that roughly 20,000 messages are being pushed out per second during peak hours. The PL processes show up with 0 CPU and 0 RAM usage when I run top, sometimes 1% CPU :) They hardly register even i aggregate, the server would be running 99% idle if it was only running the WS server. To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis (raw disk is also supported if Redis is not available), this is also in effect in production and is working flawlessly since months. This is how I start the WS server in production: (load pl-web/pl-web.l) (setq *Mobj (new '(+Redis) pl-ws-)) (undef 'app) (setq *WsAuth '((notifications ((send (put your password/key here)) (de app () (splitPath) (wsApp) (bye)) (de go () (wsServer) (server 9090) ) -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Websockets now considered stable
Hi all, after over a month without any of the prior issues I now consider the websockets part of pl-web stable: https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU usage and zombie processes. With Alex's help the main web server is now more stable (he made me throw away a few throws in favour of a few byes). The throws were causing the zombies. I was also including dbg.l (it was causing hung processes at 100% CPU), it's basically been deprecated or something, I'll leave it up to him to elaborate. It's just something I've been including by habit since years ago when at some point I needed to include it to do some kind of debugging. Anyway atm the WS router is regularly routing up to 40 messages per second to upwards 300-500 clients which means that roughly 20,000 messages are being pushed out per second during peak hours. The PL processes show up with 0 CPU and 0 RAM usage when I run top, sometimes 1% CPU :) They hardly register even i aggregate, the server would be running 99% idle if it was only running the WS server. To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis (raw disk is also supported if Redis is not available), this is also in effect in production and is working flawlessly since months. This is how I start the WS server in production: (load pl-web/pl-web.l) (setq *Mobj (new '(+Redis) pl-ws-)) (undef 'app) (setq *WsAuth '((notifications ((send (put your password/key here)) (de app () (splitPath) (wsApp) (bye)) (de go () (wsServer) (server 9090) ) -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets now considered stable
Hi Andreas. 1) As far as I remember from a discussion with Alex it's a hard limit (OS related). 2) Line 369 - 372 here: https://bitbucket.org/hsarvell/pl-web/src/c445ca3861159d0b28ea779a183572c91b7b8458/pl-web.l?at=default On Fri, Jun 26, 2015 at 9:50 PM, andr...@itship.ch wrote: Hi Henrik Awesome! That's really cool, thank you for your effort and for sharing the code :-) 20k message with nearly zero server load sounds very impressive. Question: To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis 1) Where comes this limit from? POSIX IPC? PicoLisp IPC ? 2) I couldn't find the redis part in the code, maybe you can give me a hint where to look? Thanks, your work on websockets will definitely help me in in the future :-) - beneroth - Original Message - From: Henrik Sarvell [mailto:hsarv...@gmail.com] To: picolisp@software-lab.de Sent: Fri, 26 Jun 2015 21:30:58 +0200 Subject: Hi all, after over a month without any of the prior issues I now consider the websockets part of pl-web stable: https://bitbucket.org/hsarvell/pl-web Gone are the days of 100% CPU usage and zombie processes. With Alex's help the main web server is now more stable (he made me throw away a few throws in favour of a few byes). The throws were causing the zombies. I was also including dbg.l (it was causing hung processes at 100% CPU), it's basically been deprecated or something, I'll leave it up to him to elaborate. It's just something I've been including by habit since years ago when at some point I needed to include it to do some kind of debugging. Anyway atm the WS router is regularly routing up to 40 messages per second to upwards 300-500 clients which means that roughly 20,000 messages are being pushed out per second during peak hours. The PL processes show up with 0 CPU and 0 RAM usage when I run top, sometimes 1% CPU :) They hardly register even i aggregate, the server would be running 99% idle if it was only running the WS server. To work around the inter-process limit of 4096 byte long messages the router now supports storing the messages in Redis (raw disk is also supported if Redis is not available), this is also in effect in production and is working flawlessly since months. This is how I start the WS server in production: (load pl-web/pl-web.l) (setq *Mobj (new '(+Redis) pl-ws-)) (undef 'app) (setq *WsAuth '((notifications ((send (put your password/key here)) (de app () (splitPath) (wsApp) (bye)) (de go () (wsServer) (server 9090) ) -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Wiki: Websockets with PicoLisp, 404
Yes it can, I just haven't gotten around to updating the links. Here is Jose's new bitbucket identity: https://bitbucket.org/iromero91/ On Mon, Feb 2, 2015 at 4:33 PM, Jon Kleiser jon.klei...@fsat.no wrote: Hi, Just want to mention that the wiki page http://picolisp.com/wiki/?Websockets contains a link to web.l's websocket demo https://bitbucket.org/cyborgar/web.l/src/default/webtest.l?at=default that now gives a 404. Does anybody know if this demo code can be found elsewhere? /Jon-- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Wiki: Websockets with PicoLisp, 404
Hi, Just want to mention that the wiki page http://picolisp.com/wiki/?Websockets contains a link to web.l's websocket demo https://bitbucket.org/cyborgar/web.l/src/default/webtest.l?at=default that now gives a 404. Does anybody know if this demo code can be found elsewhere? /Jon-- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
FWIW, in my case I fall back to polling every 10s in case websockets are not supported. However, as soon as IE9 penetration drops to an insignificant level I will stop with fallbacks. On Tue, Jul 15, 2014 at 7:11 PM, andr...@itship.ch wrote: Seems like we have a similar goal, Amaury! Cool :) On Mon, Jul 14, 2014 at 03:52:42AM -0700, Amaury Hernández Águila wrote: Yeah that would be nice. So, isn't that a good reason to have websockets in PocoLisp? I would not say so. In a video game you have so much continuous communication going on (most notably the stream of image frames), that you don't need an extra channel. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe Online games don't work by feeding single frames to clients. There are now a few companies trying to get this alive - like video streaming for games - but this not working fluently as the amount of data to transfer when rendering images on the server is just too big to have a reasonable latency for real-time-games (in opposite to turn based games). The continuous communication in a online game is chat and game logic messages - user input going up to the server, and results of that inputs together with results of behaviour from other agents going down to client
Re: Doubts about the benefits of WebSockets
FWIW, in my case I fall back to polling every 10s in case websockets are not supported. However, as soon as IE9 penetration drops to an insignificant level I will stop with fallbacks. Make a user agent statistic from your users, or try to obtain data about your target audience. Rumour is, like half of Asia is still running on pirated WinXP using old versions of IE (even IE6), so its surely good to have fallbacks around. Of course, this is probably insignificant, but still interesting how stubborn this old stuff sticks around. A point for trying to make software well from the start, you never know how long it will haunt you or others after release... Mobile support of websockets in the current client versions seems to be pretty good: http://caniuse.com/websockets Question is what version the user-base actually is using... --- Just some random sources for my statements, not exactly well researched stuff: http://www.techinasia.com/windows-xp-now-dead-but-200-million-machines-in-china-still-using-it/ http://www.troyhunt.com/2010/08/aye-pirates-be-reason-ie6-just-wont-die.html -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
How reliable are the statistics from w3schools? ( http://www.w3schools.com/browsers/browsers_stats.asp). It is stated in this page http://www.w3schools.com/browsers/browsers_explorer.asp that IE6 users are about 0.1% of the total Internet users. When I first began web development I always tried to make my websites compatible with every browser. But now I'm getting tired of that (particularly, tired of IE) and now I just care about the latest versions of Chrome and FF. At least this is true for personal projects. I understand that clients might require their products to be fully compatible with every browser. 2014-07-15 14:41 GMT-07:00 andr...@itship.ch: FWIW, in my case I fall back to polling every 10s in case websockets are not supported. However, as soon as IE9 penetration drops to an insignificant level I will stop with fallbacks. Make a user agent statistic from your users, or try to obtain data about your target audience. Rumour is, like half of Asia is still running on pirated WinXP using old versions of IE (even IE6), so its surely good to have fallbacks around. Of course, this is probably insignificant, but still interesting how stubborn this old stuff sticks around. A point for trying to make software well from the start, you never know how long it will haunt you or others after release... Mobile support of websockets in the current client versions seems to be pretty good: http://caniuse.com/websockets Question is what version the user-base actually is using... --- Just some random sources for my statements, not exactly well researched stuff: http://www.techinasia.com/windows-xp-now-dead-but-200-million-machines-in-china-still-using-it/ http://www.troyhunt.com/2010/08/aye-pirates-be-reason-ie6-just-wont-die.html -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Doubts about the benefits of WebSockets
Hi Henrik, first of all, thank you for the article and the good work! However, I must say that I have doubts about the benefits of WebSockets in general. I cannot see that they are worth the overhead. They introduce a complicated machinery, which is not just a simple protocol extension, but a fundamental change in the HTTP transaction principles. As that, it violates the PicoLisp philosophy of minimizing the number of concepts. Second, they require a browser which can handle them. This excludes, for example, text browsers like 'w3m'. Third, I'm still waiting for the killer-application which really needs them. All scenarios I have seen so far can be handled with the standard PicoLisp framework much more easily - without explicit maintenance tasks lile clearing out disconnected clients and sending a ping ... to keep the connection alive. All this is already done automatically. To keep with the typical realtime chat, a minimal setup could be: #!/usr/bin/pil (load @lib/http.l @lib/xhtml.l @lib/form.l) (de chat () (app) (action (html 0 Chat @lib.css NIL (form NIL (gui 'log '(+FileField) chat.log 60 20) (--) (gui 'msg '(+TextField) 60) (gui '(+Click +Auto +Button) 2000 'This 2000 Chat '(when (val (: home msg)) (out +chat.log (prinl @)) (clr (: home msg)) ) ) ) ) ) ) (server 8080 !chat) (wait) Put this into a file called chat, set it to executable, and start it as: $ ./chat Then point some browser windows to http://localhost:8080;. Everything typed into the 'msg' text field (and even in the 'log' field above it) appears in all other connected clients after maximally 2 seconds. Note that also text browsers like w3m can be used. Just the automatic firing of the button won't happen without JavaScript, so the user must press the Chat button whenever he wants an update. Also note that due to the same-origin-policy of JavaScrpt (repeatedly discussed here) the Chat buttun must be pressed in the beginning once, unless - as usually recommended - 'httpGate' is running and the client connected directly to http://localhost;. Other features like a notification system can be implemented even simpler. You could, for example, clone the existing 'ping' and 'ping' functions in @lib/xhtml.l and extend them to carry a payload. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
Hi Alex, doesn't all that polling you're doing introduce a lot of unnecessary requests to the server. There can be up to 200 persons logged in at the same time at the site where I'm using websockets now, that would be 100 HTTP POSTS per second with full HTTP headers etc just to check for notifications that perhaps 1 or 2 of them would get per 10 seconds. See figure 3 here: http://www.websocket.org/quantum.html On Mon, Jul 14, 2014 at 2:44 PM, Alexander Burger a...@software-lab.de wrote: Hi Henrik, first of all, thank you for the article and the good work! However, I must say that I have doubts about the benefits of WebSockets in general. I cannot see that they are worth the overhead. They introduce a complicated machinery, which is not just a simple protocol extension, but a fundamental change in the HTTP transaction principles. As that, it violates the PicoLisp philosophy of minimizing the number of concepts. Second, they require a browser which can handle them. This excludes, for example, text browsers like 'w3m'. Third, I'm still waiting for the killer-application which really needs them. All scenarios I have seen so far can be handled with the standard PicoLisp framework much more easily - without explicit maintenance tasks lile clearing out disconnected clients and sending a ping ... to keep the connection alive. All this is already done automatically. To keep with the typical realtime chat, a minimal setup could be: #!/usr/bin/pil (load @lib/http.l @lib/xhtml.l @lib/form.l) (de chat () (app) (action (html 0 Chat @lib.css NIL (form NIL (gui 'log '(+FileField) chat.log 60 20) (--) (gui 'msg '(+TextField) 60) (gui '(+Click +Auto +Button) 2000 'This 2000 Chat '(when (val (: home msg)) (out +chat.log (prinl @)) (clr (: home msg)) ) ) ) ) ) ) (server 8080 !chat) (wait) Put this into a file called chat, set it to executable, and start it as: $ ./chat Then point some browser windows to http://localhost:8080;. Everything typed into the 'msg' text field (and even in the 'log' field above it) appears in all other connected clients after maximally 2 seconds. Note that also text browsers like w3m can be used. Just the automatic firing of the button won't happen without JavaScript, so the user must press the Chat button whenever he wants an update. Also note that due to the same-origin-policy of JavaScrpt (repeatedly discussed here) the Chat buttun must be pressed in the beginning once, unless - as usually recommended - 'httpGate' is running and the client connected directly to http://localhost;. Other features like a notification system can be implemented even simpler. You could, for example, clone the existing 'ping' and 'ping' functions in @lib/xhtml.l and extend them to carry a payload. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
On Mon, Jul 14, 2014 at 11:41:08AM +0200, Alexander Burger wrote: Not such a big problem. If I measure the described chat client, pinging every 2 seconds, I get 335 Bytes per second on the average. This amounts to 65 kB per second for 200 clients. Not a big problem today. Typically Oops, no! It is even less than that. The size I wrote above includes log information like time stamps and process IDs. Without them, there are only 218 Bytes per second, giving 44 KB per second. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
Hi Tomas, thing like the example from Alex, then the amount of work on the server seems rather small and avoiding sending HTTP headers seems like pointless micro-optimization. True. The posts caused by the +Auto button are POST /55319/29110032894590418~!jsForm?!chat?*Menu=+0*Tab=+1*ID= HTTP/1.1 Host: localhost User-Agent: Mozilla Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8 Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip, deflate Referer: http://localhost/ Content-Length: 51 Content-Type: text/plain; charset=UTF-8 Connection: keep-alive Pragma: no-cache Cache-Control: no-cache *Gui:-3=Chat*Get=1*Form=2*Evt=0*Gui:1=*Gui:2= with a size of 449 bytes. This is less than the default TCP packet size of 1152 bytes, so a headerless protcoll wouldn't save anything here. What about putting the log field into an iframe with automatic reload instead of the +Auto button which needs javascript? Would that not be even better? I don't know, I have never use iframes. But with them we again introduce a new concept and new dependencies, which I try to avoid. The +Auto button is more general, and I use it quite often. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
How about a browser videogame? Developing videogames in PicoLisp would be great. I think I'll start one tomorrow. El jul 14, 2014 3:21 AM, Alexander Burger a...@software-lab.de escribió: On Mon, Jul 14, 2014 at 12:02:10PM +0200, Alexander Burger wrote: with a size of 449 bytes. This is less than the default TCP packet size of 1152 bytes, so a headerless protcoll wouldn't save anything here. Sorry, forget that! I think there is no default TCP packet size :) Anyway, we are not talking about huge amounts of data here. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
Yeah that would be nice. So, isn't that a good reason to have websockets in PocoLisp? El jul 14, 2014 3:49 AM, Alexander Burger a...@software-lab.de escribió: Hi Amaury, On Mon, Jul 14, 2014 at 03:26:15AM -0700, Amaury Hernández Águila wrote: How about a browser videogame? Developing videogames in PicoLisp would be great. I think I'll start one tomorrow. Really? That would be great! I suspect there are many people here interested to help. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
On Mon, Jul 14, 2014 at 03:52:42AM -0700, Amaury Hernández Águila wrote: Yeah that would be nice. So, isn't that a good reason to have websockets in PocoLisp? I would not say so. In a video game you have so much continuous communication going on (most notably the stream of image frames), that you don't need an extra channel. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
44KB / second is far from insignificant IMO, it works out to 0.35 Mbit/s if I'm not mistaken, we're paying 20 EUR per month per 1Mbit at our current co-location. Well worth spending a couple of days to avoid permanently. Plus, the goal is to have much much more people logged in in the future. On Mon, Jul 14, 2014 at 4:52 PM, Alexander Burger a...@software-lab.de wrote: On Mon, Jul 14, 2014 at 11:41:08AM +0200, Alexander Burger wrote: Not such a big problem. If I measure the described chat client, pinging every 2 seconds, I get 335 Bytes per second on the average. This amounts to 65 kB per second for 200 clients. Not a big problem today. Typically Oops, no! It is even less than that. The size I wrote above includes log information like time stamps and process IDs. Without them, there are only 218 Bytes per second, giving 44 KB per second. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
Hi Henrik, 44KB / second is far from insignificant IMO, it works out to 0.35 Mbit/s if I'm not mistaken, we're paying 20 EUR per month per 1Mbit at our current co-location. Well worth spending a couple of days to avoid permanently. wow, 20 EUR per 1Mbit? 160 EUR per 1MByte? In 2014? I can't believe that. My home connection costs 24 EUR a month without trafic restrictions iirc and is much faster than that:-) Cheers, Tomas -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Doubts about the benefits of WebSockets
Yes but that's a residential subscription, before we moved to co-location we used fasthosts.co.uk (highly recommended if you don't do the kind of realtime stuff I do at work). With fasthosts you get unlimited speed and transfers but when you do what we do you quickly realize that it doesn't work good because you have no control over how your boxes are spaced out in the facility. If some other customer is transferring massive amounts of data between one of their boxes the quality of your LAN connection could start suffering. At one point another customer of theirs got attacked by some kind of denial of service attack which affected us too since it's all shared. In the co-lo where we are now we rent space, a whole rack actually, so our machines are next to each other served by our own switches. This has resulted in zero network related outages as opposed to the situation at fasthosts. To the co-location facility there are 3 geographically separated uplinks which we buy dedicated space on, ie we have unlimited amount of transfer per month of course but we have a roof on how much we can transfer per second (up and down), this roof can be raised by 20 EUR per Mbit/s per month. We don't share this with anyone else so the speed is 100% guaranteed, if we don't utilize it that space is not used, the actions of other customers in the co-location facility can't affect us as they are also similarly limited in how much they use. This is working remarkably well, we're not experiencing any network related issues at all anymore. But it costs... On Tue, Jul 15, 2014 at 2:00 AM, Tomas Hlavaty t...@logand.com wrote: Hi Henrik, 44KB / second is far from insignificant IMO, it works out to 0.35 Mbit/s if I'm not mistaken, we're paying 20 EUR per month per 1Mbit at our current co-location. Well worth spending a couple of days to avoid permanently. wow, 20 EUR per 1Mbit? 160 EUR per 1MByte? In 2014? I can't believe that. My home connection costs 24 EUR a month without trafic restrictions iirc and is much faster than that:-) Cheers, Tomas -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
Hi Andreas. I don't put httpGate behind Apache since I don't need to use port 443, in any case without that websocket proxy module I'm not sure it would even be possible. Yes, making those two numbers configurable is probably a good idea and trivial to implement, will fix that asap. On Sun, Jul 13, 2014 at 10:00 PM, andr...@itship.ch wrote: Hi Henrik Thank you for publishing your work and the nice write up! I will definitively study it in detail and probably reuse it at some point in the future. About 1.) and 2.): I think this is highly depended on the actual use case, and has to be configured individually for actual applications (e.g. when the usual message interval is high, ping is probably not necessary). Good you pointed this configurations out! One question: You write you use this in a LAMP environment, am I correct in assuming clients connect directly to httpGate and then pl-web? Or do you put httpGate behind Apache, and if so, why? Thanks :) The past few days I've been working on websockets and the write up is here: http://picolisp.com/wiki/?Websockets There are two settings that could be discussed right away on this mailing list: 1.) The appropriateness of clearing out disconnected clients from the tag and id lists every 5 seconds, maybe it should be higher, maybe lower. 2.) The appropriateness of sending a ping from each client process to each client every 10 seconds to keep the connection alive, it could probably be higher. Finally note that I'm not using the words subscribe, publish or channel in the post (or the code) is that I don't want to confuse what I've done with WAMP: http://wamp.ws/ since it's not an implementation of that sub-protocol. Enjoy! -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Websockets
The past few days I've been working on websockets and the write up is here: http://picolisp.com/wiki/?Websockets There are two settings that could be discussed right away on this mailing list: 1.) The appropriateness of clearing out disconnected clients from the tag and id lists every 5 seconds, maybe it should be higher, maybe lower. 2.) The appropriateness of sending a ping from each client process to each client every 10 seconds to keep the connection alive, it could probably be higher. Finally note that I'm not using the words subscribe, publish or channel in the post (or the code) is that I don't want to confuse what I've done with WAMP: http://wamp.ws/ since it's not an implementation of that sub-protocol. Enjoy! -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
On further examination I have decided that websockets are overkill for my purpose, it seems that XHR v2 is a better fit for file uploads. Here is an example: https://github.com/Integralist/XHR2-Multiple-File-Upload--with-PHP- That code makes me believe that I will be able to accomplish a progressbar without having to change my current PL web server code at all. To be continued after I actually try it... On Mon, May 28, 2012 at 1:41 PM, Henrik Sarvell hsarv...@gmail.com wrote: Thanks Jose, awesome! Will check this out more thoroughly tonight. What do you mean with xor descrambling, the ut8 decode function? It has a comment labelling it as slow. On Mon, May 28, 2012 at 12:54 AM, José Romero j...@2.71828.com.ar wrote: On Sun, 27 May 2012 18:53:31 +0700 Henrik Sarvell hsarv...@gmail.com wrote: Google couldn't show me any prior discussions of websockets and picolisp. Is it too early to start thinking about this maybe, seems like the spec/ref is still changing a little bit too fast/much atm? http://en.wikipedia.org/wiki/WebSocket http://tools.ietf.org/html/rfc6455 C lib: http://git.warmcat.com/cgi-bin/cgit/libwebsockets/ It came up today when I started to think about how crude file uploads via http is with no way of knowing the progress, but they can be used for much much more than fancy uploading interfaces of course. Having looked a bit at the ref ( http://tools.ietf.org/html/rfc6455 ) it seems to me that it looks quite straight forward. The question is, what is easier/more efficient, implementing this from scratch or with the help of the C lib and native? That is the question that maybe someone more experienced in C/in general than me can answer? I guess i should have posted this here before: https://bitbucket.org/cyborgar/web.l/src/249fe9f5c7d8/web/sockets.l That implements the RFC almost completely (there's no much implementation of the error code thingamajig because actually browsers don't even implement ping/pong correctly yet!). The implementation is in pure lisp, but it could be accelerated with a C library (actually the only thing to accelerate is the xor descrambling, could be done in an 'in-like environment). The webtest.l in the root of the repo provides a (ugly) chat server example using my web.l framework. Sorry if the code is not very elegant but I didn't devote that much time to that example. Here's the documentation of that module: https://bitbucket.org/cyborgar/web.l/wiki/Sockets Also, the spec is pretty much set in stone, at least as far as the IETF is concerned, RFC 6455 is a standards track document. Cheers, José -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
Hi Henrik, That code makes me believe that I will be able to accomplish a progressbar without having to change my current PL web server code at all. If it is only a progress bar what you are after, note that something similar is already existent in the standard PicoLisp release. It (ab)uses the Drag/Drop button for that. When you drag a document onto that button, the button's label changes to a percent display which counts up till 100% while transferring the file. You might consider using that directly, or modifying it to pop up a progress dialog instead of the simple percent text. Cheers, - Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
Thanks Jose, awesome! Will check this out more thoroughly tonight. What do you mean with xor descrambling, the ut8 decode function? It has a comment labelling it as slow. On Mon, May 28, 2012 at 12:54 AM, José Romero j...@2.71828.com.ar wrote: On Sun, 27 May 2012 18:53:31 +0700 Henrik Sarvell hsarv...@gmail.com wrote: Google couldn't show me any prior discussions of websockets and picolisp. Is it too early to start thinking about this maybe, seems like the spec/ref is still changing a little bit too fast/much atm? http://en.wikipedia.org/wiki/WebSocket http://tools.ietf.org/html/rfc6455 C lib: http://git.warmcat.com/cgi-bin/cgit/libwebsockets/ It came up today when I started to think about how crude file uploads via http is with no way of knowing the progress, but they can be used for much much more than fancy uploading interfaces of course. Having looked a bit at the ref ( http://tools.ietf.org/html/rfc6455 ) it seems to me that it looks quite straight forward. The question is, what is easier/more efficient, implementing this from scratch or with the help of the C lib and native? That is the question that maybe someone more experienced in C/in general than me can answer? I guess i should have posted this here before: https://bitbucket.org/cyborgar/web.l/src/249fe9f5c7d8/web/sockets.l That implements the RFC almost completely (there's no much implementation of the error code thingamajig because actually browsers don't even implement ping/pong correctly yet!). The implementation is in pure lisp, but it could be accelerated with a C library (actually the only thing to accelerate is the xor descrambling, could be done in an 'in-like environment). The webtest.l in the root of the repo provides a (ugly) chat server example using my web.l framework. Sorry if the code is not very elegant but I didn't devote that much time to that example. Here's the documentation of that module: https://bitbucket.org/cyborgar/web.l/wiki/Sockets Also, the spec is pretty much set in stone, at least as far as the IETF is concerned, RFC 6455 is a standards track document. Cheers, José -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Websockets
On Sun, 27 May 2012 18:53:31 +0700 Henrik Sarvell hsarv...@gmail.com wrote: Google couldn't show me any prior discussions of websockets and picolisp. Is it too early to start thinking about this maybe, seems like the spec/ref is still changing a little bit too fast/much atm? http://en.wikipedia.org/wiki/WebSocket http://tools.ietf.org/html/rfc6455 C lib: http://git.warmcat.com/cgi-bin/cgit/libwebsockets/ It came up today when I started to think about how crude file uploads via http is with no way of knowing the progress, but they can be used for much much more than fancy uploading interfaces of course. Having looked a bit at the ref ( http://tools.ietf.org/html/rfc6455 ) it seems to me that it looks quite straight forward. The question is, what is easier/more efficient, implementing this from scratch or with the help of the C lib and native? That is the question that maybe someone more experienced in C/in general than me can answer? I guess i should have posted this here before: https://bitbucket.org/cyborgar/web.l/src/249fe9f5c7d8/web/sockets.l That implements the RFC almost completely (there's no much implementation of the error code thingamajig because actually browsers don't even implement ping/pong correctly yet!). The implementation is in pure lisp, but it could be accelerated with a C library (actually the only thing to accelerate is the xor descrambling, could be done in an 'in-like environment). The webtest.l in the root of the repo provides a (ugly) chat server example using my web.l framework. Sorry if the code is not very elegant but I didn't devote that much time to that example. Here's the documentation of that module: https://bitbucket.org/cyborgar/web.l/wiki/Sockets Also, the spec is pretty much set in stone, at least as far as the IETF is concerned, RFC 6455 is a standards track document. Cheers, José -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
