Re: Solved: Tell mechanism issue (perhaps OT)
On Fri, Aug 08, 2014 at 11:56:10AM +0700, Henrik Sarvell wrote: > The fix has been running now for roughly 24 hours with PIDs both below > and above 16bit with roughly 2-30 requests per second depending on the > hour, works flawlessly. > > One reset from 98304 to 300 has already happened without issues. Thanks Henrik! ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
The fix has been running now for roughly 24 hours with PIDs both below and above 16bit with roughly 2-30 requests per second depending on the hour, works flawlessly. One reset from 98304 to 300 has already happened without issues. Using pil64. On Thu, Aug 7, 2014 at 2:19 PM, Alexander Burger wrote: > Hi all, > > On Thu, Aug 07, 2014 at 09:00:21AM +0200, Alexander Burger wrote: >> Fixed pil32 too. I hope I didn't break anything. Tests are welcome! > > For the records: > > The handling of PIDs in both the 32-bit and the 64-bit versions of > PicoLisp itself was never a problem. They are handled in full 'pid_t' > size. > > The problem was the internal IPC protocol used by the 'tell' function, > when an (optional) PID is passed for the addressee of the message. Here, > a field of only 16 bits was used to store the PID. This is fixed now in > pil32 and pil64. > > ♪♫ Alex > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Hi all, On Thu, Aug 07, 2014 at 09:00:21AM +0200, Alexander Burger wrote: > Fixed pil32 too. I hope I didn't break anything. Tests are welcome! For the records: The handling of PIDs in both the 32-bit and the 64-bit versions of PicoLisp itself was never a problem. They are handled in full 'pid_t' size. The problem was the internal IPC protocol used by the 'tell' function, when an (optional) PID is passed for the addressee of the message. Here, a field of only 16 bits was used to store the PID. This is fixed now in pil32 and pil64. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
On Thu, Aug 07, 2014 at 08:01:50AM +0200, Alexander Burger wrote: > Hi Jorge, > > > I’m getting pids well above 64k on my laptop (OS X). OK Fixed pil32 too. I hope I didn't break anything. Tests are welcome! I've uploaded a new version to http://software-lab.de/picoLisp.tgz ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Oh, you mean the 32 bit version of pico lisp? Thats right… On 06 Aug 2014, at 23:23, Jorge Acereda Maciá wrote: > It’s a 64bit 8 GB machine. BSDs work differently: > > http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/basics-processeshtml > > Maximum seems to be 9, at least on FreeBSD. > > On 06 Aug 2014, at 23:09, Henrik Sarvell wrote: > >> Hi Jorge, how much RAM does it have, is it a 64bit machine? >> >> I have checked the number Alex mentioned on some of our servers, all >> running Ubuntu 12.04, servers below 64GB RAM have that number set to >> 32768 per default, machines with 128GB got 98304. >> >> On Thu, Aug 7, 2014 at 12:49 PM, Jorge Acereda Maciá >> wrote: >>> I’m getting pids well above 64k on my laptop (OS X). >>> >>> On 06 Aug 2014, at 22:33, Alexander Burger wrote: >>> Hi Randall, > I believe that modern Linux and FreeBSD implementations use 32 bit > ints for the pid_t. Right. > There will never be that many processes on a 32 bit > OS, but since they just go forward until they wrap, getting a pid bigger > than 16 bits is probably even to be expected. However, they don't plainly wrap. There is a system limit in the kernel, controlled via "/proc/sys/kernel/pid_max". Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit number), PIDs don't get up to such huge numbers. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >>> >>> -- >>> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe >> -- >> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe > > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
That’s not true. The kernel can boot in 64 bit and 32 bit mode (there are keyboard combos for that). Applications will run in 64 bit mode even when the kernel is in 32 bit mode. On 06 Aug 2014, at 23:19, Henrik Sarvell wrote: > Jorge, forget that question, I just got told that osx only have the > 32bit version. > > On Thu, Aug 7, 2014 at 1:09 PM, Henrik Sarvell wrote: >> Hi Jorge, how much RAM does it have, is it a 64bit machine? >> >> I have checked the number Alex mentioned on some of our servers, all >> running Ubuntu 12.04, servers below 64GB RAM have that number set to >> 32768 per default, machines with 128GB got 98304. >> >> On Thu, Aug 7, 2014 at 12:49 PM, Jorge Acereda Maciá >> wrote: >>> I’m getting pids well above 64k on my laptop (OS X). >>> >>> On 06 Aug 2014, at 22:33, Alexander Burger wrote: >>> Hi Randall, > I believe that modern Linux and FreeBSD implementations use 32 bit > ints for the pid_t. Right. > There will never be that many processes on a 32 bit > OS, but since they just go forward until they wrap, getting a pid bigger > than 16 bits is probably even to be expected. However, they don't plainly wrap. There is a system limit in the kernel, controlled via "/proc/sys/kernel/pid_max". Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit number), PIDs don't get up to such huge numbers. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >>> >>> -- >>> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
On Thu, Aug 07, 2014 at 01:19:16PM +0700, Henrik Sarvell wrote: > Jorge, forget that question, I just got told that osx only have the > 32bit version. Yes, but the question is valid. The machine itself might be 64-bit with a large RAM, and many processes, still running pil32. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
It’s a 64bit 8 GB machine. BSDs work differently: http://www.freebsd.org/doc/en_US.ISO8859-1/books/handbook/basics-processeshtml Maximum seems to be 9, at least on FreeBSD. On 06 Aug 2014, at 23:09, Henrik Sarvell wrote: > Hi Jorge, how much RAM does it have, is it a 64bit machine? > > I have checked the number Alex mentioned on some of our servers, all > running Ubuntu 12.04, servers below 64GB RAM have that number set to > 32768 per default, machines with 128GB got 98304. > > On Thu, Aug 7, 2014 at 12:49 PM, Jorge Acereda Maciá > wrote: >> I’m getting pids well above 64k on my laptop (OS X). >> >> On 06 Aug 2014, at 22:33, Alexander Burger wrote: >> >>> Hi Randall, >>> I believe that modern Linux and FreeBSD implementations use 32 bit ints for the pid_t. >>> >>> Right. >>> There will never be that many processes on a 32 bit OS, but since they just go forward until they wrap, getting a pid bigger than 16 bits is probably even to be expected. >>> >>> However, they don't plainly wrap. There is a system limit in the kernel, >>> controlled via "/proc/sys/kernel/pid_max". >>> >>> Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit >>> number), PIDs don't get up to such huge numbers. >>> >>> ♪♫ Alex >>> -- >>> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >> >> -- >> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Jorge, forget that question, I just got told that osx only have the 32bit version. On Thu, Aug 7, 2014 at 1:09 PM, Henrik Sarvell wrote: > Hi Jorge, how much RAM does it have, is it a 64bit machine? > > I have checked the number Alex mentioned on some of our servers, all > running Ubuntu 12.04, servers below 64GB RAM have that number set to > 32768 per default, machines with 128GB got 98304. > > On Thu, Aug 7, 2014 at 12:49 PM, Jorge Acereda Maciá > wrote: >> I’m getting pids well above 64k on my laptop (OS X). >> >> On 06 Aug 2014, at 22:33, Alexander Burger wrote: >> >>> Hi Randall, >>> I believe that modern Linux and FreeBSD implementations use 32 bit ints for the pid_t. >>> >>> Right. >>> There will never be that many processes on a 32 bit OS, but since they just go forward until they wrap, getting a pid bigger than 16 bits is probably even to be expected. >>> >>> However, they don't plainly wrap. There is a system limit in the kernel, >>> controlled via "/proc/sys/kernel/pid_max". >>> >>> Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit >>> number), PIDs don't get up to such huge numbers. >>> >>> ♪♫ Alex >>> -- >>> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe >> >> -- >> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Hi Jorge, how much RAM does it have, is it a 64bit machine? I have checked the number Alex mentioned on some of our servers, all running Ubuntu 12.04, servers below 64GB RAM have that number set to 32768 per default, machines with 128GB got 98304. On Thu, Aug 7, 2014 at 12:49 PM, Jorge Acereda Maciá wrote: > I’m getting pids well above 64k on my laptop (OS X). > > On 06 Aug 2014, at 22:33, Alexander Burger wrote: > >> Hi Randall, >> >>> I believe that modern Linux and FreeBSD implementations use 32 bit >>> ints for the pid_t. >> >> Right. >> >>> There will never be that many processes on a 32 bit >>> OS, but since they just go forward until they wrap, getting a pid bigger >>> than 16 bits is probably even to be expected. >> >> However, they don't plainly wrap. There is a system limit in the kernel, >> controlled via "/proc/sys/kernel/pid_max". >> >> Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit >> number), PIDs don't get up to such huge numbers. >> >> ♪♫ Alex >> -- >> UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe > > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subjectUnsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Hi Jorge, > I’m getting pids well above 64k on my laptop (OS X). Oops. I see. Hmm ... -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
I’m getting pids well above 64k on my laptop (OS X). On 06 Aug 2014, at 22:33, Alexander Burger wrote: > Hi Randall, > >> I believe that modern Linux and FreeBSD implementations use 32 bit >> ints for the pid_t. > > Right. > >> There will never be that many processes on a 32 bit >> OS, but since they just go forward until they wrap, getting a pid bigger >> than 16 bits is probably even to be expected. > > However, they don't plainly wrap. There is a system limit in the kernel, > controlled via "/proc/sys/kernel/pid_max". > > Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit > number), PIDs don't get up to such huge numbers. > > ♪♫ Alex > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
Hi Randall, > I believe that modern Linux and FreeBSD implementations use 32 bit > ints for the pid_t. Right. > There will never be that many processes on a 32 bit > OS, but since they just go forward until they wrap, getting a pid bigger > than 16 bits is probably even to be expected. However, they don't plainly wrap. There is a system limit in the kernel, controlled via "/proc/sys/kernel/pid_max". Even on 64-bit machines (where pid_t is also an 'int', i.e. a 64-bit number), PIDs don't get up to such huge numbers. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Solved: Tell mechanism issue (perhaps OT)
I believe that modern Linux and FreeBSD implementations use 32 bit ints for the pid_t. There will never be that many processes on a 32 bit OS, but since they just go forward until they wrap, getting a pid bigger than 16 bits is probably even to be expected. > On Aug 6, 2014, at 11:43 AM, Alexander Burger wrote: > I've fixed it now for pil64. I do not want to fix it for pil32, because > it is (1) more difficult and error-prone in C than in assembly, and (2) > because I believe that on 32-bit machines a 16-bit PID will always be > sufficient. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Solved: Tell mechanism issue (perhaps OT)
Hi all, On Mon, Aug 04, 2014 at 01:29:04PM +0700, Henrik Sarvell wrote: > To sum up some IRC communication, it's not the parent id that gets > told, it's a sibling PID. > > I will be back with more when the problem arises again. After further discussion in IRC, Henrik pointed me to the fact that 'tell' failed whenever a PID was greater than 65535. So: Sorry, my fault! I wrongly assumed that PIDs fit always into 16 bits in Unix. This is far from true. On large machines this limit may be well exceeded. Fortunately, this assumption was made only in the internal packaging of 'tell' messages. All other places where a PID is handled in PicoLisp, it has the full integer size. I've fixed it now for pil64. I do not want to fix it for pil32, because it is (1) more difficult and error-prone in C than in assembly, and (2) because I believe that on 32-bit machines a 16-bit PID will always be sufficient. I've uploaded a new picoLisp.tgz to the download site. After more tests by Henrik I'll check it into the repo. ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Tell mechanism issue (perhaps OT)
To sum up some IRC communication, it's not the parent id that gets told, it's a sibling PID. I will be back with more when the problem arises again. On Mon, Aug 4, 2014 at 12:30 PM, Alexander Burger wrote: > Hi Henrik, > >> When using the new websocket server ( >> http://picolisp.com/wiki/?Websockets ) in production (tell ParentPid >> 'func bla bla) stops working after a while. > > I haven't checked the details, but let me note the 'tell'ing to the > parent process is not possible. > > The reference of 'tell' says > >Send .. to .. all children of the current process, and all other >children of the parent process > > The parent process is the one thay relays all messages between the > processes. It cannot be the addressee of a messages. > > Therefore, as a general rule, the parent should never be assigned any > application tasks (especially not database manipulations, as these are > also synchronized via the 'tell' mechanisms). It should be as > lightweight as possible, just waiting for events and delegating work to > the children. > > > For messages to the parent, you might consider the "@lib/boss.l" > mechanism (I know you have used that in the past), calling 'boss' in the > children and 'hear' in the parent. > > > So this is to be expected: > >> parent process has problems preventing it from responding to tell, so > > > But still, however, I have no explanation for > >> I've had the server running on a test machine for weeks now without >> issues whereas it fails in production after 1-2 days so it has > > ♪♫ Alex > -- > UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Re: Tell mechanism issue (perhaps OT)
Hi Henrik, > When using the new websocket server ( > http://picolisp.com/wiki/?Websockets ) in production (tell ParentPid > 'func bla bla) stops working after a while. I haven't checked the details, but let me note the 'tell'ing to the parent process is not possible. The reference of 'tell' says Send .. to .. all children of the current process, and all other children of the parent process The parent process is the one thay relays all messages between the processes. It cannot be the addressee of a messages. Therefore, as a general rule, the parent should never be assigned any application tasks (especially not database manipulations, as these are also synchronized via the 'tell' mechanisms). It should be as lightweight as possible, just waiting for events and delegating work to the children. For messages to the parent, you might consider the "@lib/boss.l" mechanism (I know you have used that in the past), calling 'boss' in the children and 'hear' in the parent. So this is to be expected: > parent process has problems preventing it from responding to tell, so But still, however, I have no explanation for > I've had the server running on a test machine for weeks now without > issues whereas it fails in production after 1-2 days so it has ♪♫ Alex -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
Tell mechanism issue (perhaps OT)
This question might be OT or self-evident for someone with better basic understanding of Linux than me. When using the new websocket server ( http://picolisp.com/wiki/?Websockets ) in production (tell ParentPid 'func bla bla) stops working after a while. Here comes the weird part, the most probable cause would be that the parent process has problems preventing it from responding to tell, so restarting the server should fix the issue. It doesn't, only rebooting the machine makes tell work again. I've had the server running on a test machine for weeks now without issues whereas it fails in production after 1-2 days so it has something to do with the higher activity on the production box but I can't understand what the cause might be. -- UNSUBSCRIBE: mailto:picolisp@software-lab.de?subject=Unsubscribe
