Re: [U2] [u2] Parallel processing in Universe (Unclassified)
Only outside of U2 using UniObjects can you achieve any type of parallel activity. We have through UniObjects got 80 processes working from a single Eclipse session through the use of threads in Java. UniObjects creates individual uvapi_slave or udapi_slave for each of these processes but the system or in this case the udapi_server or uvadpi_server cannot handle as many threads as we would like. We never ran out of memory on our 8GB Windows 2008R2 Server nor did SSD 120GB drive fail to keep up with the 80 ANALYZE.FILES or the 80 RESIZE commands we were issuing on from our XLr8Resizer product within Eclipse. The only way we got this working was to set the retries to 1000 on reopening the connections. Although that number seems high it helped and get us from our previous best of 39 process to 80 process. When we have a lot of time and cannot think of anything better to do we will try for 500 process. Regards, Doug www.u2logic.com Eclipse based tools for the U2 programmer -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto: u2-users-boun...@listserver.u2ug.org] On Behalf Of HENDERSON MIKE, MR Sent: Tuesday, 2 October 2012 1:18 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified) I have often thought about this - mostly in an idle moment or as a displacement activity for something less amusing that I ought to be doing. ;-) First of all, Universe is already extremely parallel: there's a separate O/S thread for each TTY and for each phantom, and you can't get more parallel than that for interactive processing. So you want more parallelism for your batch processes. Different applications have different degrees of inherent parallelism. For example in utility billing systems there is frequently the concept of a group of premises - based on the old concept of a foot-borne meter reader with a 'book' of readings to get. Each 'book' can be processed independently of every other. In payroll, each employee's record can be processed independently. Other areas of commerce have different characteristics. I think that whatever unit of parallelism you settle for, you'd need three processes: a 'dispatcher' that selects records for processing and queues them into some structure for processing; a set of 'workers' that take queued work items, process them, mark them as processed and put the results in some common store; and a 'monitor' that looks for unprocessed records and indications of stuck processes, and collates the results for final output. I've seen a couple of versions of this, one for electricity billings and another for overnight batch-processing of report requests, both well over a decade ago, and neither still in use although their underlying packages are still being run. The major issue is that these days the whole entity in the general commercial world is far more likely to be I/O limited than CPU limited, and therefore introducing parallelism will be no help at all if the I/O system is already choked. Even if the system is currently CPU-limited, multi-threading may not produce much improvement without very careful design of the record locking philosophy - introducing parallelism will be no help if all the threads end up contending serially for one record lock or a small set of locks. If you want it to go faster, buy the CPU with the fastest clock you can get (not the one with the most cores), and put your database on SSD like Ross said. The Power7+ chips being announced any day now are rumoured to go to 5GHz+, maybe even more if you have half the cores on the chip disabled. Regards Mike -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris Sent: Tuesday, 2 October 2012 3:50 p.m. To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe If the file were big enough, and already had part files, then I believe that you could have a phantom process each of the individual parts. Failing that, get an SSD relatively cheap, and will give your processing a reasonable kick along!! Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: Tuesday, 2 October 2012 8:47 AM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe OK - I was trying to create a 'smoother use' of the disk and 'read ahead' -- this example the disk would be chattering from the heads moving all over the place. I was trying to find a way to make this process more 'orderly' -- is there one? -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben Sent: Monday, October 01, 2012 4:48 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel
Re: [U2] [u2] Parallel processing in Universe (Unclassified)
Only outside of U2 using UniObjects can you achieve any type of parallel activity. We have through UniObjects got 80 processes working from a single Eclipse session through the use of threads in Java. UniObjects creates individual uvapi_slave or udapi_slave for each of these processes but the system or in this case the udapi_server or uvadpi_server cannot handle as many threads as we would like. We never ran out of memory on our 8GB Windows 2008R2 Server nor did SSD 120GB drive fail to keep up with the 80 ANALYZE.FILES or the 80 RESIZE commands we were issuing on from our XLr8Resizer product within Eclipse. The only way we got this working was to set the retries to 1000 on reopening the connections. Although that number seems high it helped and get us from our previous best of 39 process to 80 process. When we have a lot of time and cannot think of anything better to do we will try for 500 process. Regards, Doug www.u2logic.com/tools.html Eclipse based tools for the U2 programmer ___ U2-Users mailing list U2-Users@listserver.u2ug.org http://listserver.u2ug.org/mailman/listinfo/u2-users
Re: [U2] [u2] Parallel processing in Universe (Unclassified)
I have often thought about this - mostly in an idle moment or as a displacement activity for something less amusing that I ought to be doing. ;-) First of all, Universe is already extremely parallel: there's a separate O/S thread for each TTY and for each phantom, and you can't get more parallel than that for interactive processing. So you want more parallelism for your batch processes. Different applications have different degrees of inherent parallelism. For example in utility billing systems there is frequently the concept of a group of premises - based on the old concept of a foot-borne meter reader with a 'book' of readings to get. Each 'book' can be processed independently of every other. In payroll, each employee's record can be processed independently. Other areas of commerce have different characteristics. I think that whatever unit of parallelism you settle for, you'd need three processes: a 'dispatcher' that selects records for processing and queues them into some structure for processing; a set of 'workers' that take queued work items, process them, mark them as processed and put the results in some common store; and a 'monitor' that looks for unprocessed records and indications of stuck processes, and collates the results for final output. I've seen a couple of versions of this, one for electricity billings and another for overnight batch-processing of report requests, both well over a decade ago, and neither still in use although their underlying packages are still being run. The major issue is that these days the whole entity in the general commercial world is far more likely to be I/O limited than CPU limited, and therefore introducing parallelism will be no help at all if the I/O system is already choked. Even if the system is currently CPU-limited, multi-threading may not produce much improvement without very careful design of the record locking philosophy - introducing parallelism will be no help if all the threads end up contending serially for one record lock or a small set of locks. If you want it to go faster, buy the CPU with the fastest clock you can get (not the one with the most cores), and put your database on SSD like Ross said. The Power7+ chips being announced any day now are rumoured to go to 5GHz+, maybe even more if you have half the cores on the chip disabled. Regards Mike -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris Sent: Tuesday, 2 October 2012 3:50 p.m. To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe If the file were big enough, and already had part files, then I believe that you could have a phantom process each of the individual parts. Failing that, get an SSD relatively cheap, and will give your processing a reasonable kick along!! Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: Tuesday, 2 October 2012 8:47 AM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe OK - I was trying to create a 'smoother use' of the disk and 'read ahead' -- this example the disk would be chattering from the heads moving all over the place. I was trying to find a way to make this process more 'orderly' -- is there one? -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben Sent: Monday, October 01, 2012 4:48 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe Create an index on a dict pointing at the first character of the key, and have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9) -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: October-01-12 2:43 PM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe So how would a user 'chop up' a file for parallel processing? Ideally, if here was a Mod 10001 file (or whatever) it would seem like it would be 'ideal' to assign 2000 groups to 5 phantoms -- but I don't know how 'start a BASIC select at Group 2001 or 4001' ... -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen Sent: Monday, October 01, 2012 3:29 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe 0001: OPENSEQ /tmp/pipetest TO F.PIPE ELSE STOP NO PIPE 0002: LOOP 0003:READSEQ LINE FROM F.PIPE ELSE CONTINUE 0004:PRINT LINE 0005: REPEAT 0006: STOP 0007: END Although, not sure if you might need to sleep a litte between the READSEQ's ELSE and CONTINUE Might suck up cpu time when nothing is writing to the file. Then you could setup a printer in UV that did a cat - /tmp/pipetest Now your phantom
Re: [U2] [u2] Parallel processing in Universe (Unclassified)
Interestingly I'm currently trying to find a definitive answer/correlation between clock speed performance on a single core/thread on Intel CPU's to confirm, or deny, that for grunt batch work if a 4C Intel running @ 3.4Ghz will actually be faster than an 8C running @ 2.7Ghz -- the answer isn't as straight forward (or as easy to find) as I would have hoped, as even within the same family (e5-2600) there can be architectural differences that come into play and if anyone has a definitive answer, please feel free to share ! Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of HENDERSON MIKE, MR Sent: Tuesday, 2 October 2012 1:18 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified) I have often thought about this - mostly in an idle moment or as a displacement activity for something less amusing that I ought to be doing. ;-) First of all, Universe is already extremely parallel: there's a separate O/S thread for each TTY and for each phantom, and you can't get more parallel than that for interactive processing. So you want more parallelism for your batch processes. Different applications have different degrees of inherent parallelism. For example in utility billing systems there is frequently the concept of a group of premises - based on the old concept of a foot-borne meter reader with a 'book' of readings to get. Each 'book' can be processed independently of every other. In payroll, each employee's record can be processed independently. Other areas of commerce have different characteristics. I think that whatever unit of parallelism you settle for, you'd need three processes: a 'dispatcher' that selects records for processing and queues them into some structure for processing; a set of 'workers' that take queued work items, process them, mark them as processed and put the results in some common store; and a 'monitor' that looks for unprocessed records and indications of stuck processes, and collates the results for final output. I've seen a couple of versions of this, one for electricity billings and another for overnight batch-processing of report requests, both well over a decade ago, and neither still in use although their underlying packages are still being run. The major issue is that these days the whole entity in the general commercial world is far more likely to be I/O limited than CPU limited, and therefore introducing parallelism will be no help at all if the I/O system is already choked. Even if the system is currently CPU-limited, multi-threading may not produce much improvement without very careful design of the record locking philosophy - introducing parallelism will be no help if all the threads end up contending serially for one record lock or a small set of locks. If you want it to go faster, buy the CPU with the fastest clock you can get (not the one with the most cores), and put your database on SSD like Ross said. The Power7+ chips being announced any day now are rumoured to go to 5GHz+, maybe even more if you have half the cores on the chip disabled. Regards Mike -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris Sent: Tuesday, 2 October 2012 3:50 p.m. To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe If the file were big enough, and already had part files, then I believe that you could have a phantom process each of the individual parts. Failing that, get an SSD relatively cheap, and will give your processing a reasonable kick along!! Ross Ferris Stamina Software Visage Better by Design! -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: Tuesday, 2 October 2012 8:47 AM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe OK - I was trying to create a 'smoother use' of the disk and 'read ahead' -- this example the disk would be chattering from the heads moving all over the place. I was trying to find a way to make this process more 'orderly' -- is there one? -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben Sent: Monday, October 01, 2012 4:48 PM To: U2 Users List Subject: Re: [U2] [u2] Parallel processing in Universe Create an index on a dict pointing at the first character of the key, and have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9) -Original Message- From: u2-users-boun...@listserver.u2ug.org [mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton Sent: October-01-12 2:43 PM To: 'U2 Users List' Subject: Re: [U2] [u2] Parallel processing in Universe So how would a user 'chop up' a file
