Interestingly I'm currently trying to find  a definitive answer/correlation 
between "clock speed" & "performance" on a single core/thread on Intel CPU's to 
confirm, or deny, that for grunt batch work if a 4C Intel running @ 3.4Ghz will 
actually be faster than an 8C running @ 2.7Ghz --> the answer isn't as straight 
forward (or as easy to find) as I would have hoped, as even within the same 
family (e5-2600) there can be "architectural differences" that come into play 
.... and if anyone has a definitive answer, please feel free to share !

Ross Ferris
Stamina Software
Visage > Better by Design!


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org 
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of HENDERSON MIKE, MR
Sent: Tuesday, 2 October 2012 1:18 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified)

I have often thought about this - mostly in an idle moment or as a displacement 
activity for something less amusing that I ought to be doing. ;-)


First of all, Universe is already extremely parallel: there's a separate O/S 
thread for each TTY and for each phantom, and you can't get more parallel than 
that for interactive processing.

So you want more parallelism for your batch processes.
Different applications have different degrees of inherent parallelism.
For example in utility billing systems there is frequently the concept of a 
group of premises - based on the old concept of a foot-borne meter reader with 
a 'book' of readings to get. Each 'book' can be processed independently of 
every other. In payroll, each employee's record can be processed independently. 
Other areas of commerce have different characteristics.

I think that whatever unit of parallelism you settle for, you'd need three 
processes: a 'dispatcher' that selects records for processing and queues them 
into some structure for processing; a set of 'workers' that take queued work 
items, process them, mark them as processed and put the results in some common 
store; and a 'monitor' that looks for unprocessed records and indications of 
stuck processes, and collates the results for final output.
I've seen a couple of versions of this, one for electricity billings and 
another for overnight batch-processing of report requests, both well over a 
decade ago, and neither still in use although their underlying packages are 
still being run.

The major issue is that these days the whole entity in the general commercial 
world is far more likely to be I/O limited than CPU limited, and therefore 
introducing parallelism will be no help at all if the I/O system is already 
choked.
Even if the system is currently CPU-limited, multi-threading may not produce 
much improvement without very careful design of the record locking philosophy - 
introducing parallelism will be no help if all the threads end up contending 
serially for one record lock or a small set of locks.


If you want it to go faster, buy the CPU with the fastest clock you can get 
(not the one with the most cores), and put your database on SSD like Ross said.
The Power7+ chips being announced any day now are rumoured to go to
5GHz+, maybe even more if you have half the cores on the chip disabled.


Regards


Mike

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris
Sent: Tuesday, 2 October 2012 3:50 p.m.
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe

If the file were big enough, and already had part files, then I believe that 
you could have a phantom process each of the individual parts.
Failing that, get an SSD .... relatively cheap, and will give your processing a 
reasonable kick along!!

Ross Ferris
Stamina Software
Visage > Better by Design!


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton
Sent: Tuesday, 2 October 2012 8:47 AM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe

OK - I was trying to create a 'smoother use' of the disk and 'read ahead' -- 
this example the disk would be chattering from the heads moving all over the 
place. I was trying to find a way to make this process more 'orderly' -- is 
there one?

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben
Sent: Monday, October 01, 2012 4:48 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe

Create an index on a dict pointing at the first character of the key, and have 
each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9)

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton
Sent: October-01-12 2:43 PM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe

So how would a user 'chop up' a file for parallel processing?  Ideally, if here 
was a Mod 10001 file (or whatever) it would seem like it would be 'ideal' to 
assign 2000 groups to 5 phantoms -- but I don't know how 'start a BASIC select 
at Group 2001 or 4001' ...

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen
Sent: Monday, October 01, 2012 3:29 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe

0001: OPENSEQ "/tmp/pipetest" TO F.PIPE ELSE STOP "NO PIPE"
0002: LOOP
0003:    READSEQ LINE FROM F.PIPE ELSE CONTINUE
0004:    PRINT LINE
0005: REPEAT
0006: STOP
0007: END

Although, not sure if you might need to sleep a litte between the READSEQ's 
ELSE and CONTINUE
   Might suck up cpu time when nothing is writing to the file.

Then you could setup a printer in UV that did a  "cat - >> /tmp/pipetest"

Now your phantom just needs to print to that printer.

George

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen
Sent: Monday, October 01, 2012 4:16 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe

The only thing about a pipe is that once it's closed, I believe it has to be 
re-opened by both Ends again. So if point a opens one end, and point b opens 
the other end, once either end closes, It closes for both sides, and both sides 
would have to reopen again to use.

To eliminate this, you could have one end open a file, and have the other sides 
do a ">>" append To that file - just make sure you include some kind of 
dataheader so the reading side knows which Process just wrote the data.

-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of u2ug
Sent: Monday, October 01, 2012 4:11 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe

pipes


-----Original Message-----
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Monday, October 01, 2012 4:05 PM
To: u2-users@listserver.u2ug.org
Subject: [U2] [u2] Parallel processing in Universe


What's the largest dataset in the Universe user world?
In terms of number of records.

I'm wondering if we have any potential for utilities that map-reduce.
I suppose you would spawn phantoms but how do they communicate back to the 
master node?
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users


_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
The information contained in this Internet Email message is intended for the 
addressee only and may contain privileged information, but not necessarily the 
official views or opinions of the New Zealand Defence Force.
If you are not the intended recipient you must not use, disclose, copy or 
distribute this message or the information in it.

If you have received this message in error, please Email or telephone the 
sender immediately.
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users
_______________________________________________
U2-Users mailing list
U2-Users@listserver.u2ug.org
http://listserver.u2ug.org/mailman/listinfo/u2-users

Reply via email to