A Spanner deployment is called a Universe
-Original Message-
From: Robert Colquhoun robert.colquh...@gmail.com
To: U2 Users List u2-users@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 9:13 pm
Subject: Re: [U2] [u2] Parallel processing in Universe
On Tue, Oct 2, 2012 at 5:58 PM
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: 01 October 2012 21:05
To: u2-users@listserver.u2ug.org
Subject: [U2] [u2] Parallel processing in Universe
What's the largest dataset in the Universe user world?
In terms of number of records.
I'm wondering if we have any
On 01/10/12 22:47, Robert Houben wrote:
Create an index on a dict pointing at the first character of the key, and
have each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9)
Actually, this is a very BAD way of chopping up a file into five even
chunks.
I'm not sure of the stats, but on any
On 02/10/12 03:49, Ross Ferris wrote:
If the file were big enough, and already had part files, then I believe that
you could have a phantom process each of the individual parts. Failing that,
get an SSD relatively cheap, and will give your processing a reasonable
kick along!!
Just be
: [U2] [u2] Parallel processing in Universe
On 02/10/12 03:49, Ross Ferris wrote:
If the file were big enough, and already had part files, then I believe that
you could have a phantom process each of the individual parts. Failing that,
get an SSD relatively cheap, and will give your
List
Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified)
I have often thought about this - mostly in an idle moment or as a
displacement activity for something less amusing that I ought to be doing.
;-)
First of all, Universe is already extremely parallel: there's
Of Wols Lists
Sent: Tuesday, October 02, 2012 4:20 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] [u2] Parallel processing in Universe
On 02/10/12 03:49, Ross Ferris wrote:
If the file were big enough, and already had part files, then I believe that
you could have a phantom process
Only outside of U2 using UniObjects can you achieve any type of parallel
activity. We have through UniObjects got 80 processes working from a single
Eclipse session through the use of threads in Java.
UniObjects creates individual uvapi_slave or udapi_slave for each of these
processes but the
] [u2] Parallel processing in Universe
On 01/10/12 22:47, Robert Houben wrote:
Create an index on a dict pointing at the first character of the key, and
have
each phantom take two digits. (0-1, 2-3, 4-5, 6-7, 8-9)
Actually, this is a very BAD way of chopping up a file into five even
chunks
Of David Taylor
Sent: Monday, October 01, 2012 6:10 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
Or, let's suppose you wanted to process repetitive segments of one very
large record using the same logic in a separate phantom process for each
segment, how large
.
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton
Sent: Tuesday, October 02, 2012 11:43 AM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe
In my example, I would grab
Of George Gallen
Sent: Tuesday, October 02, 2012 10:52 AM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
What if you created a duplicate file, did a SELECT and saved the list
(non-sorted).
Each of the phantoms would do a getlist and loop through using
readlist/readu
Wolverton
Sent: Monday, October 01, 2012 4:47 PM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe
OK - I was trying to create a 'smoother use' of the disk and 'read ahead' --
this example the disk would be chattering from the heads moving all over the
place. I was trying
...@listserver.u2ug.org] On Behalf Of Daniel McGrath
Sent: Tuesday, October 02, 2012 12:05 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
You've highlighted one problem here.
By having multiple processes accessing the disk in different locations, you
destroy cache
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Daniel McGrath
Sent: Tuesday, October 02, 2012 12:05 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
You've highlighted one problem here.
By having multiple processes accessing the disk in different locations, you
@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 10:32 am
Subject: Re: [U2] [u2] Parallel processing in Universe
Yes, SSD will definitely help. Just keep in mind, it doesn't prevent all
negatives in regards to I/O, particularly with regards to caching.
Disk caching in a modern system is fairly complex
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Tuesday, October 02, 2012 1:35 PM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] [u2] Parallel processing in Universe
The idea of the phantoms would be to read the file in order, not randomly, just
inorder from five
u2-users@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 10:39 am
Subject: Re: [U2] [u2] Parallel processing in Universe
If 5 phantoms were running, and read in order but from 5 different starting
points, the records would
Essentially still be processed in a random order, if you were to layout
Users List u2-users@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 10:39 am
Subject: Re: [U2] [u2] Parallel processing in Universe
If 5 phantoms were running, and read in order but from 5 different starting
points, the records would
Essentially still be processed in a random order, if you were
To: U2 Users List u2-users@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 10:39 am
Subject: Re: [U2] [u2] Parallel processing in Universe
If 5 phantoms were running, and read in order but from 5 different starting
points, the records would Essentially still be processed in a random order,
if you were
@listserver.u2ug.org
Sent: Tue, Oct 2, 2012 10:59 am
Subject: Re: [U2] [u2] Parallel processing in Universe
Which was my question -- was there a way to 'jump to' a group or 'BASIC
SELECT' with a'starting/ending' group -- so that again, 10001 moduo, one
phantom does 'groups' 1-2000, next phantom does
...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Wednesday, 3 October 2012 3:42 AM
To: u2-users@listserver.u2ug.org
Subject: Re: [U2] [u2] Parallel processing in Universe
The point of the caching concern is related to the read ahead, and you will
still get some benefit from this, if your five
Sent: Wednesday, 3 October 2012 3:19 AM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe
Great point!! I think we can agree that 'spinning media latency' is the enemy
and having phantoms increasing the 'head dance' can make things worse, not
better!
Many problems go
On Tue, Oct 2, 2012 at 5:58 PM, Symeon Breen syme...@gmail.com wrote:
However map reduce and hadoop are pretty horrible things. Even Google have
moved away from it with Caffiene etc.
Going OT a little, i think Google is replacing BigTable which was part of
Caffeine in 2010 with Spanner now.
What's the largest dataset in the Universe user world?
In terms of number of records.
I'm wondering if we have any potential for utilities that map-reduce.
I suppose you would spawn phantoms but how do they communicate back to the
master node?
___
pipes
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent: Monday, October 01, 2012 4:05 PM
To: u2-users@listserver.u2ug.org
Subject: [U2] [u2] Parallel processing in Universe
What's the largest dataset
...@listserver.u2ug.org] On Behalf Of u2ug
Sent: Monday, October 01, 2012 4:11 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
pipes
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Wjhonson
Sent
Of George Gallen
Sent: Monday, October 01, 2012 4:16 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
The only thing about a pipe is that once it's closed, I believe it has to be
re-opened by both
Ends again. So if point a opens one end, and point b opens the other end, once
: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of George Gallen
Sent: Monday, October 01, 2012 3:29 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
0001: OPENSEQ /tmp/pipetest TO F.PIPE ELSE STOP NO PIPE
0002: LOOP
0003
-01-12 2:43 PM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe
So how would a user 'chop up' a file for parallel processing? Ideally, if here
was a Mod 10001 file (or whatever) it would seem like it would be 'ideal' to
assign 2000 groups to 5 phantoms -- but I don't know
...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben
Sent: Monday, October 01, 2012 4:48 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
Create an index on a dict pointing at the first character of the key, and
have each phantom take two
-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Robert Houben
Sent: Monday, October 01, 2012 4:48 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
Create an index on a dict pointing at the first character of the key, and
have each
Subject: Re: [U2] [u2] Parallel processing in Universe
The only thing about a pipe is that once it's closed, I believe it has
to be re-opened by both Ends again. So if point a opens one end, and
point b opens the other end, once either end closes, It closes for both
sides, and both sides would have
...@listserver.u2ug.org] On Behalf Of George Gallen
Sent: Monday, October 01, 2012 3:29 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
0001: OPENSEQ /tmp/pipetest TO F.PIPE ELSE STOP NO PIPE
0002: LOOP
0003:READSEQ LINE FROM F.PIPE ELSE CONTINUE
0004:PRINT LINE
0005
by Design!
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of David Wolverton
Sent: Tuesday, 2 October 2012 8:47 AM
To: 'U2 Users List'
Subject: Re: [U2] [u2] Parallel processing in Universe
OK - I was trying to create
-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of Ross Ferris
Sent: Tuesday, 2 October 2012 3:50 p.m.
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe
If the file were big enough, and already had part files, then I believe
that you
!
-Original Message-
From: u2-users-boun...@listserver.u2ug.org
[mailto:u2-users-boun...@listserver.u2ug.org] On Behalf Of HENDERSON MIKE, MR
Sent: Tuesday, 2 October 2012 1:18 PM
To: U2 Users List
Subject: Re: [U2] [u2] Parallel processing in Universe (Unclassified)
I have often thought
37 matches
Mail list logo
