Given that there's a lot in your head about your application that I don't
know, perhaps I should stop guessing about what you're trying to do and just
make sure the principles are clear:
1. The more data you have to serialize and send to other processes, the slower
it will be. The key thing about a SharedArray is that it uses a couple tricks
to serialize the "container" without needing to serialize the data---but
that's the only thing special about it.
2. A type that contains non-immutable types (e.g., Arrays) only stores
references to those objects. So
type MyType{T}
A::SharedArray{T,2}
b::SharedArray{T,1}
end
should serialize very nicely, because a "MyType" object is, in itself, tiny
(if you know C, think of it essentially as 2 pointers) and the big parts (the
SharedArrays) serialize nicely.
I worry, though, that your array-of-tuples will prove troublesome, because
each one of those is in turn a reference to an object which would need to be
serialized. If they are all of the same size and you need to serialize it, you
should consider a different structure (e.g., a SharedArray of immutables).
--Tim
On Wednesday, May 07, 2014 12:52:59 PM Bob Quazar wrote:
> I don't need to modify any of the data, so I could use immutable. Would you
> recommend using tuples then, as fields in my immutable composite type,
> rather than arrays? I tried that, but one of my arrays has about 3,000,000
> entries (and each entry is itself a tuple); ntuple doesn't seem designed to
> handle this much data.
>
> w2 = ntuple(length(myArray), (j) -> myArray[j])
> ERROR: stack overflow
> in ntuple at tuple.jl:30 (repeats 74810 times)
> in ntuple at tuple.jl:29
>
> Is there an alternative way to constructing immutable arrays?
>
> Thanks much, Bob
>
> On Wednesday, May 7, 2014 3:38:45 AM UTC-7, Tim Holy wrote:
> > Given your current design, reinterpret (which is indeed the same thing as
> > casting) is not going to work for you, as least not without a lot of
> > hacking
> > on your part. Can you use an immutable? The key point is that an immutable
> > has
> > a predictable layout in memory. For that to work, your arrays would have
> > to be
> > of fixed size.
> >
> > In the memory layout of immutables, one thing you have to be aware of is
> > that
> > sometimes "gaps" are introduced for the purpose of memory-alignment.
> > (These
> > can also be platform-dependent, I believe, but if you're not transferring
> > data
> > from one machine to another this shouldn't be a problem.) Here's a demo,
> > in
> > case it's of any value to you:
> >
> > julia> immutable MyArray
> >
> > x::Uint16
> > y::Uint16
> > z::Uint16
> >
> > end
> >
> > julia> immutable MyType
> >
> > a::MyArray
> > c::Char
> > d::Float64
> >
> > end
> >
> > julia> sizeof(MyType)
> > 24
> >
> > julia> r = [uint8(1:24)]
> >
> > 24-element Array{Uint8,1}:
> > 0x01
> > 0x02
> > 0x03
> > 0x04
> > 0x05
> > 0x06
> > 0x07
> > 0x08
> > 0x09
> > 0x0a
> >
> > ⋮
> >
> > 0x0f
> > 0x10
> > 0x11
> > 0x12
> > 0x13
> > 0x14
> > 0x15
> > 0x16
> > 0x17
> > 0x18
> >
> > julia> reinterpret(MyType, r)
> >
> > 1-element Array{MyType,1}:
> > MyType(MyArray(0x0201,0x0403,0x0605),'\Uc0b0a09',1.2650169649295773e-192)
> >
> > On Tuesday, May 06, 2014 09:57:27 PM Bob Quazar wrote:
> > > Tim, Thanks for the response.
> > >
> > > How can I determine whether my structure (a composite type) is a
> >
> > contiguous
> >
> > > block of memory? The composite type has a few fields. One of the fields
> >
> > has
> >
> > > type Array{Array{Uint16,1},1} and another has type
> > > Array{Array{Float64,2},1}. Am I out of luck?
> > >
> > > What does it mean to reinterpret a composite type as a
> > > SharedArray{Uint8,1}? Is that something like casting?
> > >
> > > On Tuesday, May 6, 2014 7:32:47 PM UTC-7, Tim Holy wrote:
> > > > If your structure is a contiguous block of memory, you could
> >
> > presumably
> >
> > > > use a
> > > > SharedArray{Uint8,1} and then reinterpret it as your custom object.
> >
> > But if
> >
> > > > you
> > > > want independently garbage-collectable constituents, this is not
> >
> > likely to
> >
> > > > be
> > > > feasible.
> > > >
> > > > --Tim
> > > >
> > > > On Tuesday, May 06, 2014 03:03:01 PM Bob Quazar wrote:
> > > > > In Julia, is it possible to share read-only data structures between
> > > > > threads/processes running on different processors, without making a
> >
> > copy
> >
> > > > of
> > > >
> > > > > the entire data structure for each thread? The data structure I need
> >
> > to
> >
> > > > > share isn't of "bit type", so I can't use a SharedArray. And each
> >
> > worker
> >
> > > > > needs to read random parts from the full data structure, so I can't
> >
> > use
> >
> > > > a
> > > >
> > > > > distributed array. I also tried just putting "@parallel" before a
> >
> > for
> >
> > > > loop,
> > > >
> > > > > whose body accesses the data structure I want to share, but I
> >
> > believe
> >
> > > > the
> > > >
> > > > > whole data structure gets copied at each iteration. (I could be
> >
> > wrong
> >
> > > > about
> > > >
> > > > > that, but it's certainly very slow.) Does Julia have some
> >
> > alternative to
> >
> > > > a
> > > >
> > > > > copy-on-write for sharing data between threads?
> > > > >
> > > > > Thanks in advance!
> > > > >
> > > > > --Bob
