Have you looked at StrPack.jl? It may have a packed option. Julia uses the platform ABI padding rules for easy interop with C. On Nov 22, 2014 7:17 AM, "Joshua Adelman" <[email protected]> wrote:
> For testing/evaluation purposes, it's actually the case that I don't need > to actually use any of the fixed length string fields. They are in the > data, but I have numerical encodings for most of the important ones in > other fields. So in playing around I found that I could just create a > bitstype of the appropriate size to consume them when using test data. > > However the caveat to this strategy is that numpy's internal memory layout > for a record array (and when it's saved to a file), is that the records are > packed structs. So a record with fields float64, int32, float32, int16 has > an itemsize of 18 bytes. If I build an immutable composite type in Julia: > > immutable TestType > a::Float64 > b::Int32 > c::Float32 > d::Int16 > end > > And then query the size of each field and the the aggregate as a whole: > > println(sizeof(Float64)) > println(sizeof(Int32)) > println(sizeof(Float32)) > println(sizeof(Int16)) > println(sizeof(TestType)) > > I get: > > 8 > 4 > 4 > 2 > 24 > > So it looks like Julia is padding the internal layout of `TestType` (I was > hoping this wasn't the case based on some of the language in: > http://julialang.org/blog/2013/03/efficient-aggregates/). > > If I put dummy fields to mimic the padding when I create some test data, I > can get everything to work just fine on the Julia side. However, if for > real data I assume that I get a file and can't pick packed vs padded, is > there any way on the Julia side to specified that my immutable type's > memory layout should be packed? Or perhaps some other workaround? I > couldn't find any promising leads on this reading the documentation and > searching through the Github issues. In cython this is trivial to do since > you can define an actual struct as `cdef packed struct`. > > I forgot to mention it when I posted originally, but I'm currently using > Julia v0.3.2 on OS X. > > Any suggestions would be appreciated. > > Josh > > > On Friday, November 21, 2014 3:11:32 PM UTC-5, Tim Holy wrote: >> >> You'll see why if you type `methods(mmap_array)`: the dims has to be >> represented as a tuple. >> >> Currently, the only way I know of to create a fixed-sized buffer as an >> element >> of a "struct" in julia is via immutables with one field per object. >> Here's one >> example: >> https://github.com/JuliaGPU/CUDArt.jl/blob/1742a19b35a52ecec4ee14cfbec823 >> f8bcb22e0f/gen/gen_libcudart_h.jl#L403-L660 >> >> It has not escaped notice that this is less than ideal :-). >> >> --Tim >> >> On Friday, November 21, 2014 11:57:10 AM Joshua Adelman wrote: >> > I'm playing around with Julia for the first time in an attempt to see >> if I >> > can replace a Python + Cython component of a system I'm building. >> Basically >> > I have a file of bytes representing a numpy structured/recarray (in >> memory >> > this is an array of structs). This gets memory mapped into a numpy >> array as >> > (Python code): >> > >> > f = open(data_file, 'r+') >> > cmap = mmap.mmap(f.fileno(), nbytes) >> > data_array = np.ndarray(size, dtype=dtype, buffer=cmap) >> > >> > >> > where dtype=[('x', np.int32), ('y', np.float64), ('name', 'S17')]. >> > >> > In cython I would create a C packed struct and to deal with the fixed >> > length string elements, I would specify them as char[N] arrays: >> > >> > cdef packed struct atype: >> > np.int32_t x >> > np.float64 y >> > char[17] name >> > >> > I'm trying to figure out how I would accomplish something similar in >> Julia. >> > Setting aside the issue of the fixed length strings for a moment, I >> thought >> > to initially create a composite type: >> > >> > immutable AType >> > x::Int32 >> > y::Float64 >> > name::??? >> > end >> > >> > and then if I had an file containing 20 records use: >> > >> > f = open("test1.dat", "r") >> > data = mmap_array(AType, 20, f) >> > >> > but I get an error: >> > >> > ERROR: `mmap_array` has no method matching mmap_array(::Type{AType}, >> > >> > ::Int64, ::IOStream) >> > >> > Is there a way to memory map a file into an array of custom >> > records/composite types in Julia? And if there is, how should one >> represent >> > the fixed length string fields? >> > >> > Any suggestions would be much appreciated. >> > >> > Josh >> >>
