Re: [julia-users] Re: Help reading structured binary data files
Ohhh... shiny! This looks really nice, thanks.
It's certainly fast enough for my needs.
On Saturday, October 3, 2015 at 6:26:29 PM UTC-4, Patrick O'Leary wrote:
>
> Going back to StrPack, there's syntax for that:
>
> @struct type SomeThings
> anInt::Int32
> aVectorWithSixElements::Vector{Int32}(6)
> aStringOfEightBytes::ASCIIString(8)
> end
>
> On Friday, October 2, 2015 at 9:01:29 AM UTC-5, David McInnis wrote:
>>
>> @Tom : I couldn't figure out how to do arrays or strings of a specific
>> number of bytes.
>>
>>
>> On Thursday, October 1, 2015 at 6:41:30 PM UTC-4, Tom Breloff wrote:
>>>
>>> This is exactly what my packedStruct macro is for... define the type
>>> (composed of bitstypes) and the macro will set up helper methods to read
>>> raw bytes into the fields. I'm curious if it suits your needs, and what is
>>> missing.
>>>
>>> On Thu, Oct 1, 2015 at 3:39 PM, David McInnis
>>> wrote:
>>>
Related follow-on question..
Is there a nice way to get the data into a type format rather than a
dict ?
Here's the form I'm using now..
function DNP(myfile::String)
dnp = { "File" => myfile }
fh = open(myfile, "r")
dnp["Label"] = bytestring(readbytes(fh, 4))
dnp["Version"] = read(fh, Uint32)
dnp["Revision"] = read(fh, Uint32)
dnp["Date"] = bytestring(readbytes(fh, 28))
dnp["FileFormat"] = read(fh, Uint32)
dnp["FileType"] = bytestring(readbytes(fh,4))
dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
dnp["Annotate"] = bytestring(readbytes(fh,84))
dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
dnp["LaserWavelengthMicrons"] = read(fh, Float64)
dnp["LaserNullDoubling"] = read(fh, Uint32)
dnp["Padding"] = read(fh, Uint32)
dnp["DispersionConstantXc"] = read(fh, Float64)
dnp["DispersionConstantXm"] = read(fh, Float64)
dnp["DispersionConstantXb"] = read(fh, Float64)
dnp["NumChan"] = read(fh, Uint32)
dnp["InterferogramSize"] = read(fh, Uint32)
dnp["ScanDirection"] = read(fh, Uint32)
dnp["ACQUIREMODE"] = read(fh, Uint32)
dnp["EMISSIWITY"] = read(fh, Uint32)
dnp["APODIZATION"] = read(fh, Uint32)
dnp["ZEROFILL"] = read(fh, Uint32)
dnp["RUNTIMEMATH"] = read(fh, Uint32)
dnp["FFTSize"] = read(fh, Uint32)
dnp["NumberOfCoAdds"] = read(fh, Uint32)
dnp["SingleSided"] = read(fh, Uint32)
dnp["ChanDisplay"] = read(fh, Uint32)
dnp["AmbTemperature"] = read(fh, Float64)
dnp["InstTemperature"] = read(fh, Float64)
dnp["WBBTemperature"] = read(fh, Float64)
dnp["CBBTemperature"] = read(fh, Float64)
dnp["TEMPERATURE_DWR"] = read(fh, Float64)
dnp["EMISSIVITY_DWR"] = read(fh, Float64)
dnp["LaserTemperature"] = read(fh, Float64)
dnp["SpareI"] = read(fh, Uint32,10)
dnp["SpareF"] = read(fh, Float64,10)
dnp["SpareNA"] = bytestring(readbytes(fh,68))
dnp["SpareNB"] = bytestring(readbytes(fh,68))
dnp["SpareNC"] = bytestring(readbytes(fh,68))
dnp["SpareND"] = bytestring(readbytes(fh,68))
dnp["SpareNE"] = bytestring(readbytes(fh,68))
dnp["End"] = bytestring(readbytes(fh,4))
dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
dnp["NumberOfCoAdds"])
fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
dnp["Spectrum"] = read(fh, Float32, fft_size)
close(fh)
wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
spectral_range = wavelength_range / 2
spectral_binsize = spectral_range / fft_size
x_fft = [0:fft_size-1] * spectral_binsize
m = dnp["DispersionConstantXm"]
b = dnp["DispersionConstantXb"]
c = dnp["DispersionConstantXc"]
x_corrected = x_fft + c + exp10(m * x_fft + b)
dnp["WL_cm"] = x_corrected
dnp["WL_microns"] = 1.0 ./ x_corrected
return dnp
end
Which works fine, but it leaves me with the data in a form that's ugly
(to me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
I can create an appropriate type like:
type DNP
Label
Version
Revision
Date
etc etc
end
..but I can't figure out a good way to get the data there.Well,
other than keeping my current function, defining the type, and then having
another function to copy the data into the type... ugly.
I read all the docs I could find on types but never saw
Re: [julia-users] Re: Help reading structured binary data files
On Saturday, October 3, 2015 at 6:38:44 PM UTC-5, Tom Breloff wrote: > > Thanks Patrick. Yes StrPack is the way to go then. My only warning is > that StrPack has a bunch of logic for endianness, etc which slows it down a > little, but for most purposes it should work well. (Also, it might have > changed since I last tried it ~8 months ago, so ymmv) > Trust me, it hasn't changed :D Those are totally valid points; I never really tried to wring out all the performance stuff, and Julia itself has changed a lot since I put it together--but it's not something I see myself getting to anytime soon.
Re: [julia-users] Re: Help reading structured binary data files
Thanks Patrick. Yes StrPack is the way to go then. My only warning is
that StrPack has a bunch of logic for endianness, etc which slows it down a
little, but for most purposes it should work well. (Also, it might have
changed since I last tried it ~8 months ago, so ymmv)
On Sat, Oct 3, 2015 at 5:26 PM, Patrick O'Leary
wrote:
> Going back to StrPack, there's syntax for that:
>
> @struct type SomeThings
> anInt::Int32
> aVectorWithSixElements::Vector{Int32}(6)
> aStringOfEightBytes::ASCIIString(8)
> end
>
>
> On Friday, October 2, 2015 at 9:01:29 AM UTC-5, David McInnis wrote:
>>
>> @Tom : I couldn't figure out how to do arrays or strings of a specific
>> number of bytes.
>>
>>
>> On Thursday, October 1, 2015 at 6:41:30 PM UTC-4, Tom Breloff wrote:
>>>
>>> This is exactly what my packedStruct macro is for... define the type
>>> (composed of bitstypes) and the macro will set up helper methods to read
>>> raw bytes into the fields. I'm curious if it suits your needs, and what is
>>> missing.
>>>
>>> On Thu, Oct 1, 2015 at 3:39 PM, David McInnis
>>> wrote:
>>>
Related follow-on question..
Is there a nice way to get the data into a type format rather than a
dict ?
Here's the form I'm using now..
function DNP(myfile::String)
dnp = { "File" => myfile }
fh = open(myfile, "r")
dnp["Label"] = bytestring(readbytes(fh, 4))
dnp["Version"] = read(fh, Uint32)
dnp["Revision"] = read(fh, Uint32)
dnp["Date"] = bytestring(readbytes(fh, 28))
dnp["FileFormat"] = read(fh, Uint32)
dnp["FileType"] = bytestring(readbytes(fh,4))
dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
dnp["Annotate"] = bytestring(readbytes(fh,84))
dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
dnp["LaserWavelengthMicrons"] = read(fh, Float64)
dnp["LaserNullDoubling"] = read(fh, Uint32)
dnp["Padding"] = read(fh, Uint32)
dnp["DispersionConstantXc"] = read(fh, Float64)
dnp["DispersionConstantXm"] = read(fh, Float64)
dnp["DispersionConstantXb"] = read(fh, Float64)
dnp["NumChan"] = read(fh, Uint32)
dnp["InterferogramSize"] = read(fh, Uint32)
dnp["ScanDirection"] = read(fh, Uint32)
dnp["ACQUIREMODE"] = read(fh, Uint32)
dnp["EMISSIWITY"] = read(fh, Uint32)
dnp["APODIZATION"] = read(fh, Uint32)
dnp["ZEROFILL"] = read(fh, Uint32)
dnp["RUNTIMEMATH"] = read(fh, Uint32)
dnp["FFTSize"] = read(fh, Uint32)
dnp["NumberOfCoAdds"] = read(fh, Uint32)
dnp["SingleSided"] = read(fh, Uint32)
dnp["ChanDisplay"] = read(fh, Uint32)
dnp["AmbTemperature"] = read(fh, Float64)
dnp["InstTemperature"] = read(fh, Float64)
dnp["WBBTemperature"] = read(fh, Float64)
dnp["CBBTemperature"] = read(fh, Float64)
dnp["TEMPERATURE_DWR"] = read(fh, Float64)
dnp["EMISSIVITY_DWR"] = read(fh, Float64)
dnp["LaserTemperature"] = read(fh, Float64)
dnp["SpareI"] = read(fh, Uint32,10)
dnp["SpareF"] = read(fh, Float64,10)
dnp["SpareNA"] = bytestring(readbytes(fh,68))
dnp["SpareNB"] = bytestring(readbytes(fh,68))
dnp["SpareNC"] = bytestring(readbytes(fh,68))
dnp["SpareND"] = bytestring(readbytes(fh,68))
dnp["SpareNE"] = bytestring(readbytes(fh,68))
dnp["End"] = bytestring(readbytes(fh,4))
dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
dnp["NumberOfCoAdds"])
fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
dnp["Spectrum"] = read(fh, Float32, fft_size)
close(fh)
wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
spectral_range = wavelength_range / 2
spectral_binsize = spectral_range / fft_size
x_fft = [0:fft_size-1] * spectral_binsize
m = dnp["DispersionConstantXm"]
b = dnp["DispersionConstantXb"]
c = dnp["DispersionConstantXc"]
x_corrected = x_fft + c + exp10(m * x_fft + b)
dnp["WL_cm"] = x_corrected
dnp["WL_microns"] = 1.0 ./ x_corrected
return dnp
end
Which works fine, but it leaves me with the data in a form that's ugly
(to me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
I can create an appropriate type like:
type DNP
Label
Version
Revision
Date
etc etc
end
..but I can't figure out a good way to get the data there.Well,
other than keeping my current function, defining the type,
Re: [julia-users] Re: Help reading structured binary data files
Going back to StrPack, there's syntax for that:
@struct type SomeThings
anInt::Int32
aVectorWithSixElements::Vector{Int32}(6)
aStringOfEightBytes::ASCIIString(8)
end
On Friday, October 2, 2015 at 9:01:29 AM UTC-5, David McInnis wrote:
>
> @Tom : I couldn't figure out how to do arrays or strings of a specific
> number of bytes.
>
>
> On Thursday, October 1, 2015 at 6:41:30 PM UTC-4, Tom Breloff wrote:
>>
>> This is exactly what my packedStruct macro is for... define the type
>> (composed of bitstypes) and the macro will set up helper methods to read
>> raw bytes into the fields. I'm curious if it suits your needs, and what is
>> missing.
>>
>> On Thu, Oct 1, 2015 at 3:39 PM, David McInnis wrote:
>>
>>> Related follow-on question..
>>>
>>> Is there a nice way to get the data into a type format rather than a
>>> dict ?
>>>
>>> Here's the form I'm using now..
>>> function DNP(myfile::String)
>>>
>>> dnp = { "File" => myfile }
>>> fh = open(myfile, "r")
>>>
>>> dnp["Label"] = bytestring(readbytes(fh, 4))
>>> dnp["Version"] = read(fh, Uint32)
>>> dnp["Revision"] = read(fh, Uint32)
>>> dnp["Date"] = bytestring(readbytes(fh, 28))
>>> dnp["FileFormat"] = read(fh, Uint32)
>>> dnp["FileType"] = bytestring(readbytes(fh,4))
>>> dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
>>> dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
>>> dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
>>> dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
>>> dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
>>> dnp["Annotate"] = bytestring(readbytes(fh,84))
>>> dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
>>> dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
>>> dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
>>> dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
>>> dnp["LaserWavelengthMicrons"] = read(fh, Float64)
>>> dnp["LaserNullDoubling"] = read(fh, Uint32)
>>> dnp["Padding"] = read(fh, Uint32)
>>> dnp["DispersionConstantXc"] = read(fh, Float64)
>>> dnp["DispersionConstantXm"] = read(fh, Float64)
>>> dnp["DispersionConstantXb"] = read(fh, Float64)
>>> dnp["NumChan"] = read(fh, Uint32)
>>> dnp["InterferogramSize"] = read(fh, Uint32)
>>> dnp["ScanDirection"] = read(fh, Uint32)
>>> dnp["ACQUIREMODE"] = read(fh, Uint32)
>>> dnp["EMISSIWITY"] = read(fh, Uint32)
>>> dnp["APODIZATION"] = read(fh, Uint32)
>>> dnp["ZEROFILL"] = read(fh, Uint32)
>>> dnp["RUNTIMEMATH"] = read(fh, Uint32)
>>> dnp["FFTSize"] = read(fh, Uint32)
>>> dnp["NumberOfCoAdds"] = read(fh, Uint32)
>>> dnp["SingleSided"] = read(fh, Uint32)
>>> dnp["ChanDisplay"] = read(fh, Uint32)
>>> dnp["AmbTemperature"] = read(fh, Float64)
>>> dnp["InstTemperature"] = read(fh, Float64)
>>> dnp["WBBTemperature"] = read(fh, Float64)
>>> dnp["CBBTemperature"] = read(fh, Float64)
>>> dnp["TEMPERATURE_DWR"] = read(fh, Float64)
>>> dnp["EMISSIVITY_DWR"] = read(fh, Float64)
>>> dnp["LaserTemperature"] = read(fh, Float64)
>>> dnp["SpareI"] = read(fh, Uint32,10)
>>> dnp["SpareF"] = read(fh, Float64,10)
>>> dnp["SpareNA"] = bytestring(readbytes(fh,68))
>>> dnp["SpareNB"] = bytestring(readbytes(fh,68))
>>> dnp["SpareNC"] = bytestring(readbytes(fh,68))
>>> dnp["SpareND"] = bytestring(readbytes(fh,68))
>>> dnp["SpareNE"] = bytestring(readbytes(fh,68))
>>> dnp["End"] = bytestring(readbytes(fh,4))
>>>
>>>
>>> dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
>>> dnp["NumberOfCoAdds"])
>>> fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
>>> dnp["Spectrum"] = read(fh, Float32, fft_size)
>>>
>>> close(fh)
>>>
>>> wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
>>> spectral_range = wavelength_range / 2
>>> spectral_binsize = spectral_range / fft_size
>>> x_fft = [0:fft_size-1] * spectral_binsize
>>> m = dnp["DispersionConstantXm"]
>>> b = dnp["DispersionConstantXb"]
>>> c = dnp["DispersionConstantXc"]
>>> x_corrected = x_fft + c + exp10(m * x_fft + b)
>>> dnp["WL_cm"] = x_corrected
>>> dnp["WL_microns"] = 1.0 ./ x_corrected
>>>
>>>
>>> return dnp
>>> end
>>>
>>> Which works fine, but it leaves me with the data in a form that's ugly
>>> (to me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
>>> Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
>>>
>>> I can create an appropriate type like:
>>> type DNP
>>> Label
>>> Version
>>> Revision
>>> Date
>>> etc etc
>>> end
>>>
>>> ..but I can't figure out a good way to get the data there.Well,
>>> other than keeping my current function, defining the type, and then having
>>> another function to copy the data into the type... ugly.
>>>
>>> I read all the docs I could find on types but never saw anything that
>>> hinted at a solution..maybe a function/type hybrid??
>>> I tried creating the type within the function but didn't get anywhere.
>>> Ideas?
>>>
>>
>>
Re: [julia-users] Re: Help reading structured binary data files
@Tom : I couldn't figure out how to do arrays or strings of a specific
number of bytes.
On Thursday, October 1, 2015 at 6:41:30 PM UTC-4, Tom Breloff wrote:
>
> This is exactly what my packedStruct macro is for... define the type
> (composed of bitstypes) and the macro will set up helper methods to read
> raw bytes into the fields. I'm curious if it suits your needs, and what is
> missing.
>
> On Thu, Oct 1, 2015 at 3:39 PM, David McInnis > wrote:
>
>> Related follow-on question..
>>
>> Is there a nice way to get the data into a type format rather than a
>> dict ?
>>
>> Here's the form I'm using now..
>> function DNP(myfile::String)
>>
>> dnp = { "File" => myfile }
>> fh = open(myfile, "r")
>>
>> dnp["Label"] = bytestring(readbytes(fh, 4))
>> dnp["Version"] = read(fh, Uint32)
>> dnp["Revision"] = read(fh, Uint32)
>> dnp["Date"] = bytestring(readbytes(fh, 28))
>> dnp["FileFormat"] = read(fh, Uint32)
>> dnp["FileType"] = bytestring(readbytes(fh,4))
>> dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
>> dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
>> dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
>> dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
>> dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
>> dnp["Annotate"] = bytestring(readbytes(fh,84))
>> dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
>> dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
>> dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
>> dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
>> dnp["LaserWavelengthMicrons"] = read(fh, Float64)
>> dnp["LaserNullDoubling"] = read(fh, Uint32)
>> dnp["Padding"] = read(fh, Uint32)
>> dnp["DispersionConstantXc"] = read(fh, Float64)
>> dnp["DispersionConstantXm"] = read(fh, Float64)
>> dnp["DispersionConstantXb"] = read(fh, Float64)
>> dnp["NumChan"] = read(fh, Uint32)
>> dnp["InterferogramSize"] = read(fh, Uint32)
>> dnp["ScanDirection"] = read(fh, Uint32)
>> dnp["ACQUIREMODE"] = read(fh, Uint32)
>> dnp["EMISSIWITY"] = read(fh, Uint32)
>> dnp["APODIZATION"] = read(fh, Uint32)
>> dnp["ZEROFILL"] = read(fh, Uint32)
>> dnp["RUNTIMEMATH"] = read(fh, Uint32)
>> dnp["FFTSize"] = read(fh, Uint32)
>> dnp["NumberOfCoAdds"] = read(fh, Uint32)
>> dnp["SingleSided"] = read(fh, Uint32)
>> dnp["ChanDisplay"] = read(fh, Uint32)
>> dnp["AmbTemperature"] = read(fh, Float64)
>> dnp["InstTemperature"] = read(fh, Float64)
>> dnp["WBBTemperature"] = read(fh, Float64)
>> dnp["CBBTemperature"] = read(fh, Float64)
>> dnp["TEMPERATURE_DWR"] = read(fh, Float64)
>> dnp["EMISSIVITY_DWR"] = read(fh, Float64)
>> dnp["LaserTemperature"] = read(fh, Float64)
>> dnp["SpareI"] = read(fh, Uint32,10)
>> dnp["SpareF"] = read(fh, Float64,10)
>> dnp["SpareNA"] = bytestring(readbytes(fh,68))
>> dnp["SpareNB"] = bytestring(readbytes(fh,68))
>> dnp["SpareNC"] = bytestring(readbytes(fh,68))
>> dnp["SpareND"] = bytestring(readbytes(fh,68))
>> dnp["SpareNE"] = bytestring(readbytes(fh,68))
>> dnp["End"] = bytestring(readbytes(fh,4))
>>
>>
>> dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
>> dnp["NumberOfCoAdds"])
>> fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
>> dnp["Spectrum"] = read(fh, Float32, fft_size)
>>
>> close(fh)
>>
>> wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
>> spectral_range = wavelength_range / 2
>> spectral_binsize = spectral_range / fft_size
>> x_fft = [0:fft_size-1] * spectral_binsize
>> m = dnp["DispersionConstantXm"]
>> b = dnp["DispersionConstantXb"]
>> c = dnp["DispersionConstantXc"]
>> x_corrected = x_fft + c + exp10(m * x_fft + b)
>> dnp["WL_cm"] = x_corrected
>> dnp["WL_microns"] = 1.0 ./ x_corrected
>>
>>
>> return dnp
>> end
>>
>> Which works fine, but it leaves me with the data in a form that's ugly
>> (to me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
>> Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
>>
>> I can create an appropriate type like:
>> type DNP
>> Label
>> Version
>> Revision
>> Date
>> etc etc
>> end
>>
>> ..but I can't figure out a good way to get the data there.Well, other
>> than keeping my current function, defining the type, and then having
>> another function to copy the data into the type... ugly.
>>
>> I read all the docs I could find on types but never saw anything that
>> hinted at a solution..maybe a function/type hybrid??
>> I tried creating the type within the function but didn't get anywhere.
>> Ideas?
>>
>
>
Re: [julia-users] Re: Help reading structured binary data files
This is exactly what my packedStruct macro is for... define the type
(composed of bitstypes) and the macro will set up helper methods to read
raw bytes into the fields. I'm curious if it suits your needs, and what is
missing.
On Thu, Oct 1, 2015 at 3:39 PM, David McInnis wrote:
> Related follow-on question..
>
> Is there a nice way to get the data into a type format rather than a
> dict ?
>
> Here's the form I'm using now..
> function DNP(myfile::String)
>
> dnp = { "File" => myfile }
> fh = open(myfile, "r")
>
> dnp["Label"] = bytestring(readbytes(fh, 4))
> dnp["Version"] = read(fh, Uint32)
> dnp["Revision"] = read(fh, Uint32)
> dnp["Date"] = bytestring(readbytes(fh, 28))
> dnp["FileFormat"] = read(fh, Uint32)
> dnp["FileType"] = bytestring(readbytes(fh,4))
> dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
> dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
> dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
> dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
> dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
> dnp["Annotate"] = bytestring(readbytes(fh,84))
> dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
> dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
> dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
> dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
> dnp["LaserWavelengthMicrons"] = read(fh, Float64)
> dnp["LaserNullDoubling"] = read(fh, Uint32)
> dnp["Padding"] = read(fh, Uint32)
> dnp["DispersionConstantXc"] = read(fh, Float64)
> dnp["DispersionConstantXm"] = read(fh, Float64)
> dnp["DispersionConstantXb"] = read(fh, Float64)
> dnp["NumChan"] = read(fh, Uint32)
> dnp["InterferogramSize"] = read(fh, Uint32)
> dnp["ScanDirection"] = read(fh, Uint32)
> dnp["ACQUIREMODE"] = read(fh, Uint32)
> dnp["EMISSIWITY"] = read(fh, Uint32)
> dnp["APODIZATION"] = read(fh, Uint32)
> dnp["ZEROFILL"] = read(fh, Uint32)
> dnp["RUNTIMEMATH"] = read(fh, Uint32)
> dnp["FFTSize"] = read(fh, Uint32)
> dnp["NumberOfCoAdds"] = read(fh, Uint32)
> dnp["SingleSided"] = read(fh, Uint32)
> dnp["ChanDisplay"] = read(fh, Uint32)
> dnp["AmbTemperature"] = read(fh, Float64)
> dnp["InstTemperature"] = read(fh, Float64)
> dnp["WBBTemperature"] = read(fh, Float64)
> dnp["CBBTemperature"] = read(fh, Float64)
> dnp["TEMPERATURE_DWR"] = read(fh, Float64)
> dnp["EMISSIVITY_DWR"] = read(fh, Float64)
> dnp["LaserTemperature"] = read(fh, Float64)
> dnp["SpareI"] = read(fh, Uint32,10)
> dnp["SpareF"] = read(fh, Float64,10)
> dnp["SpareNA"] = bytestring(readbytes(fh,68))
> dnp["SpareNB"] = bytestring(readbytes(fh,68))
> dnp["SpareNC"] = bytestring(readbytes(fh,68))
> dnp["SpareND"] = bytestring(readbytes(fh,68))
> dnp["SpareNE"] = bytestring(readbytes(fh,68))
> dnp["End"] = bytestring(readbytes(fh,4))
>
>
> dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
> dnp["NumberOfCoAdds"])
> fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
> dnp["Spectrum"] = read(fh, Float32, fft_size)
>
> close(fh)
>
> wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
> spectral_range = wavelength_range / 2
> spectral_binsize = spectral_range / fft_size
> x_fft = [0:fft_size-1] * spectral_binsize
> m = dnp["DispersionConstantXm"]
> b = dnp["DispersionConstantXb"]
> c = dnp["DispersionConstantXc"]
> x_corrected = x_fft + c + exp10(m * x_fft + b)
> dnp["WL_cm"] = x_corrected
> dnp["WL_microns"] = 1.0 ./ x_corrected
>
>
> return dnp
> end
>
> Which works fine, but it leaves me with the data in a form that's ugly (to
> me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
> Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
>
> I can create an appropriate type like:
> type DNP
> Label
> Version
> Revision
> Date
> etc etc
> end
>
> ..but I can't figure out a good way to get the data there.Well, other
> than keeping my current function, defining the type, and then having
> another function to copy the data into the type... ugly.
>
> I read all the docs I could find on types but never saw anything that
> hinted at a solution..maybe a function/type hybrid??
> I tried creating the type within the function but didn't get anywhere.
> Ideas?
>
[julia-users] Re: Help reading structured binary data files
Related follow-on question..
Is there a nice way to get the data into a type format rather than a
dict ?
Here's the form I'm using now..
function DNP(myfile::String)
dnp = { "File" => myfile }
fh = open(myfile, "r")
dnp["Label"] = bytestring(readbytes(fh, 4))
dnp["Version"] = read(fh, Uint32)
dnp["Revision"] = read(fh, Uint32)
dnp["Date"] = bytestring(readbytes(fh, 28))
dnp["FileFormat"] = read(fh, Uint32)
dnp["FileType"] = bytestring(readbytes(fh,4))
dnp["OriginalFileName"] = bytestring(readbytes(fh,68))
dnp["ReferenceFileName"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameA"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameB"] = bytestring(readbytes(fh,68))
dnp["RelatedFileNameC"] = bytestring(readbytes(fh,68))
dnp["Annotate"] = bytestring(readbytes(fh,84))
dnp["InstrumentModel"] = bytestring(readbytes(fh,36))
dnp["InstrumentSerialNumber"] = bytestring(readbytes(fh,36))
dnp["SoftwareVersionNumber"] = bytestring(readbytes(fh,36))
dnp["CrystalMaterial"] = bytestring(readbytes(fh,36))
dnp["LaserWavelengthMicrons"] = read(fh, Float64)
dnp["LaserNullDoubling"] = read(fh, Uint32)
dnp["Padding"] = read(fh, Uint32)
dnp["DispersionConstantXc"] = read(fh, Float64)
dnp["DispersionConstantXm"] = read(fh, Float64)
dnp["DispersionConstantXb"] = read(fh, Float64)
dnp["NumChan"] = read(fh, Uint32)
dnp["InterferogramSize"] = read(fh, Uint32)
dnp["ScanDirection"] = read(fh, Uint32)
dnp["ACQUIREMODE"] = read(fh, Uint32)
dnp["EMISSIWITY"] = read(fh, Uint32)
dnp["APODIZATION"] = read(fh, Uint32)
dnp["ZEROFILL"] = read(fh, Uint32)
dnp["RUNTIMEMATH"] = read(fh, Uint32)
dnp["FFTSize"] = read(fh, Uint32)
dnp["NumberOfCoAdds"] = read(fh, Uint32)
dnp["SingleSided"] = read(fh, Uint32)
dnp["ChanDisplay"] = read(fh, Uint32)
dnp["AmbTemperature"] = read(fh, Float64)
dnp["InstTemperature"] = read(fh, Float64)
dnp["WBBTemperature"] = read(fh, Float64)
dnp["CBBTemperature"] = read(fh, Float64)
dnp["TEMPERATURE_DWR"] = read(fh, Float64)
dnp["EMISSIVITY_DWR"] = read(fh, Float64)
dnp["LaserTemperature"] = read(fh, Float64)
dnp["SpareI"] = read(fh, Uint32,10)
dnp["SpareF"] = read(fh, Float64,10)
dnp["SpareNA"] = bytestring(readbytes(fh,68))
dnp["SpareNB"] = bytestring(readbytes(fh,68))
dnp["SpareNC"] = bytestring(readbytes(fh,68))
dnp["SpareND"] = bytestring(readbytes(fh,68))
dnp["SpareNE"] = bytestring(readbytes(fh,68))
dnp["End"] = bytestring(readbytes(fh,4))
dnp["Interferograms"] = read(fh, Int16, dnp["InterferogramSize"],
dnp["NumberOfCoAdds"])
fft_size = dnp["FFTSize"] * dnp["ZEROFILL"] * 512
dnp["Spectrum"] = read(fh, Float32, fft_size)
close(fh)
wavelength_range = 1.0 / dnp["LaserWavelengthMicrons"]
spectral_range = wavelength_range / 2
spectral_binsize = spectral_range / fft_size
x_fft = [0:fft_size-1] * spectral_binsize
m = dnp["DispersionConstantXm"]
b = dnp["DispersionConstantXb"]
c = dnp["DispersionConstantXc"]
x_corrected = x_fft + c + exp10(m * x_fft + b)
dnp["WL_cm"] = x_corrected
dnp["WL_microns"] = 1.0 ./ x_corrected
return dnp
end
Which works fine, but it leaves me with the data in a form that's ugly (to
me) in calculations: dnp["InterferogramSize"] * dnp["NumberOfCoAdds"]
Instead of:dnp.InterferogramSize * dnp.NumberOfCoAdds
I can create an appropriate type like:
type DNP
Label
Version
Revision
Date
etc etc
end
..but I can't figure out a good way to get the data there.Well, other
than keeping my current function, defining the type, and then having
another function to copy the data into the type... ugly.
I read all the docs I could find on types but never saw anything that
hinted at a solution..maybe a function/type hybrid??
I tried creating the type within the function but didn't get anywhere.
Ideas?
Re: [julia-users] Re: Help reading structured binary data files
Ah, thank you.. that's much nicer.
@Tom and Patrick
I like the idea but don't understand how to specify say a string. ??
field1::Uint16 # makes sense but how to do multiple numbers?
field2::asciistringofXbytes -or- Xbytes and I'll convert it later
whee, a fire alarm.
On Wednesday, September 30, 2015 at 12:54:24 PM UTC-4, Jameson wrote:
>
> For the bitstypes, you can do `[read(fh, UInt16)]` to be a bit more
> concise.
>
>
> On Wed, Sep 30, 2015 at 12:31 PM David McInnis > wrote:
>
>> Sorry for the slow response, was called away.
>>
>> As a starting place I'll try to stick with the builtin routines first.
>> With Stefan's idea I've got something that works though I don't see a way
>> to make it more.. ummm... elegant.
>>
>> Here's where I'm at:
>> myfile = "dnp.sam"
>> dnp = { "File" => myfile }
>>
>> fh = open(myfile, "r")
>>
>> dnp["Label"] = bytestring(readbytes(fh, 4))
>> dnp["Version"] = reinterpret( Uint16, readbytes(fh, 2) )
>> dnp["Revision"] = reinterpret( Uint16, readbytes(fh, 2) )
>> dnp["Date"] = bytestring(readbytes(fh, 28))
>> # and so on for 30 other variables
>>
>> close(fh)
>>
>>
>> Any suggestions?
>>
>> @Tom : I love how clean your code looks.
>> @ : We may be after the same thing.
>>
>>
>>
>>
>> On Thursday, September 17, 2015 at 9:09:15 PM UTC-4, David McInnis wrote:
>>>
>>> I'm in the process of switching from python to julia and have gotten
>>> stuck for a couple of days trying to read a, for me, typical data file.
>>>
>>> In python I'd create a C-style format, open the file and read the data.
>>> I don't see an equivalent method in Julia.
>>>
>>> Ex:
>>> Using a data structure of something like:"<4sii28s4i"
>>> I'd figure out the size of the structure, point to the beginning byte,
>>> and then unpack it.
>>>
>>> In Julia it looks like *maybe* I could make a data type to do this, but
>>> I can't figure out how.
>>> There's also StrPack.jl, but it too is a little beyond what I
>>> understand.
>>>
>>> I work with a lot of different instruments, each with its own file
>>> format. Usually I only need to read these files. After processing I'll
>>> save everything into an hdf5 file.
>>>
>>> Thanks, David.
>>>
>>>
Re: [julia-users] Re: Help reading structured binary data files
For the bitstypes, you can do `[read(fh, UInt16)]` to be a bit more concise.
On Wed, Sep 30, 2015 at 12:31 PM David McInnis wrote:
> Sorry for the slow response, was called away.
>
> As a starting place I'll try to stick with the builtin routines first.
> With Stefan's idea I've got something that works though I don't see a way
> to make it more.. ummm... elegant.
>
> Here's where I'm at:
> myfile = "dnp.sam"
> dnp = { "File" => myfile }
>
> fh = open(myfile, "r")
>
> dnp["Label"] = bytestring(readbytes(fh, 4))
> dnp["Version"] = reinterpret( Uint16, readbytes(fh, 2) )
> dnp["Revision"] = reinterpret( Uint16, readbytes(fh, 2) )
> dnp["Date"] = bytestring(readbytes(fh, 28))
> # and so on for 30 other variables
>
> close(fh)
>
>
> Any suggestions?
>
> @Tom : I love how clean your code looks.
> @ : We may be after the same thing.
>
>
>
>
> On Thursday, September 17, 2015 at 9:09:15 PM UTC-4, David McInnis wrote:
>>
>> I'm in the process of switching from python to julia and have gotten
>> stuck for a couple of days trying to read a, for me, typical data file.
>>
>> In python I'd create a C-style format, open the file and read the data.
>> I don't see an equivalent method in Julia.
>>
>> Ex:
>> Using a data structure of something like:"<4sii28s4i"
>> I'd figure out the size of the structure, point to the beginning byte,
>> and then unpack it.
>>
>> In Julia it looks like *maybe* I could make a data type to do this, but I
>> can't figure out how.
>> There's also StrPack.jl, but it too is a little beyond what I understand.
>>
>> I work with a lot of different instruments, each with its own file
>> format. Usually I only need to read these files. After processing I'll
>> save everything into an hdf5 file.
>>
>> Thanks, David.
>>
>>
[julia-users] Re: Help reading structured binary data files
Sorry for the slow response, was called away.
As a starting place I'll try to stick with the builtin routines first.
With Stefan's idea I've got something that works though I don't see a way
to make it more.. ummm... elegant.
Here's where I'm at:
myfile = "dnp.sam"
dnp = { "File" => myfile }
fh = open(myfile, "r")
dnp["Label"] = bytestring(readbytes(fh, 4))
dnp["Version"] = reinterpret( Uint16, readbytes(fh, 2) )
dnp["Revision"] = reinterpret( Uint16, readbytes(fh, 2) )
dnp["Date"] = bytestring(readbytes(fh, 28))
# and so on for 30 other variables
close(fh)
Any suggestions?
@Tom : I love how clean your code looks.
@ : We may be after the same thing.
On Thursday, September 17, 2015 at 9:09:15 PM UTC-4, David McInnis wrote:
>
> I'm in the process of switching from python to julia and have gotten stuck
> for a couple of days trying to read a, for me, typical data file.
>
> In python I'd create a C-style format, open the file and read the data.
> I don't see an equivalent method in Julia.
>
> Ex:
> Using a data structure of something like:"<4sii28s4i"
> I'd figure out the size of the structure, point to the beginning byte, and
> then unpack it.
>
> In Julia it looks like *maybe* I could make a data type to do this, but I
> can't figure out how.
> There's also StrPack.jl, but it too is a little beyond what I understand.
>
> I work with a lot of different instruments, each with its own file format.
> Usually I only need to read these files. After processing I'll save
> everything into an hdf5 file.
>
> Thanks, David.
>
>
