[bug #23017] Multidimensional numpy arrays are not being stored as IEEE 754 arrays in the XML state and results files.
URL:
http://gna.org/bugs/?23017
Summary: Multidimensional numpy arrays are not being stored
as IEEE 754 arrays in the XML state and results files.
Project: relax
Submitted by: bugman
Submitted on: Tue 02 Dec 2014 07:52:23 PM UTC
Category: relax's source code
Specific analysis category: All analyses
Priority: 5 - Normal
Severity: 3 - Normal
Status: None
Assigned to: bugman
Originator Name:
Originator Email:
Open/Closed: Open
Release: 3.3.3
Discussion Lock: Any
Operating System: All systems
___
Details:
The following relax code demonstrates the problem:
from numpy import array
pipe.create('test', 'mf')
cdp.test = array([1, 2, 3.])
cdp.test2 = array([[1, 2, 3.], [4, 5, 6]])
state.save('test', force=True)
Opening the test.bz2 file, the XML for the numpy arrays is:
test type=dtype('float64')
value
array([ 1., 2., 3.])
/value
ieee_754_byte_array
[[0, 0, 0, 0, 0, 0, 240, 63], [0, 0, 0, 0, 0, 0, 0, 64],
[0, 0, 0, 0, 0, 0, 8, 64]]
/ieee_754_byte_array
/test
test2 type=dtype('float64')
value
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
/value
/test2
The test2 tag should also be represented with the non-truncating
ieee_754_byte_array tag.
___
Reply to this item at:
http://gna.org/bugs/?23017
___
Message sent via/by Gna!
http://gna.org/
___
relax (http://www.nmr-relax.com)
This is the relax-devel mailing list
relax-devel@gna.org
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel
Re: Implement handles for nmrglue in relax
Hi Troels,
I have just fixed the numarray storage in the relax data store. This
was actually a regression, as multidimensional numpy arrays were
previously stored as IEEE 754 byte arrays
(https://gna.org/bugs/?23017). I've now fixed it in the trunk, so it
can be ported to your nmrglue branch using the svnmerge.py script.
Then your spectral data storage should be of much higher quality and
not suffer from floating point truncation artefacts.
Regards,
Edward
On 2 December 2014 at 20:20, Edward d'Auvergne edw...@nmr-relax.com wrote:
Hi Jonathan,
Thank you for your reply to our public mailing list. It can take a
while to appear (http://news.gmane.org/gmane.science.nmr.relax.devel),
as the mailing list maintainer, Chris MacRaild, is in a very different
time zone
(http://www.monash.edu.au/pharm/research/researchers/profile.html?sid=47804pid=5391).
Those not signed up to the list need to wait for the maintainer to
accept the message. Anyway, I'm sure Troels will expand on this
relax-nmrglue topic in the nmrglue forums. For your points, please
see below:
A few points regarding nmrglue in this discussion. The main nmrglue
website is www.nmrglue.com, links to that site are best in any documentation
or credits. The source code and issue tracker are on GitHub at
https://github.com/jjhelmus/nmrglue.
The relax mailing lists (relax-announce, relax-users, relax-devel, and
relax-commits, see http://www.nmr-relax.com/communication.html) are
archived locally, with Gmane (http://gmane.org/), The Mail Archive
(https://www.mail-archive.com/), and MARC (http://marc.info/). So the
nmrglue links will be amplified. This should really help nmrglue pop
up higher in search engine results. Crosslinking can further improve
the rankings. Anyway, Troels is giving you plenty of links, see the
recent changes by Troels at
http://news.gmane.org/gmane.science.nmr.relax.scm
(http://article.gmane.org/gmane.science.nmr.relax.scm/24567 for
example).
The representation of NMR data is in NumPy multi-dimenstional ndarray
objects [1]. These can be serialized to strings using the .tostring() or
.tobytes() methods. Spectral metadata is stored in a Python dictionary,
there are a number of options for serializing these. Also both objects can
be pickled.
Troels, the relax data store XML representation can handle numpy
ndarrays as is. In relax, try:
relax from numpy import array
relax pipe.create('numpy_test', 'mf')
relax cdp.test = array([[1, 2, 3.], [4, 5, 6]])
relax state.save('test')
The 'test.bz2' XML file can then be loaded via state.load without
problem and cdp.test will be the correct numpy array. Note there is a
truncation artefact present, as the XML looks like:
[snip]
pipe desc=The contents of a relax data pipe name=numpy_test
type=mf
global desc=Global data located in the top level of the data pipe
test type=dtype('float64')
value
array([[ 1., 2., 3.],
[ 4., 5., 6.]])
/value
/test
/global
[snip]
Ideally I would modify the data store to_xml() methods use the numpy
tobytes() methods and then store proper byte arrays using Gary
Thompson's float.py module to store the byte arrays, such as we
currently do with floats:
relax pipe.create('numpy_test', 'mf')
relax cdp.test = 1.23456
relax state.save('test', force=True)
Which gives:
test type=float
value
1.23456
/value
ieee_754_byte_array
[56, 50, 143, 252, 193, 192, 243, 63]
/ieee_754_byte_array
/test
The value is just there for users to read. On the other hand, the
loading of the state file reads the data from the
ieee_754_byte_array XML tag. Tell me if you have any truncation
artefact problems when saving states with spectral nmrglue data, and
I'll look into this full precision float support. Due to horrible
compatibility problems, non-truncating pickling (Python or numpy) is
not an option.
Python 3 support is available in the development version of nmrglue,
0.5-dev, available on GitHub. There is an issue with installing the package
in a Python 3 environment which I need to work out but then I plan or
releasing version 0.5 which will be the first release to support Python 3.
Expect it out and available on PyPI by the end of the week.
I've been watching this. There are still a lot of type() calls that
need to be replaced by isinstance(). And the unfortunate set()
function which rightfully no longer exists. The following command
picks up a lot of the problems:
$ 2to3 -j 4 -w -f buffer -f idioms -f set_literal -f ws_comma -x
except -x import -x imports -x long -x numliterals -x xrange .
If you need help, have a look at the relax compat.py module
(http://svn.gna.org/viewcvs/relax/trunk/lib/compat.py?view=log,
Re: Implement handles for nmrglue in relax
I uploaded the nmrglue logo in a number of formats including a SVG and PS from which high-res raster images can be created to https://github.com/jjhelmus/nmrglue_logo. Feel free to include these or files generated from these in relax. - Jonathan Helmus On 12/01/2014 12:37 PM, Jonathan Helmus wrote: Edward and Troels, A few points regarding nmrglue in this discussion. The main nmrglue website is www.nmrglue.com, links to that site are best in any documentation or credits. The source code and issue tracker are on GitHub at https://github.com/jjhelmus/nmrglue. The representation of NMR data is in NumPy multi-dimenstional ndarray objects [1]. These can be serialized to strings using the .tostring() or .tobytes() methods. Spectral metadata is stored in a Python dictionary, there are a number of options for serializing these. Also both objects can be pickled. Python 3 support is available in the development version of nmrglue, 0.5-dev, available on GitHub. There is an issue with installing the package in a Python 3 environment which I need to work out but then I plan or releasing version 0.5 which will be the first release to support Python 3. Expect it out and available on PyPI by the end of the week. I have an SVG file of the nmrglue logo from which the small jpg was created. I'll need to find where it is stored on my computer at home but I'll send it along with a higher resolution png/jpg once I locate it. Best of luck on getting relax to play nicely with nmrglue. If there is anything I can do to help on the nmrglue side feel free to ask. nmrglue is BSD licensed and I would like to keep it that way, if you send any code my way I'd appreciate if it could be licensed under the same terms. Cheers, - Jonathan Helmus [1] http://docs.scipy.org/doc/numpy/reference/generated/numpy.ndarray.html On 11/27/2014 07:30 AM, Edward d'Auvergne wrote: Hi Troels, This appears to be the main nmrglue website http://www.nmrglue.com/ . I can see here that there is an nmrglue icon (http://www.nmrglue.com/images/nmrglue_logo.jpg). This may be useful for the relax GUI graphics for the spectrum.nmrglue_* user function menu entries and wizard pages. The wizard graphics (in the graphics/wizard directory) have a width of 200 px though. You should ask on the nmrglue forum (http://groups.google.com/group/nmrglue-discuss) about the use of such a graphic and if there is a higher resolution version. Cheers, Edward On 27 November 2014 at 12:16, Edward d'Auvergne edw...@nmr-relax.com wrote: Hi, This sounds good. And rather simple, the backend of the spectrum.nmrglue_read user function could simply be the single line of the tutorial http://nmrglue.readthedocs.org/en/latest/tutorial.html#reading-nmr-files: import nmrglue as ng dic,data = ng.pipe.read(test.fid) For the *.ft2 files, how big are they? I wonder if you can downsample the original *.fid files so that the *.ft2 files are within 10 to 100 kB (using the nmrPipe extract command, etc.). Anyway, just add a system test, and we can then discuss more of the design before implementing the nmrglue interface. From your aims, it sounds pretty straightforward. If the number of relax spectrum.nmrglue_* user functions expand significantly to access more of the nmrglue functionality, maybe we can create a dedicated nmrglue user function class later on. Anyway, we just have to make sure that relax continues to operate smoothly when the scipy/matplotlib/nmrglue packages are absent. One problem will be storage of the nmrglue data. This will probably not fit into the relax data store, as it absolutely requires to_xml() and from_xml() methods to allow the data to be saved and read again. Any ideas? We could create a special class in lib.spectrum.nmrglue which uses the singleton design pattern (https://en.wikipedia.org/wiki/Singleton_pattern), as is used extensively in relax. Just 'grep -i' for 'singleton'. The data storage would then be temporary. Or, ideally, we could modify the nmrglue data structures to be convertible to and from XML. Some ideas should be discussed with the nmrglue authors though (which is why I have CCed Jonathan, so he knows we are developing a simple relax-nmrglue interface). At the moment, I don't know much about the nmrglue designs. For the user functions, do you know if nmrglue has an icon that we can use for the menus and user function wizard pages in the GUI? Cheers, Edward On 27 November 2014 at 11:54, Troels Emtekær Linnet tlin...@gmail.com wrote: Hi Edward. To: test_suite/shared_data/dispersion/repeated_analysis/SOD1/cpmg_disp_sod1d90a_060521/cpmg_disp_sod1d90a_060521_normal.fid/ft2_data I will add the corresponding .ft2 files. To limit size use, I will add just very few .ft2 files. I will add wrapper functions which will: - read a ft2 file - read a range of ft2 files according to a glob of file names - plot the spectrum manipulate the numpy arrray,
