Hi Troels,
If you need help with any of this XML stuff, just say. It's a rather
different concept, but the code itself is not complex. Adding classes
to the relax data store simply involves using the base classes with
to_xml() and from_xml() methods in data_store.data_classes, and then
adding something to the pipe or spin container from_xml() method to
recreate the class in the data store.
The base64 functions are insanely quick. Try the following script on
your branch to see:
"""
import base64
import numpy as np
pipe.create(pipe_name='mf', pipe_type='mf', bundle=None)
spectrum.nmrglue_read(file='freq_real.ft2',
dir='/data/relax/branches/nmrglue/extern/nmrglue/nmrglue_0_4/tests/pipe_proc_tests',
nmrglue_id='test')
a = cdp.nmrglue_data['test']
s = base64.b64encode(a)
file = open('test', 'w')
file.write(s)
r = base64.decodestring(s)
q=np.frombuffer(r,dtype=np.float64)
"""
The same logic would be used to output the string 's' into the XML
file, and convert it back. This creates a file of 8.0 Mb for a total
time of ~1 second (with half of this spent on the
spectrum.nmrglue_read user function).
Regards,
Edward
On 8 December 2014 at 10:11, Edward d'Auvergne <edw...@nmr-relax.com> wrote:
> Here is the Python documentation for this:
>
> https://docs.python.org/2/library/base64.html
>
> Regards,
>
> Edward
>
>
>
> On 8 December 2014 at 10:09, Edward d'Auvergne <edw...@nmr-relax.com> wrote:
>> Hi Troels,
>>
>> This is a standard problem of using text files and XML for universal
>> compatibility. However built into the XML format is the ability to
>> store any text between the tags. For example in this case we probably
>> need Base64:
>>
>> http://www.xml.com/pub/a/98/07/binary/binary.html
>> https://commons.apache.org/proper/commons-codec/apidocs/org/apache/commons/codec/binary/Base64.html
>>
>> This is many orders of magnitude quicker. This is the text format
>> used for all binary email attachments, for example. The only problem
>> here is converting the nmrglue numpy format to Base64, and back again.
>> Maybe you could ask Jonathan about it at
>> https://groups.google.com/forum/#!forum/nmrglue-discuss ? Although a
>> quick Google search finds this very simple Python solution:
>>
>> https://stackoverflow.com/questions/6485790/numpy-array-to-base64-and-back-to-numpy-array-python
>>
>> The relax data store to_xml() and from_xml() methods would need to be
>> modified to catch this data type. The to_xml() would have to set the
>> XML type attribute to 'base64', or '%s, base64'%repr(value.dtype).
>> See lib.xml.object_to_xml(). And the from_xml() method modified to
>> catch this and decode the Base64 string. This will require shifting
>> the lib.spectrum.object.Assignment class into
>> data_store.nmrglue.Assignment and then writing a custom to_xml()
>> method (maybe by a helper function in lib.xml). The decoding of the
>> Base64 numpy ndarray data could be part of the general function in
>> lib.xml.xml_to_object(). We should not make this general for all
>> numpy data, as the relax data store Python and XML structures are
>> designed for power users to be able read, and to be able to parse and
>> manipulate. However Base64 can be used for certain large data
>> structures for speed and size efficiency, with nmrglue data being the
>> prime example.
>>
>> Regards,
>>
>> Edward
>>
>>
>> On 7 December 2014 at 23:43, <tlin...@nmr-relax.com> wrote:
>>> Author: tlinnet
>>> Date: Sun Dec 7 23:43:06 2014
>>> New Revision: 27007
>>>
>>> URL: http://svn.gna.org/viewcvs/relax?rev=27007&view=rev
>>> Log:
>>> Change to systemtest Nmrglue.test_save_state.
>>>
>>> This now tries to save a 8MB numpy array to a state file.
>>> The numpy array has shape 512x4096.
>>>
>>> numpy.save store the array instantly, while saving the state, hungs up the
>>> computer for a long time.
>>>
>>> This shows that relax is really in-efficient in storing data.
>>> This both accounts for simple numpy array, and for storing class instances.
>>>
>>> Task #7873 (https://gna.org/task/index.php?7873): Write wrapper function to
>>> nmrglue, to read .ft2 files and process them.
>>> Homepage: http://www.nmrglue.com/
>>> Link to nmrglue discussion:
>>> https://groups.google.com/forum/#!forum/nmrglue-discuss
>>> The code is develop at Github: https://github.com/jjhelmus/nmrglue/
>>> Documentation: http://nmrglue.readthedocs.org/en/latest/index.html
>>>
>>> Modified:
>>> branches/nmrglue/test_suite/system_tests/nmrglue.py
>>>
>>> Modified: branches/nmrglue/test_suite/system_tests/nmrglue.py
>>> URL:
>>> http://svn.gna.org/viewcvs/relax/branches/nmrglue/test_suite/system_tests/nmrglue.py?rev=27007&r1=27006&r2=27007&view=diff
>>> ==============================================================================
>>> --- branches/nmrglue/test_suite/system_tests/nmrglue.py (original)
>>> +++ branches/nmrglue/test_suite/system_tests/nmrglue.py Sun Dec 7 23:43:06
>>> 2014
>>> @@ -373,7 +373,7 @@
>>> print("Filesize of .npy file is: %i"%(data_numpy_size) )
>>>
>>> # Delete the large data array, for faster saving.
>>> - cdp.nmrglue_data[sp_id] = 0
>>> + #cdp.nmrglue_data[sp_id] = 0
>>>
>>> # Store the directory path, before reset of the controller.
>>> dirpath = ds.tmpdir
>>>
>>>
>>> _______________________________________________
>>> relax (http://www.nmr-relax.com)
>>>
>>> This is the relax-commits mailing list
>>> relax-comm...@gna.org
>>>
>>> To unsubscribe from this list, get a password
>>> reminder, or change your subscription options,
>>> visit the list information page at
>>> https://mail.gna.org/listinfo/relax-commits
_______________________________________________
relax (http://www.nmr-relax.com)
This is the relax-devel mailing list
relax-devel@gna.org
To unsubscribe from this list, get a password
reminder, or change your subscription options,
visit the list information page at
https://mail.gna.org/listinfo/relax-devel
