[Matplotlib-users] matplotlib figure serialization
Hello, I am a relatively new user of matplotlib; thank you to the matplotlib team for this excellent package. I have a question about serializing matplotlib figures. I have searched for serialization options for matplotlib figures but have not found much information. I am interested to hear about serialization use cases and the approaches others use in these cases. Here is the reason I am asking: My use case for serialization is that I want to build a CouchDB database of matplotlib figures. The database could be accessed from a web application (in my case I want to build a django app to create, edit and manage figures) or desktop gui, or whatever. For storage of the figures in CouchDB, I am working on JSON representations of matplotlib figures. The JSON could be run through simple python functions to regenerate the matplotlib figures. I have very simple working examples, but to more completely test out this approach I would attempt to recreate the plots in the matplotlib gallery using JSON representations and a small set of (hopefully) very simple python functions which would process the JSON markup. Before I get too far, I wanted to see what others have done for similar use cases, make sure I am not missing existing approaches, etc. I am getting ahead of myself now, but if there is broader interest in this approach, and no other better solutions exist, I would set up a project on Google Code or some other site to work on this. Your feedback is very much appreciated. Thanks! Rich -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] matplotlib figure serialization
What is the advantage of JSON (is this specific case) over Python source code? matplotlib is designed around it and it's more flexible. Unless you're planning on automatically manipulating the JSON, I don't see why you wouldn't just use Python source. Mike Rich Krauter wrote: Hello, I am a relatively new user of matplotlib; thank you to the matplotlib team for this excellent package. I have a question about serializing matplotlib figures. I have searched for serialization options for matplotlib figures but have not found much information. I am interested to hear about serialization use cases and the approaches others use in these cases. Here is the reason I am asking: My use case for serialization is that I want to build a CouchDB database of matplotlib figures. The database could be accessed from a web application (in my case I want to build a django app to create, edit and manage figures) or desktop gui, or whatever. For storage of the figures in CouchDB, I am working on JSON representations of matplotlib figures. The JSON could be run through simple python functions to regenerate the matplotlib figures. I have very simple working examples, but to more completely test out this approach I would attempt to recreate the plots in the matplotlib gallery using JSON representations and a small set of (hopefully) very simple python functions which would process the JSON markup. Before I get too far, I wanted to see what others have done for similar use cases, make sure I am not missing existing approaches, etc. I am getting ahead of myself now, but if there is broader interest in this approach, and no other better solutions exist, I would set up a project on Google Code or some other site to work on this. Your feedback is very much appreciated. Thanks! Rich -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users -- Michael Droettboom Science Software Branch Operations and Engineering Division Space Telescope Science Institute Operated by AURA for NASA -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] matplotlib figure serialization
Michael Droettboom wrote: What is the advantage of JSON (is this specific case) over Python source code? matplotlib is designed around it and it's more flexible. Unless you're planning on automatically manipulating the JSON, I don't see why you wouldn't just use Python source. Indeed. There have been a few threads about this topic, and I think the consensus is that the way to auto-generate figures is with python. I don't think that there is any technical reason that one couldn't create a serialized version of an MPL figure in XML, or JSON, (or, for that matter, a python data structure), but it would be a fair bit of effort to write the code, and I don't think you'd get any real advantage over just using scripts -- you need a python script to create a figure in the first place, why not serialize that? -Chris -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/ORR(206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception chris.bar...@noaa.gov -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] matplotlib figure serialization
Rich Krauter wrote: Hello, I am a relatively new user of matplotlib; thank you to the matplotlib team for this excellent package. I have a question about serializing matplotlib figures. I have searched for serialization options for matplotlib figures but have not found much information. I am interested to hear about serialization use cases and the approaches others use in these cases. Here is the reason I am asking: My use case for serialization is that I want to build a CouchDB database of matplotlib figures. The database could be accessed from a web application (in my case I want to build a django app to create, edit and manage figures) or desktop gui, or whatever. For storage of the figures in CouchDB, I am working on JSON representations of matplotlib figures. The JSON could be run through simple python functions to regenerate the matplotlib figures. I have very simple working examples, but to more completely test out this approach I would attempt to recreate the plots in the matplotlib gallery using JSON representations and a small set of (hopefully) very simple python functions which would process the JSON markup. Before I get too far, I wanted to see what others have done for similar use cases, make sure I am not missing existing approaches, etc. I am getting ahead of myself now, but if there is broader interest in this approach, and no other better solutions exist, I would set up a project on Google Code or some other site to work on this. On Wed, Mar 24, 2010 at 1:15 PM, Michael Droettboom md...@stsci.edu wrote: What is the advantage of JSON (is this specific case) over Python source code? matplotlib is designed around it and it's more flexible. Unless you're planning on automatically manipulating the JSON, I don't see why you wouldn't just use Python source. Mike Mike, I don't know that there is much of a benefit to JSON outside of my use case or similar use cases. I want to manipulate the JSON representation of a figure within a javascript-based web interface to provide dynamic plotting through a web page. I also want to be able to store and query JSON representations using CouchDB. I am probably not exactly clear on what you mean by using python source to represent a figure. Is there a standard agreed upon way to do this? I do have python source code representations of figures. i.e. I have dict representations of matplotlib figures. The dicts have a required internal structure. I feed the dict to a function which regenerates the figure graphic from that structure. If I want to update the plot, I just change the contents of the dict data structure representing the plot, not the source code that is used to generate the figure. If I instead had a JSON object representation of a figure, I would convert it to a python dict and use the same function as before to produce the figure. I haven't found much discussion about serialization of matplotlib figures, but I probably have not searched well enough, or maybe it is not a high interest topic. The discussion I have found seems to suggest using the script you used to create the figure as the serialization of that figure. To modify the figure, you modify the script an rerun it. What I would like to have (and what I have some very preliminary examples for) are versioned data structures that can be converted to matplotlib figures without modifying any python source code (other than the structured representation of the figure itself.) However, I don't know how much the matplotlib API changes, and an approach like this may be very sensitive to those changes. Rich -- Download Intel#174; Parallel Studio Eval Try the new software tools for yourself. Speed compiling, find bugs proactively, and fine-tune applications for parallel performance. See why Intel Parallel Studio got high marks during beta. http://p.sf.net/sfu/intel-sw-dev ___ Matplotlib-users mailing list Matplotlib-users@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] matplotlib figure serialization
Rich Krauter wrote: Rich Krauter wrote: Hello, I am a relatively new user of matplotlib; thank you to the matplotlib team for this excellent package. I have a question about serializing matplotlib figures. I have searched for serialization options for matplotlib figures but have not found much information. I am interested to hear about serialization use cases and the approaches others use in these cases. Here is the reason I am asking: My use case for serialization is that I want to build a CouchDB database of matplotlib figures. The database could be accessed from a web application (in my case I want to build a django app to create, edit and manage figures) or desktop gui, or whatever. For storage of the figures in CouchDB, I am working on JSON representations of matplotlib figures. The JSON could be run through simple python functions to regenerate the matplotlib figures. I have very simple working examples, but to more completely test out this approach I would attempt to recreate the plots in the matplotlib gallery using JSON representations and a small set of (hopefully) very simple python functions which would process the JSON markup. Before I get too far, I wanted to see what others have done for similar use cases, make sure I am not missing existing approaches, etc. I am getting ahead of myself now, but if there is broader interest in this approach, and no other better solutions exist, I would set up a project on Google Code or some other site to work on this. On Wed, Mar 24, 2010 at 1:15 PM, Michael Droettboom md...@stsci.edu wrote: What is the advantage of JSON (is this specific case) over Python source code? matplotlib is designed around it and it's more flexible. Unless you're planning on automatically manipulating the JSON, I don't see why you wouldn't just use Python source. Mike Mike, I don't know that there is much of a benefit to JSON outside of my use case or similar use cases. I want to manipulate the JSON representation of a figure within a javascript-based web interface to provide dynamic plotting through a web page. I also want to be able to store and query JSON representations using CouchDB. I am probably not exactly clear on what you mean by using python source to represent a figure. Is there a standard agreed upon way to do this? In general, most matplotlib users write Python scripts to generate their plots. These scripts usually read in data from an external file in any number of formats (the format tends to be domain-specific, but matplotlib provides support for a number of CSV formats, Numpy itself supports a number of ways of reading arrays etc.) matplotlib tends to be agnostic about data (as long as you can convert it to a Numpy array somehow, it's happy), but has a clearly defined API for plot types and styles. I do have python source code representations of figures. i.e. I have dict representations of matplotlib figures. The dicts have a required internal structure. I feed the dict to a function which regenerates the figure graphic from that structure. If I want to update the plot, I just change the contents of the dict data structure representing the plot, not the source code that is used to generate the figure. If I instead had a JSON object representation of a figure, I would convert it to a python dict and use the same function as before to produce the figure. I guess I have trouble seeing why a dictionary representation which is then interpreted to convert it to function calls is better than just making the function calls directly. That's the interface to matplotlib that is known and tested. The only use case I can imagine where a dictionary might be preferable would be if an external tool needs to read in the dictionary, modify it and spit it back out. Reading arbitrary Python code is of course extremely hairy, whereas the JSON dictionary could be defined to be a more limited and manageable subset. Another possible advantage may be security related -- if you need to run untrusted plot code, you certainly don't want to be running untrusted Python code. I haven't found much discussion about serialization of matplotlib figures, but I probably have not searched well enough, or maybe it is not a high interest topic. The discussion I have found seems to suggest using the script you used to create the figure as the serialization of that figure. To modify the figure, you modify the script an rerun it. Yes -- that's the general consensus (at least among the core developers) when the discussion comes up. There have been discussions and experiments using enthought.Traits that might make plots serializable and malleable, but it's a significant refactoring of matplotlib to take such an approach, for a fairly minor gain. It's also extremely difficult to invent a serialization that would survive version upgrades to
Re: [Matplotlib-users] matplotlib figure serialization
On Wed, Mar 24, 2010 at 1:37 PM, Chris Barker chris.bar...@noaa.gov wrote:
Michael Droettboom wrote:
What is the advantage of JSON (is this specific case) over Python source
code? matplotlib is designed around it and it's more flexible. Unless
you're planning on automatically manipulating the JSON, I don't see why
you wouldn't just use Python source.
Indeed. There have been a few threads about this topic, and I think the
consensus is that the way to auto-generate figures is with python.
I don't think that there is any technical reason that one couldn't
create a serialized version of an MPL figure in XML, or JSON, (or, for
that matter, a python data structure), but it would be a fair bit of
effort to write the code, and I don't think you'd get any real advantage
over just using scripts -- you need a python script to create a figure
in the first place, why not serialize that?
Chris,
To answer your question, because I can't think of a way to build a
web-based user interface to let users make incremental changes to the
plot produced by that script. Or some other plot that was generated
using a different script.
ISTM if I have a defined serialization structure (whether it be in
XML, JSON, or a python data structure) I can more easily build a
web-based user interface for manipulating that structure. Below is an
example figure structured as a python dict and a rendering function.
Not sure if this clarifies what I am trying to do ...
import matplotlib
matplotlib.use('Agg')
import matplotlib.pyplot as plt
plot = {
'metadata': {
'description': 'This is a sample plot representation',
'matplotlib_version': '0.99.0',
'author': 'RMK',
'last_updated': [2010, 3, 24, 13, 25, 0],
'type': 'lineplot'
},
'figure':
{'methods':
[
['set_size_inches', [10,4], {} ]
]
},
'axes':
{
121:
{'datasets':[
{
'data': [ [1,2,3], [4,5,6] ],
'options': {'linewidth':4, 'label': 'Source 1'},
},
{
'data': [ [1,2,3], [12,13,14] ],
'options': {'linewidth':4, 'label': 'Source 2',
'marker':'*', 'visible': True},
}
],
'methods': [
['set_xlabel', [Testing ...], {} ],
['legend', [], {} ]
]
},
122:
{ 'datasets': [
{
'data': [ [1,2,3], [7,8,9] ],
'options':{'linewidth':4, 'label': 'Source 3'},
}
],
'methods': [
['set_xlabel', [Label ...], {} ],
['legend', [], {} ]
]
}
}
}
def generate(plot,figname):
fig = plt.figure()
methods = plot['figure']['methods']
for method, args, kwds in methods:
getattr(fig, method)(*args, **kwds)
for axes in plot['axes']:
ax = plt.subplot(axes)
datasets = plot['axes'][axes]['datasets']
for dataset in datasets:
plt.plot(*(dataset['data']), **(dataset['options']))
for method, args, kwds in plot['axes'][axes]['methods']:
getattr(ax, method)(*args, **kwds)
plt.savefig(figname)
if __name__ == '__main__':
generate(plot, 'junk.png')
Rich
--
Download Intel#174; Parallel Studio Eval
Try the new software tools for yourself. Speed compiling, find bugs
proactively, and fine-tune applications for parallel performance.
See why Intel Parallel Studio got high marks during beta.
http://p.sf.net/sfu/intel-sw-dev
___
Matplotlib-users mailing list
Matplotlib-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/matplotlib-users
Re: [Matplotlib-users] matplotlib figure serialization
On Wed, Mar 24, 2010 at 4:19 PM, Michael Droettboom md...@stsci.edu wrote: Rich Krauter wrote: Rich Krauter wrote: Hello, I am a relatively new user of matplotlib; thank you to the matplotlib team for this excellent package. I have a question about serializing matplotlib figures. I have searched for serialization options for matplotlib figures but have not found much information. I am interested to hear about serialization use cases and the approaches others use in these cases. Here is the reason I am asking: My use case for serialization is that I want to build a CouchDB database of matplotlib figures. The database could be accessed from a web application (in my case I want to build a django app to create, edit and manage figures) or desktop gui, or whatever. For storage of the figures in CouchDB, I am working on JSON representations of matplotlib figures. The JSON could be run through simple python functions to regenerate the matplotlib figures. I have very simple working examples, but to more completely test out this approach I would attempt to recreate the plots in the matplotlib gallery using JSON representations and a small set of (hopefully) very simple python functions which would process the JSON markup. Before I get too far, I wanted to see what others have done for similar use cases, make sure I am not missing existing approaches, etc. I am getting ahead of myself now, but if there is broader interest in this approach, and no other better solutions exist, I would set up a project on Google Code or some other site to work on this. On Wed, Mar 24, 2010 at 1:15 PM, Michael Droettboom md...@stsci.edu wrote: What is the advantage of JSON (is this specific case) over Python source code? matplotlib is designed around it and it's more flexible. Unless you're planning on automatically manipulating the JSON, I don't see why you wouldn't just use Python source. Mike Mike, I don't know that there is much of a benefit to JSON outside of my use case or similar use cases. I want to manipulate the JSON representation of a figure within a javascript-based web interface to provide dynamic plotting through a web page. I also want to be able to store and query JSON representations using CouchDB. I am probably not exactly clear on what you mean by using python source to represent a figure. Is there a standard agreed upon way to do this? In general, most matplotlib users write Python scripts to generate their plots. These scripts usually read in data from an external file in any number of formats (the format tends to be domain-specific, but matplotlib provides support for a number of CSV formats, Numpy itself supports a number of ways of reading arrays etc.) matplotlib tends to be agnostic about data (as long as you can convert it to a Numpy array somehow, it's happy), but has a clearly defined API for plot types and styles. I do have python source code representations of figures. i.e. I have dict representations of matplotlib figures. The dicts have a required internal structure. I feed the dict to a function which regenerates the figure graphic from that structure. If I want to update the plot, I just change the contents of the dict data structure representing the plot, not the source code that is used to generate the figure. If I instead had a JSON object representation of a figure, I would convert it to a python dict and use the same function as before to produce the figure. I guess I have trouble seeing why a dictionary representation which is then interpreted to convert it to function calls is better than just making the function calls directly. That's the interface to matplotlib that is known and tested. Here are my reasons why a structured representation (dict, JSON, XML, ...) is useful: - I want to access the same plot representation through both python and through javascript. I need to access it in python to run MPL and create plot images, and I want to use javascript to build the user interface. - I want to separate the plot content from the plot generation. I can serialize a data structure containing plot contents more easily than I can record the commands a user might call to generate a plot. The content of the plot is not python specific, only the generation of the MPL plot is. I need to be able to serialize the content to support later modifications. The only use case I can imagine where a dictionary might be preferable would be if an external tool needs to read in the dictionary, modify it and spit it back out. Reading arbitrary Python code is of course extremely hairy, whereas the JSON dictionary could be defined to be a more limited and manageable subset. Another possible advantage may be security related -- if you need to run untrusted plot code, you certainly don't want to be running untrusted Python code. I haven't found much discussion about serialization of matplotlib
