Re: Most Pythonic way to store (small) configuration
On Thursday 06 August 2015 10:07, Chris Angelico wrote:
On Thu, Aug 6, 2015 at 9:43 AM, Tim Chase python.l...@tim.thechases.com
wrote:
Significant whitespace? Not usually simple (just stuck touching a
project where someone committed with tons of trailing whitespaces.
grumble), so strip 'em off as if they're an error condition. I've
never had a config-file where I wanted leading/trailing whitespace as
significant.
If you're configuring a prompt, sometimes you need to be able to
include a space at the end of it.
Sometimes? What sort of filthy perv doesn't separate their prompts from
the user input with at least one space? Disgusting, I call it.
this is a prompt and this isn't
versus
this is a promptand this isn't
Come the revolution, anyone who writes the second will be taken out and
shot.
The right solution to that is to have the display function add a space to
the end of the prompt if there isn't already one, that way you don't need to
care about putting a space at the end of the prompt yourself.
Since trailing whitespace on a line
in the file itself is a bad idea, you need some way of marking it.
I'm partial to one of two schemes:
- strings need to be quoted, so leading and trailing spaces are easy:
key = this has both leading and trailing spaces
- strings don't need to be quoted, and spaces are significant *except*
at the ends of the string:
key = this has no leading or trailing spaces
If you need spaces at the end, use an escape sequence:
key = \sthis has both leading and trailing spaces\s
say, or ^SP or \N{SPACE} or whatever floats your boat. Unless your
requirements are limited to only printable ASCII strings, you're going to
need to provide some sort of escape sequence anyway, to allow (say)
newlines.
--
Steve
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Thu, Aug 6, 2015 at 5:33 PM, Steven D'Aprano
steve+comp.lang.pyt...@pearwood.info wrote:
On Thursday 06 August 2015 10:07, Chris Angelico wrote:
On Thu, Aug 6, 2015 at 9:43 AM, Tim Chase python.l...@tim.thechases.com
wrote:
Significant whitespace? Not usually simple (just stuck touching a
project where someone committed with tons of trailing whitespaces.
grumble), so strip 'em off as if they're an error condition. I've
never had a config-file where I wanted leading/trailing whitespace as
significant.
If you're configuring a prompt, sometimes you need to be able to
include a space at the end of it.
Sometimes? What sort of filthy perv doesn't separate their prompts from
the user input with at least one space? Disgusting, I call it.
this is a prompt and this isn't
versus
this is a promptand this isn't
Come the revolution, anyone who writes the second will be taken out and
shot.
Oh, well, there are these people in backwards nations that still have
prompts that look like this:
C:\
and have no space after them. I *was* trying to be courteous, but
maybe courtesy is wasted on those who'll be shot come the revolution
anyway.
The right solution to that is to have the display function add a space to
the end of the prompt if there isn't already one, that way you don't need to
care about putting a space at the end of the prompt yourself.
That's an option that solves one specific instance of the problem, but
I'm sure there are other places where you may or may not want a
trailing space, so it needs a marker.
Since trailing whitespace on a line
in the file itself is a bad idea, you need some way of marking it.
I'm partial to one of two schemes:
- strings need to be quoted, so leading and trailing spaces are easy:
key = this has both leading and trailing spaces
Yeah, this is fine for a lot of cases. It does add small complexity,
though, to the simple case where you're setting simple tokens as
values.
- strings don't need to be quoted, and spaces are significant *except*
at the ends of the string:
key = this has no leading or trailing spaces
If you need spaces at the end, use an escape sequence:
key = \sthis has both leading and trailing spaces\s
say, or ^SP or \N{SPACE} or whatever floats your boat. Unless your
requirements are limited to only printable ASCII strings, you're going to
need to provide some sort of escape sequence anyway, to allow (say)
newlines.
This is starting to get toward a full-blown DSL, which is costly. At
this point, you may as well go for something well-known, which is what
this thread's all about.
A compromise might be: Unquoted strings get trimmed and are restricted
to one line, but if you want leading/trailing spaces, or
leading/trailing quote characters, or embedded newlines, or Unicode
escapes, you must first put quotes around the string.
server = 127.0.0.1
message = Hi there!\nTwo line message being delivered.
windows_path = c:\users\adam
That way, you pay the complexity cost of full string parsing only if
you actually need it.
ChrisA
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Rustom Mody rustompm...@gmail.com writes: Does yaml have comments? Yes, the same syntax as Python's. ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wednesday 05 August 2015 05:59, Ben Finney wrote: marco.naw...@colosso.nl writes: Why not use Python files itself as configuration files? Because configuration data will be user-editable. (If it's not user-editable, that is itself a poor design choice.) If you allow executable code to be user-edited, that opens your program to arbitrary injection of executable code. Your program becomes wide open for security exploits, whether through malicious or accidental bugs, and simple human error can lead to arbitrary-scope damage to the user's system. Yeah... it's almost as if you allowed the user to edit the source code of your application, or even write their own code. And nobody would do that! *wink* I'm not entirely disagreeing, just adding some nuance to the discussion. My own personal feeling is that using code as config is a little disquieting. It's a bit of a code smell. Do you really need that much power just to allow people to set some configuration settings? Using a Turing Complete programming language just to set a bunch of name:value pairs seems to be gross overkill. But on the other hand, config systems do tend to grow in power. People often want conditional and computed settings. Rather than invent your own (buggy) mini-language to allow conf like this: if $PLATFORM = 'Windows' set location = 'C:\The Stuff'; if $PLATFORM = 'Linux' set location = $baselocation + '/thestuff'; using Python seems like a much better idea. Code smell or not, I don't think that Python code as config is that much of a problem, for most applications. Yes, the user can insert arbitrary code into their config file, and have it run... but it's their user account running it, they presumably could just insert that code into a file and run it with python regardless. There's (usually) no security implications. Although I can think of some exceptions. For example, the company I work for has a Linux desktop designed for untrusted, hostile users (prisoners in jail), and using .py files as config would *definitely* not be allowed for that. But I think that's an exceptional case. On another dimension, configuration files specifying the behaviour of the system are much more useful if their format is easily parsed and re-worked by tools the user chooses. Your program should not be the only tool (and Python should not be the only language) that can read and/or write the configuration data with straightfoward data manipulation. Python is an open standard that anyone can use, or write their own should they wish. If you don't like CPython, use Jython or PyPy or MyFirstPythonInterpreter to pass the config data. A better argument may be that *running arbitrary code* should not be required in order to parse a config file. I'm sympathetic to that argument too. But if that's your worry, and you absolutely must read a .py config file, you could write your own stripped down interpreter of a limited subset of the language, and use that. No, it won't be able to determine all the config settings from arbitrary .py files, but it may give you a practical solution which is good enough for what you need. So a complex full-blown programming language like Python is a poor choice for configuration data for that reason, too. Much better to choose a tightly-defined, limited-scope configuration data format that you can be confident any user cannot employ to run arbitrary code. Sure, if your config requirements are met by such a data format. -- Steve -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wed, Aug 5, 2015 at 6:32 PM, Steven D'Aprano steve+comp.lang.pyt...@pearwood.info wrote: My own personal feeling is that using code as config is a little disquieting. It's a bit of a code smell. Do you really need that much power just to allow people to set some configuration settings? Using a Turing Complete programming language just to set a bunch of name:value pairs seems to be gross overkill. But on the other hand, config systems do tend to grow in power. People often want conditional and computed settings. Rather than invent your own (buggy) mini-language to allow conf like this: if $PLATFORM = 'Windows' set location = 'C:\The Stuff'; if $PLATFORM = 'Linux' set location = $baselocation + '/thestuff'; using Python seems like a much better idea. I often have config files where the example is really simple, like this: https://github.com/Rosuav/LetMeKnow/blob/master/keys_sample.py https://github.com/Rosuav/Yosemite/blob/master/config.py but the in-production versions sometimes have a teensy bit more code in them - maybe an if-expression, maybe pulling something from os.environ. In both of those cases, any person editing the config file will also have full power to tamper with the source code, so the security issue doesn't apply; and apart from requiring quotes around string values, it's pretty much the level of simplicity you'd get out of any other config file format. You get comments, you get freedom to add whitespace anywhere you like (or not add it, if you so desire), it's great! Right? Well, there is another subtlety, and that's that there's no error checking. If you duplicate an entry, nobody will issue even a warning. For small config files, that's fine; but what if you're editing something the size of postgresql.conf? You can't keep it all on screen or in your head, and if you simply search the file for some keyword, would you notice that there are two of them? So, I'd recommend this if - and ONLY if - the files are normally going to be small enough to fit on a single page. Otherwise, consider carefully whether you'd later on want to add error checking like that, and consider how hard it would be to hack that in. Drink responsibly. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 2015-08-02 12:11, Cecil Westerhof wrote:
There are a lot of ways to store configuration information:
- conf file
- xml file
- database
- json file
- and possible a lot of other ways
I want to write a Python program to display cleaned log files. I do
not think I need a lot of configuration to be stored:
- some things relating to the GUI
- default behaviour
- default directory
- log files to display, including some info
- At least until where it was displayed
Because of this I think a human readable file would be best.
Yet another mostly-built-in option is to just have a simple file of
key/value pairs, optionally with comments. This can be read with
something like
config = {}
with open('config.ini') as f:
for row in f:
row = row.strip()
if not row or row.startswith(('#', ';')):
continue
k, _, v = row.partition('=')
config[k.strip().upper()] = v.lstrip()
which is pretty straight-forward and easy format to edit.
-tkc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wednesday, August 5, 2015 at 6:58:01 PM UTC+5:30, Tim Chase wrote:
On 2015-08-02 12:11, Cecil Westerhof wrote:
There are a lot of ways to store configuration information:
- conf file
- xml file
- database
- json file
- and possible a lot of other ways
I want to write a Python program to display cleaned log files. I do
not think I need a lot of configuration to be stored:
- some things relating to the GUI
- default behaviour
- default directory
- log files to display, including some info
- At least until where it was displayed
Because of this I think a human readable file would be best.
Yet another mostly-built-in option is to just have a simple file of
key/value pairs, optionally with comments. This can be read with
something like
config = {}
with open('config.ini') as f:
for row in f:
row = row.strip()
if not row or row.startswith(('#', ';')):
continue
k, _, v = row.partition('=')
config[k.strip().upper()] = v.lstrip()
which is pretty straight-forward and easy format to edit.
-tkc
JSON handles basic types like this:
from json import loads
loads({anInt:1, aString:2})
{'aString': '2', 'anInt': 1}
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Sunday, August 2, 2015 at 3:44:51 PM UTC+5:30, Cecil Westerhof wrote: There are a lot of ways to store configuration information: - conf file - xml file - database - json file - and possible a lot of other ways One that I dont think has been mentioned: ast.literal_eval -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 2015-08-05, Michael Torrie torr...@gmail.com wrote: On 08/04/2015 01:59 PM, Ben Finney wrote: marco.naw...@colosso.nl writes: Why not use Python files itself as configuration files? Because configuration data will be user-editable. (If it's not user-editable, that is itself a poor design choice.) If you allow executable code to be user-edited, that opens your program to arbitrary injection of executable code. Your program becomes wide open for security exploits, whether through malicious or accidental bugs, and simple human error can lead to arbitrary-scope damage to the user's system. We need to state the context here. The only context in which having a Python config file is dangerous is when the python program runs as a different user/privilege than the owner of the config file. If the user owns the python files as well as the config file then none of this matters. Yes, it does. We're not just talking about intentional, malicious damange, we're also talking about _accidental_ damage caused by an incorrect edit of a configuration files. It's much harder to cause damage by mis-editing an ini format file that's parsed with the config file library than it is by mis-editing a Python file that's imported. -- Grant Edwards grant.b.edwardsYow! Clear the laundromat!! at This whirl-o-matic just had gmail.coma nuclear meltdown!! -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wed, 5 Aug 2015 11:46 pm, Rustom Mody wrote: On Sunday, August 2, 2015 at 3:44:51 PM UTC+5:30, Cecil Westerhof wrote: There are a lot of ways to store configuration information: - conf file - xml file - database - json file - and possible a lot of other ways One that I dont think has been mentioned: ast.literal_eval Probably because it doesn't work :-) py import ast py s = x = 23 # pretend I read this line from a file py ast.literal_eval(s) Traceback (most recent call last): File stdin, line 1, in module File /usr/local/lib/python3.3/ast.py, line 47, in literal_eval node_or_string = parse(node_or_string, mode='eval') File /usr/local/lib/python3.3/ast.py, line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File unknown, line 1 x = 23 ^ SyntaxError: invalid syntax You might be able to build Yet Another Config Format using literal_eval as a building block, for evaluating values, but *in and of itself* it isn't a way to store config information, any more than int float or any other function which takes a string and evaluates it as a Python primitive or built-in type. However, there are at least config formats in the standard library which I believe we've missed: shelve, and plistlib. help(shelve) A shelf is a persistent, dictionary-like object. The difference with dbm databases is that the values (not the keys!) in a shelf can be essentially arbitrary Python objects -- anything that the pickle module can handle. This includes most class instances, recursive data types, and objects containing lots of shared sub-objects. The keys are ordinary strings. help(plistlib) The property list (.plist) file format is a simple XML pickle supporting basic object types, like dictionaries, lists, numbers and strings. Usually the top level object is a dictionary. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wednesday, August 5, 2015 at 7:38:46 PM UTC+5:30, Steven D'Aprano wrote:
On Wed, 5 Aug 2015 11:46 pm, Rustom Mody wrote:
On Sunday, August 2, 2015 at 3:44:51 PM UTC+5:30, Cecil Westerhof wrote:
There are a lot of ways to store configuration information:
- conf file
- xml file
- database
- json file
- and possible a lot of other ways
One that I dont think has been mentioned:
ast.literal_eval
Probably because it doesn't work :-)
In the way you describe... yes
Works alright if you give it *literals*
from ast import literal_eval
literal_eval('{x:hello, y:2, z:3.142}')
{'z': 3.142, 'x': 'hello', 'y': 2}
which is identical the the json.loads behavior
loads('{x:hello, y:2, z:3.142}')
{'z': 3.142, 'x': 'hello', 'y': 2}
of course the natural method of use would of course get the string from
the open-ing of a config file -- something along the lines
configuration = literal_eval(open(~/.fooconfig))
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 2015-08-05 06:37, Rustom Mody wrote:
config = {}
with open('config.ini') as f:
for row in f:
row = row.strip()
if not row or row.startswith(('#', ';')):
continue
k, _, v = row.partition('=')
config[k.strip().upper()] = v.lstrip()
which is pretty straight-forward and easy format to edit.
-tkc
JSON handles basic types like this:
from json import loads
loads({anInt:1, aString:2})
{'aString': '2', 'anInt': 1}
But requires the person hand-editing the file to make sure that
opening braces close, that quoted text is properly opened/closed, has
peculiarities regarding things following back-slashes, etc.
There's a certain simplicity to simply having key/value pairs
separated by an = and then letting the application do whatever it
needs/wants with those key/value strings.
-tkc
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Tim Chase python.l...@tim.thechases.com: There's a certain simplicity to simply having key/value pairs separated by an = and then letting the application do whatever it needs/wants with those key/value strings. That trap has lured in a lot of wildlife. What to do with lists? Is whitespace significant? Case in point, systemd configuration files: URL: http://www.freedesktop.org/software/systemd/man/systemd.servic e.html#Command%20lines It specifies all kinds of application-do-whatever-it-needs syntax: * backslash escapes * double quotes and single quotes * line continuations * percent specifiers * dollar substitution with or without braces * double-dollar escape * together with undocumented quirks (for example, how do you specify a command whose pathname contains whitespace?) All of which makes it all but impossible to algorithmically escape a literal command into the ExecStart= entry. When you start with something that's too simple, you end up with layers of ever-increasing complexity but never attain the expressive power of JSON, S-expressions and the like (I don't have the stomach to mention XML). Marko -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Thursday, August 6, 2015 at 6:32:03 AM UTC+5:30, Rustom Mody wrote: By contrast here is a more friendly error message (had put a comma where a colon required) Traceback (most recent call last): File stdin, line 1, in module File /usr/lib/python3.4/ast.py, line 46, in literal_eval node_or_string = parse(node_or_string, mode='eval') File /usr/lib/python3.4/ast.py, line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File unknown, line 2 i, 1} Uh... The more helpfulness not evident as the most crucial line not cut-pasted Heres the full bt Traceback (most recent call last): File stdin, line 1, in module File /usr/lib/python3.4/ast.py, line 46, in literal_eval node_or_string = parse(node_or_string, mode='eval') File /usr/lib/python3.4/ast.py, line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File unknown, line 2 i, 1} ^ SyntaxError: invalid syntax -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Thursday, August 6, 2015 at 2:31:52 AM UTC+5:30, Tim Chase wrote:
On 2015-08-05 06:37, Rustom Mody wrote:
config = {}
with open('config.ini') as f:
for row in f:
row = row.strip()
if not row or row.startswith(('#', ';')):
continue
k, _, v = row.partition('=')
config[k.strip().upper()] = v.lstrip()
which is pretty straight-forward and easy format to edit.
-tkc
JSON handles basic types like this:
from json import loads
loads({anInt:1, aString:2})
{'aString': '2', 'anInt': 1}
But requires the person hand-editing the file to make sure that
opening braces close, that quoted text is properly opened/closed, has
peculiarities regarding things following back-slashes, etc.
There's a certain simplicity to simply having key/value pairs
separated by an = and then letting the application do whatever it
needs/wants with those key/value strings.
-tkc
I just checked that literal_eval accepts comments.
So thats one plus for that.
However I must admit that in checking that out I was faced with more than
(I) expected unfriendly error messages like
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/lib/python3.4/ast.py, line 84, in literal_eval
return _convert(node_or_string)
File /usr/lib/python3.4/ast.py, line 62, in _convert
in zip(node.keys, node.values))
File /usr/lib/python3.4/ast.py, line 61, in genexpr
return dict((_convert(k), _convert(v)) for k, v
File /usr/lib/python3.4/ast.py, line 83, in _convert
raise ValueError('malformed node or string: ' + repr(node))
ValueError: malformed node or string: _ast.Name object at 0x7fe1173ebac8
By contrast here is a more friendly error message (had put a comma where a colon
required)
Traceback (most recent call last):
File stdin, line 1, in module
File /usr/lib/python3.4/ast.py, line 46, in literal_eval
node_or_string = parse(node_or_string, mode='eval')
File /usr/lib/python3.4/ast.py, line 35, in parse
return compile(source, filename, mode, PyCF_ONLY_AST)
File unknown, line 2
i, 1}
So overall whether ast.literal_eval is a good idea is not clear to me
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 2015-08-06 00:47, Marko Rauhamaa wrote: There's a certain simplicity to simply having key/value pairs separated by an = and then letting the application do whatever it needs/wants with those key/value strings. That trap has lured in a lot of wildlife. What to do with lists? Is whitespace significant? As I said, simple. Lists? Not in a simple config format. Though I suppose, just like the .ini format, you could use a convention of comma or space-separated list items. Or quoted white-space-separated items like shlex handles nicely. Or if you really need, pass the value portion of the key/value pair through ast.literal_eval() Significant whitespace? Not usually simple (just stuck touching a project where someone committed with tons of trailing whitespaces. grumble), so strip 'em off as if they're an error condition. I've never had a config-file where I wanted leading/trailing whitespace as significant. Case in point, systemd configuration files: The antithesis of simplicity ;-) -tkc -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Thu, Aug 6, 2015 at 9:43 AM, Tim Chase python.l...@tim.thechases.com wrote: Significant whitespace? Not usually simple (just stuck touching a project where someone committed with tons of trailing whitespaces. grumble), so strip 'em off as if they're an error condition. I've never had a config-file where I wanted leading/trailing whitespace as significant. If you're configuring a prompt, sometimes you need to be able to include a space at the end of it. Since trailing whitespace on a line in the file itself is a bad idea, you need some way of marking it. That might mean quoting the string, or having a Unicode or byte escape like \x20 that means space, or something like that. If you define that spaces around your equals sign are insignificant, you need the same sort of system to cope with the possibility of actual leading whitespace, too. Case in point, systemd configuration files: The antithesis of simplicity ;-) Ehh... I reckon they're pretty simple. They're somewhat more powerful than Upstart config files, and pay some complexity cost for that; but they're a lot simpler than sysvinit config files (which are actually shell scripts), especially with all the dependency handling cruft that goes into encoded comments. Frankly, I do my best to avoid ever touching those. I'm not sure what else to compare them against (never used any init system other than the three just mentioned), so I don't see systemd files as being particularly complex. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
- Original Message - From: Cecil Westerhof ce...@decebal.nl To: python-list@python.org Sent: Sunday, 2 August, 2015 12:11:28 PM Subject: Most Pythonic way to store (small) configuration There are a lot of ways to store configuration information: - conf file - xml file - database - json file - and possible a lot of other ways I want to write a Python program to display cleaned log files. I do not think I need a lot of configuration to be stored: - some things relating to the GUI - default behaviour - default directory - log files to display, including some info - At least until where it was displayed Because of this I think a human readable file would be best. Personally I do not find XML very readable. So a conf or json file looks the most promising to me. And I would have a slight preference for a json file. Any comments, thoughts or tips? -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list Did you consider using serpent ? It's a python serializer than can produce human readable text format. If your configuration is a dictionary for instance, there's possibly no work to do. see for yourself at : https://pypi.python.org/pypi/serpent Regards, JM -- IMPORTANT NOTICE: The contents of this email and any attachments are confidential and may also be privileged. If you are not the intended recipient, please notify the sender immediately and do not disclose the contents to any other person, use it for any purpose, or store or copy the information in any medium. Thank you. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Sunday, August 2, 2015 at 12:14:51 PM UTC+2, Cecil Westerhof wrote:
There are a lot of ways to store configuration information:
- conf file
- xml file
- database
- json file
- and possible a lot of other ways
I want to write a Python program to display cleaned log files. I do
not think I need a lot of configuration to be stored:
- some things relating to the GUI
- default behaviour
- default directory
- log files to display, including some info
- At least until where it was displayed
Because of this I think a human readable file would be best.
Personally I do not find XML very readable. So a conf or json file
looks the most promising to me. And I would have a slight preference
for a json file.
Any comments, thoughts or tips?
--
Cecil Westerhof
Senior Software Engineer
LinkedIn: http://www.linkedin.com/in/cecilwesterhof
Why not use Python files itself as configuration files?
I have done so several times and I am very pleased with
it. I have tried both INI and JSON, but in the end always
settled on using Python itself. If I have to share the
configuration information with other languages (like C++
or Javascript) I use JSON as the
For those who are interested, I provide a short summary
below on how I handle configuration files.
The scenario I want to support is that there is a hierarchical
set of configuration files (as typically found in Unix like
systems) that can be stored in a system wide folder (like
/etc/app), in a user folder (like /home/user/.config/app) and
in a local folder.
Each configuration file has a fixed name, not necessarily ending
in .py. So for example, the name of the configuration file
is config.myapp.
Next I start looking for the existence of these files and once
I find them I start merging all the configuration settings into
one Python dictionary. The function that I use can be found below.
(you need proper import statements, but this is standard Python).
def load_config(config_file, checked=False):
'''Small utility function to load external configuration files.
'''
if checked is False:
if not isfile(config_file) or islink(config_file):
msg = 'Config file %s does not exist.'
LOGGER.warning(msg)
return {}
# Load the module object; we use the imp.load_source function because
# the module configuration typically has a non-standard file extension
# (by default the file is called config.myapp)
module_name = 'm%s' % str(uuid.uuid1()).replace('-', '')
# Prevent bytecode generation for configuration files
sys.dont_write_bytecode = True
config_module = imp.load_source(module_name, config_file)
# Continue with bytecode generation for other normal modules
sys.dont_write_bytecode = False
if not hasattr(config_module, '__config__'):
msg = 'Missing __config__ attribute in configuration file.'
LOGGER.warning(msg)
return {}
else:
return config_module.__config__
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Dear Cecil, I subscribed to late to answer to your top-post in that thread. I had the same topic for myself in the last months and tried a lot of things. In your situation I would prefere the INI-file format, too. But doen't user 'configparser' for that. As other fellows described it's a bad solution because option order and comments are not preserved when writing back. It is a quite incomplete implementation of human-readable-config files. Use 'configobj'[1][2] which can be handled very easy, too. You can have comments and everything you want in there. [1] https://pypi.python.org/pypi/configobj [2] https://github.com/DiffSK/configobj -- -BEGIN PGP PUBLIC KEY BLOCK- Version: GnuPG v1 mQENBFQIluABCACfPwAhRAwFD3NXgv5CtVUGSiqdfJGVViVBqaKd+14E0pASA0MU G0Ewj7O7cGy/ZIoiZ0+lIEZmzJKHfuGwYhXjR/PhnUDrQIHLBvh9WuD6JQuULXfH kXtVm/i9wm76QAcvr2pwYgNzhcJntUHl2GcgnInYbZDeVmg+p9yIPJjuq73/lRS3 0/McgNoFOBhKK/S6STQuFyjr9OyJyYd1shoM3hmy+kg0HYm6OgQBJNg92WV9jwGe GzlipvEp2jpLwVsTxYir2oOPhfd9D1fC9F/l/3gXbfjd5GIIVrZFq2haZmoVeJ33 LJxo3RA5Tf9LoUeels1b4s9kFz6h7+AHERUpABEBAAG0IUNocmlzdGlhbiBCdWh0 eiA8YnVodHpAcG9zdGVvLmRlPokBPgQTAQIAKAUCVAiW4AIbAwUJAeEzgAYLCQgH AwIGFQgCCQoLBBYCAwECHgECF4AACgkQZLsXsAdRqOxNUAf/V/hDA5zGDpySuCEj DhjiVRK74J9Wd8gfH0WAf1Co5HZ24wZH8rgOIVIgXw8rWkOw/VA6xfdfT+64xjTY Fhkpbrk199nDzp72F7Jc4NC+x8xac2e3rK5ifSWhZx7L5A32pGYE+d16m3EEqImK D4gcZl38x9zdUnD4hHyXkIPz1uCfuMuGgWEnaUk4Wbj41CBZr3O0ABue6regV15U jaes8r+B8iCcY+0yP2kse+3iaCaMqNv5FgQZ9+b2Cql8pFkZJVtBVUw4GW3DWZJi du0O/YrC9TgS+xY9ht/MD2qSHwjcK1sdImjqBO7xP8TIOwKeYyDvGKnSO3EJ/sSA UPGEPrkBDQRUCJbgAQgA0k/Qg67CCUJE2/zuxBEoK4wLJpDRJzh8CQPZpjWx8VP0 KL892jwfxymXn8KNhuy1SgCBFSeV9jg4VZNWDlUGJc2lo82ajr9PzIsrQwu4lf0B zrUWV5hWepKu/kb8uSjx58YYfx0SFz4+9akX3Wwu9TUHntzL5Gk3Q26nnsr1xEJ+ VEumvCH9AE0Tk0K7dQpJ2/JcLuO+uhrpd/lHFDYVN5NsG3P015uFOkDI6N/xNFCj v95XNR93QlfKpK3qWlFGescfG+o/7Ub6s67/i/JoNbw0XgPEHmQfXpD7IHO4cu+p +ETb11cz+1mmi96cy98ID+uTiToJ8G//yD9rmtyxoQARAQABiQElBBgBAgAPBQJU CJbgAhsMBQkB4TOAAAoJEGS7F7AHUajs6sQH/iKs6sPc0vkRJLfbwrijZeecwCWF blo/jzIQ8jPykAj9SLjV20Xwqg3XcJyko8ZU6/zuRJq9xjlv9pZr/oVudQAt6v+h 2Cf4rKEjmau483wjMV2xjTXQhZi9+ttDbia4fgdmGtKsOicn5ae2fFXcXNPu3RiW sZKifWdokA6xqMW6iIG9YjjI5ShxngHWp2xfPscBFMDRtFOMags/Yx+YvwoyEZ4A dURYMFHFqpwILEc8hIzhRg1gq40AHbOaEdczS1Rr3T7/gS6eBs4u6HuY5g2Bierm lLjpspFPjMXwJAa/XLOBjMF2vsHPrZNcouNKkumQ36yq/Pm6DFXAseQDxOk= =PGP9 -END PGP PUBLIC KEY BLOCK- -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 08/04/2015 08:44 PM, random...@fastmail.us wrote: On Tue, Aug 4, 2015, at 21:32, Michael Torrie wrote: In many of my projects I put basic config variables in a file like config.py and import that in each module that needs it. The config module doubles as a global namespace for sharing between modules as well. What about JSONP? That is, a file consisting exactly of config_data = [JSON object] That would get you some of the benefits of having your config file exist as a python module, but still allow it to be examined by other tools, written out, etc. But I don't need it to be examined by other tools. So the added complication of yet another layer isn't worth it or needed. Python's syntax is simple enough that a person with a text editor can certainly do it. Again, context is everything. My programs are written for mostly my own use. If I was writing a system that was like, say, Apache, I would certainly do a DSL with a robust error checking and reporting system that could clearly help people with syntax errors, etc. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 08/04/2015 08:19 PM, Cameron Simpson wrote: So on the whole I am against python code as the config file format. Really, who needs a Turing complete configuration file? In Django's case, since you're intimately referring to certain classes and methods, particularly in the url mapping section, I think using straight Python is the way to go. It's the most flexibility for the least amount of work. And someone working with Django already has familiarity with both Django and Python. I'm sure everything could be put in a DSL, but then you'd be inventing a subset of Python anyway. And like I say, since the same person editing the config file is also editing the other Django project files, the issue of code injection is moot; he's already doing that! So for the OP's case, if his needs are simple like mine are, then a simple python file he imports could be just fine. I'm hard pressed to make a case for something different if the needs are simple like that. Go with something simple, both in syntax and semantics. It is easier to read, easier to write/modify, and easier to access from multiple tools. Python as configuration is simpler for my purposes, in both ways. And multiple tools for me == text editors. I have no need of machine-generated config files. Maybe Cecil doesn't either. Such tools are of limited use anyway in my experience. For example webadmin never cut it for me. Neither did the old web-based Samba configurater. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Wednesday, August 5, 2015 at 10:19:11 AM UTC+5:30, Michael Torrie wrote: On 08/04/2015 08:44 PM, wrote: On Tue, Aug 4, 2015, at 21:32, Michael Torrie wrote: In many of my projects I put basic config variables in a file like config.py and import that in each module that needs it. The config module doubles as a global namespace for sharing between modules as well. What about JSONP? That is, a file consisting exactly of config_data = [JSON object] That would get you some of the benefits of having your config file exist as a python module, but still allow it to be examined by other tools, written out, etc. But I don't need it to be examined by other tools. So the added complication of yet another layer isn't worth it or needed. Python's syntax is simple enough that a person with a text editor can certainly do it. Again, context is everything. My programs are written for mostly my own use. If I was writing a system that was like, say, Apache, I would certainly do a DSL with a robust error checking and reporting system that could clearly help people with syntax errors, etc. I think the main point is Cameron's So on the whole I am against python code as the config file format. Really, who needs a Turing complete configuration file? stated more strongly: I sure dont want to use something Turing complete from something that is inherently more trivial. And if the cost is of matching the trivial format to the trivial-izing reader (say json) it seems like a small price to pay. In the past Ive always recommended yaml even though the cost is higher -- a separate install [yeah Ive had clients tell me that yaml's out because of that] [Does yaml have comments?] -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Tue, Aug 4, 2015, at 21:32, Michael Torrie wrote: In many of my projects I put basic config variables in a file like config.py and import that in each module that needs it. The config module doubles as a global namespace for sharing between modules as well. What about JSONP? That is, a file consisting exactly of config_data = [JSON object] That would get you some of the benefits of having your config file exist as a python module, but still allow it to be examined by other tools, written out, etc. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 08/04/2015 01:59 PM, Ben Finney wrote: marco.naw...@colosso.nl writes: Why not use Python files itself as configuration files? Because configuration data will be user-editable. (If it's not user-editable, that is itself a poor design choice.) If you allow executable code to be user-edited, that opens your program to arbitrary injection of executable code. Your program becomes wide open for security exploits, whether through malicious or accidental bugs, and simple human error can lead to arbitrary-scope damage to the user's system. We need to state the context here. The only context in which having a Python config file is dangerous is when the python program runs as a different user/privilege than the owner of the config file. If the user owns the python files as well as the config file then none of this matters. In most cases, I've never bought the argument you and others are making here about security and demanding yet another DSL. In fact I find the argument to be rather circular in that we're dealing with programs that aren't compiled but written in Python anyway. I can open and edit any python file in the project that I want and make arbitrary, possibly malicious changes to it! Oh no! The idea that a malicious user could inject python code in this instance and somehow deliberately harm the system is kind of silly if you think about it. It's me that's running the python code in the first place. I could open any file and change it. I'm already running arbitrary code. If I'm talking about a system service that is doing things for non-root users, then yeah I'll agree with your argument completely. But not for most other situations. Even a system service, if the config file is owned by root, I'm okay with using python as configuration. Because if root's compromised, all bets are off anyway and all the python scripts could be modified. In fact python as configuration works very well for my purposes, and it works well for Django, and there are many other projects that also do this. I don't think you'd want to use any other mechanism for configuring Django, frankly. I'm a bit surprised no one has mentioned Django in this discussion yet. In many of my projects I put basic config variables in a file like config.py and import that in each module that needs it. The config module doubles as a global namespace for sharing between modules as well. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 4-8-2015 16:53, marco.naw...@colosso.nl wrote: Why not use Python files itself as configuration files? It could create a security risk if the config files are user-editable. (it will make it easy to inject code into your application) Irmen -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Tuesday, August 4, 2015 at 7:06:33 PM UTC+2, Irmen de Jong wrote: On 4-8-2015 16:53, marco.naw...@colosso.nl wrote: Why not use Python files itself as configuration files? It could create a security risk if the config files are user-editable. (it will make it easy to inject code into your application) Irmen Yes, I am aware of the security risk, but right now I am not too worried about it. To be honest, JSON would be my preferred configuration file format, but as others already mentioned, there is no standard way of adding comments to a JSON file. Marco -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
marco.naw...@colosso.nl writes: Why not use Python files itself as configuration files? Because configuration data will be user-editable. (If it's not user-editable, that is itself a poor design choice.) If you allow executable code to be user-edited, that opens your program to arbitrary injection of executable code. Your program becomes wide open for security exploits, whether through malicious or accidental bugs, and simple human error can lead to arbitrary-scope damage to the user's system. On another dimension, configuration files specifying the behaviour of the system are much more useful if their format is easily parsed and re-worked by tools the user chooses. Your program should not be the only tool (and Python should not be the only language) that can read and/or write the configuration data with straightfoward data manipulation. So a complex full-blown programming language like Python is a poor choice for configuration data for that reason, too. Much better to choose a tightly-defined, limited-scope configuration data format that you can be confident any user cannot employ to run arbitrary code. -- \ “For myself, I am an optimist — it does not seem to be much use | `\ being anything else.” —Winston Churchill, 1954-11-09 | _o__) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 04Aug2015 19:32, Michael Torrie torr...@gmail.com wrote: On 08/04/2015 01:59 PM, Ben Finney wrote: marco.naw...@colosso.nl writes: Why not use Python files itself as configuration files? Because configuration data will be user-editable. (If it's not user-editable, that is itself a poor design choice.) If you allow executable code to be user-edited, that opens your program to arbitrary injection of executable code. [...] We need to state the context here. The only context in which having a Python config file is dangerous is when the python program runs as a different user/privilege than the owner of the config file. If the user owns the python files as well as the config file then none of this matters. [...] If I'm talking about a system service that is doing things for non-root users, then yeah I'll agree with your argument completely. But not for most other situations. Even a system service, if the config file is owned by root, I'm okay with using python as configuration. [...] In fact python as configuration works very well for my purposes, and it works well for Django, and there are many other projects that also do this. I don't think you'd want to use any other mechanism for configuring Django, frankly. I'm a bit surprised no one has mentioned Django in this discussion yet. [...] Django crossed my mind as well, but my take is very different. I'm of two minds on Django's use of Python files for config. On the one hand it is very flexible, it can express comoplicated/subtle(/fragile?) setups as needs and, if you are writing the Django app instead of just using it, it is in the same language as the app. On the other hand, if you want to consult that config from _outside_ the app with another tool not necessarily written in Python you're screwed. Many things can read a .ini file. Very few things can read arbitrary Python code, and any that can _must_ expose themselves to whatever that code may do. If the consulting code is in a different security context (system config monitors, admin scripts, etc) then we're back in the risk zone. So on the whole I am against python code as the config file format. Really, who needs a Turing complete configuration file? Go with something simple, both in syntax and semantics. It is easier to read, easier to write/modify, and easier to access from multiple tools. Cheers, Cameron Simpson c...@zip.com.au A protocol is complete when there is nothing left to remove. - Stuart Cheshire, Apple Computer, regarding Zero Configuration Networking -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 03/08/2015 14:38, Steven D'Aprano wrote: On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote: Well, I have at least some non-zero chance of reading and writing JSON or XML by hand. Can the same be said for a sqlite database? Real programmers edit their SQL databases directly on the hard drive platter using a magnetised needle and a steady hand. I'd stick with SQLiteStudio or something similar if I were you. The last time I tried the above, having consumed 10 bottles of Newcastle Brown, the results were disastrous. I pricked my finger. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote: Well, I have at least some non-zero chance of reading and writing JSON or XML by hand. Can the same be said for a sqlite database? Real programmers edit their SQL databases directly on the hard drive platter using a magnetised needle and a steady hand. -- Steven -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Mon, Aug 3, 2015 at 11:38 PM, Steven D'Aprano st...@pearwood.info wrote: On Mon, 3 Aug 2015 02:02 pm, Dan Sommers wrote: Well, I have at least some non-zero chance of reading and writing JSON or XML by hand. Can the same be said for a sqlite database? Real programmers edit their SQL databases directly on the hard drive platter using a magnetised needle and a steady hand. REAL programmers use emacs, with the C-x M-x M-butterfly command. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Sun, 02 Aug 2015 16:11:14 -0500, Tim Chase wrote: On 2015-08-02 21:54, Ben Finney wrote: So, both XML and JSON should be considered write-only, and produced only for consumption by a computer; they are a poor choice for presenting to a human. [snip] I second Ben's thoughts against XML JSON -- they *can* be edited by hand, but put the onus on the user to make perfect XML/JSON. Config files (.ini) are more forgiving. [snip] An additional option is using a sqlite database. The sqlite library is part of the stdlib, and advantages include being a single file, expandability, consistent/reliable character encoding, cross-platform portability, and atomicity (utilities that read/write are blocked from getting/creating incomplete data seen by the other file). Well, I have at least some non-zero chance of reading and writing JSON or XML by hand. Can the same be said for a sqlite database? ;-) -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 02/08/2015 12:54, Ben Finney wrote: Cecil Westerhof ce...@decebal.nl writes: Because of this I think a human readable file would be best. The “INI” format as handled by the Python ‘configparser’ module is what I would recommend for a simple flat configuration file. It is more intuitive to edit, and has a conventional commenting format. +1 -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Sunday 2 Aug 2015 13:54 CEST, Ben Finney wrote: Cecil Westerhof ce...@decebal.nl writes: Because of this I think a human readable file would be best. I agree with that criterion; in the absence of compelling reasons otherwise, human-readable and -editable text is a good default. Personally I do not find XML very readable. So a conf or json file looks the most promising to me. And I would have a slight preference for a json file. XML and JSON should both be considered data serialisation formats only. JSON is human-readable to an extent, but it is quite brittle, and there are no comments permitted in the syntax. So, both XML and JSON should be considered write-only, and produced only for consumption by a computer; they are a poor choice for presenting to a human. The “INI” format as handled by the Python ‘configparser’ module is what I would recommend for a simple flat configuration file. It is more intuitive to edit, and has a conventional commenting format. Well, I would use nested data. (A file will have extra fields besides the name.) That is why I was thinking about json. But I will look into it. -- Cecil Westerhof Senior Software Engineer LinkedIn: http://www.linkedin.com/in/cecilwesterhof -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On Sun, Aug 2, 2015 at 8:11 PM, Cecil Westerhof ce...@decebal.nl wrote: Because of this I think a human readable file would be best. Personally I do not find XML very readable. So a conf or json file looks the most promising to me. And I would have a slight preference for a json file. Any comments, thoughts or tips? I'd agree with your analysis. XML is not readable; the two best options would be JSON, if you need the potential for deep structure, or the simple config file (INI file) if you don't. ChrisA -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Cecil Westerhof ce...@decebal.nl writes: Because of this I think a human readable file would be best. I agree with that criterion; in the absence of compelling reasons otherwise, human-readable and -editable text is a good default. Personally I do not find XML very readable. So a conf or json file looks the most promising to me. And I would have a slight preference for a json file. XML and JSON should both be considered data serialisation formats only. JSON is human-readable to an extent, but it is quite brittle, and there are no comments permitted in the syntax. So, both XML and JSON should be considered write-only, and produced only for consumption by a computer; they are a poor choice for presenting to a human. The “INI” format as handled by the Python ‘configparser’ module is what I would recommend for a simple flat configuration file. It is more intuitive to edit, and has a conventional commenting format. If it were more complex I might recommend YAML, which is a very widespread, flexible hierarchical format, that still allows fairly natural editing and commenting. Its main downside (as opposed to INI format) is that the Python standard library does not support it. -- \ “Simplicity and elegance are unpopular because they require | `\ hard work and discipline to achieve and education to be | _o__)appreciated.” —Edsger W. Dijkstra | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 02Aug2015 18:51, Cecil Westerhof ce...@decebal.nl wrote:
On Sunday 2 Aug 2015 13:54 CEST, Ben Finney wrote:
Cecil Westerhof ce...@decebal.nl writes:
Because of this I think a human readable file would be best.
I agree with that criterion; in the absence of compelling reasons
otherwise, human-readable and -editable text is a good default.
[...]
The “INI” format as handled by the Python ‘configparser’ module is
what I would recommend for a simple flat configuration file. It is
more intuitive to edit, and has a conventional commenting format.
Well, I would use nested data. (A file will have extra fields besides
the name.) That is why I was thinking about json. But I will look into
it.
Like others, I also recommend an INI file. You can always move to something
more complex (and harder to edit) if it doesn't work out.
Note that nested data does not rule out an INI file unless you mean
recursive data or quite a deep nesting of structure. Shallow structure (one
or two levels) is very easy to embed in the INI format.
Eg:
[main]
this = that
size = 10
directory = /path/to/default/dir
[gui]
foreground = green
style = baroque
[*.log]
stuff about log files...
[/path/to/file1]
title = something
param1 = 5
param2 = 6
fields = column2, column7
[column2:/path/to/file1]
format = numeric
scale = logarithmic
[column7:/path/to/file1]
format = roman
scale = linear
See that by being a little tricky about the section names you can incorporate a
fair degree of structure? I would not advocating going too fair down that
rabbit hole, but for basic stuff it works well.
And of course you can put structure within a section:
[foo]
complexity = {'blah': 'blah blah', 'frib': {'frob': 1, 'frab': 2}}
format_width = 10
format_style = right-justified
format_dialect = quoted
i.e. you could put (small) JSON snippet in some fields, or present a few
parameters to describe format.
There are several ways to do this kind of thing before reaching for more
complex syntaxes. Just don't let it get too weird - that would probably be a
sign you do want to reach for JSON or YAML.
But start with INI. Simple and easy.
Cheers,
Cameron Simpson c...@zip.com.au
A squealing tire is a happy tire.
- Bruce MacInnes, Skip Barber Driving School instructor
--
https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Cecil Westerhof ce...@decebal.nl writes: Well, I would use nested data. (A file will have extra fields besides the name.) That is why I was thinking about json. But I will look into it. An alternative, very similar to JSON but with some good cherries picked from YAML is AXON, which is my preferite these days for this kind of things. http://intellimath.bitbucket.org/axon/ ciao, lele. -- nickname: Lele Gaifax | Quando vivrò di quello che ho pensato ieri real: Emanuele Gaifas | comincerò ad aver paura di chi mi copia. l...@metapensiero.it | -- Fortunato Depero, 1929. -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
Cecil Westerhof ce...@decebal.nl writes: On Sunday 2 Aug 2015 13:54 CEST, Ben Finney wrote: So, both XML and JSON should be considered write-only, and produced only for consumption by a computer; they are a poor choice for presenting to a human. Well, I would use nested data. (A file will have extra fields besides the name.) In that case, your needs are more complex than “store some simple configuration”. You need a human-editable format (so, not JSON and not XML) which handles structured data well, and is widely implemented. For nested configuration data, you would be better served by YAML URL:https://en.wikipedia.org/wiki/YAML, and the PyYAML library URL:https://pypi.python.org/pypi/PyYAML/ is the one to use. -- \學而不思則罔，思而不學則殆。 | `\(To study and not think is a waste. To think and not study | _o__) is dangerous.) —孔夫子 Confucius (551 BCE – 479 BCE) | Ben Finney -- https://mail.python.org/mailman/listinfo/python-list
Re: Most Pythonic way to store (small) configuration
On 2015-08-02 21:54, Ben Finney wrote: So, both XML and JSON should be considered write-only, and produced only for consumption by a computer; they are a poor choice for presenting to a human. The “INI” format as handled by the Python ‘configparser’ module is what I would recommend for a simple flat configuration file. It is more intuitive to edit, and has a conventional commenting format. I second Ben's thoughts against XML JSON -- they *can* be edited by hand, but put the onus on the user to make perfect XML/JSON. Config files (.ini) are more forgiving. However, the .ini format (or at least the stdlib implementation in ConfigParser.py) is not without its faults, mostly when you read a file, then write it back out: - comments and blank lines get lost in the process: [section] # set to local configuration location=path/to/foo will get written out as [section] location=path/to/foo - the order of options is not preserved: [section] thing=1 other=2 may get written back out as [section] other=2 thing=1 though this has improved once ConfigParser started attempting to use an OrderedDict by default for internal storage. - a single key can only appear once in a section: [section] option=one option=two gets written back out as [section] option=two - implicit encoding (is it UTF-8, Latin-1, etc?) When you understand that the underlying internal storage is a dict (ordered or unordered, depending on availability), a lot of the above makes sense. But it still makes me wish for the power of git's config-file format that seems to preserve original config files much better. An additional option is using a sqlite database. The sqlite library is part of the stdlib, and advantages include being a single file, expandability, consistent/reliable character encoding, cross-platform portability, and atomicity (utilities that read/write are blocked from getting/creating incomplete data seen by the other file). -tkc -- https://mail.python.org/mailman/listinfo/python-list
