Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-10 Thread Zefir Kurtisi
David,

I do not disagree, and in fact I was not proposing a new configuration format.

What I tried to point out was
a) maintaining a 'meta-configuration' for some config overlay very quickly 
becomes
very challenging and resource consuming
b) if the format has to be changed, in OpenWRT JSON should be the obvious choice
from a pragmatic point: most of the configuration today already is converted to
JSON at runtime to be passed to the various ubus services.

As for the inherent problem of standardizing a continuously changing
configuration, that's what I described as the 'heaviest part'. Considering that
there are snmp MIBs released decades ago and (with gradual extensions) still 
used
today, difficult but doable.


Anyway, uci is not bad as is generally and does not need a replacement. We have
our JSON config overlay and it is doing ok. Alas, it would be a waste of 
resources
if multiple parties were cooking their proprietary overlay system when everybody
is trying to reach similar goals.


On 03/07/2015 12:00 AM, David Lang wrote:
 One other thought. If there is a desire to change the format, let's pick a 
 format
 that's already in use for configuring system rather than inventing yet another
 one. We should survey the tools that are being written for managing software
 defined datacenters (openstack and similar) and see if we can use one of those
 config languages (either as-is or with an automated conversion back and forth)
 
 David Lang
 
 On Fri, 6 Mar 2015, David Lang wrote:
 
 changing the format doesn't solve the problem, it just causes churn in all 
 the
 software.

 The problem is the lack of standardization and the fact that the way things 
 are
 configured changes over time. Not everything is configured in UCI (and this 
 will
 always be the case because it's not worth trying to change every piece of
 software available)

 When you try to design a standardized config space that will work in the
 future for equipment and software that nobody has imagined yet, you are 
 either
 going to go crazy-complicated to try and cover everything and have people 
 going
 cross-eyed trying to understand the spec, or you are going to lock things 
 down
 to be so ridged that new things will be working around your config spec. The
 best that you can do is to setup a framework to keep the configs from 
 stepping
 on each other and try to converge similar software to use the same terms in 
 the
 config where it makes sense (remember, the newcomer may have a better way of
 looking at the problem, so trying to force it to define it's config the way 
 the
 legacy things did it can cause big problems)

 I'm all for tools to convert the format to whatever you want to use, and (as 
 I
 posted earlier), to have tools to assemble/generate the configs, but 
 requiring
 changes to every piece of software to just change the format requires more 
 than
 every langauage can understand JSON

 David Lang

___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-06 Thread David Lang
changing the format doesn't solve the problem, it just causes churn in all the 
software.


The problem is the lack of standardization and the fact that the way things are 
configured changes over time. Not everything is configured in UCI (and this will 
always be the case because it's not worth trying to change every piece of 
software available)


When you try to design a standardized config space that will work in the 
future for equipment and software that nobody has imagined yet, you are either 
going to go crazy-complicated to try and cover everything and have people going 
cross-eyed trying to understand the spec, or you are going to lock things down 
to be so ridged that new things will be working around your config spec. The 
best that you can do is to setup a framework to keep the configs from stepping 
on each other and try to converge similar software to use the same terms in the 
config where it makes sense (remember, the newcomer may have a better way of 
looking at the problem, so trying to force it to define it's config the way the 
legacy things did it can cause big problems)


I'm all for tools to convert the format to whatever you want to use, and (as I 
posted earlier), to have tools to assemble/generate the configs, but requiring 
changes to every piece of software to just change the format requires more than 
every langauage can understand JSON


David Lang

On Fri, 6 Mar 2015, Zefir Kurtisi wrote:


Hello Matthias,

can't provide qualified feedback to your proposal, but at least want to post my
experience and hope that it is valuable for the discussion.

Our company develops highly specialized WLAN products for decades, for a long 
time
based on OpenWRT. The niche market we are in has a strict requirement for
long-term availability and compatibility. In the context of configuration, in
addition to the default Web-UI configurability, we must support SNMP and stay
compatible with MIBs we initially released like 10 years ago.

Consequently, for a long time we are following an approach similar to what you 
are
proposing. In the beginning, the config overlay was a simple mapper between
backend (uci) and frontend (snmp). Later, when config space and complexity grew,
we defined and used a 'meta-config-space' that was generated at runtime from uci
to simplify the mapping to snmp.

Alas, what looked like a reasonable and clever approach, over time turned into
maintenance-hell. We were faced with two moving targets (uci config gradually
changing, MIB extensions) that need continuous synchronization and render the
approach error-prone and resource-demanding. From that experience, I tend to
believe that an overlay- or meta-configuration is only practicable as interim
solution during a transition towards a standardized config space.

If I had a chance to go back and do things better, this would be my approach:

1) define standardized config space
The hardest of all parts - config space must be generic and complete enough to
cover each and every potential setup, while at the same time it has to be
extensible, backwards-compatible, and well defined. Whoever is familiar with
standardization processes knows this will be no picnic. On the good side, with
ubus gradually becoming the core framework for OpenWRT, the format for the new
config to be JSON is more or less given. As for the structure, in [1] an
announcement for a generic network configuration standard in JSON was posted 
that
might well serve as starting point.
2) implement interim overlay
Once the config space in 1) is standardized, implement an temporary overlay that
provides transparent access to the configuration via legacy uci and new JSON
config spaces. From that point on, new applications / packages must not use uci
any more.
3) gradually move to new config space
Over a defined (and relatively short) transition period, existing packages has 
to
be converted to use the JSON config. At the end of that period, uci should not 
be
used any more.


I have developed a few private tools for the overlay-config approach (like
conversion from uci to JSON), which are not suitable for upstreaming (C++,
proprietary dependencies). From there, my estimate for the implementation 
workload
for the transition (after the config space was standardized) is around 3 
man-months.


Good Luck,
Zefir


[1] https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html


On 03/05/2015 01:36 PM, Matthias Schiffer wrote:

Hi,
during the development of our Freifunk firmware framework Gluon we've
come to the conclusion that the current UCI configuration format using
static files doesn't always fit our needs. Therefore we propose a new
feature we've called UCI overlays.

The basic idea is that we'd like to provide UCI configuration by other
means than the static files in /etc/config, for example from a database
or generated by scripts on the fly (the latter being our main usecase -
we'd like to generate configuration for netifd and fw3 based on

Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-06 Thread David Lang
One other thought. If there is a desire to change the format, let's pick a 
format that's already in use for configuring system rather than inventing yet 
another one. We should survey the tools that are being written for managing 
software defined datacenters (openstack and similar) and see if we can use one 
of those config languages (either as-is or with an automated conversion back and 
forth)


David Lang

On Fri, 6 Mar 2015, David Lang wrote:

changing the format doesn't solve the problem, it just causes churn in all 
the software.


The problem is the lack of standardization and the fact that the way things 
are configured changes over time. Not everything is configured in UCI (and 
this will always be the case because it's not worth trying to change every 
piece of software available)


When you try to design a standardized config space that will work in the 
future for equipment and software that nobody has imagined yet, you are 
either going to go crazy-complicated to try and cover everything and have 
people going cross-eyed trying to understand the spec, or you are going to 
lock things down to be so ridged that new things will be working around your 
config spec. The best that you can do is to setup a framework to keep the 
configs from stepping on each other and try to converge similar software to 
use the same terms in the config where it makes sense (remember, the newcomer 
may have a better way of looking at the problem, so trying to force it to 
define it's config the way the legacy things did it can cause big problems)


I'm all for tools to convert the format to whatever you want to use, and (as 
I posted earlier), to have tools to assemble/generate the configs, but 
requiring changes to every piece of software to just change the format 
requires more than every langauage can understand JSON


David Lang

On Fri, 6 Mar 2015, Zefir Kurtisi wrote:


Hello Matthias,

can't provide qualified feedback to your proposal, but at least want to 
post my

experience and hope that it is valuable for the discussion.

Our company develops highly specialized WLAN products for decades, for a 
long time

based on OpenWRT. The niche market we are in has a strict requirement for
long-term availability and compatibility. In the context of configuration, 
in
addition to the default Web-UI configurability, we must support SNMP and 
stay

compatible with MIBs we initially released like 10 years ago.

Consequently, for a long time we are following an approach similar to what 
you are

proposing. In the beginning, the config overlay was a simple mapper between
backend (uci) and frontend (snmp). Later, when config space and complexity 
grew,
we defined and used a 'meta-config-space' that was generated at runtime 
from uci

to simplify the mapping to snmp.

Alas, what looked like a reasonable and clever approach, over time turned 
into
maintenance-hell. We were faced with two moving targets (uci config 
gradually
changing, MIB extensions) that need continuous synchronization and render 
the
approach error-prone and resource-demanding. From that experience, I tend 
to
believe that an overlay- or meta-configuration is only practicable as 
interim

solution during a transition towards a standardized config space.

If I had a chance to go back and do things better, this would be my 
approach:


1) define standardized config space
The hardest of all parts - config space must be generic and complete enough 
to

cover each and every potential setup, while at the same time it has to be
extensible, backwards-compatible, and well defined. Whoever is familiar 
with
standardization processes knows this will be no picnic. On the good side, 
with
ubus gradually becoming the core framework for OpenWRT, the format for the 
new

config to be JSON is more or less given. As for the structure, in [1] an
announcement for a generic network configuration standard in JSON was 
posted that

might well serve as starting point.
2) implement interim overlay
Once the config space in 1) is standardized, implement an temporary overlay 
that
provides transparent access to the configuration via legacy uci and new 
JSON
config spaces. From that point on, new applications / packages must not use 
uci

any more.
3) gradually move to new config space
Over a defined (and relatively short) transition period, existing packages 
has to
be converted to use the JSON config. At the end of that period, uci should 
not be

used any more.


I have developed a few private tools for the overlay-config approach (like
conversion from uci to JSON), which are not suitable for upstreaming (C++,
proprietary dependencies). From there, my estimate for the implementation 
workload
for the transition (after the config space was standardized) is around 3 
man-months.



Good Luck,
Zefir


[1] 
https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html



On 03/05/2015 01:36 PM, Matthias Schiffer wrote:

Hi,
during the development of our Freifunk firmware 

Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-06 Thread Zefir Kurtisi
Hello Matthias,

can't provide qualified feedback to your proposal, but at least want to post my
experience and hope that it is valuable for the discussion.

Our company develops highly specialized WLAN products for decades, for a long 
time
based on OpenWRT. The niche market we are in has a strict requirement for
long-term availability and compatibility. In the context of configuration, in
addition to the default Web-UI configurability, we must support SNMP and stay
compatible with MIBs we initially released like 10 years ago.

Consequently, for a long time we are following an approach similar to what you 
are
proposing. In the beginning, the config overlay was a simple mapper between
backend (uci) and frontend (snmp). Later, when config space and complexity grew,
we defined and used a 'meta-config-space' that was generated at runtime from uci
to simplify the mapping to snmp.

Alas, what looked like a reasonable and clever approach, over time turned into
maintenance-hell. We were faced with two moving targets (uci config gradually
changing, MIB extensions) that need continuous synchronization and render the
approach error-prone and resource-demanding. From that experience, I tend to
believe that an overlay- or meta-configuration is only practicable as interim
solution during a transition towards a standardized config space.

If I had a chance to go back and do things better, this would be my approach:

1) define standardized config space
The hardest of all parts - config space must be generic and complete enough to
cover each and every potential setup, while at the same time it has to be
extensible, backwards-compatible, and well defined. Whoever is familiar with
standardization processes knows this will be no picnic. On the good side, with
ubus gradually becoming the core framework for OpenWRT, the format for the new
config to be JSON is more or less given. As for the structure, in [1] an
announcement for a generic network configuration standard in JSON was posted 
that
might well serve as starting point.
2) implement interim overlay
Once the config space in 1) is standardized, implement an temporary overlay that
provides transparent access to the configuration via legacy uci and new JSON
config spaces. From that point on, new applications / packages must not use uci
any more.
3) gradually move to new config space
Over a defined (and relatively short) transition period, existing packages has 
to
be converted to use the JSON config. At the end of that period, uci should not 
be
used any more.


I have developed a few private tools for the overlay-config approach (like
conversion from uci to JSON), which are not suitable for upstreaming (C++,
proprietary dependencies). From there, my estimate for the implementation 
workload
for the transition (after the config space was standardized) is around 3 
man-months.


Good Luck,
Zefir


[1] https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html


On 03/05/2015 01:36 PM, Matthias Schiffer wrote:
 Hi,
 during the development of our Freifunk firmware framework Gluon we've
 come to the conclusion that the current UCI configuration format using
 static files doesn't always fit our needs. Therefore we propose a new
 feature we've called UCI overlays.
 
 The basic idea is that we'd like to provide UCI configuration by other
 means than the static files in /etc/config, for example from a database
 or generated by scripts on the fly (the latter being our main usecase -
 we'd like to generate configuration for netifd and fw3 based on
 meta-configuration data). This should work transparently, without
 needing changes in the config consumers (applications).
 
 The overlay-provided configuration packages should be merged with the
 config read from /etc/config. We'd like to generate both config sections
 which may be overruled by corresponding options in /etc/config, and
 read-only sections which can't be changed by the user through UCI.
 
 We see two different ways to implement this:
 
 (1) Make the overlay an alternative backend which uses the file
 backend to merge generated config with the one from files
 
 (2) Introduce overlays as a new concept in libuci
 
 
 While the first option would need less changes in libuci, it doesn't
 allow stacking different overlays, so we're in favour of option 2.
 
 Both ways would need a config file (/etc/uci.conf?) to configure the
 overlays being used. Our plan is to implement overlays as dlopen-able
 plugins loaded from somewhere like /lib/uci/overlay so it is possible
 for other packages to provide overlay implementations.
 
 In addition, we'd like to add a new attribute readonly to the
 uci_element structure so changing read-only configuration will fail as
 soon as someone tries to uci_set it.
 
 Does this sound reasonable? We can develop the needed extensions
 ourselves, but of course we'd like to get this feature upstream to avoid
 carrying the patches downstream indefinitly, so we're eager to know what
 you 

Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-06 Thread Matthias Schiffer
On 03/05/2015 08:25 PM, David Lang wrote:
 On Thu, 05 Mar 2015 13:36:10 +0100, Matthias Schiffer wrote:
 Hi,
 during the development of our Freifunk firmware framework Gluon we've
 come to the conclusion that the current UCI configuration format using
 static files doesn't always fit our needs. Therefore we propose a new
 feature we've called UCI overlays.

 The basic idea is that we'd like to provide UCI configuration by other
 means than the static files in /etc/config, for example from a database
 or generated by scripts on the fly (the latter being our main usecase -
 we'd like to generate configuration for netifd and fw3 based on
 meta-configuration data). This should work transparently, without
 needing changes in the config consumers (applications).

 The overlay-provided configuration packages should be merged with the
 config read from /etc/config. We'd like to generate both config sections
 which may be overruled by corresponding options in /etc/config, and
 read-only sections which can't be changed by the user through UCI.

 We see two different ways to implement this:

 (1) Make the overlay an alternative backend which uses the file
 backend to merge generated config with the one from files

 (2) Introduce overlays as a new concept in libuci


 While the first option would need less changes in libuci, it doesn't
 allow stacking different overlays, so we're in favour of option 2.

 Both ways would need a config file (/etc/uci.conf?) to configure the
 overlays being used. Our plan is to implement overlays as dlopen-able
 plugins loaded from somewhere like /lib/uci/overlay so it is possible
 for other packages to provide overlay implementations.

 In addition, we'd like to add a new attribute readonly to the
 uci_element structure so changing read-only configuration will fail as
 soon as someone tries to uci_set it.

 Does this sound reasonable? We can develop the needed extensions
 ourselves, but of course we'd like to get this feature upstream to avoid
 carrying the patches downstream indefinitly, so we're eager to know what
 you think of this proposal.
 
 Should this be built in to uci, or should this be a set of utilities
 that take /etc/config/wireless.d/* and create /etc/config/wireless when
 run (which then gets used normally)?
 
 You can then have scripts, database tools, config management tools
 (puppet, chef, etc) drop things in such directories and the utilities
 can run in cron (or use inotify in an advanced setup) to notice that
 there was a change and do the appropriate thing to re-read the config.
 
 Building it into uci is more elegant, but I worry that having it that
 integrated will mean that each new way to create config info will end up
 requiring changes to uci. File based is a little less elegant, but it
 makes it much easier for people to add new ways to do things.
 
 If you are assembling /etc/config/wireless from /etc/config/wireless.d/*
 you can just say that the resulting file is the combination of all the
 included files, and it's up to the admin to avoid conflicts between the
 pieces. If you take /etc/config/wireless and overlay other things on it,
 you have the question of how do you negate something that was in the
 base file.
 
 
 
 I do think that something along these lines should be done. It would be
 very nice to be able to break up existing config files a little bit (for
 example, /etc/config/network split raw interfaces from bridge configs
 from switch configs). And it would be great to be able to have a config
 management tool be able to just add-on to the base config, replacing
 part of it, without the config management tool needing to manage
 everything in the file.
 
 Over on the CeroWRT list we are discussing what things are in CeroWRT
 that have not yet been unstreamed into OpenWRT, and most of them come
 down to config items. Some of them are 'fairly' small (like renaming
 interfaces by function, so you have an interface named WAN rather than
 eth0), while others are very fundamental (routing between each wifi
 interface and the wired network instead of bridging). An overlay or
 name.d approach would make it much easier for these different approaches
 to be presented as packages that can be added to OpenWRT without forcing
 everyone to change to them.
 
 David Lang


At the moment we use scripts to edit /etc/config/* to our needs. There
are multiple reasons why we'd like to get rid of this:

* Changing specific parts of meta-configuration may change large parts
of network and firewall configuration. We'd like to avoid writing to the
flash more than necessary.
* As long as the UCI config is written to files, admins think they can
change them, not anticipating that parts of the config may be rewritten,
leading to all kinds of funny misconfigurations. Not actually having the
config written out, but generated on the fly, is also an attempt to make
the configuration more fool-proof, by leaving the meta-config as the
only way to make changes to the 

Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-05 Thread David Lang

On Thu, 05 Mar 2015 13:36:10 +0100, Matthias Schiffer wrote:

Hi,
during the development of our Freifunk firmware framework Gluon 
we've
come to the conclusion that the current UCI configuration format 
using

static files doesn't always fit our needs. Therefore we propose a new
feature we've called UCI overlays.

The basic idea is that we'd like to provide UCI configuration by 
other
means than the static files in /etc/config, for example from a 
database
or generated by scripts on the fly (the latter being our main usecase 
-

we'd like to generate configuration for netifd and fw3 based on
meta-configuration data). This should work transparently, without
needing changes in the config consumers (applications).

The overlay-provided configuration packages should be merged with the
config read from /etc/config. We'd like to generate both config 
sections

which may be overruled by corresponding options in /etc/config, and
read-only sections which can't be changed by the user through UCI.

We see two different ways to implement this:

(1) Make the overlay an alternative backend which uses the file
backend to merge generated config with the one from files

(2) Introduce overlays as a new concept in libuci


While the first option would need less changes in libuci, it doesn't
allow stacking different overlays, so we're in favour of option 2.

Both ways would need a config file (/etc/uci.conf?) to configure the
overlays being used. Our plan is to implement overlays as dlopen-able
plugins loaded from somewhere like /lib/uci/overlay so it is possible
for other packages to provide overlay implementations.

In addition, we'd like to add a new attribute readonly to the
uci_element structure so changing read-only configuration will fail 
as

soon as someone tries to uci_set it.

Does this sound reasonable? We can develop the needed extensions
ourselves, but of course we'd like to get this feature upstream to 
avoid
carrying the patches downstream indefinitly, so we're eager to know 
what

you think of this proposal.


Should this be built in to uci, or should this be a set of utilities 
that take /etc/config/wireless.d/* and create /etc/config/wireless when 
run (which then gets used normally)?


You can then have scripts, database tools, config management tools 
(puppet, chef, etc) drop things in such directories and the utilities 
can run in cron (or use inotify in an advanced setup) to notice that 
there was a change and do the appropriate thing to re-read the config.


Building it into uci is more elegant, but I worry that having it that 
integrated will mean that each new way to create config info will end up 
requiring changes to uci. File based is a little less elegant, but it 
makes it much easier for people to add new ways to do things.


If you are assembling /etc/config/wireless from 
/etc/config/wireless.d/* you can just say that the resulting file is the 
combination of all the included files, and it's up to the admin to avoid 
conflicts between the pieces. If you take /etc/config/wireless and 
overlay other things on it, you have the question of how do you negate 
something that was in the base file.




I do think that something along these lines should be done. It would be 
very nice to be able to break up existing config files a little bit (for 
example, /etc/config/network split raw interfaces from bridge configs 
from switch configs). And it would be great to be able to have a config 
management tool be able to just add-on to the base config, replacing 
part of it, without the config management tool needing to manage 
everything in the file.


Over on the CeroWRT list we are discussing what things are in CeroWRT 
that have not yet been unstreamed into OpenWRT, and most of them come 
down to config items. Some of them are 'fairly' small (like renaming 
interfaces by function, so you have an interface named WAN rather than 
eth0), while others are very fundamental (routing between each wifi 
interface and the wired network instead of bridging). An overlay or 
name.d approach would make it much easier for these different approaches 
to be presented as packages that can be added to OpenWRT without forcing 
everyone to change to them.


David Lang
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel


Re: [OpenWrt-Devel] UCI overlay proposal

2015-03-05 Thread Shankar Unni
You even make the conflicts reliably handled by processing the contents of, 
say, /etc/config/wireless.d/* in sort order (the old 01-blah, 02-foo 
trick).  This way, if one package drops in a config file that you want to 
override, you only have to make sure that your override comes after the file 
it's overriding.

 On Mar 5, 2015, at 11:25 AM, David Lang da...@lang.hm wrote:
 
 On Thu, 05 Mar 2015 13:36:10 +0100, Matthias Schiffer wrote:
 Hi,
 during the development of our Freifunk firmware framework Gluon we've
 come to the conclusion that the current UCI configuration format using
 static files doesn't always fit our needs. Therefore we propose a new
 feature we've called UCI overlays.
 
 The basic idea is that we'd like to provide UCI configuration by other
 means than the static files in /etc/config, for example from a database
 or generated by scripts on the fly (the latter being our main usecase -
 we'd like to generate configuration for netifd and fw3 based on
 meta-configuration data). This should work transparently, without
 needing changes in the config consumers (applications).
 
 The overlay-provided configuration packages should be merged with the
 config read from /etc/config. We'd like to generate both config sections
 which may be overruled by corresponding options in /etc/config, and
 read-only sections which can't be changed by the user through UCI.
 
 We see two different ways to implement this:
 
 (1) Make the overlay an alternative backend which uses the file
 backend to merge generated config with the one from files
 
 (2) Introduce overlays as a new concept in libuci
 
 
 While the first option would need less changes in libuci, it doesn't
 allow stacking different overlays, so we're in favour of option 2.
 
 Both ways would need a config file (/etc/uci.conf?) to configure the
 overlays being used. Our plan is to implement overlays as dlopen-able
 plugins loaded from somewhere like /lib/uci/overlay so it is possible
 for other packages to provide overlay implementations.
 
 In addition, we'd like to add a new attribute readonly to the
 uci_element structure so changing read-only configuration will fail as
 soon as someone tries to uci_set it.
 
 Does this sound reasonable? We can develop the needed extensions
 ourselves, but of course we'd like to get this feature upstream to avoid
 carrying the patches downstream indefinitly, so we're eager to know what
 you think of this proposal.
 
 Should this be built in to uci, or should this be a set of utilities that 
 take /etc/config/wireless.d/* and create /etc/config/wireless when run (which 
 then gets used normally)?
 
 You can then have scripts, database tools, config management tools (puppet, 
 chef, etc) drop things in such directories and the utilities can run in cron 
 (or use inotify in an advanced setup) to notice that there was a change and 
 do the appropriate thing to re-read the config.
 
 Building it into uci is more elegant, but I worry that having it that 
 integrated will mean that each new way to create config info will end up 
 requiring changes to uci. File based is a little less elegant, but it makes 
 it much easier for people to add new ways to do things.
 
 If you are assembling /etc/config/wireless from /etc/config/wireless.d/* you 
 can just say that the resulting file is the combination of all the included 
 files, and it's up to the admin to avoid conflicts between the pieces. If you 
 take /etc/config/wireless and overlay other things on it, you have the 
 question of how do you negate something that was in the base file.
 
 
 
 I do think that something along these lines should be done. It would be very 
 nice to be able to break up existing config files a little bit (for example, 
 /etc/config/network split raw interfaces from bridge configs from switch 
 configs). And it would be great to be able to have a config management tool 
 be able to just add-on to the base config, replacing part of it, without the 
 config management tool needing to manage everything in the file.
 
 Over on the CeroWRT list we are discussing what things are in CeroWRT that 
 have not yet been unstreamed into OpenWRT, and most of them come down to 
 config items. Some of them are 'fairly' small (like renaming interfaces by 
 function, so you have an interface named WAN rather than eth0), while others 
 are very fundamental (routing between each wifi interface and the wired 
 network instead of bridging). An overlay or name.d approach would make it 
 much easier for these different approaches to be presented as packages that 
 can be added to OpenWRT without forcing everyone to change to them.
 
 David Lang
 ___
 openwrt-devel mailing list
 openwrt-devel@lists.openwrt.org
 https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org

[OpenWrt-Devel] UCI overlay proposal

2015-03-05 Thread Matthias Schiffer
Hi,
during the development of our Freifunk firmware framework Gluon we've
come to the conclusion that the current UCI configuration format using
static files doesn't always fit our needs. Therefore we propose a new
feature we've called UCI overlays.

The basic idea is that we'd like to provide UCI configuration by other
means than the static files in /etc/config, for example from a database
or generated by scripts on the fly (the latter being our main usecase -
we'd like to generate configuration for netifd and fw3 based on
meta-configuration data). This should work transparently, without
needing changes in the config consumers (applications).

The overlay-provided configuration packages should be merged with the
config read from /etc/config. We'd like to generate both config sections
which may be overruled by corresponding options in /etc/config, and
read-only sections which can't be changed by the user through UCI.

We see two different ways to implement this:

(1) Make the overlay an alternative backend which uses the file
backend to merge generated config with the one from files

(2) Introduce overlays as a new concept in libuci


While the first option would need less changes in libuci, it doesn't
allow stacking different overlays, so we're in favour of option 2.

Both ways would need a config file (/etc/uci.conf?) to configure the
overlays being used. Our plan is to implement overlays as dlopen-able
plugins loaded from somewhere like /lib/uci/overlay so it is possible
for other packages to provide overlay implementations.

In addition, we'd like to add a new attribute readonly to the
uci_element structure so changing read-only configuration will fail as
soon as someone tries to uci_set it.

Does this sound reasonable? We can develop the needed extensions
ourselves, but of course we'd like to get this feature upstream to avoid
carrying the patches downstream indefinitly, so we're eager to know what
you think of this proposal.

Regards,
Matthias



signature.asc
Description: OpenPGP digital signature
___
openwrt-devel mailing list
openwrt-devel@lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel