Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
Am Montag, den 04.05.2009, 22:01 +0100 schrieb Sérgio Almeida:
Gentoo Dev's,
My name is Sérgio Almeida, I am Portuguese and I am a student for this
year's Google SoC coding the Universal Select Tool project for Gentoo
being Sébastien Fabbro (bicatali) my mentor.
Abstract:
Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from modules, softenv, and
both eselect and eclectic.
My initial proposal does not get in-depth with implementation details
and I need to make some decisions as soon as possible. Implementation
language will be python as it is easy to maintain, easy to code and
faster and more flexible than bash. See attachment for more details.
Besides introducing myself, the purpose of this e-mail is a
call-to-ideas to all Gentoo developers, mainly all eselect-* and
*-config developers.
Here are the main interest ideas:
* keep eselect structure of modules - actions
* symlinking, environment and aliases actions can consist of something
like:
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action system moo
description Moo Action Description
symlink regexp regexp
env regexp regexp
alias regexp regexp
# end moo
These should get the job done for most of the modules and opens the door
to automatic module creation prior to a successful emerge (if some USE
flag set)
* Actions that consist of code blocks that support any scripting
language (what about binaries?) to do more complex actions (full module
example):
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action user moo
description Example Module will moo for any user
type runnable
runner /bin/bash
# file moo.bash
#!/bin/bash
do_moo() {
echo This is the Example Module mooing
}
do_moo()
# end moo.bash
# end moo
* actions can be run system-wide and per-user:
# action user moo
# action system moo
* automatic module loading and profile management can be managed by some
env.d python scripts that are user-aware and follow some database
I've been given this difficult task of unifying all of these tools
together and as you all can understand, I won't be having the time to
read through all eselect-* modules and *-config utilities code.
Please drop me a line here or at freenode if you have anything to add to
these ideas or have any further ideas that can help me on this project.
Thank you all in advance.
What I'd like to see is the possibility to
... localize messages (will be difficult since modules need translations
as well, but maybe you find a way :)
... encapsulation of methods to set/list/change such that instead of a
CLI- a NCurses- or GUI-Frontend could be developed.
Cheers,
Tiziano
--
Tiziano Müller
Gentoo Linux Developer, Council Member
Areas of responsibility:
Samba, PostgreSQL, CPP, Python, sysadmin, GLEP Editor
E-Mail : dev-z...@gentoo.org
GnuPG FP : F327 283A E769 2E36 18D5 4DE2 1B05 6A63 AE9C 1E30
signature.asc
Description: Dies ist ein digital signierter Nachrichtenteil
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
On Tue, 2009-05-26 at 22:04 +0200, Tiziano Müller wrote:
Am Montag, den 04.05.2009, 22:01 +0100 schrieb Sérgio Almeida:
Gentoo Dev's,
My name is Sérgio Almeida, I am Portuguese and I am a student for this
year's Google SoC coding the Universal Select Tool project for Gentoo
being Sébastien Fabbro (bicatali) my mentor.
Abstract:
Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from modules, softenv, and
both eselect and eclectic.
My initial proposal does not get in-depth with implementation details
and I need to make some decisions as soon as possible. Implementation
language will be python as it is easy to maintain, easy to code and
faster and more flexible than bash. See attachment for more details.
Besides introducing myself, the purpose of this e-mail is a
call-to-ideas to all Gentoo developers, mainly all eselect-* and
*-config developers.
Here are the main interest ideas:
* keep eselect structure of modules - actions
* symlinking, environment and aliases actions can consist of something
like:
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action system moo
description Moo Action Description
symlink regexp regexp
env regexp regexp
alias regexp regexp
# end moo
These should get the job done for most of the modules and opens the door
to automatic module creation prior to a successful emerge (if some USE
flag set)
* Actions that consist of code blocks that support any scripting
language (what about binaries?) to do more complex actions (full module
example):
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action user moo
description Example Module will moo for any user
type runnable
runner /bin/bash
# file moo.bash
#!/bin/bash
do_moo() {
echo This is the Example Module mooing
}
do_moo()
# end moo.bash
# end moo
* actions can be run system-wide and per-user:
# action user moo
# action system moo
* automatic module loading and profile management can be managed by some
env.d python scripts that are user-aware and follow some database
I've been given this difficult task of unifying all of these tools
together and as you all can understand, I won't be having the time to
read through all eselect-* modules and *-config utilities code.
Please drop me a line here or at freenode if you have anything to add to
these ideas or have any further ideas that can help me on this project.
Thank you all in advance.
What I'd like to see is the possibility to
... localize messages (will be difficult since modules need translations
as well, but maybe you find a way :)
... encapsulation of methods to set/list/change such that instead of a
CLI- a NCurses- or GUI-Frontend could be developed.
Cheers,
Tiziano
Hello,
These are the reasons of my current prototype refactoring phase. All
this will be possible.
Thanks!
Cheers,
Sérgio
signature.asc
Description: This is a digitally signed message part
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
Michael,
Eventually, we should pass something like -Dhostname=superhost as
cmdline parameter to uselect...
Didn't want to get into that detail. Afterall `hostname` will always
return the same regarding the host therefore it can be retrieved from
uselect's engine.
I don't like to have anything interpreted after the usual and well-known
comment-marker, this just feels hackish.
Agreed, it was just an easy to read example. Will adopt {} for those.
I do have something like this content-syntax in mind for .uselect (or
eventually some .uprofile) file:
888
version 0.1
include path/to/another/uselect/file
profile generic solaris {
module A actionAX valueAX1 valueAX2
module B actionBX valueBX1 valueBX2
}
profile generic host {
module C actionCX valueCX1
}
profile superhost {
inherit profile generic solaris
inherit profile generic host
module C actionCX newvalueCX1
module A actionAX newvalueAX1 newvalueAX2
module D actionDY valueBY1 valueBY2
}
profile generic user {
module E actionEX valueEX1
}
profile user haubi {
inherit profile generic user
module F actionFX valueFX1
}
profile any user on superhost {
module G actionGX valueGX1
}
profile fallback host {
inherit profile generic host
module A actionAX valueAX1 valueAX2
}
activation superhost activation {
in category host
on fallback level 0
when $hostname matches string superhost
activate profile superhost
}
activation haubi on superhost {
in category user
on fallback level 0
when $username matches string haubi
when $hostname matches string superhost
activate profile any user on superhost
activate profile user haubi
}
activation haubi {
in category user
on fallback level 1
when $username matches string haubi
activate profile user haubi
}
activation gentoo host {
in category host
on fallback level 1
when $hostname matches regex .*\.gentoo\.org
activate profile fallback host
}
888
I'm not sure to have an ascending fallback level or descending
priority value, but both should allow for negative integer values.
I'm not quite shure if this hierarchy fallback method is the most
adequate one. Again, remember that the profiles will be automatically
created and manipulation needs to be easy. I guess we can arrange for a
better way of managing this.
The selection of one or more profiles is controlled by activation
entries, independent for each category. The order and selection of
categories is done via a commandline argument, fex --categories.
We are mixing some more complex concepts that don't need to be managed
this way. Let's see.
Types of Profiles:
something.uprofile - local file profile
.uprofile - folder profile
/usr/share/uprofiles/*.uprofile - template profiles
Adding categories to a folder profile is the same as having several file
profiles into that directory and load them on demand. Why specify to
uselect --categories=something when you can uselelect loadprofile
something (loads .uselect folder profile first, and then overrides
settings as specified in something.uprofile).
Profiles are selected when all of the when entries (besides category
and fallback level) in activation {} do match.
Selected are *all* of the matching profiles in the *lowest* fallback
level *only*, which does have at least one matching profile.
And I'm thinking of something like this commandline:
$ uselect \
--categories host,user \
--define hostname=`hostname` \
--define username=`whoami`
Remember that profile creation/storing/managing should be automatic and
nothing would need to be written by hand.
Yes, would be fine. When using something like above configuration file
syntax, some GUIding tool would be useful, at least to see which
profiles are selected for specific cmdline args.
Mmm... later feature for more complex profiling. Let us stop now and
focus on simple profiling with easy managing.
By other words, create a basic
profile automatically using your currently running settings and then
alter the profile with the util to add constrains to it.
Sounds interesting...
Remember that
all the machines that this profile is read would need to have the same
uselect modules into it's /usr/share/uselect/modules or similar.
Indeed, yes.
Ha! This kind of inheritance tree could be a solution for my long
standing bug here to simplify our way too complex environment script...
This is a basic idea from softenv so it has has it's place into uselect.
Don't know softenv. Found http://www.teragrid.org/userinfo/softenv/
Do you mean this one?
Indeed.
The question now is, bloat it all into uselect or create a uprofile
util? I like the idea of having to use only uselect for basic profiling
and using uprofile for further managing.
That's indeed the question.
Mmmm... As you wrote I realized: Complain if module versions are
different is not a
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
On Wed, 2009-05-13 at 16:40 +0100, Sérgio Almeida wrote:
This .uselect
should not contain symlinks, but plain (text?) files only.
Do we really need more than one file?
No, at least not to define the _values_ of a profile.
Checklist:
* Hostname
* Uname
* {$chost}
Mmm... Maybe we can simplify this with a parameter like:
# eval something
eval hostname superhost
what
to
do
# end something
Then if command hostname outputs superhost we know what-to-do.
Eventually, we should pass something like -Dhostname=superhost as
cmdline parameter to uselect...
This way it would get ultra versatile.
What if there is some hierarchy in .uselect/ much like the profiles in
gentoo-x86 tree, using some kind of 'parent' files to inherit/override
settings for this one project, where 'parent' can contain something like
$(CHOST), $(UNAME), $(HOSTNAME), $(USERNAME)?
Would this really be necessary? We can define hierarchy into a
single .uselect file. Even the use of folders (folder .uselect/) isn't
necessary. I think a single file can handle all this...
Eventually yes.
Lets see an
example:
# profile something
I don't like to have anything interpreted after the usual and well-known
comment-marker, this just feels hackish.
do this 3 - override the overridden =)
The actions to be done like do this 3 are a simple call to uselect
using module do and action this with 3 as parameters.
fine.
I do have something like this content-syntax in mind for .uselect (or
eventually some .uprofile) file:
888
version 0.1
include path/to/another/uselect/file
profile generic solaris {
module A actionAX valueAX1 valueAX2
module B actionBX valueBX1 valueBX2
}
profile generic host {
module C actionCX valueCX1
}
profile superhost {
inherit profile generic solaris
inherit profile generic host
module C actionCX newvalueCX1
module A actionAX newvalueAX1 newvalueAX2
module D actionDY valueBY1 valueBY2
}
profile generic user {
module E actionEX valueEX1
}
profile user haubi {
inherit profile generic user
module F actionFX valueFX1
}
profile any user on superhost {
module G actionGX valueGX1
}
profile fallback host {
inherit profile generic host
module A actionAX valueAX1 valueAX2
}
activation superhost activation {
in category host
on fallback level 0
when $hostname matches string superhost
activate profile superhost
}
activation haubi on superhost {
in category user
on fallback level 0
when $username matches string haubi
when $hostname matches string superhost
activate profile any user on superhost
activate profile user haubi
}
activation haubi {
in category user
on fallback level 1
when $username matches string haubi
activate profile user haubi
}
activation gentoo host {
in category host
on fallback level 1
when $hostname matches regex .*\.gentoo\.org
activate profile fallback host
}
888
I'm not sure to have an ascending fallback level or descending
priority value, but both should allow for negative integer values.
The selection of one or more profiles is controlled by activation
entries, independent for each category. The order and selection of
categories is done via a commandline argument, fex --categories.
Profiles are selected when all of the when entries (besides category
and fallback level) in activation {} do match.
Selected are *all* of the matching profiles in the *lowest* fallback
level *only*, which does have at least one matching profile.
And I'm thinking of something like this commandline:
$ uselect \
--categories host,user \
--define hostname=`hostname` \
--define username=`whoami`
Remember that profile creation/storing/managing should be automatic and
nothing would need to be written by hand.
Yes, would be fine. When using something like above configuration file
syntax, some GUIding tool would be useful, at least to see which
profiles are selected for specific cmdline args.
By other words, create a basic
profile automatically using your currently running settings and then
alter the profile with the util to add constrains to it.
Sounds interesting...
Remember that
all the machines that this profile is read would need to have the same
uselect modules into it's /usr/share/uselect/modules or similar.
Indeed, yes.
Ha! This kind of inheritance tree could be a solution for my long
standing bug here to simplify our way too complex environment script...
This is a basic idea from softenv so it has has it's place into uselect.
Don't know softenv. Found http://www.teragrid.org/userinfo/softenv/
Do you mean this one?
The question now is, bloat it all into uselect or create a uprofile
util? I like the idea of having to use only uselect for basic profiling
and using uprofile for further managing.
That's indeed the question.
Mmmm... As you wrote I realized: Complain if module versions are
different is not a bad idea at all...
Indeed, not only the configuration needs to be
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
On Tue, 2009-05-12 at 21:45 +0100, Sérgio Almeida wrote:
How about having the profile switch automatic whenever
you call uselect something getting the current profile settings from
current dir's .uselect folder?
Yeah, this indeed could work!
But there is one restriction:
This .uselect folder must be able to be checked in into some VCS, so it
should not contain symlinks, but plain (text?) files only.
We also want to use this on Windows based filesystems where symlinks
don't work at all.
Ohw, I do have another requirement:
We do check-out/compile/develop/run/test the same package on different
hosts and platforms. Each of these hosts might require slightly
different settings. ATM, the package's environment script handles this
by acting differently based on `hostname` and `uname` or ${chost}.
Ohw, it even does different settings based on the username (`id -un` or
`whoami`), because in production these projects usually run under a
specific user with less restrictive settings than for developers.
What if there is some hierarchy in .uselect/ much like the profiles in
gentoo-x86 tree, using some kind of 'parent' files to inherit/override
settings for this one project, where 'parent' can contain something like
$(CHOST), $(UNAME), $(HOSTNAME), $(USERNAME)?
I'm unsure if managing different settings based on hostname, uname/chost
and username (the inheritance tree) is uselect's job. It eventually
should optionally listen to some cmdline-parameter or environment-
variable where to look for the uselect settings instead, which could be
stored and regenerated using such profile hierarchy mechanism/utility.
Or even the whole uselect settings optionally come from stdin (and go to
stdout), to be managed/stored within that profile hierarchy.
Ha! This kind of inheritance tree could be a solution for my long
standing bug here to simplify our way too complex environment script...
Ah, don't forget to put a version number as the very first value into
the uselect settings, to avoid backwards compatibility issues in the
future. And when uselect settings can be read from stdin (stored in
simple text files), this version number could occur multiple times
there, as the stored files simply will be concatenated. And subsequent
values might override previous ones then.
Uh, so many strange ideas!
More of them?
Thanks!
/haubi/
--
Michael Haubenwallner
Gentoo on a different level
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
If there is a different place for requirement discussion of Gentoo specific GSoC projects, please tell, and sorry for bothering here then. On Mon, 2009-05-04 at 22:01 +0100, Sérgio Almeida wrote: Here are the main interest ideas: * actions can be run system-wide and per-user: # action user moo # action system moo Are there any thoughts to support something more fine granular settings than system and user? What I'm after: We are developing multiple source projects with multiple release branches on the same host here. These projects do have some script to set up the project specific environment. One os-user is working on multiple projects or branches, entering a project's environment by sourcing its environment script. Naturally, these projects do have different requirements to - for example - the java-vm version. The project admin needs to select the java-vm on a per-project basis, and does want to use the system-vm as fallback, never wants to use the user-vm. With current eselect (and java-config-2), this would be something like setting $HOME on the per-project basis - or not using java-config at all. Would it make sense to explicitly set the system-vm (whatever version it is or will be changed to), while leaving it unset would fall back to the user-vm? More toughts? Thanks! /haubi/ -- Michael Haubenwallner Gentoo on a different level
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
On Tue, 2009-05-12 at 15:32 +0100, Sérgio Almeida wrote: On Mon, 2009-05-04 at 22:01 +0100, Sérgio Almeida wrote: Here are the main interest ideas: * actions can be run system-wide and per-user: # action user moo # action system moo Are there any thoughts to support something more fine granular settings than system and user? Indeed, user is not for all the users. It's an action that can be run by the users to change it's own settings without touching the system's fallback default. Seems you didn't really understand my problem yet - another try: The *one* and *same* user does need different settings at the *same* time depending on the project he's currently working on in different shells. The actual setting needs to be set up by the project's environment script (when the users enters the project), not the user's profile script (when the user logs in). That's the point of the uselect tool. Per-System settings, Per-User settings (2 different ssh sessions of the same user can still have different environment settings too). This maybe is what I'm after: Question still is how to easily set up the different environment? I work as a sysadmin and manage mainly multi-user gentoo environments where people develop calculus c/python/fortran/R/whatever code using wathever utilities we can imagine. Everytime a new project is beeing done people need to set the environment variables themselves when they are kind enough not to ask me. That's mainly the whole purpose of this uselect tool, easy multi user environments managing. Seems like we have a similar job ;) Maybe it's the wording only, but in addition to the multi-user dimension I'm still missing the multi-project dimension for one user. /haubi/ -- Michael Haubenwallner Gentoo on a different level
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
Michael, Seems you didn't really understand my problem yet - another try: The *one* and *same* user does need different settings at the *same* time depending on the project he's currently working on in different shells. The actual setting needs to be set up by the project's environment script (when the users enters the project), not the user's profile script (when the user logs in). Ok, now I fully understand you and I agree. A new feature? Mmm... This can be managed by profile creation and switching. Imagine, user can create a profile with it's current uselect settings, select a default profile, select the active profile on-the-fly, and on and on... Works, but not automatic... Maybe we can optimize this by having a similar behavior of .svn folders on svn settings. How about having the profile switch automatic whenever you call uselect something getting the current profile settings from current dir's .uselect folder? I am starting the plans on profile managing today. This now gets interesting. Thanks! Cheers, Sérgio -- Sérgio Almeida meph...@gmail.com mephx @ freenode
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
Sérgio Almeida meph...@gmail.com said: Abstract: Universal Select Tool is an utility to manage system configuration. This tool is similar to the unmaintained eselect utility of Gentoo or Exherbo's eclectic. The idea is to create a tool that manages both system settings and user settings with profile creation possibilities. The utility will use mostly concepts from modules, softenv, and both eselect and eclectic. I guess this is a very high level question...but do we need yet another eselect? Why can't we enhance or fix what we already have rather than creating everything from scratch? -- Mark Loeser email - halcy0n AT gentoo DOT org email - mark AT halcy0n DOT com web - http://www.halcy0n.com pgpUGXMzGTQ7a.pgp Description: PGP signature
[gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
Gentoo Dev's,
My name is Sérgio Almeida, I am Portuguese and I am a student for this
year's Google SoC coding the Universal Select Tool project for Gentoo
being Sébastien Fabbro (bicatali) my mentor.
Abstract:
Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from modules, softenv, and
both eselect and eclectic.
My initial proposal does not get in-depth with implementation details
and I need to make some decisions as soon as possible. Implementation
language will be python as it is easy to maintain, easy to code and
faster and more flexible than bash. See attachment for more details.
Besides introducing myself, the purpose of this e-mail is a
call-to-ideas to all Gentoo developers, mainly all eselect-* and
*-config developers.
Here are the main interest ideas:
* keep eselect structure of modules - actions
* symlinking, environment and aliases actions can consist of something
like:
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action system moo
description Moo Action Description
symlink regexp regexp
env regexp regexp
alias regexp regexp
# end moo
These should get the job done for most of the modules and opens the door
to automatic module creation prior to a successful emerge (if some USE
flag set)
* Actions that consist of code blocks that support any scripting
language (what about binaries?) to do more complex actions (full module
example):
# module moo comments
description Example Module description
version Example Module Version
author m...@farm.moo
# action user moo
description Example Module will moo for any user
type runnable
runner /bin/bash
# file moo.bash
#!/bin/bash
do_moo() {
echo This is the Example Module mooing
}
do_moo()
# end moo.bash
# end moo
* actions can be run system-wide and per-user:
# action user moo
# action system moo
* automatic module loading and profile management can be managed by some
env.d python scripts that are user-aware and follow some database
I've been given this difficult task of unifying all of these tools
together and as you all can understand, I won't be having the time to
read through all eselect-* modules and *-config utilities code.
Please drop me a line here or at freenode if you have anything to add to
these ideas or have any further ideas that can help me on this project.
Thank you all in advance.
Cheers,
Sérgio Almeida
mephx @ freenode
Universal Select Tool
*Abstract
Universal Select Tool is an utility to manage system configuration.
This tool is similar to the unmaintained eselect utility of Gentoo or
Exherbo's eclectic. The idea is to create a tool that manages both
system settings and user settings with profile creation possibilities.
The utility will use mostly concepts from modules, softenv, and
both eselect and eclectic.
*Objective
The objective of this project is to create a unified configuration
utility for gentoo. Why this new tool and not eselect? Gentoo has a
set of configuration utilities (eselect-* *-config) that work with
global environment settings regarding the system and therefore needed
to be contradicted by hand by users to fulfill their need to change
it's own defaults. This is a common problem within clusters,
shared-servers, etc, basically all multi-user, multi-profile,
multi-use environments. Creating this utility makes environment
profiles, slots, virtuals and multi-implementations easier to manage,
create and manipulate by developers, sysadmins and users. The current
available solutions do not work this way and therefore enlarging the
gap between configuration unification, profiling and usability.
*General Ideas
Some modules that this project wants to unite do much more than just
update environment variables. Further discussion should be done with
all participants in such projects to check if a general tool can be
created and if not, what can be split as a module and what can not.
Modules will need to be thought in a way that no further *-config and
eselect-* utilities need to exist anymore. These utilities will be
replaced by a module of it's own that allows the universal select tool
to manipulate all available configuration that is specified by the
module.
Modules will not only specify to the universal select tool what can be
changed but also how to make those changes.
Modules will be indexed in it's own database file for faster
searching, switching, and profile creation and managing. This database
will need to be updated by
Re: [gentoo-dev] Google SoC @ Gentoo - Universal Select Tool
2009/5/4 Sérgio Almeida meph...@gmail.com: Please drop me a line here or at freenode if you have anything to add All I have to add is thanks a lot. I wished other SoC students would also introduce themselves and their project on their list. Denis.
