On 11/20/14 03:57, Atgeirr Rasmussen wrote:
Hi everyone!
First, on the original topic of this thread: I think the new repo is
fine, it may be a useful alternative to consider placing such things in
a utilities/ directory on opm-core instead to reduce the number of
repositories though.
Second, about gentoo, development platforms etc.: all developers have
their preferences, and we basically have scratched our own backs when it
comes to platform/distro support.
Yes, I agree. Each Developer should have their own fiefdom (repo, branch
whatever).
I make sure it works for me (Mac OS X), others make sure it works on
SUSE, Ubuntu, RHEL etc. There is as far as I can tell only one
compromise made (in the sense of restricting the developers) for
compatibility, and that is that we should support gcc 4.4 (which means
that certain C++11 features are not to be used for example). I see that
gentoo is making some people very enthusiastic, and may try it in the
future, but platform support will come from people who actually use it
of course. When it comes to compilers, I use a recent clang, so do some
others. I know people are using gcc versions from 4.4 to 4.9. So I think
our platform support is broad enough from a developer's point of view.
This is also an good statement. What general architectural documents
provide, for one thing, is the scope and usage of the standard
languages that are acceptable to the project's core; as you have
partially illuminated above. This is a great start to part of the
general documents, that are needed. When you want to use another
language, say R, you just put your work into a separate module and
document the compile time and runtime issues that must be satisfied.
Modular design now and there are no restrictions for new modules and codes.
Third, about user friendliness and support: this year, work on such
things (in particular creating a new proper release) has taken a back
seat to completing the work on the fully implicit black-oil simulator
(opm-autodiff's sim_fibo_ad as it is currently named). This is quite
unfortunate, but it is both a question of resources and priorities, and
also about not advertising a release until the new things in it are
complete. The second point is probably a bit controversial: it is
against the 'release early, release often' mantra that is often
considered desirable. It is a bit psychologically motivated, I think: if
I feel that the complete version is just around the corner, I'd like to
wait just a little bit before releasing. Of course that may turn into a
problem when things take longer than expected, as they often do (and did
in this case as well).
Wow, I think you completely missed my point. Your points made above
are very valid, in a Microsoft or Apple world. They miss the point
of open source development, completely, imho.
Users are not second grade kids, but mostly over-educated professionals
who if they choose to, can code, modify codes, and ask deeply troubling
questions to the developers. These sort of (technical) users, are the
reason software becomes robust. What you have articulated so far, does
not encourage others to "join in" and become serious users or
contributors, both fundamental reasons for open source projects.
Part of the user-facing complexity is the existence of several modules
and their non-trivial dependence relations. Also the fact that different
modules may require different third-party libraries, some of which may
be optional.
The dependencies should be robustly handled with compile time flags and
run time flags. These can all be brought out into the CLI or passed on
as settable flags (passed parameters) to other codes. What I'm proposing
here is the robust usage of flag settings, so an array of current
choices and a myriad of future choices, are standardized in how they are
optioned, selected and the resulting codes compiled. Let's get one
thing straight. I'm not promoting Gentoo here. I have used dozens of
operating system and languages over the decades and as such am agnostic
to those decisions. In fact, for an open source project to become
robust, it should be readily compilable on a variety of platforms. That
said, Gentoo has solved this quandary that complex software development
creates and gives users a choice of package managers (portage and
Paludis). In fact there is nothing preventing yet another package manager.
What Gentoo does if force the tough decisions to be solved, before
rolling out a binary mess. Cleaning up the "mess" periodically and
calling it a "release" is a very weak method of for modern management
of codes. What most robust (rolling if you like) linux distros offer now
is stable and testing releases that are timed by the individual
developer. They are available along side the established (stable)
versions that have been previously released. The user merely selects
at compile time which stable or testing version of a module to include.
There will always be a tension between wanting everything to be in one
big module for simplicity and wanting each piece to be as independent
from others as possible.
Boy, I do not know where to begin with this statement. If you write,
debug, test and make the patches available, folks will frequently
update (sync), test and verify the code is good, particularly the devs
of other modules. There is no every 6 months or so (nightmare) to clean
up. Furthermore, astute develops will keep their code clean and
discernible, so technically astute users will offer up patches to that
dev, because they are a *team*.
Rolling a bunch of small problems into one giant mess has never
simplified resolution; the incited finger pointing just gets worse.
That is does not simplify anything, it adds overt, unnecessary
complexity. It obscures bad decisions that much later need to be
reversed. It complicates accountability for codes that are fundamentally
flawed. Beside different coding styles have to coalesce; so without
pulling a code review of the current codes, I'd be willing to bet there
are no current coding standards for this project. In fact I'd be
surprised if any of the repos, where 3 or more folks are currently
contributing has an established and published coding standard?
Toss as much a possible into a single core module ? Very bad idea.
My current views are:
- There are currently too many modules, and it is bad that opm-core is
not the root of the tree. Suggestion: combine opm-parser and opm-core
into opm-core.
I'm not going to challenge this. I'm going to blindly trust that the
devs that have written most of the "core codes", whatever that is,
should agree on how to organize the code. However there are a few
fundamentals to keep in mind:
(A) The core (base) code should compile and install on many different
platforms as reasonable possible. 2 or 3 is not sufficient to be robust.
(B) The core code should be designed with the express intention of
creating an environment where many different modules can interact with
the core code. Multiple API, methods for data movement and interaction
with other softwares are keen design goals of all open source softwares.
(C) Everything that can be extracted into another separate module,
should be developed that way. Grids and solvers are but two of a myriad
of codes that need to have a straightforward method to allow
for competing modules. Surely advance languages, with unique
capabilities, such as but not limited to R, Scala, and Haskell will
bring forth robust (read more and faster modules with many more
features in new modules that will make OPM very, very attractive to a
much wider audience. So do not force things into the core for the sake
of making it a massive singular kludge. Modular software design
methodologies abound, for good reason; read about them (if necessary)
and keep the core clean, minimize and robustly well documented.
(D) A, B and C are not a comprehensive list, merely a starting point for
the OPM architects and USER communities to discuss and formalize.
- We do not want everything in one module, we should have a fairly low
threshold for adding a new module with new functionality.
YES!
But we should allow the core (or a few modules close to it) to grow to
include such things as they mature.
Why? To make debugging a nightmare? To encourage crusted up, old codes
to not be re-worked? If you go and look and study any of the really
big, important projects, that have survive more than a decade, you'll
find most all of that original code has been replaced, often many times.
It's very difficult to write robust code the first few renditions, as
everyone knows. You create a mega-monolith, the code becomes too
expensive to maintain or innovate with and the project dies, or requires
a world class expertto debug.
Just read up on "heartbleed"/openssh [2] or ffmpeg for some illuminations.
- Adding third party dependencies (such as Eigen) to the core is
acceptable if it is easy to satisfy this prerequisite on the most
important platforms (Ubuntu and RHEL in my mind), easy meaning 'can be
done with a maximum of 2 commands'.
Properly written codes (modules) should compile stand alone. Maybe a
small amount of wrapper code is needed to it can be used, stand alone.
Confusing OS syntax with code quality is what you are doing here. A
module needs to compile with no errors (hopefully warning are a result
of options not set and other trivia). And be usefule by itself.
So it should be no more complicated than adding a PPA and apt-get. And
it should not be hard (requiring manual tweaking) on any of the
platforms we support or work on
Here we go again. Designing open source software, crippled from the
beginning with the semantics (short comings) of a particular distro
is a certain recipe for failure (code too difficult to maintaining,
troubleshoot or extend). Many folks use gentoo ebuilds as a guide to
port codes to other OSes. It's a well establish practice. There is
nothing obscure about it, other than a well documented set of tools that
enhance standard linux shell commands. It reads like a manual of how to
compile a given code, including the compile time and run time
dependencies. Gentoo will facilitate porting codes to MacOS, windows,
BSD and a plethora of chip architectures, by the very nature of working
on Gentoo.
(I'd like to get rid of SuperLU usage for this reason). A new code
having a new third-party dependency is often a good reason for putting
it in its own module.
- Using branches instead of repos seems very alien to me, I do not
think it brings much benefit, but it does require more attention and
care from users and developers. If I want to use code together that
resides in two different branches, I do not see a simple way to do this,
other than having multiple working copies of the repositorym each
checked out to a different branch.
OK, let me try to articulate this with some greater granularity.
Branches, repos, various codng (style) semantics, languages, data IO,
etc etc, do not matter. Let each of the Devs have their fiefdom, as they
deserve that level of respect and support. However that said, these
individual devs become burdened, as they should be, with the
responsibility to ensure that their work compiles into and work with
the (base) core codes as well as those other established modules.
What does matter is that the "core" is defined as above, with more
design constraints that effect openness and interoperability with as
wide a variety of codes as possible. The more modules, the more keenly
the base (core) codes become, tested, debugged, extended, maintained and
extended to support *ALL* of the companion modules.
Bring code in is as simple as compile time and run time options, that
the user selects when they build OPM. The mechanism is compile time and
and run time flags. Gentoo solves this by supporting multiple "stable"
versions of codes. We use a mechanism referred to as "slots" to
distinguish the various stable versions of codes. And a given stable
version of a code may or may not support the same list of flag setting.
The ebuild for that stable version of a code lists those constraints of
the other module versions it can inter-operate with. The user simple
selects the flags they want. There are commands and places to read what
each flag does on each code
- As everyone else, I agree that the current situation with the
build-system being duplicated everywhere is not good. I do not think we
have reached a consensus about how to replace it though. I'd like for
all of it to be in opm-core only, and other modules only having a way to
find opm-core, and adding their own file lists etc.
Huh? New folks should be able to download code, run make, and execute
binaries on a wide variety of platforms. Not having design document that
insist on this "openness" is the exact reason we have a kludge now.
We cannot even publish a stable version beccause of this sort of
pathetic posturing. You write code, it should be compilable every few
days, or a week at the most. If is usable, folks on at least (6) other
platforms should be able to download, compile and test, particular other
devs and technical users. If that is not your semantic, your semantic is
fundamentally flawed, imho.
The reason we are all waisting time on this now, it because the project
was not robustly organized with conceptualization constructs and
criteria that prevent this sort of aloofness. If we fix this once, now,
then we'll save an incredible amount of(wo)man-years in efforts down the
line. Gentoo is not a distro for the average computer user. It is a
distro where everything, including compilers are built 100% from the
ground up. Gentoo fleshes out many problems before the consumer uses the
binaries, so as to make the developers look like extremely accomplished
developers. Technical users are a fundamental component
if for no other reason than to robustly provide a diverse testing
environment. It forces openness, inter- interoperability and provides a
well established path for technical users to become developers. Gentoo
is ported and encourages porting to an enormously wide variety of
hardware platforms; this substantiates it's semantics for code
development, compiling and maintenance and yes forking.
If you look at the wide variety of competing hardware that users should
be able to run OPM on in the future (arm, SIMD, MIMD, HPC, clusters,
clouds, x86 and derivatives just to mention a few), you also solve the
future portability problems by properly implementing a robust, open,
well defined, modular coding semantic now.
Otherwise OPM become a useless kludge that will be so miserable to
maintain, that it will be prematurely abandoned, because no new
developers will be attracted to this project. It's a boys club now,
let us make it an open source project, we can all be very proud of?
There are other ways to solve these problems, but their solution has
already been achieved by Gentoo. Folks that develop on gentoo have
their work readily ported to any other OS out there. Most of what is
developed elsewhere is a major pain to port to different OS. Truth be
know, I know lots of folks that develop codes on gentoo and then
purposely port to target hardware, like embedded systems, SOC, FPGA,
Asic and wider variety of commonly know Operating Systems; gentoo is a
well kept secret among technical users and developers.
Gentoo has a livedvd folks can play with now [2]. Critiques
about gentoo [3] abound and are correct concerning novices. For
developers, a traditional gentoo install takes a few hours, with
moderate, periodic intervention. Afterwards, it is always upgradeable
and is fantastic for installing the latest codes from a variety of
sources, developing software, and having access to an incredible cadre
of experts, many of whom will debug and write code snippets for you, on
the fly. Gentoo does become "an addiction" for technical users,
because there is no other distro that is as powerful, and puts the
*USER* in the driver's seat. YMMV.
And I'll have a usb image so folks can run gentoo, install and compile
codes for saving. That way, you can keep gentoo on a usbstick, on your
keyring! Do wish me luck?
If the codes and the project are properly structured, via proper desing
that is well thought out and inclusive of diverse platforms, gentoo
is but one of many. If you find yourself struggling for that "nirvana"
then I invite you take gentoo for a test-drive.
Atgeirr Flø Rasmussen
peace,
James
[1] http://www.theepochtimes.com/news/5-7-5/30084.html
[2] https://www.gentoo.org/news/20140826-livedvd.xml
[3] http://www.linuxinsider.com/story/81207.html
_______________________________________________
Opm mailing list
Opm@opm-project.org
http://www.opm-project.org/mailman/listinfo/opm
