On 30/11/06, Simon Laws <[EMAIL PROTECTED]> wrote:
>
> On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> >
> > On 30/11/06, Andrew Borley <[EMAIL PROTECTED]> wrote:
> > >
> > > On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > > On 30/11/06, Simon Laws <[EMAIL PROTECTED]> wrote:
> > > > >
> > > > > On 11/30/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > > > >
> > > > > > On 30/11/06, Pete Robbins <[EMAIL PROTECTED]> wrote:
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On 30/11/06, Simon Laws <[EMAIL PROTECTED]>
wrote:
> > > > > > > >
> > > > > > > > On 11/30/06, Pete Robbins <[EMAIL PROTECTED]>
wrote:
> > > > > > > > >
> > > > > > > > > On 30/11/06, Pete Robbins <[EMAIL PROTECTED]>
> wrote:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On 30/11/06, Jean-Sebastien Delfino <
> > [EMAIL PROTECTED]>
> > > > > > wrote:
> > > > > > > > > > >
> > > > > > > > > > > Pete Robbins wrote:
> > > > > > > > > > > > Our current method of packaging and loading an
> > extension
> > > is
> > > > > > > > fairly
> > > > > > > > > > > > simple:
> > > > > > > > > > > > we load all schema and libraries in the
extensions
> > path.
> > > > > This
> > > > > > > > has a
> > > > > > > > > > > > number
> > > > > > > > > > > > of problems.
> > > > > > > > > > > >
> > > > > > > > > > > > 1. An extension may consist of more than one
library
> > e.g
> > > .
> > > > > > > > > > > > libmy_extension.so
> > > > > > > > > > > > and libmy_extension_utils.so. Our current
loading
> > scheme
> > > > > will
> > > > > > > > > attempt
> > > > > > > > > > > to
> > > > > > > > > > > > load both of these and may fail on the one that
> > doesn't
> > > > > > provide
> > > > > > > > the
> > > > > > > > > > > > extension initialize method. On MacOS the output
of
> > our
> > > > > build
> > > > > > > > > produces
> > > > > > > > > > > a
> > > > > > > > > > > > libx.dylib and a series of symlinks to this
called
> > > > > > libx.0.dylib,
> > > > > > > > > > > > libx.0.0.dylib etc.. ur runtime loads ALL of
these
> > which
> > > > > > doesn't
> > > > > > > > > cause
> > > > > > > > > > > > problems as they are all the sam library and
just
> > > repeatedly
> > > > > > > > > register
> > > > > > > > > > > to
> > > > > > > > > > > > handle the same requests. Very inefficient
though!
> > > > > > > > > > > >
> > > > > > > > > > > > 2. Control of whether or not to load an
extension
> > > library is
> > > > > > > > > currently
> > > > > > > > > > > by
> > > > > > > > > > > > renaming the library so the runtime doesn't find
it.
> > An
> > > > > > example
> > > > > > > > is
> > > > > > > > > > > > that we
> > > > > > > > > > > > ship our python extension as
> > > > > libtuscany_sca_python.so.diabled.
> > > > > > > > This
> > > > > > > > > is
> > > > > > > > > > >
> > > > > > > > > > > > horrible and error prone.
> > > > > > > > > > > >
> > > > > > > > > > > > We could improve this by having a system
> configuration
> > > file
> > > > > > that
> > > > > > > > > lists
> > > > > > > > > > > > the
> > > > > > > > > > > > required extensions but the I like the self
> contained
> > > > > package
> > > > > > > > > approach
> > > > > > > > > > > > that
> > > > > > > > > > > > we have now. I'd like to implement an improved
> scheme
> > > for
> > > > > > > > packaging
> > > > > > > > > an
> > > > > > > > > > > > extension by introducing a per extension
> configuration
> > > file.
> > > > > > > > > Something
> > > > > > > > > > > > like:
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > tuscany_sca_ws_binding.extension
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > > <scacpp:extension name="ws binding"
enabled="true">
> > > > > > > > > > > > <library name="tuscany_sca_ws_reference"/>
> > > > > > > > > > > > <library name="tuscany_sca_ws_service"/>
> > > > > > > > > > > > </scacppp:extension>
> > > > > > > > > > > >
> > > > > > > > > > > > So the package would look like:
> > > > > > > > > > > >
> > > > > > > > > > > > extensions/
> > > > > > > > > > > > ws/
> > > > > > > > > > > > tusany_sca_ws_binding.extension
> > > > > > > > > > > > lib/
> > > > > > > > > > > > xsd/
> > > > > > > > > > > > other_folder/
> > > > > > > > > > > > ...
> > > > > > > > > > > >
> > > > > > > > > > > > The .extension configuration file is saying to
load
> > the
> > > > > > library
> > > > > > > > > which
> > > > > > > > > > > is
> > > > > > > > > > > > located somewhere in the package... the runtime
will
> > > find
> > > > > > it...
> > > > > > > > no
> > > > > > > > > > > > need to
> > > > > > > > > > > > specify a path.
> > > > > > > > > > > >
> > > > > > > > > > > > Taking this further the configuration file could
> list
> > > the
> > > > > > schema
> > > > > > > > to
> > > > > > > > > be
> > > > > > > > > > > > loaded. Currently the runtime will just load any
it
> > > finds
> > > > > but
> > > > > > > > these
> > > > > > > > > > > > may not
> > > > > > > > > > > > be needed by the runtime e.g. the schema may be
for
> > some
> > > > > > > > extension
> > > > > > > > > > > > implementation specific purpose.
> > > > > > > > > > > >
> > > > > > > > > > > > I think it would also be good for the extension
> > > > > > initialization()
> > > > > > > > > > > > method to
> > > > > > > > > > > > take as a parameter the root of the extension
e.g.
> > > > > > > > > > > > extension("/tuscany/extensions/ws"). This would
> allow
> > > the
> > > > > > > > extension
> > > > > > > > > > > > package
> > > > > > > > > > > > to contain any configuration information that it
> > needs.
> > > > > > > > > > > >
> > > > > > > > > > > > I'd like to start by at least introducing the
> > .extension
> > > > > file
> > > > > > > > for
> > > > > > > > > each
> > > > > > > > > > > > extension and loading only the specified
> library(ies)
> > if
> > > the
> > > > > > > > > extension
> > > > > > > > > > > is
> > > > > > > > > > > > enabled.
> > > > > > > > > > > >
> > > > > > > > > > > > Any thoughts?
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > > Two thoughts:
> > > > > > > > > > > - convention over configuration
> > > > > > > > > > > - the runtime should be consumable without
having to
> go
> > > tweak
> > > > > > XML
> > > > > > > > > > > configuration files
> > > > > > > > > > >
> > > > > > > > > > > If I remember correctly, renaming the dlls to
> .disabled
> > > was a
> > > > > > last
> > > > > > > >
> > > > > > > > > > > minute change to work around DLL loading errors
with
> our
> > > M2
> > > > > > > > release on
> > > > > > > > > > > Windows.
> > > > > > > > > > >
> > > > > > > > > > > I agree that we should do better than renaming to
> > > .disabled,
> > > > > but
> > > > > > > > I'd
> > > > > > > > > > > like to understand better the actual issues
that we
> > faced
> > > > > before
> > > > > > > > > > > inventing yet another XML based
runtime configuration
> > > > > language
> > > > > > :)
> > > > > > > > > > >
> > > > > > > > > > > I'm aware of the following issues:
> > > > > > > > > > > 1. We need the runtime to load extension libs
only,
> not
> > > other
> > > > > > libs
> > > > > > > > > under
> > > > > > > > > > >
> > > > > > > > > > > the extension directory which are not actual
> extensions
> > > > > > > > > > > 2. Same for XSDs, we need to load XSDs that
contribute
> > to
> > > > > SCDL,
> > > > > > > > and
> > > > > > > > > > > leave other XSDs under the extension directory
alone
> > > > > > > > > > > 3. Extensions that cannot be loaded because
some of
> > their
> > > > > > > > dependencies
> > > > > > > > > > > are not there should no break the runtime
> > > > > > > > > > > 4. The system admin / installer should be able to
> > disable
> > > > > > > > extensions
> > > > > > > > > > > that won't load because their dependencies are not
> there
> > > (I'm
> > > > > > not
> > > > > > > > yet
> > > > > > > > > > > convinced that this is still an issue if we manage
to
> > > solve
> > > > > > issue
> > > > > > > > #3)
> > > > > > > > > > >
> > > > > > > > > > > Did we run into any other big issues?
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > No, that's about it.
> > > > > > > > > >
> > > > > > > > > > 2. XSD loading can be done by convention (schemas
for
> the
> > > > > runtime
> > > > > > > > are
> > > > > > > > > > always in a folder called 'xsd'
> > > > > > > > > > 1. Could be solved in a similar way by only loading
> > > libraries in
> > > > > a
> > > > > > > > > folder
> > > > > > > > > > called ???
> > > > > > > > > > 3. Can be solved by just ignoring the load
> errors/issuing
> > a
> > > > > > warning
> > > > > > > > > rather
> > > > > > > > > > than giving up
> > > > > > > > > > 4. Can be solved by solution to 3.
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > > > I recall now why 3. was a big problem. Windows
sometimes
> > > throws up
> > > > > > an
> > > > > > > > > error
> > > > > > > > > dialog when the load fails so it was not just a
case of
> the
> > > > > runtime
> > > > > > > > > handling
> > > > > > > > > the load failure so we had to "disable" the extension.
> > > > > > > > >
> > > > > > > > > Cheers,
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Pete
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > --
> > > > > > > > > Pete
> > > > > > > > >
> > > > > > > > > Pete, just summarizing so that I understand. It seems
> there
> > > are
> > > > > two
> > > > > > > > parts
> > > > > > > > to this.
> > > > > > > >
> > > > > > > > A/ Building and installing an extension in order that it
can
> > be
> > > > > > consumed
> > > > > > > > by
> > > > > > > > the runtime. Building and installing extensions is
> potentially
> > a
> > > > > > > > separate
> > > > > > > > activity from building and installing the runtime
itself.
> > > Looking at
> > > > > > the
> > > > > > > > M2
> > > > > > > > download they all come together at the moment.
> > > > > > >
> > > > > > >
> > > > > > > Yes. It just happens that we are the only providers of
> > extensions
> > > at
> > > > > the
> > > > > > > moment.
> > > > > > >
> > > > > > > B/ Optionally consuming/enabling an extension in the
runtime,
> > once
> > > the
> > > > > > > > extension has been installed, in such a way that the
runtime
> > is
> > > able
> > > > > > to
> > > > > > > > find
> > > > > > > > and load it.
> > > > > > >
> > > > > > >
> > > > > > > Yes.
> > > > > > >
> > > > > > > If I understand the previous posts (and the current
code) A/
> is
> > > > > achieved
> > > > > > > > by
> > > > > > > > building the extension in the deployed folder layout
> > > > > > > > extension type/
> > > > > > > > bin
> > > > > > > > lib
> > > > > > > > xsd
> > > > > > >
> > > > > > >
> > > > > > > The current runtime does not mandate any layout. It will
seek
> > and
> > > > > > > destroy.. I mean seek and load any library it finds and
any
> .xsd
> > > it
> > > > > > finds.
> > > > > > > That layout is just how we package extensions in ou
build at
> the
> > > > > moment.
> > > > > > >
> > > > > > >
> > > > > > > and Pete you are proposing that this is adjusted so that
there
> > are
> > > > > > > > directories that hold just the stuff that the
contains the
> > > extension
> > > > > > > > rather
> > > > > > > > than the things it depends on.
> > > > > > > > extension type/
> > > > > > > > bin
> > > > > > > > extensionbin?
> > > > > > > > lib
> > > > > > > > xsd
> > > > > > > > extensionxsd?
> > > > > > > >
> > > > > > > > You are also proposing that B/ is achieved by ensuring
that
> > this
> > > > > > > > directory
> > > > > > > > structure be placed in a location that the the
runtime can
> > > search
> > > > > for
> > > > > > > > active
> > > > > > > > extensions (as it does at the moment in the deploy
> directory)
> > > > > > >
> > > > > > >
> > > > > > > it currently loads any extensions from under the
> > > > > > > <sca_install_dir>/extensions folder
> > > > > > >
> > > > > > > and that the
> > > > > > > > runtime ignores any badly configured extensions if
possible.
> > > > > Removing
> > > > > > an
> > > > > > > > extension's directory from the deploy
> > > > > > >
> > > > > > >
> > > > > > > extensions not deploy
> > > > > > >
> > > > > > > directory structure has the effect of
> > > > > > > > disabling it so I guess this is the fallback if the
runtime
> > > can't
> > > > > > > > continue.
> > > > > > >
> > > > > > >
> > > > > > > That is true in today's implementation.
> > > > > > >
> > > > > > > Anything more complex and, as Jean-Sebastien suggests, you
> start
> > > > > getting
> > > > > > > > into the full blown package and dependency management
> problem
> > > that
> > > > > > many
> > > > > > > > other systems try to solve in different ways, e.g.
rpm or
> > pear.
> > > > > > >
> > > > > > >
> > > > > > > Agreed... I'll give up on that plan ;-)
> > > > > > >
> > > > > > >
> > > > > > > What we need is to enable the runtime to identify which
> > extension
> > > > > > > libraries to load and which schema to load. My suggestion
is:
> > > > > > >
> > > > > > > my_extension/
> > > > > > > bin/
> > > > > > > extension/
> > > > > > > library_that_will_be_loaded_by_the_runtime.so
> > > > > > > lib/
> > > > > > > lib_my_extension.so
> > > > > > > include/
> > > > > > > ... some headers maybe
> > > > > > > xsd/
> > > > > > > my_binding.xsd
> > > > > > > any_other_folder/
> > > > > > > any_stuff_I_like.xsd
> > > > > > >
> > > > > > > So the runtime will load any xsd that is in a folder named
xsd
> > and
> > > > > > attempt
> > > > > > > to load any library in a folder called extension. It would
not
> > > attempt
> > > > > > to
> > > > > > > load the any_stuff_I_like.xsd or the lib_my_extension.so
> > > > > > >
> > > > > > >
> > > > > > > Cheers,
> > > > > > >
> > > > > > > --
> > > > > > > Pete
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Pete
> > > > > >
> > > > > > Sounds like the right idea to me. What goes in the bin dir?.
> > > >
> > > >
> > > >
> > > > Anything you like!
> > >
> > >
> > > On windows, .exe's and .dll's are often found in bin, whereas the
> > > .lib's (used for linking against at compile time) are generally
found
> > > in lib. Some projects (e.g. Axis2C) put dll's and lib's in lib
(which
> > > is more unix-like, and, IMO, nicer)
> > >
> > > >
> > > > Should the
> > > > > lib be...
> > > > >
> > > > > lib/
> > > > > lib_my_extension.lib
> > > >
> > > >
> > > > lif you like... again... anything you like.
> > > >
> > > > Maybe you could stick with bin,include, lib,xsd and then have
your
> > > "other"
> > > > > directory have everything else in it that the extension might
rely
> > on.
> > > > > Depends what you now anticipate being in bin and lib over and
> above
> > > the
> > > > > actual library exposing the extension.
> > > > >
> > > > > Will you continue to mandate a specific place where extensions
are
> > > > > installed, i.e. from you post <sca_install_dir>/extensions
folder?
> > > >
> > > >
> > > > Yes
> > > >
> > > > So that
> > > > > you can move an extension out of there if you don't want it to
> load.
> > > >
> > > >
> > > > Well the idea here is you don't need to remove the whole
> package..just
> > > the
> > > > library from the extension dir
> > > >
> > > > Simon
> > > > >
> > > > >
> > > > The layout of the extension package is whatever the package
creater
> > > wants to
> > > > make it. We don't care. All we will care about is loading any
> library
> > in
> > > a
> > > > folder anywhere in the extension package that is in a folder
called
> > > > 'extension' and expecting it to implement the extension
initialize
> > > method.
> > > > We will also load any schema files in any folder called
'xsd'. An
> > > extension
> > > > only needs the xsd folder if it has schema required for the
runtime
> > > model.
> > > >
> > > > As an example here is the ws binding extension implemented for
> axis2c:
> > > >
> > > > <sca_install>/extensions/
> > > > ... somewhere in this tree... could be anywhere...
> > > > ws/
> > > > xsd/
> > > > ws-binding.xsd
> > > > reference/
> > > > lib/
> > > > libtuscany_sca_reference.so
> > > > service/
> > > > lib/
> > > > libtuscany_sca_service.so
> > > >
> > > >
> > > > OK... today the runtime would load the ws-binding.xsd as it
loads
> any
> > > xsd it
> > > > finds. It would also load any .so it finds so both reference
aand
> > > service
> > > > extensions would be loaded/initialized.
> > > >
> > > > With my proposed change the xsd would be loaded (as it's in a
folder
> > > named
> > > > 'xsd' ) but the libraries would not. To get the extensions
loaded
> you
> > > would
> > > > have to package with an 'extension' folder ... again anywhere in
the
> > > tree...
> > > > which "contains" the libraries to load eg.
> > > >
> > > > ws/
> > > > extension/
> > > > libload_this.so ->
reference/lib/libtuscany_sca_reference.so
> > > > xsd/
> > > > ws-binding.xsd
> > > > reference/
> > > > lib/
> > > > libtuscany_sca_reference.so
> > > > service/
> > > > lib/
> > > > libtuscany_sca_service.so
> > > >
> > > > Here the runtime loads anything in any folder named
'extension' so
> the
> > > > reference extension would be loaded/initialized.
> > >
> > > Windows doesn't really do symlinks - I guess on windows we'd have
the
> > > tuscany_sca_reference.dll in extension/ and nothing in
reference/lib/
> > > or reference/bin/? For extensions that have dependencies on other
> > > libraries, those dll's could go in lib/
> >
> >
> > I would actually have a copy of the dll in extension on windows and
> delete
> > it to disable. For a large extension it would be very wasy to make a
> tiny
> > dll which just implements the initialise interface expected by the
> runtime
> > and leave everything else in a separate dll in bin/lib.
> >
> > > I would package the cpp language extension as
> > > >
> > > > cpp/
> > > > bin/
> > > > lib/
> > > > include/
> > > > xsd/
> > > > extension/
> > > >
> > > > The bin, lib, include are exactly what you would expect from a
> package
> > > that
> > > > you might want to build against. The extension/ folder would
hold
> the
> > > > library that implements the extension interface and xsd holds...
> well
> > > xsds!
> > > >
> > >
> > > All sounds good to me. +1
> > >
> > > Andy
> > >
> > >
---------------------------------------------------------------------
> > > To unsubscribe, e-mail: [EMAIL PROTECTED]
> > > For additional commands, e-mail: [EMAIL PROTECTED]
> > >
> > >
> >
> >
> > --
> > Pete
> >
> > Ok, so I think I get it. My concern is that giving the developer
the
> flexibility to put "extension" and "xsd" anywhere they like means that
> they
> will accidentally duplicate these directories as these are pretty
common
> names for directories. So we could ask them always to have them at the
top
> level. Doesn't remove the possibility for duplication but reduces the
risk
> a
> little. Or we could make the names a little more exotic with some
> qualification "sca_extension", "sca_xsd" for example, and maintain the
> flexibility you want.
>
> Simon
>
>
Yes... the names are always the hardest thing to come up with! They
should
not be sca_ as this is Tuscany specific.
so... we could get away with just the one folder tuscany_extension that
contains the schema and libraries:
cpp/
bin/
lib/
include/
xsd/
tuscany_extension/ <- schema and libraries implementing extension i/f
live under this tree
xsd -> ../xsd <- could even be a symlink to where they really are
I like that better. Runtime will load library and schema from any
subtree
under <sca_install>/extensions that is named 'tuscany_extension' and
ignore
anything else.
Nice and flexible.
--
Pete
That sounds like a plan. Nice one.
