Kimbro Staken wrote:
>
> We need to get new development kick started here. To start I'm trying to
> collect all the bits of information/features/ideas/questions into a
> document for everyone to see. I've posted the start here
> http://www.xindice.org/papers/planning.html
>
> This is basically a brain dump of things that have been looked at/asked
> for in the project. It isn't complete, but there's already a lot of stuff
> there.
>
> Number 1 on the list is "Critical to bring in more developers". This is
> very important, if you're interested in getting involved here, now is the
> time.
>
> My plan for the immediate future is to continue filling in this outline
> and to collect feedback and expand on the items. I know that right now it
> probably isn't exactly clear what a lot of this stuff is, but I wanted to
> get the general ideas down first and then expand and detail. There's a LOT
> missing so please help me fill it in.
Ok, here are my full reviews of the TODO list:
Community
-----------
> Critical to bring in more developers
> Need some community oriented applications to develop
> The cocoon projects blog effort is probably a good start.
> Forrest is also a good effort
> Focus on cross project communication
> Evangelize the use of our code to other projects
I would like to share some thoughts on this (which I believe its the
most important area and where a mistake could slow the effort down a
lot)
I'm currently focusing on making Forrest a reality.
Forrest (cvs checkout xml-forrest, no web site yet) is an effort to
build the new web publishing and development infrastructure for
xml.apache.org.
It will run on top of Cocoon, it will be a dynamic web site, unlike
current one which is static.
This dynamicity will allow us to write stuff like:
1) graphs of logs (downloads, cvs commits, mail list messages, mail
list subscribers)
2) weblogs
3) project-related calendars
4) user-extensible in-place documents (as php.net does)
we could be using file systems or CVS directly, but Forrest will need a
database behind Cocoon.
Given the highly semi-structured nature of the data that forrest will
publish, a relational DB can't do everything without serious performance
limitations.
Result: I plan to use XIndice.
I strongly believe that Forrest might do good for *all* the above points
so I would welcome any partecipation in the [EMAIL PROTECTED]
mail list (shameless plug, but I think it's a good thing for everyone
under xml.apache.org if Forrest gets successful)
Project management
------------------
> Need to determine release roadmap
> Should probably split on two paths 1.0 and 2.0 series
Let me give you my personal suggestion: whenever possible, continue
incrementally.
Rewriting software from scratch normally causes *lots* of pain and is
much harder to manage since forking friction might develop.
Cocoon 2.0 took two full years to happen, with more than 5 different
developers helping out directly. This project has two (yes, I see this
growing, but only two people really know the code today and this is a
fact that must not be ignored).
My suggestion: use the 1.x series to move to 2.0 slowly and one feature
at a time, instead of doing a major rewrite for 2.0.
For Cocoon, we didn't do this because it was impossible: Cocoon 1.x and
Cocoon 2.x are so different that they don't share a single line of code.
I strongly doubt that the planned XIndice 2.0 will not be able to share
anything from XIndice 1.x, so my warm suggestion is to keep going with
the 1.x series, attacking one thing at a time.
Release early and release often.
It's the good old OSS tune, but it also the same thing that teaches
eXtreme Programming. My experience tells me that the '2.0 syndrome' has
normally wasted more energy than it ever saved later on.
And don't think about internal design: incremental evolution will bring
more people in faster, and you'll have more time to think about the next
step carefully and with more input from the outside.
Database Core
--------------
> Transactions
+0, my vision is clearly document-oriented so I won't plan to use them,
but I see no harm in having this in the core.
> Better support of different encodings
Absolutely!
> Schema validation in core
-0: I'm with Tom here: I don't think this should reside in the database
core, but if others find it valuable enough to contribute it and if this
doesn't cause internal performance degradation (both speed and memory),
I'd say we accept it.
Just keep in mind that without namespace-aware validation, we are going
nowhere, so I'd be -1 on DTD validation in the core.
> Focus on roundtripping?
What do you exactly mean by 'roundtripping'?
Canonical XML?
> Better support for large documents.
+1
> Incorporate XNode functionality.
-0
> Full text indexing is essential
+1
> Should explore other types of indexing options
don't know here
> Update jars to keep as current as possible
of course :)
> We should probably look at including some connectivity to other data sources
> Maybe a MySQL backend to start.
Hmmm, I'm curious here: why?
> Linking
> To keep and mature or to eliminate?
big +1 for keeping and maturing the concept!
> Expanded in database meta-data
> Need to see exactly what we do and do not have here.
metadata is vital for a decently fast use of XIndice. I would say this
is #1 priority in future development since Forrest might not be possible
for XIndice without the ability to cache resources and without exposing
metadata.
> Triggers
> Been planned for a while, probably still worth adding
Absolutely!
The fastest way to implement a cache system on top of a database is
having an list of valid resources on database caller's side and having
the database update that list after being triggered by an update.
This *inverted* caching scheme is *extremely* fast and effective. The
ability to perform this would turn XIndice into *the* XML database of
choice. this is my #2 priority.
> XMLObjects
> Are these really necessary as a separate concept?
> Especially questionable if we're embedding in Tomcat.
don't know what you are referring to here. can you elaborate a bit?
> Command Line Tools
> Simpler interface
> Maybe replace with interactive interpreter and SixDML implementation
A simple yet powerful command line interface is a must, expecially since
databases are normally managed over the wire, via ssh.
It must be said that with java it is entirely possible to use RMI over a
crypted socket, thus having GUI even for remote servers, but system
administrators simply don't like that so I don't think it's worth the
effort.
> Graphical Tools
> Bring in browser projects as part of the core?
Hmmm, I don't know: I would say so at first, then see if it's ok to
spawn another internal subproject (like Velocity or Avalon do on
Jakarta) when the release cycles of the two efforts start to get out of
synch.
> Graphical admin tool?
Might be useful only if capable of going over the wire in a crypted
fashion (or, at least, exchange passwords using digest challenging and
without passing the password over the wire.
> APIs and Remote Access
> Drop CORBA
+1
> Add HTTP access to documents
yes, this goes along with the WebDAV indication below. (WebDAV is, in
fact, nothing than HTTP++)
> Should strive for a much more consistent URI centric approach
+1
> Should we virtualize things more so that the collection path and any XPath
> can be treated as one unit?
+1!!!!! yes yes yes
> Would allow easy retrieval of bits of XML documents
which is a must for an XML database (otherwise, what's the point of
abandoning the relational model?)
> The database would appear logically as one big XML document, while physically
> being a different structure.
yes yes yes, this is, IMO, the ideal solution, even because it could
allow the creation of an easy database 'dump' as one big XML file (which
is going to be *extremely* useful for many situations like moving your
data to back-incompatible versions of the database)
> Could enable either the collection centric or document centric view to be
> used.
Yes, in fact, the 'collection' idea can be used at different levels: a
book is a collection of chapters, a chapter a collection of sections, a
section a collection of blocks, etc...
I would love to have a database that might logically appear as one big
persistent document, as a XML-oriented file system (with folders, files
and access control) or with other logical views, because that would
allow easy access to the system from the different realms of data
inserting and data query.
Admittedly, this is a document-centric view, but since this is very
likely to be one of the areas where XML databases make more sense, I
think it's very important to consider these things right from the design
phase.
> Add XML-RPC or SOAP based API
> Need to determine main API?
> XML-RPC is very simple and lightweight
> SOAP provides abilities for alternative encodings to be used. Could result in
> more > efficient in system use.
> Labrador proposed as possible solution
> Is this the right way to go?
I honestly don't know what to say here, but I think Sam might want to
add a few comments here.
> APIs need to expose more meta data about objects in the database.
yes, absolutely.
> Client API will evolve with SixDML becoming an XML:DB project and marging
> portions > with the current XML:DB API.
> Schemas should also be considered within the context of the API.
> Need to make sure documents with associated schemas and DTDs can be stored
> cleanly > in the database.
This is another interesting point against DTDs: they can't be stored
consistently in an XML database, unlike XMLSchema and RelaxNG.
> WebDAV support
> Either built in or a simple integration with an existing WebDAV impl?
> Current best route seems existing impl.
My personal suggestion would be to extend the Tomcat WebDAV Servlet (as
Gianugo already proposed). Slide is too complex and WebDAV is a very
simple protocol to implement if you don't mess into every detail.
Security
--------
> Access control
> Should be considered within the context of where we're going to go with the
> server framework
probably
> For embedded apps should probably be possible to get it completely out of the
> way.
Hmmm, I dare to disagree here:
One of my feature dreams of a native XML database is the ability to turn
parts of the tree transparent during my query: for example, suppose you
have something like
<data db:owner="xindice-dev">
<datum db:owner="stefano" value="10"/>
<datum db:owner="kimbro" value="20"/>
<datum value="30"/>
</data>
and suppose you have the information that there is a dependency of
owners, since both 'stefano' and 'kimbro' both belong to 'xindice-dev'.
Now, suppose you make a Connection to this database, submitting a 'role'
and authenticating with a password (or any other authentication
mechanism). Now the database knows *who* is making the query. So,
performing the same query, but with a different username, might return
different results, totally transparent.
xpath query: /data/datum
will return:
1) xindice-dev -> <datum value="30"/>
2) stefano -> <datum value="10"/> <datum value="30"/>
3) kimbro -> <datum value="20"/> <datum value="30"/>
4) root -> <datum value="10"/> <datum value="20"/> <datum value="30"/>
I see this as an *incredibly* powerful way to add different 'views' of
the same data, depending on 'who' makes the query ('who' might not be
which software, but which user is requesting data thru that sofware)
> Authorization
> Encryption?
> With HTTP based protocols and integrated SSL network level encryption should
> be > relatively simple. Depends on where we go for server framework
I don't think XIndice should mess with any crypto stuff at this level.
> Query Facility
> Add SixDML
what is this?
> Add XQuery
> Lots'o work - especially with typing
> Very important regardless
Yes.
> Shouldn't be a Xindice only effort.
I think the Xalan folks are interested in this (as Tom said, I know that
Scott Boag is)
> Queries should be able to be applied to a single collection or a collection
> and > all child collections.
I think it should follow thru nested collections, otherwise, there is no
single-document view, which is expecially useful during querying.
> Deprecate XUpdate?
> With SixDML XUpdate isn't strictly necessary so we should decide whether or
> not to > keep it around.
+1 for deprecation. I've never liked XUpdate.
Server Framework
----------------
> Move to Avalon for most framework services?
+1
> What exactly do we need from this?
> What are the benefits and the costs?
Expect high costs: Avalon is not only a server framework, but is also a
way of componentizing your application. Porting an application under
Avalon, normally is a hard operation, but pays off *a lot* later on
since refactored code is simpler cleaner and better interoperable with
rest of the avalon realms.
> Use Tomcat for the runtime environment eliminating all the old Juggernaut
> code?
what would you need tomcat for?
> Overall stronger focus on embedding and less on standalone server?
I don't know, but one thing is for sure: Cocoon doesn't currently ship
XIndice because it's too big and heavy with Corba support and all that.
Once CORBA is removed and Avalon is used, we would love to ship Xindice
with Cocoon so that you already have it when you install the thing and
you don't need anything else.... but in order to do this, you should
focus a little more on embedded operation, or, at least, in making the
layer the thinner possible.
> Need to look at other Java mechanisms to see if any fit. i.e. leveraging
> things > like JMX when it makes sense.
don't know about this, but I wouldn't dilute the effort too much: if the
need will emerge, somebody will add the feature themselves.
Regression Tests
----------------
> Current tests are based on the CORBA API
> We need new tests and a commitment to maintain them.
> Tests should exist for all levels of the system from the database core
> through client tools.
> Need to establish culture of test focus and passing the regression suite
> before any commit.
Absolutely.
Documentation
-------------
> More howtos
> More higher level applying the technology type information
> Need application case studies
> Add docs for internal APIs
+1
I also like to add:
*) versioning at the core level
*) full XML export/import of the database as a single big file (full
means that it should be possible to reimport the entire database in one
shot) [this will also help to see the logical structure of the data
stored in the database]
*) node-granular metadata and access control
Hope these comments help.
--
Stefano Mazzocchi One must still have chaos in oneself to be
able to give birth to a dancing star.
<[EMAIL PROTECTED]> Friedrich Nietzsche
--------------------------------------------------------------------
