On Saturday, February 16, 2002, at 01:30 PM, Tom Bradford wrote:
On Saturday, February 16, 2002, at 01:49 AM, Kimbro Staken wrote:We need to get new development kick started here. To start I'm trying to collect all the bits of information/features/ideas/questions into a document for everyone to see. I've posted the start here
http://www.xindice.org/papers/planning.html
3. Database Core 1. Transactions 1. What's the strategy to implement this? 2. Could bootstrap off Berkley DB
We definitely need transactional capabilities. I had some ideas for this early on, but they've oozed out of my head since then. I'll think about this a little more deeply when I get the chance. I share the same concerns regarding Berkley DB, and would prefer to avoid it for now.
Well my point wasn't that we should avoid it, but that we should consider using it as a starting point. It would allow us to focus initially on the higher level transactional concepts and figure out how we make the overall system transactional before diving into the really difficult stuff. I don'
t see a problem with telling people for the short term that if you need transactions you have to go download some sourceforge project to get them.
I'm not suggesting it as a long term solution, just as a bootstrap.
4. Focus on roundtripping? 1. How important is this in the real world?
Not sure it's so important... But it depends on how compliant with the W3C XML data model/infoset we want to be.
The Roundtripping I'm talking about doesn't really have so much to do with the infoset, it's more syntax related. I believe for some applications it can be pretty important that what goes into the database is the same as what comes out. Maybe the solution there is simply saying if you need to be able to roundtrip, store canonical XML.
5. Better support for large documents.
The DTSM spec, combined with streaming capability going in and out of Filers will make this possible, and should be incredibly hard to do. I also want to explore using Java 1.4's new I/O system for more efficient paged IO.
It will be a long time before we can depend on 1.4 though. I doubt it will happen within even the 2.0 release cycle.
8. Should explore other types of indexing options
We need indexes based on a data model that takes into account that documents may have several different contexts with the same element or attribute name which are actually bound to different types. We can't have an optimally performing schema system without this.
10. We should probably look at including some connectivity to other data sources
1. Maybe a MySQL backend to start.
We should look into possibly merging Ari's DBDOM stuff as a backend.
Is Ari still working on it?
11. Linking 1. To keep and mature or to eliminate?
I'm for eliminating it.
I'm still not sure about this. It's not a priority either way, but I'm not prepared to dismiss the idea yet.
12. Expanded in database meta-data 1. Need to see exactly what we do and do not have here.
It's something I've wanted. It's hard to implement consistently if we expand out to several backends, but it's workable.
What are we tracking today?
13. Triggers 1. Been planned for a while, probably still worth adding
I agree.
14. XMLObjects 1. Are these really necessary as a separate concept? 2. Especially questionable if we're embedding in Tomcat.
Not so sure. I would still lean toward having them. If we were to incorporate Labrador, XMLObjects would be a free. We'd only have to write enough code to load and instantiate classes, but not much else.
If you're embedded in Tomcat why do you need a separate concept? Why not just build on something that already exists?
4. Command Line Tools 1. Simpler interface 2. Maybe replace with interactive interpreter and SixDML implementation
I like the idea of an interative SixDML interpreter. Especially for people who might be writing batch scripts and such.
5. Graphical Tools 1. Bring in browser projects as part of the core? 2. Graphical admin tool?
There should be a single admin/browser tool, and I'm leaning toward saying that it should be web-based, rather than a standalone GUI.
In general I agree, but given a choice between pulling in an existing project and trying to bootstrap a new one I'd probably choose the former.
4. Should we virtualize things more so that the collection path and any XPath can be treated as one unit?
1. Would allow easy retrieval of bits of XML documents
2. The database would appear logically as one big XML document, while physically being a different structure.
3. Could enable either the collection centric or document centric view to be used.
The problem with this is the same thing that I pointed out in our SixDML discussions. It creates a potential for ambiguity that isn't easily overcome, and will lead to non-deterministic behavior. This type of thing isn't something I'd recommend.
Except it may be more logical and friendly to the user and because of that is worth exploring. Right now I don't know if it's a good thing or not, just something worth looking into.
5. Add XML-RPC or SOAP based API
1. Need to determine main API?
1. XML-RPC is very simple and lightweight
2. SOAP provides abilities for alternative encodings to be used. Could result in more efficient in system use.
2. Labrador proposed as possible solution
1. Is this the right way to go?
Of course it is!
I have several concerns with Labrador and I'm not at all convinced it is what we need or the way we should go. Here's the issues I have with labrador, note my issues are mostly non-technical and have nothing to do with my opinion of Labrador on a technical level. My comments only apply to its use within the scope of the Xindice project.
1. It overlaps in goal with Axis, http://cvs.apache.org/viewcvs.cgi/~checkout~/xml-
axis/java/docs/requirements.html Axis may not support XML-RPC today, but then Labrador doesn't support SOAP either. SOAP is a lot harder then XML-RPC.
2. It is very immature and missing large amounts of functionality.
3. It only has one developer who is also a primary developer on this project. (and it must be mentioned is also working for a new company that will consume large amounts of his time)
4. There is no community supporting it, this makes it very risky.
5. The framework is relatively complex. While it is simple to plug in basic services the immaturity of the framework it self and the lack of community will necessitate our needing to fix issues in the core code. This brings a lot of baggage and risk.
Basically, if we use Labrador and you decide/are forced to stop working on it for whatever reason this community will have no choice but to pick up maintenance. This is not a safe thing to depend on. I'd prefer to simply depend on Axis and Apache XML-RPC directly and defer all development on protocol support to those communities. If Labrador develops momentum and a community then we can look at refactoring to leverage it. Until then I do not think using it is the right decision for this project.
2. Add XQuery
1. Lots'o work - especially with typing
1. Very important regardless
2. Shouldn't be a Xindice only effort.
3. Queries should be able to be applied to a single collection or a collection and all child collections.
4. Deprecate XUpdate?
XQuery is something I spoke to Scott Boag about before he got ill and was unavilable for a month. We should assume that this will be a joint effort between the Xalan and Xindice projects (along with any other projects who need to be involved)
Yes, this will probably be a pretty broad project.
I'm for deprecating XUpdate and possibly implementing one of the proposed update syntaxes for XQuery. We can't expose a query language that only allows you to update using static values (XUpdate). Otherwise, we need to extend XUpdate to support calculated expressions.
Well to start SixDML supports updates in a similar manner to XUpdate so that can replace it. I agree on also working with one of the proposed update syntaxes for XQuery.
9. Server Framework 1. Move to Avalon for most framework services?
Yes... In my experimental Xindice stuff, I've almost completely eliminated all of the Juggernaut code except for Services. These can easily be translated to Avalon blocks.
What functionality are we getting from Avalon?
3. Overall stronger focus on embedding and less on standalone server?
We should try to strike a nice balance between the two. I think this shouldn't be too hard to do based on the work that I've done.
Yeah, i didn't mean we should focus on it completely, just that we don't really focus on it very much right now. I think embedding will be a very common use so I want to make sure it's clear how to do it and that it becomes a focused goal while developing.
-- Tom Bradford - http://www.tbradford.org Architect - XQRL (XQuery Engine) - http://www.xqrl.com Apache Xindice (Native XML Database) - http://xml.apache.org Project Labrador (Web Services Framework) - http://notdotnet.org
Kimbro Staken XML Database Software, Consulting and Writing http://www.xmldatabases.org/
