I've discussed this email below with Mikhail. We discussed 2 points:
* How to support the following:
{velocity}
<div>
<h1>Hello $customer.Name!</h1>
<table>
#foreach( $mud in $mudsOnSpecial )
#if ( $customer.hasPurchased($mud) )
<tr>
<td>
* [link>$flogger.getPromo( $mud )]
</td>
</tr>
#end
#end
</table>
</div>
{/velocity}
The solution we've come to is the following for rendering the above:
a) When parsed with wikimodel, the velocity macro is called
b) the velocity macro uses an XML parser to parse its content (by
adding a top level xml wrapping element for example)
c) for each text content found during the XML parsing, call wikimodel
on it so that the wiki syntax can be interpreted
Same for the {html} macro. We could also use parameters to decide if
XML parsing should be done, if wiki syntax should be interpreted,
etc. For example: {html xml=true|false wikisyntax=true|false ...},
{velocity xml=true|false wikisyntax=true|false ...}.
* How to speed up document rendering?
We propose several caches:
- level 1: Parse documents using a wikisyntax DOM parser which
produces a DOM tree which is cached. This tree is composed of nodes
representing macros (unparsed or XML-parsed in a XML DOM tree which
is cached).
- level 2: Cache the non macro blocks after they are rendered (XHTML)
since they are static.
- level 3: Caching at the level of the rendered XHTML. This would be
a timed-cache (the cache is refreshed every N minutes). This is
absolutely required for heavy sites (Imagine Apache.org using XWiki).
I think we would need to cache pages for users not logged in at
least. Not sure how to cache pages when users are logged in.
Note1: level 2 and level 3 caches do not require a DOM tree.
Note2: we need the DOM tree to speed up all macros that act on
content, like the TOC macro as otherwise it means the content will
need to be parsed (which is hard and slow where traversing the DOM
tree is easy and fast). Since the DOM tree is cached the rendering
for these macros will be fast.
Note3: The TOC macro reminds me that we need somehow to support
macros that should be rendered last since they operate on the full
DOM tree. This is easy to do with a DOM tree but would quite harder
if we were only using a stream.
Any comment? Do you agree? Any other idea?
Thanks
-Vincent
On Oct 1, 2007, at 12:51 PM, Mikhail Kotelnikov wrote:
Hi! Excuse me please for the late response...
On 9/26/07, Vincent Massol <[EMAIL PROTECTED]> wrote:
Hi xwiki devs,
This is a summary of the decisions so far and the remaining
questions. This is also the outcome of my discussion of today with
Mikhail on skype.
1) We'll be able to import all syntaxes.
Yes
2) An XWiki instance will use a single syntax at a time. Once the
database is created using that syntax it won't be possible to
change it (except by doing an export and reimport).
We also need to decide what syntax we use by default OOB. I propose
we use the current xwiki syntax for some time and then switch to
the Wikimodel one (Common Syntax) later on.
I have no choice - I have to vote for the "CommonSyntax" :-)
3) All pages will be able to be exported to any syntax. Some
elements have no equivalent in other syntaxes and when this happens
a warning will be displayed and the elements in question ignored.
Yes. You can loose data only when you *export* your data from the
CommonSyntax, not when you *import* it. The WikiModel/CommonSyntax
support a super-set of elements defined in other wikis.
BTW: I think that it would be useful to involve Max Völkel (http://
xam.de/) in the discussion about wiki imports/exports. Max is one
of authors of the Semantic Media WIki. He proposed a Wiki
Interchange Format (WIF) <http://eyaloren.org/pubs/semwiki2006-
wif.pdf> which covers the topic. At least it would be useful to
know his opinion.
4) Mikhail has agreed to modify wikiModel to add macro block
recognition.The syntax isn't fully defined yet but it'll be
something like:
{xxx param1=value1 param2=value2}
...
{/xxx}
(param1='value value' and param1="value value" will also be supported)
This means we'll be able to have a common syntax for xwiki's macros
and also for groovy code:
{groovy}
...
{/groovy}
And also for HTML blocks:
{html}
...
{/html}
It is already done and committed. You can use "embedded" macro
blocks like
{xxx param1=value1 param2="long value 2" param3='this is a
parameter 3'}
... {yyy} ... {/yyy} ...
{/xxx}
Even embedded elements with the same names are possible:
{xxx}
...
{xxx}
... {yyy} ... {/yyy} ...
{/xxx}
...
{/xxx}
5) We need to decide if we want:
A) No velocity block but document properties/metadata to tell xwiki
to render the page using velocity. A user putting velocity code in
a page will have to check a box somewhere to say that this is a
velocity page.
Pros:
* Slightly easier to enter velocity code
Cons:
* Exception case compared to groovy, macros, etc
* User must not forget to check the checkbox saying the page
contains velocity code
OR
B) Velocity blocks same as what exists for groovy/macros/html, namely:
{velocity}
... velocity code with wiki syntax allowed
{/velocity}
Note1: For B) we would allow putting wiki syntax inside the
velocity block. Technically we'll apply the velocity rendering and
then re-apply wikimodel on the result.
Note2: For backward compatibility we can have a config flag
( xwiki.compatibility = 1) that automatically adds the {velocity}{/
velocity} marker around the whole page. The only downside is that
it'll be as slow as it currently is (actually it'll be faster since
wikimodel is going to be faster than radeox)
Pros:
* Speed. Since we know the blocks that use velocity we can cache
all the wiki syntax not inside the velocity/macros/groovy blocks
which will speed up considerably the rendering of pages
* Consistency with macros and groovy blocks.
My preferences goes to B) and I'm proposing to use that.
My +1 for B)
6) Mikhail is going to add support for recognizing XML tags in the
wikimodel parser so that onOpenXmlTag()/onCloseXmlTag() events are
called in listeners). This is needed for point 7 below.
Hmm... Yes, I said that I'll do it. Technically it is simple
(simpler than to add "macro" blocks). But conceptually it breaks
everything.
Explanations: Imagine that the listener already has the onOpenXmlTag
(String name, WikiParameters params)/onCloseXmlTag(String name)
methods.
Then the text:
----------------------
<table>...
This is a content of the table
...</table>
----------------------
will be reported as following:
----------------------
- onBeginParagraph
- onOpenXmlTag: => "table"
- onEndParagraph
- onBeginParagraph
- onWord/onSpace => "This is a content of the table"
- onEndParagraph
- onBeginParagraph
- onCloseXmlTag: => "table"
- onEndParagraph
----------------------
It means that the calls onOpenXmlTag/onCloseXmlTag will cross the
borders of multiple wiki paragraphs. And this is a BIG problem.
It means that we have to chose one of the following:
(A) Report HTML tags "as is" inside of wiki structural elements; In
the example above opening and closing "table" tags are reported in
two separate wiki paragraphs.
pro: It is the simplest solution.
con: if somebody want to treat these elements and create a well
formed document then it is up to the him/her to do it by hands and
to ignore non-appropriate structural wiki elements;
(B) Ignore wiki formatting inside of XML tags. In the example above
all wiki paragraphs will be skipped and only HTML "table" tags will
be taken into account; in this case WikiModel can not guaranties
that each opening element was really closed.
pro: It is doable.
con: a) It breaks completely the idea of the WikiModel - to give
access to a well-formed structure of wiki documents; b) the grammar
will be bigger; c) It is not so simple to implement
(C) Add some HTML tags as markup elements for the CommonSyntax. In
this case each '<table>...</table>' tag pair will be interpreted in
the same way as normal wiki tables. The same for "ol/ul/li/dd/dl/dt/
p/span/div/..." elements.
pro: you can mix your wiki and HTML markup with the same meaning
and all of them will be reported in the same way to the listeners.
con: a) the grammar will be bigger; b) I have to do much more
additional validations of documents by hand to guarantee that the
document is well-formed; c) the parsing will be much slower (as the
consequence of a and b); d) it is much more difficult to
implement; e) the number of possible HTML elements have to be fixed
in advance
From my point of view neither option is good.
One another approach to resolve this problem - see below, in the
response to the next question.
7) We need to allow intermixing velocity/HTML and wiki syntax
easily. For this our listener (the code that listens to {velocity}
events) will evaluate the content using velocity and will call wiki
model again on the resulting code. Since wikimodel requires HTML to
be in a block ({html} for us) we'll use a different wikimodel
listener that intercepts the onOpenXmlTag/onCloseXmlTag so that
it'll output XML tags with no modifications (the standard HTML
listener generates < and > for < and >). This will allow
writing:
{velocity}
<div>
<h1>Hello $customer.Name!</h1>
<table>
#foreach( $mud in $mudsOnSpecial )
#if ( $customer.hasPurchased($mud) )
<tr>
<td>
* [link>$flogger.getPromo( $mud )]
</td>
</tr>
#end
#end
</table>
</div>
{/velocity}
I propose to interpret the content of such blocks in the following
manner:
- The given text is interpreted as a "relaxed-XML" where all
opening XML tags have to be closed (or a tag has to be empty)
- If an XML tag has a text content (not only spaces) then this
content is interpreted as a wiki syntax. In this case the wiki
content can be handled by a normal wiki parser.
In this case the scenario of usage can be following:
1. You parse your initial wiki document and create a backbone
structure of the document containing macro blocks
2. All macro blocks corresponding to template blocks are handled by
the corresponding engine (ie velocity)
3. The output from template engines is used as an input for such a
"relaxed-XML" parser
4. A not-empty text content of tags is interpreted as wiki content
using separate wiki parser instances; the content formed by these
wiki blocks should be inserted in the document from the step 3.
pro:
* It seems that it is the simplest solution and it resolves
problems with inter-mixing of the wiki syntax/XML/HTML from the
previous point (if such inter-mixing is available only in macro
blocks).
* For such a "relaxed-XML" content existing XML parsers can be
used. It is possible just to add the "<xml>...</xml>" pair around
the content and use directly a normal XML parser. But there is a
risk that this document is not a well-formed XML. Or an HTML
cleaner (JTidy/NekoHTML/...) can be used to get a well-formed XHTML
before the XML parsing.
* All steps gives well-formed structures
con: More parsers in the chain of the page treatment => the
treatment is slower (this can be partially compensated by
additional caches in the future)
Personally I definitely prefer this approach. And from my point of
view it resolves the problems with onOpenXmlTag/onCloseXmlTag
mentioned above.
8) Documents are stored in textual format in the DB (i.e. as the
user sees them). Portions of them will be cached after they're
rendered for the first time (see option B above for the best
caching option).
Are you ok on these points and especially about using the 5B solution?
Anything else I've forgotten?
If we agree, then my next steps are:
* Understand the wikimodel API in more details
* Understand the doxia API too (it's a "competitor" to wikimodel).
The reason is that I'd like to see two implementations to ensure
that the XWiki Interfaces can be implemented using different
implementations so that XWiki is independent of the underlying
rendering/parsing framework used. Jason Van Zyl is also interested
in implementing the doxia part for XWiki in the future.
I think that in any case it is a good idea to have a specialized
interface (API) for each type of functionalities to isolate the
core of the system from external libraries/frameworks.
BTW: Thank you for the pointer on Doxia! I will see it more in
details. After a brief look I saw that WikiModel contains very
similar modules with the "Sink" object of Doxia. I think that it is
very simple to create a WikiModel/Doxia bridge. And it seems
possible to add the APT ( http://maven.apache.org/doxia/references/
apt-format.html) format support directly to the WikiModel.
Best regards
Mikhail
* Propose a XWiki API
* Propose an integration path
* Implement it using WikiModel
Thanks
-Vincent
_______________________________________________
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs
