Re: [xwiki-devs] [Brainstorm] Notifications filters capabilities and performances.
The two main problems that I see are that you're mixing two separate things: 1. Tags which are stored in one place, and events which are stored in a different place 2. Again tags, and document locations. They may seem similar, but they are completely different as they are implemented. That's why saying "not in this wiki, except for this tag" cannot ever be implemented in a sane way since the exception is on a different plane. The logical approach would be to also store the tags in the events, but this is not a good thing to do. Where do we stop? For future-proofing filters, we would have to store the entire document in the event, in case someone wants to add a filter on "documents with a specific type of object attached", or "documents with a specific value for a specific property of a specific object", or "documents with attachments ending in .extension". On 06/28/2018 04:44 PM, Guillaume Delhumeau wrote: > For the tags filter, I can also: > > * perform a query to fetch all documents that correspond to the given tags > (it doesn't need to be permanent, but it would be better to cache it) > * add a HQL filter on these pages (OR document IN :list_of_pages). > > It's a little variation of solution A. It's ugly but it could do the job. It's the sanest proposal so far, but still impossible. Since "not in LOCATION except WITH TAG" doesn't work because it mixes two types of information in the same query fragment, how about making this a one-dimensional query as "only WITH TAG". And since this information is in a different place, pre-querying it is needed. However, if there are thousands or tens/hundreds of thousands of documents with a tag, won't the query crash anyway? > 2018-06-27 12:58 GMT+02:00 Guillaume Delhumeau < > guillaume.delhum...@xwiki.com>: > >> Hi developers. >> >> I am trying to add a new filter to the notifications to be able to follow >> pages >> that are marked with a given tag. And it leads me to some questions about >> the >> technical implementation of the notifications. >> >> To remind the context: notifications are computed on top of the events >> recorded >> by the event stream (a.k.a activity stream). We take events from the event >> stream SQL table, we apply some transformations on them, and we display >> them to >> the user. >> >> Then we have implemented the ability to filter on these events: for >> example >> "don't show events concerning the document A nor the wiki B". Filters are >> implemented with 2 distinct ways: >> >> 1/ SQL injections: each filter can add SQL elements in the query we make >> to >> fetch the events from the event stream table. We made this mechanism >> so we >> can let the database do a lot of the filtering process. After all, >> it's its >> job and it's supposed to perform well. To be precise, Clement has >> even >> created an Abstract Syntax Tree (AST) so it's easier to inject some >> content >> in the query and it creates an abstraction over the SQL language so >> we can >> even consider to change the storage of the event stream someday. >> >> The bad thing is that some complex filtering are difficult to write >> with >> the SQL language (event with the AST) or even impossible. >> >> 2/ Post-filtering: after the events have been fetched from the database, >> each >> filter can still decide to keep or filter them. This is useful for >> complex filtering that cannot be expressed with the SQL language. It >> is >> also needed by the real-time notification email sender, because it >> takes >> the events immediately when they occurs without fetching them in the >> database (so SQL filters are bypassed). >> >> The bad thing is that some events are loaded in the memory to finally >> be >> rejected, and these filters can perform costly operations such as >> loading >> documents. >> >> Until now, this double mechanism was working quite well, with each >> mechanism >> filling the lacks of the other. >> >> However, we still have technical limitations in our design: >> >> 1/ Users who have a lot of filter preferences can end up with a giant >> SQL >> query that is almost impossible to perform by the database. Actually >> we had >> a user complaining about an OutOfMemory problem in the HQL to SQL >> translator ! >> >> 2/ I cannot implement the tag filter ! >> >> The tag filter is supposed to show events that concern pages that hold a >> given >> tag, EVEN IF THE PAGE WAS EXCLUDED BY THE USER. Example of use-case: "I >> don't >> want to receive notifications about wiki A except for pages marked with >> the tag >> T". >> >> And it is not working. First because it is difficult to write a SQL query >> for >> that. It requires to make a join clause with the document and the object >> tables, >> which our SQL injection mechanism does not support. Even if it were >> possible, >> creating a SQL join with the document table will de-facto filter events >> that do >>
Re: [xwiki-devs] [Brainstorm] Notifications filters capabilities and performances.
For the tags filter, I can also: * perform a query to fetch all documents that correspond to the given tags (it doesn't need to be permanent, but it would be better to cache it) * add a HQL filter on these pages (OR document IN :list_of_pages). It's a little variation of solution A. It's ugly but it could do the job. 2018-06-27 12:58 GMT+02:00 Guillaume Delhumeau < guillaume.delhum...@xwiki.com>: > Hi developers. > > I am trying to add a new filter to the notifications to be able to follow > pages > that are marked with a given tag. And it leads me to some questions about > the > technical implementation of the notifications. > > To remind the context: notifications are computed on top of the events > recorded > by the event stream (a.k.a activity stream). We take events from the event > stream SQL table, we apply some transformations on them, and we display > them to > the user. > > Then we have implemented the ability to filter on these events: for > example > "don't show events concerning the document A nor the wiki B". Filters are > implemented with 2 distinct ways: > > 1/ SQL injections: each filter can add SQL elements in the query we make > to > fetch the events from the event stream table. We made this mechanism > so we > can let the database do a lot of the filtering process. After all, > it's its > job and it's supposed to perform well. To be precise, Clement has > even > created an Abstract Syntax Tree (AST) so it's easier to inject some > content > in the query and it creates an abstraction over the SQL language so > we can > even consider to change the storage of the event stream someday. > > The bad thing is that some complex filtering are difficult to write > with > the SQL language (event with the AST) or even impossible. > > 2/ Post-filtering: after the events have been fetched from the database, > each > filter can still decide to keep or filter them. This is useful for > complex filtering that cannot be expressed with the SQL language. It > is > also needed by the real-time notification email sender, because it > takes > the events immediately when they occurs without fetching them in the > database (so SQL filters are bypassed). > > The bad thing is that some events are loaded in the memory to finally > be > rejected, and these filters can perform costly operations such as > loading > documents. > > Until now, this double mechanism was working quite well, with each > mechanism > filling the lacks of the other. > > However, we still have technical limitations in our design: > > 1/ Users who have a lot of filter preferences can end up with a giant > SQL > query that is almost impossible to perform by the database. Actually > we had > a user complaining about an OutOfMemory problem in the HQL to SQL > translator ! > > 2/ I cannot implement the tag filter ! > > The tag filter is supposed to show events that concern pages that hold a > given > tag, EVEN IF THE PAGE WAS EXCLUDED BY THE USER. Example of use-case: "I > don't > want to receive notifications about wiki A except for pages marked with > the tag > T". > > And it is not working. First because it is difficult to write a SQL query > for > that. It requires to make a join clause with the document and the object > tables, > which our SQL injection mechanism does not support. Even if it were > possible, > creating a SQL join with the document table will de-facto filter events > that do > not concern any pages or pages that do not have any objects: so many other > filters will be broken. I also don't consider creating a SQL subquery, I > think > the whole query would became too big. So I decided to just not inject any > SQL > code for this filter and only implement the post-filtering mechanism. > > But the other filter "EXCLUDE WIKI A" generates a SQL injection such as > "WIKI <> 'WIKI A'" so the events concerning the wiki A are not fetched > from the > database. Consequence: the tag filter never see the events that it is > supposed > to keep. It would be actually possible to by-pass the first SQL injections > by > injecting something like "OR 1=1". But doing something like this is like > dropping the all SQL injections mechanism. > > I see some solutions for this problem: > > A/ For each tag, create a permanent list of pages that hold it. So I can > inject "OR document IN (that_list)". I think this is heavy. > > B/ Drop the SQL injection mechanism and only rely on the post-filtering > mechanism. It would require to load from the database A LOT of events, > but maybe we could cache this. > > C/ Don't drop the SQL injection mechanism completely but use it as > little as > possible (for example, do not use it for LOCATION filtering). Seems > hard to > determine when a filter should use this feature or not. > > D/ Don't implement the "tags" filter, since it is the root of the issue. > But > it is
[xwiki-devs] [ANN] XWiki 9.11.7 released
The XWiki development team is proud to announce the availability of XWiki . This is a bugfix release that covers important issues that we have discovered since 9.11.6 has been released. You can download it here: http://www.xwiki.org/xwiki/bin/view/Main/Download Make sure to review the release notes: http://www.xwiki.org/xwiki/bin/view/ReleaseNotes/Data/XWiki/9.11.7 Thanks for your support -The XWiki dev team
Re: [xwiki-devs] [UX][Idea] Left Panels
On Thu, Jun 28, 2018 at 2:03 PM, Marius Dumitru Florea < mariusdumitru.flo...@xwiki.com> wrote: > On Wed, Jun 20, 2018 at 12:31 PM, Ecaterina Moraru (Valica) < > vali...@gmail.com> wrote: > > > On Wed, Jun 20, 2018 at 12:09 PM, Vincent Massol > > wrote: > > > > > Hi Caty, > > > > > > > On 19 Jun 2018, at 17:29, Ecaterina Moraru (Valica) < > vali...@gmail.com > > > > > > wrote: > > > > > > > > Hi devs, > > > > > > > > Some ideas that could be applied to left panels area: > > > > > > > > * Mark "More applications" as expandable > > > > > > Seems good > > > > > > > * Provide a "Customize" panel button on hover > > > > > > I agree with the general idea. > > > > > > Some details: > > > * For admins only right? > > > * What does it do? We don’t have the concept of Panel administration > ATM. > > > What would work is edit the panel page but then we may want to go > further > > > than that. Editing needs to be an option but some panel may provide an > > > admin UI too such as the Navigation Panel or the Applications Panel. > > Maybe > > > it’s time to update the Panel XClass and add an xproperty for the > > > administration of panels, which would be used if not empty. > > > > > > > > In theory only for Admins, since they have access to Administration. > > Currently we have dedicated customizations for AppBar and Navigation, so > > having that button would help the discoverability. Being able to have it > at > > the Panel class, could be nice, but I think it's interesting to have it > at > > least as individual customizations for these 2 panels that are on the > Left. > > > > > > > > * Some styling changes > > > > > > Sounds good > > > > > > > * Demo: show sticky left panels would behave > > > > > > > I made the sticky demo mostly as an experiment. I even don't know if I > like > > it. It's problematic for the current skin, since the logo disappears on > > scrolling. > > I made the demo since we discussed once on the IRC about independent > > containers and wanted to see how it looks like. From an implementation > POV > > it's a single "position: sticky" in CSS, but it's not supported in IE11. > So > > this could be an option just for a newer skin or have JS to have the > > behavior. > > > > > > > > > > I’m also not yet 100% sure about the sticky panels but why not, we > should > > > discuss the details. What I don’t like are for cases when you have > panels > > > that span more than a screenful vertically. > > > > > > It’s possible there’s not a single valid choice depending how you want > to > > > implement your website and thus maybe the only option is to have this > > > configurable in the general Panel Admin UI (fixed panels or sticky > > panels). > > > That’s what would make the most sense to me. Once we have this, the > > default > > > should be what looks best with the default content we provide for XS. > > > > > > > > > > > > Yeap, for KB having a sticky TOC or Navigation panel would look great > > > Only if the panel content fits the window height. Both the ToC and the > Navigation Panel can take a lot of vertical space (e.g. if you have a large > wiki and you expand the navigation tree). What will be the behavior if the > panels content is larger than the window height? > With 'position: sticky' it adds individual scrolls for the independent areas: 1 for the leftPanels, the other for the content. You can see the behavior in: - http://design.xwiki.org/xwiki/bin/download/Proposal/IdeaExpandableAppBar/Scrolling.gif - and how it looks like with hidden true on http://design.xwiki.org/xwiki/bin/download/Proposal/IdeaExpandableAppBar/BigScroll.png Thanks, Caty > > > , but > > it's not the case for all the Flavors. So having it customizable would be > > preferred. > > > > Thanks, > > Caty > > > > > > > I like those ideas! :) > > > > > > Thanks > > > -Vincent > > > > > > > > > > > See the prototype in action: > > > > http://design.xwiki.org/xwiki/bin/download/Proposal/ > > > IdeaExpandableAppBar/AppBar.gif > > > > > > > > The full proposal at: > > > > http://design.xwiki.org/xwiki/bin/view/Proposal/IdeaExpandableAppBar > > > > > > > > Let me know what you think. > > > > Thanks, > > > > Caty > > > > > > > > >
Re: [xwiki-devs] [UX][Idea] Left Panels
On Wed, Jun 20, 2018 at 12:31 PM, Ecaterina Moraru (Valica) < vali...@gmail.com> wrote: > On Wed, Jun 20, 2018 at 12:09 PM, Vincent Massol > wrote: > > > Hi Caty, > > > > > On 19 Jun 2018, at 17:29, Ecaterina Moraru (Valica) > > > wrote: > > > > > > Hi devs, > > > > > > Some ideas that could be applied to left panels area: > > > > > > * Mark "More applications" as expandable > > > > Seems good > > > > > * Provide a "Customize" panel button on hover > > > > I agree with the general idea. > > > > Some details: > > * For admins only right? > > * What does it do? We don’t have the concept of Panel administration ATM. > > What would work is edit the panel page but then we may want to go further > > than that. Editing needs to be an option but some panel may provide an > > admin UI too such as the Navigation Panel or the Applications Panel. > Maybe > > it’s time to update the Panel XClass and add an xproperty for the > > administration of panels, which would be used if not empty. > > > > > In theory only for Admins, since they have access to Administration. > Currently we have dedicated customizations for AppBar and Navigation, so > having that button would help the discoverability. Being able to have it at > the Panel class, could be nice, but I think it's interesting to have it at > least as individual customizations for these 2 panels that are on the Left. > > > > > * Some styling changes > > > > Sounds good > > > > > * Demo: show sticky left panels would behave > > > > I made the sticky demo mostly as an experiment. I even don't know if I like > it. It's problematic for the current skin, since the logo disappears on > scrolling. > I made the demo since we discussed once on the IRC about independent > containers and wanted to see how it looks like. From an implementation POV > it's a single "position: sticky" in CSS, but it's not supported in IE11. So > this could be an option just for a newer skin or have JS to have the > behavior. > > > > > > I’m also not yet 100% sure about the sticky panels but why not, we should > > discuss the details. What I don’t like are for cases when you have panels > > that span more than a screenful vertically. > > > > It’s possible there’s not a single valid choice depending how you want to > > implement your website and thus maybe the only option is to have this > > configurable in the general Panel Admin UI (fixed panels or sticky > panels). > > That’s what would make the most sense to me. Once we have this, the > default > > should be what looks best with the default content we provide for XS. > > > > > > Yeap, for KB having a sticky TOC or Navigation panel would look great Only if the panel content fits the window height. Both the ToC and the Navigation Panel can take a lot of vertical space (e.g. if you have a large wiki and you expand the navigation tree). What will be the behavior if the panels content is larger than the window height? > , but > it's not the case for all the Flavors. So having it customizable would be > preferred. > > Thanks, > Caty > > > > I like those ideas! :) > > > > Thanks > > -Vincent > > > > > > > > See the prototype in action: > > > http://design.xwiki.org/xwiki/bin/download/Proposal/ > > IdeaExpandableAppBar/AppBar.gif > > > > > > The full proposal at: > > > http://design.xwiki.org/xwiki/bin/view/Proposal/IdeaExpandableAppBar > > > > > > Let me know what you think. > > > Thanks, > > > Caty > > > > >
Re: [xwiki-devs] [Proposal] Add a new method in the icon API
On Fri, Jun 15, 2018 at 12:18 PM, Adel Atallah
wrote:
> Another idea would be to add a 'getMetaData()' method to the API
> (instead of renderJSON) which would take the same arguments as
> 'render()' and 'renderHTML()' methods and would return a map with some
> metadata, particularly the url of the image or the css class depending
> of the chosen iconset. This way, we won't even need to parse a string
> to know if the icon should be rendered as an image or with a css
> class, we will just need to check for the presence of some key in the
> map. We will also be able add other useful information in the future
> as well.
>
+1
>
> We will still need the '$services.icon.use()' methods as described by
> Marius to pull the necessary resources.
>
> Thanks,
> Adel
>
>
> On Thu, Jun 14, 2018 at 1:24 PM, Marius Dumitru Florea
> wrote:
> > +1 obviously.
> >
> > I encountered this need multiple times. One such example is the Document
> > Tree which uses the jstree library which accepts as node icon an URL, a
> > path or a CSS class name. I believe it is common in the JavaScript world
> to
> > be able to specify an icon either as an URL/path or as a CSS icon. The
> > reason Adel proposed "renderJSON" is because this method is needed mainly
> > when you generate JSON and the client that receives the JSON wants to
> have
> > 100% control over the HTML that is generated (so injecting the HTML we
> get
> > from renderHTML is not an option).
> >
> > Note that besides renderJSON we would also need a method to pull the
> > resources required by the icon set. I propose something like:
> >
> > $services.icon.use() <--- pull the resources for the configured icon set
> > $services.icon.use('someIconSet') <--- pull the resources for the
> specified
> > icon set
> >
> > These methods will call ssx, jsx, etc. under the hood. This is done
> already
> > when you call render() and renderHTML() but it obviously doesn't work if
> > you call render* on an AJAX request that returns JSON.
> >
> > Thanks,
> > Marius
> >
> >
> > On Thu, Jun 14, 2018 at 12:56 PM, Adel Atallah
> > wrote:
> >
> >> Hi devs,
> >>
> >> I'm making a rest resource to get a list of pages and, for a query, I
> >> want to specify an icon (as a metadata) for each pages in the resulted
> >> json.
> >> The problem is that the icon APIs (and more specifically the
> >> IconManager class) only allow us to render the icon in HTML or
> >> velocity and this shouldn't be put inside a json response.
> >> Also we can't hardcode the icon class or image URL to be used as it
> >> depends on the iconset configured for the wiki. Another possibility
> >> would be to render the icon using javascript but it will not be very
> >> efficient.
> >>
> >> As discussed with Marius, our proposal would be to add a new method to
> >> the IconManager to get either the icon URL (e.g.
> >> http://xwiki.org/xwiki/resources/icons/silk/page.png) or the icon
> >> class (e.g. fa fa-page) depending of the specified iconset.
> >> We could then have this new property to the icon theme definition:
> >>
> >> ## Silk
> >> xwiki.iconset.render.json=$xwiki.getSkinFile("icons/silk/${icon}.png")
> >> ## FontAwesome
> >> xwiki.iconset.render.json=fa fa-$icon
> >>
> >> We could name the new method renderJSON or something more generic (if
> >> you have any idea).
> >>
> >>
> >> WDYT?
> >>
> >> Thanks,
> >> Adel
> >>
>
