Re: [xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions

2016-08-15 Thread Vincent Massol 

> On 15 Aug 2016, at 19:11, Sergiu Dumitriu  wrote:
> 
> On 08/15/2016 12:39 PM, Vincent Massol wrote:
>> Hi Sergiu,
>> 
>>> On 15 Aug 2016, at 18:18, Sergiu Dumitriu  wrote:
>>> 
>>> Why does that value have to be stored in the wiki document?
>> 
>> Sure, we could use another store. However the goal of XWiki is also to 
>> provide a generic store which can be used to store metadata (through 
>> xobjects) related to wiki pages. And install count for an extension really 
>> looks like a good metadata candidate for a wiki page. Using another store 
>> would also be more work.
>> 
>> So, I’d prefer if we could find a way to support this use case which seems 
>> legitimate. We have several such use cases, which consist in having some 
>> system user perform modifications to wiki pages:
>> * scheduler which updates scheduled jobs with the last fire time xproperty
>> * this extension scheduler which updates install counts
>> * (I’m sure we have more examples but I can’t find one right now ;))
>> 
>> My personal POV is that long term we should have:
>> * Introduce a proper reserved (you cannot create a user named after it) and 
>> virtual “System” user (and don’t use “Admin” or “superadmin” which are users 
>> which should be used by admins)
>> * Filter out System user changes by default in Activity Stream, History, etc 
>> (with options to display them)
>> * Refactor the Versioning store (and possibly get get rid of JRCS) to have a 
>> scalable system which supports an unlimited number of revisions without 
>> impacting performances
>> * Add an API to merge several revisions into one (this could be useful for 
>> auto save too). This is related to what you called “pseudoversion” too: 
>> http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workflow+Application
> 
> I agree, but as you said, this is "long term".
> 
> Perhaps I'm missing something, but I though the count is already stored
> someplace else. Is the scheduler task that updates these counts not
> fetching the data from an external place? What I'm proposing, as a quick
> solution that doesn't require reimplementing the entire storage, user
> management and activity stream, is that instead of getting the numbers
> regularly from another service and storing them in the wiki, just let
> them be fetched directly from there when requested.

I’ve checked the scheduler code and indeed it’s taking the data from ES (see 
http://extensions.xwiki.org/xwiki/bin/edit/ExtensionCode/UpdateInstalledExtensionCountScheduler).

So I guess Thomas is saving the data in XWiki for performance reasons and so 
that we can filter/sort the LT.

So indeed with some good caching what you propose could work.However, as you 
mentioned, we would miss the ability to filter/sort the LT.

FTR and FTM I’ve modified the code to save as a minor edit.

We now need to solve the version scalability problem. Let’s see what others 
think.

Thanks
-Vincent

> Personally, I'd make it a Computed Field, and always get the current
>>> value from an external service. This has the disadvantage that the
>>> livetable can't filter/order by the number of installations.
>>> 
>>> If the cost of fetching data from an external service is a concern, then
>>> add a TTL cache.
>> 
>> Thanks
>> -Vincent
>> 
>>> On 08/15/2016 06:54 AM, Vincent Massol wrote:
 Hi Thomas and all,
 
 Back from holidays! :)
 
 I’ve noticed that the new feature of counting installed extensions on 
 e.x.o is having a drawback: it saturates the activity stream, making it 
 very hard to see real edits by users. Every day the scheduler modifies 
 lots of wiki pages to set the new install count. See for example: 
 http://www.xwiki.org/xwiki/bin/view/Main/News
 
 I think a simple change would be for the scheduler to make modifications 
 as minor edits. This should prevent the edits from being visible in the AS.
 
 WDYT?
 
 Now this is going to cause another real issue very soon: pages will soon 
 start to have a lot of revisions and we know this is currently a 
 performance issue. It’s also hiding real edits in the history making the 
 history a bit less clean.
 
 I guess an option would be for the scheduler to delete the last revision 
 after it updates a page. Although not very nice, it could work for now. 
 WDYT?
 
 Thanks
 -Vincent
>>> 
>>> 
>>> -- 
>>> Sergiu Dumitriu
___
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs

Re: [xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions

2016-08-15 Thread Sergiu Dumitriu 
On 08/15/2016 12:39 PM, Vincent Massol wrote:
> Hi Sergiu,
> 
>> On 15 Aug 2016, at 18:18, Sergiu Dumitriu  wrote:
>>
>> Why does that value have to be stored in the wiki document?
> 
> Sure, we could use another store. However the goal of XWiki is also to 
> provide a generic store which can be used to store metadata (through 
> xobjects) related to wiki pages. And install count for an extension really 
> looks like a good metadata candidate for a wiki page. Using another store 
> would also be more work.
> 
> So, I’d prefer if we could find a way to support this use case which seems 
> legitimate. We have several such use cases, which consist in having some 
> system user perform modifications to wiki pages:
> * scheduler which updates scheduled jobs with the last fire time xproperty
> * this extension scheduler which updates install counts
> * (I’m sure we have more examples but I can’t find one right now ;))
> 
> My personal POV is that long term we should have:
> * Introduce a proper reserved (you cannot create a user named after it) and 
> virtual “System” user (and don’t use “Admin” or “superadmin” which are users 
> which should be used by admins)
> * Filter out System user changes by default in Activity Stream, History, etc 
> (with options to display them)
> * Refactor the Versioning store (and possibly get get rid of JRCS) to have a 
> scalable system which supports an unlimited number of revisions without 
> impacting performances
> * Add an API to merge several revisions into one (this could be useful for 
> auto save too). This is related to what you called “pseudoversion” too: 
> http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workflow+Application

I agree, but as you said, this is "long term".

Perhaps I'm missing something, but I though the count is already stored
someplace else. Is the scheduler task that updates these counts not
fetching the data from an external place? What I'm proposing, as a quick
solution that doesn't require reimplementing the entire storage, user
management and activity stream, is that instead of getting the numbers
regularly from another service and storing them in the wiki, just let
them be fetched directly from there when requested.

>> Personally, I'd make it a Computed Field, and always get the current
>> value from an external service. This has the disadvantage that the
>> livetable can't filter/order by the number of installations.
>>
>> If the cost of fetching data from an external service is a concern, then
>> add a TTL cache.
> 
> Thanks
> -Vincent
> 
>> On 08/15/2016 06:54 AM, Vincent Massol wrote:
>>> Hi Thomas and all,
>>>
>>> Back from holidays! :)
>>>
>>> I’ve noticed that the new feature of counting installed extensions on e.x.o 
>>> is having a drawback: it saturates the activity stream, making it very hard 
>>> to see real edits by users. Every day the scheduler modifies lots of wiki 
>>> pages to set the new install count. See for example: 
>>> http://www.xwiki.org/xwiki/bin/view/Main/News
>>>
>>> I think a simple change would be for the scheduler to make modifications as 
>>> minor edits. This should prevent the edits from being visible in the AS.
>>>
>>> WDYT?
>>>
>>> Now this is going to cause another real issue very soon: pages will soon 
>>> start to have a lot of revisions and we know this is currently a 
>>> performance issue. It’s also hiding real edits in the history making the 
>>> history a bit less clean.
>>>
>>> I guess an option would be for the scheduler to delete the last revision 
>>> after it updates a page. Although not very nice, it could work for now. 
>>> WDYT?
>>>
>>> Thanks
>>> -Vincent
>>
>>
>> -- 
>> Sergiu Dumitriu
>> http://purl.org/net/sergiu
> ___
> devs mailing list
> devs@xwiki.org
> http://lists.xwiki.org/mailman/listinfo/devs
> 


-- 
Sergiu Dumitriu
http://purl.org/net/sergiu
___
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs

Re: [xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions

2016-08-15 Thread Vincent Massol 
Hi Sergiu,

> On 15 Aug 2016, at 18:18, Sergiu Dumitriu  wrote:
> 
> Why does that value have to be stored in the wiki document?

Sure, we could use another store. However the goal of XWiki is also to provide 
a generic store which can be used to store metadata (through xobjects) related 
to wiki pages. And install count for an extension really looks like a good 
metadata candidate for a wiki page. Using another store would also be more work.

So, I’d prefer if we could find a way to support this use case which seems 
legitimate. We have several such use cases, which consist in having some system 
user perform modifications to wiki pages:
* scheduler which updates scheduled jobs with the last fire time xproperty
* this extension scheduler which updates install counts
* (I’m sure we have more examples but I can’t find one right now ;))

My personal POV is that long term we should have:
* Introduce a proper reserved (you cannot create a user named after it) and 
virtual “System” user (and don’t use “Admin” or “superadmin” which are users 
which should be used by admins)
* Filter out System user changes by default in Activity Stream, History, etc 
(with options to display them)
* Refactor the Versioning store (and possibly get get rid of JRCS) to have a 
scalable system which supports an unlimited number of revisions without 
impacting performances
* Add an API to merge several revisions into one (this could be useful for auto 
save too). This is related to what you called “pseudoversion” too: 
http://extensions.xwiki.org/xwiki/bin/view/Extension/XWiki+Publication+Workflow+Application

> Personally, I'd make it a Computed Field, and always get the current
> value from an external service. This has the disadvantage that the
> livetable can't filter/order by the number of installations.
> 
> If the cost of fetching data from an external service is a concern, then
> add a TTL cache.

Thanks
-Vincent

> On 08/15/2016 06:54 AM, Vincent Massol wrote:
>> Hi Thomas and all,
>> 
>> Back from holidays! :)
>> 
>> I’ve noticed that the new feature of counting installed extensions on e.x.o 
>> is having a drawback: it saturates the activity stream, making it very hard 
>> to see real edits by users. Every day the scheduler modifies lots of wiki 
>> pages to set the new install count. See for example: 
>> http://www.xwiki.org/xwiki/bin/view/Main/News
>> 
>> I think a simple change would be for the scheduler to make modifications as 
>> minor edits. This should prevent the edits from being visible in the AS.
>> 
>> WDYT?
>> 
>> Now this is going to cause another real issue very soon: pages will soon 
>> start to have a lot of revisions and we know this is currently a performance 
>> issue. It’s also hiding real edits in the history making the history a bit 
>> less clean.
>> 
>> I guess an option would be for the scheduler to delete the last revision 
>> after it updates a page. Although not very nice, it could work for now. WDYT?
>> 
>> Thanks
>> -Vincent
> 
> 
> -- 
> Sergiu Dumitriu
> http://purl.org/net/sergiu
___
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs

Re: [xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions

2016-08-15 Thread Sergiu Dumitriu 
Why does that value have to be stored in the wiki document?

Personally, I'd make it a Computed Field, and always get the current
value from an external service. This has the disadvantage that the
livetable can't filter/order by the number of installations.

If the cost of fetching data from an external service is a concern, then
add a TTL cache.

On 08/15/2016 06:54 AM, Vincent Massol wrote:
> Hi Thomas and all,
> 
> Back from holidays! :)
> 
> I’ve noticed that the new feature of counting installed extensions on e.x.o 
> is having a drawback: it saturates the activity stream, making it very hard 
> to see real edits by users. Every day the scheduler modifies lots of wiki 
> pages to set the new install count. See for example: 
> http://www.xwiki.org/xwiki/bin/view/Main/News
> 
> I think a simple change would be for the scheduler to make modifications as 
> minor edits. This should prevent the edits from being visible in the AS.
> 
> WDYT?
> 
> Now this is going to cause another real issue very soon: pages will soon 
> start to have a lot of revisions and we know this is currently a performance 
> issue. It’s also hiding real edits in the history making the history a bit 
> less clean.
> 
> I guess an option would be for the scheduler to delete the last revision 
> after it updates a page. Although not very nice, it could work for now. WDYT?
> 
> Thanks
> -Vincent


-- 
Sergiu Dumitriu
http://purl.org/net/sergiu
___
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs

[xwiki-devs] Issue with installed count on e.x.o overflowing the AS + too many versions

2016-08-15 Thread Vincent Massol 
Hi Thomas and all,

Back from holidays! :)

I’ve noticed that the new feature of counting installed extensions on e.x.o is 
having a drawback: it saturates the activity stream, making it very hard to see 
real edits by users. Every day the scheduler modifies lots of wiki pages to set 
the new install count. See for example: 
http://www.xwiki.org/xwiki/bin/view/Main/News

I think a simple change would be for the scheduler to make modifications as 
minor edits. This should prevent the edits from being visible in the AS.

WDYT?

Now this is going to cause another real issue very soon: pages will soon start 
to have a lot of revisions and we know this is currently a performance issue. 
It’s also hiding real edits in the history making the history a bit less clean.

I guess an option would be for the scheduler to delete the last revision after 
it updates a page. Although not very nice, it could work for now. WDYT?

Thanks
-Vincent

___
devs mailing list
devs@xwiki.org
http://lists.xwiki.org/mailman/listinfo/devs