Re: [xwiki-users] severe trouble with web crawlers

2013-02-21 Thread Jan-Philip Loos
Hi Vincent, the last weeks I was busy with my studies, I will try the patch/workaround next week and keep you informed. Greetings Jan-Philip -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162p7583958.html Sent from the XWiki- Users m

Re: [xwiki-users] severe trouble with web crawlers

2013-02-08 Thread Vincent Massol
Hi Jan-Philip, What would be nice is if you could apply the workaround defined at http://jira.xwiki.org/browse/XWIKI-8046?focusedCommentId=71549&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-71549 and let us know if that fixes your issue. This will have 2 benefits

Re: [xwiki-users] severe trouble with web crawlers

2013-01-18 Thread Jan-Philip Loos
We upgraded to the latest minor versions as they were released, first migration was from 3.1 to 4.2, followed by 4.2 -> 4.3 and finally 4.3 -> 4.4 The problem still occurred with 4.3 (4.4 is to new to figured it out now). In the last 3-4 month the problem occurred round about 3 times. I wrote a sm

Re: [xwiki-users] severe trouble with web crawlers

2012-09-10 Thread Jan-Philip Loos
Hi Vincent, I'm not sure, although our problem seems to be a caching problem too. But I'm not as familiar as needed with the xwiki code base to judge it right. Some points why it couldn't be the same problem (pessimistic view): - We are running XWiki 3.4, the HiddenDocuments were introduced in XW

Re: [xwiki-users] severe trouble with web crawlers

2012-09-10 Thread Vincent Massol
Hi Jan-Philip, On Sep 10, 2012, at 9:37 PM, Jan-Philip Loos wrote: > Hello, > > it's been a while since the last posting. We experienced the problem a > couple of times, but without any serve impact. > > I watched the jira issue mentioned by Sergiu Dumitriu. It's status is still > open, major

Re: [xwiki-users] severe trouble with web crawlers

2012-09-10 Thread Jan-Philip Loos
Hello, it's been a while since the last posting. We experienced the problem a couple of times, but without any serve impact. I watched the jira issue mentioned by Sergiu Dumitriu. It's status is still open, major and unassigned, the resolution is unresolved. Any off-jira updates for this issue?

Re: [xwiki-users] severe trouble with web crawlers

2012-07-13 Thread Ludovic Dubost
Hi This could have showed up because of. Changes a while ago that flush documents from the cache after save. So combined with a crawler trigerring saves it could trigger the behavior seen here where flushCache the restores everything. Maybe there is a reading failure which is not detected pro

Re: [xwiki-users] severe trouble with web crawlers

2012-07-13 Thread Jan-Philip Loos
Thank you, We will looking forward to it. If we can help to test something for you, let us know. -- View this message in context: http://xwiki.475771.n2.nabble.com/severe-trouble-with-web-crawlers-tp7442162p7580353.html Sent from the XWiki- Users mailing list archive at Nabble.com. __

Re: [xwiki-users] severe trouble with web crawlers

2012-07-13 Thread Sergiu Dumitriu
On 07/13/2012 05:31 PM, Jan-Philip Loos wrote: Hello, I have an really important update for this issue. Last month I installed the the "Admin Tools Application" as recommended by Ludovic Dubost. I ran the "Programming Rights Check" script and inspected the "User Rights Check" page. No right ano

Re: [xwiki-users] severe trouble with web crawlers

2012-07-13 Thread Jan-Philip Loos
Hello, I have an really important update for this issue. Last month I installed the the "Admin Tools Application" as recommended by Ludovic Dubost. I ran the "Programming Rights Check" script and inspected the "User Rights Check" page. No right anomalies, as far as I can see. A few minutes ago,

Re: [xwiki-users] severe trouble with web crawlers

2012-06-07 Thread Jan-Philip Loos
Sergiu Dumitriu-2 wrote > > One important question nobody asked so far: are you using any custom > authenticator or rights implementation, like LDAP, Kerberos or another > SSO? > We are using XEM with a couple of managed wikis. The authentication of the main-wiki (Manage-Wiki) and all sub-wiki

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread Sergiu Dumitriu
On 06/06/2012 06:26 PM, Jan-Philip Loos wrote: Sorry, my fault: I misunderstood the part with the delete. You meant the rights, and not the 'delete' itself. But again to the rights hint: We double-checked the rights of the crucial sites, with different test-users and guests (without sessions). W

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread Jan-Philip Loos
Sorry, my fault: I misunderstood the part with the delete. You meant the rights, and not the 'delete' itself. But again to the rights hint: We double-checked the rights of the crucial sites, with different test-users and guests (without sessions). We testes in different browsers and even in VMs on

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread Jan-Philip Loos
Ludovic Dubost wrote > > You need to make sure the delete and un delete rights are limited to non > guest > There was a bug fixed with undelete defaulting to view which Has been > fixed at some point > But our problems aren't only restricted to deletions, currently we are experiencing empty edi

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread ludo...@xwiki.com
Quick note. In the admin tools extension http://extensions.xwiki.org/xwiki/bin/view/Extension/Admin+Tools+Application There is a page allowing to report all rights from a wiki. The output of that page is very helpful to audit rights issues Ludovic Envoyé de mon iPad Le 6 juin 2012 à 23:15, J

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread ludo...@xwiki.com
You need to make sure the delete and un delete rights are limited to non guest There was a bug fixed with undelete defaulting to view which Has been fixed at some point Make sure you prevent Bots from hitting these Urls with a robots.txt also Ludovic Envoyé de mon iPad Le 6 juin 2012 à 23:15

Re: [xwiki-users] severe trouble with web crawlers

2012-06-06 Thread Jan-Philip Loos
In the last couple of weeks we had some small issues again, but no nearly complete wipe. Meanwhile, I switched on the logging of the cookies in our apache proxy and enabled the authentication logging in our XWiki as follows: added in /etc/xwiki/classes/logback.xml > level="info"/> > level="in

Re: [xwiki-users] severe trouble with web crawlers

2012-04-28 Thread jerem
For my wiki, we used to have a crawler like this (snuffing every links out there), then, due to the useless time spent trying to retrieve pages and checking rights, I wrote small scripts to provide an indexing page to the crawler. This page lists all links that I WANT the crawler to fetch (and filt

Re: [xwiki-users] severe trouble with web crawlers

2012-04-27 Thread JPL
> Well, this is indeed strange. Could you print the cookies sent with each request in the logs as well? Thanks for the fast reply. First of all i have to activate cookie-logging, thanks for the idea ;). Then i have to import our backup and have to wait for the bot. I will reply soon as possibl

Re: [xwiki-users] severe trouble with web crawlers

2012-04-27 Thread Sergiu Dumitriu
On 04/27/2012 09:18 PM, JPL wrote: Hi I'm a companion of Olaf, the last week we exposed our xwiki to the googlebot to test our configuration. Yesterday we got the same problems with the crawler and we are lost like before. I will get more precise to the problem and quote some of our logs. My r

Re: [xwiki-users] severe trouble with web crawlers

2012-04-27 Thread JPL
Hi I'm a companion of Olaf, the last week we exposed our xwiki to the googlebot to test our configuration. Yesterday we got the same problems with the crawler and we are lost like before. I will get more precise to the problem and quote some of our logs. My results of the analysis: - some criti

Re: [xwiki-users] severe trouble with web crawlers

2012-04-12 Thread O Voss
Von: O Voss >An: Vincent Massol ; XWiki Users >Gesendet: 9:09 Freitag, 6.April 2012 >Betreff: Re: [xwiki-users] severe trouble with web crawlers > >Hi Vincent, > >Many thanks for your quick answer. > >I will list the wiki scope rights that we have set here. (I hope I get t

Re: [xwiki-users] severe trouble with web crawlers

2012-04-06 Thread O Voss
12 >Betreff: Re: [xwiki-users] severe trouble with web crawlers > >Hi Olaf, > >On Apr 6, 2012, at 8:15 AM, O Voss wrote: > >> Hi, >> >> a web crawler keeps deleting documents in our Wiki. Meanwhile we have a >> robots.txt, but it seems to ignore that. > >

Re: [xwiki-users] severe trouble with web crawlers

2012-04-05 Thread Vincent Massol
Hi Olaf, On Apr 6, 2012, at 8:15 AM, O Voss wrote: > Hi, > > a web crawler keeps deleting documents in our Wiki. Meanwhile we have a > robots.txt, but it seems to ignore that. Of course, you should never allow unregistered users to delete documents, but you know that already… ;) > In the glo

[xwiki-users] severe trouble with web crawlers

2012-04-05 Thread O Voss
Hi, a web crawler keeps deleting documents in our Wiki. Meanwhile we have a robots.txt, but it seems to ignore that. In the global rights the unregistered user is explicitely denied edit, delete and admin rights. Nowhere are such rights set to allowed. What are we missing? The deleted document