Re: [xwiki-users] Lucene re-index problem
Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30 To: users@xwiki.org Subject: [xwiki-users] Lucene re-index problem After upgrading to XWiki 3.5 (from 2.6) I noticed that the Lucene search index was not working properly. On advice I got in the #xwiki IRC chat, I shut down the server (Tomcat), deleted the Lucene index directory, and restarted the server. Only about 300 of the thousands of pages were indexed at that point.I tried re-indexing using the Admin-Search page, but every time I do that the Admin-Search page gets 'stuck' (won't render). I saw no errors in the catalina.out log file, and I configured WEB-INF/classes/logback.xml to enable some more logging to a separate file... still nothing very informative. Any ideas what I can do to troubleshoot this? I'm guessing there's some page in my wiki database that is causing the re-index process to get stuck. Thanks in advance, - Josh Almost two years ago we face a similar problem while updating to XWiki Enterprise 2.4.30451. We have been not able to get Lucene working in this installation. The problem here seems related with Microsoft Office attachements. I don't remember exactly how do we reach this conclusion and it could be a kind of shot in the air, but I remember that PDF files seem to index fine, but xls and doc files causes the failure of the process. As in your case, AdminSearch won't render. We have never find a solution. This installation is still running but we are using database search instead. i keep following this thread! Cheers, Ricardo ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] Lucene re-index problem
Interesting. Is there some way to tell the Lucene not to index the attachments? On Tue, Feb 28, 2012 at 4:39 PM, ricardo.julio.rodriguez.fernan...@sergas.es wrote: Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30 To: users@xwiki.org Subject: [xwiki-users] Lucene re-index problem After upgrading to XWiki 3.5 (from 2.6) I noticed that the Lucene search index was not working properly. On advice I got in the #xwiki IRC chat, I shut down the server (Tomcat), deleted the Lucene index directory, and restarted the server. Only about 300 of the thousands of pages were indexed at that point.I tried re-indexing using the Admin-Search page, but every time I do that the Admin-Search page gets 'stuck' (won't render). I saw no errors in the catalina.out log file, and I configured WEB-INF/classes/logback.xml to enable some more logging to a separate file... still nothing very informative. Any ideas what I can do to troubleshoot this? I'm guessing there's some page in my wiki database that is causing the re-index process to get stuck. Thanks in advance, - Josh Almost two years ago we face a similar problem while updating to XWiki Enterprise 2.4.30451. We have been not able to get Lucene working in this installation. The problem here seems related with Microsoft Office attachements. I don't remember exactly how do we reach this conclusion and it could be a kind of shot in the air, but I remember that PDF files seem to index fine, but xls and doc files causes the failure of the process. As in your case, AdminSearch won't render. We have never find a solution. This installation is still running but we are using database search instead. i keep following this thread! Cheers, Ricardo ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] Lucene re-index problem
From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 22:55 To: XWiki Users Subject: Re: [xwiki-users] Lucene re-index problem Interesting. Is there some way to tell the Lucene not to index the attachments? Sorry, I don't remember this. On my side, I must go back to Jira and the email lists to regain access to this issue. Any different thing will be an absurd waste of energy and a lot of noise!!! Greetings! On Tue, Feb 28, 2012 at 4:39 PM, ricardo.julio.rodriguez.fernan...@sergas.es wrote: Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30 To: users@xwiki.org Subject: [xwiki-users] Lucene re-index problem After upgrading to XWiki 3.5 (from 2.6) I noticed that the Lucene search index was not working properly. On advice I got in the #xwiki IRC chat, I shut down the server (Tomcat), deleted the Lucene index directory, and restarted the server. Only about 300 of the thousands of pages were indexed at that point.I tried re-indexing using the Admin-Search page, but every time I do that the Admin-Search page gets 'stuck' (won't render). I saw no errors in the catalina.out log file, and I configured WEB-INF/classes/logback.xml to enable some more logging to a separate file... still nothing very informative. Any ideas what I can do to troubleshoot this? I'm guessing there's some page in my wiki database that is causing the re-index process to get stuck. Thanks in advance, - Josh Almost two years ago we face a similar problem while updating to XWiki Enterprise 2.4.30451. We have been not able to get Lucene working in this installation. The problem here seems related with Microsoft Office attachements. I don't remember exactly how do we reach this conclusion and it could be a kind of shot in the air, but I remember that PDF files seem to index fine, but xls and doc files causes the failure of the process. As in your case, AdminSearch won't render. We have never find a solution. This installation is still running but we are using database search instead. i keep following this thread! Cheers, Ricardo ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] Lucene re-index problem
In my case I am using the default value in xwiki.cfg: xwiki.work.dir=data This seems to be pointing to /home/tomcat user/data on my system. Looks like the Lucene index is in /home/tomcat user/data/lucene ... hmm. I deleted a different directory before. Anyway... yes, the user has access to this directory. -bash-3.2$ ll data total 8 drwxr-xr-x 3 tomcat60 tomcat60 4096 Feb 22 06:24 extension drwxr-xr-x 2 tomcat60 tomcat60 4096 Feb 28 18:00 lucene -bash-3.2$ ll data/lucene total 1548 -rw-r--r-- 1 tomcat60 tomcat60 638394 Feb 28 18:00 _f4.fdt -rw-r--r-- 1 tomcat60 tomcat60 3404 Feb 28 18:00 _f4.fdx -rw-r--r-- 1 tomcat60 tomcat60 11929 Feb 28 18:00 _f4.fnm -rw-r--r-- 1 tomcat60 tomcat60 125852 Feb 28 18:00 _f4.frq -rw-r--r-- 1 tomcat60 tomcat60 141104 Feb 28 18:00 _f4.nrm -rw-r--r-- 1 tomcat60 tomcat60 248013 Feb 28 18:00 _f4.prx -rw-r--r-- 1 tomcat60 tomcat60 5082 Feb 28 18:00 _f4.tii -rw-r--r-- 1 tomcat60 tomcat60 369203 Feb 28 18:00 _f4.tis -rw-r--r-- 1 tomcat60 tomcat60280 Feb 28 18:00 segments_7m -rw-r--r-- 1 tomcat60 tomcat60 20 Feb 28 18:00 segments.gen On Tue, Feb 28, 2012 at 6:13 PM, Guillaume Fenollar guillaume.fenol...@xwiki.com wrote: Hi, When the page stucks, it's often because of a write issue on xwiki work directory. Are you sure you put a static value of xwiki.work.dir in xwiki.cfg file after your upgrade? Your application server must have the right to write in there. 2012/2/28 Joshua Davis pgm...@gmail.com Interesting. Is there some way to tell the Lucene not to index the attachments? On Tue, Feb 28, 2012 at 4:39 PM, ricardo.julio.rodriguez.fernan...@sergas.es wrote: Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30 To: users@xwiki.org Subject: [xwiki-users] Lucene re-index problem After upgrading to XWiki 3.5 (from 2.6) I noticed that the Lucene search index was not working properly. On advice I got in the #xwiki IRC chat, I shut down the server (Tomcat), deleted the Lucene index directory, and restarted the server. Only about 300 of the thousands of pages were indexed at that point.I tried re-indexing using the Admin-Search page, but every time I do that the Admin-Search page gets 'stuck' (won't render). I saw no errors in the catalina.out log file, and I configured WEB-INF/classes/logback.xml to enable some more logging to a separate file... still nothing very informative. Any ideas what I can do to troubleshoot this? I'm guessing there's some page in my wiki database that is causing the re-index process to get stuck. Thanks in advance, - Josh Almost two years ago we face a similar problem while updating to XWiki Enterprise 2.4.30451. We have been not able to get Lucene working in this installation. The problem here seems related with Microsoft Office attachements. I don't remember exactly how do we reach this conclusion and it could be a kind of shot in the air, but I remember that PDF files seem to index fine, but xls and doc files causes the failure of the process. As in your case, AdminSearch won't render. We have never find a solution. This installation is still running but we are using database search instead. i keep following this thread! Cheers, Ricardo ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users -- Guillaume Fenollar XWiki SysAdmin Tel : +33 (0)1.83.62.66.15 ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users
Re: [xwiki-users] Lucene re-index problem
I tried shutting down tomcat an deleting the contents of the ~/data/lucene directory. The Search page was accessible for a little while, it said there were 1000 documents in the queue. Then, it got stuck. In the logs, I had a lot of this: 97898 [Lucene Index Updater] WARN c.x.x.plugin.lucene.AttachmentData - error getting content of attachment [zzz] for document [name = [...], type = [DOCUMENT], parent = [name = [...], type = [SPACE], parent = [name = [xwiki], type = [WIKI], parent = [null java.lang.RuntimeException: Failed to get InputStream at com.xpn.xwiki.doc.XWikiAttachmentContent.getContentInputStream(XWikiAttachmentContent.java:213) ~[xwiki-platform-legacy-oldcore-3.5.jar:na] at com.xpn.xwiki.doc.XWikiAttachment.getContentInputStream(XWikiAttachment.java:561) ~[xwiki-platform-legacy-oldcore-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AttachmentData.getContentAsText(AttachmentData.java:191) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AttachmentData.getFullText(AttachmentData.java:167) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AbstractDocumentData.getFullText(AbstractDocumentData.java:242) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AbstractDocumentData.addDocumentDataToLuceneDocument(AbstractDocumentData.java:203) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AbstractDocumentData.addDataToLuceneDocument(AbstractDocumentData.java:144) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.AttachmentData.addDataToLuceneDocument(AttachmentData.java:92) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.IndexUpdater.addToIndex(IndexUpdater.java:283) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.IndexUpdater.updateIndex(IndexUpdater.java:227) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.IndexUpdater.runMainLoop(IndexUpdater.java:173) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.plugin.lucene.IndexUpdater.runInternal(IndexUpdater.java:158) [xwiki-platform-search-lucene-3.5.jar:na] at com.xpn.xwiki.util.AbstractXWikiRunnable.run(AbstractXWikiRunnable.java:110) [xwiki-platform-legacy-oldcore-3.5.jar:na] at java.lang.Thread.run(Thread.java:662) [na:1.6.0_31] Caused by: java.io.FileNotFoundException: /opt/tomcat/temp/upload__501aef8c_135c64be6b5__8000_1015.tmp (Too many open files) at java.io.FileInputStream.open(Native Method) ~[na:1.6.0_31] at java.io.FileInputStream.init(FileInputStream.java:120) ~[na:1.6.0_31] at org.apache.commons.fileupload.disk.DiskFileItem.getInputStream(DiskFileItem.java:236) ~[commons-fileupload-1.2.2.jar:1.2.2] at com.xpn.xwiki.doc.XWikiAttachmentContent.getContentInputStream(XWikiAttachmentContent.java:211) ~[xwiki-platform-legacy-oldcore-3.5.jar:na] ... 13 common frames omitted On Tue, Feb 28, 2012 at 6:24 PM, Joshua Davis pgm...@gmail.com wrote: In my case I am using the default value in xwiki.cfg: xwiki.work.dir=data This seems to be pointing to /home/tomcat user/data on my system. Looks like the Lucene index is in /home/tomcat user/data/lucene ... hmm. I deleted a different directory before. Anyway... yes, the user has access to this directory. -bash-3.2$ ll data total 8 drwxr-xr-x 3 tomcat60 tomcat60 4096 Feb 22 06:24 extension drwxr-xr-x 2 tomcat60 tomcat60 4096 Feb 28 18:00 lucene -bash-3.2$ ll data/lucene total 1548 -rw-r--r-- 1 tomcat60 tomcat60 638394 Feb 28 18:00 _f4.fdt -rw-r--r-- 1 tomcat60 tomcat60 3404 Feb 28 18:00 _f4.fdx -rw-r--r-- 1 tomcat60 tomcat60 11929 Feb 28 18:00 _f4.fnm -rw-r--r-- 1 tomcat60 tomcat60 125852 Feb 28 18:00 _f4.frq -rw-r--r-- 1 tomcat60 tomcat60 141104 Feb 28 18:00 _f4.nrm -rw-r--r-- 1 tomcat60 tomcat60 248013 Feb 28 18:00 _f4.prx -rw-r--r-- 1 tomcat60 tomcat60 5082 Feb 28 18:00 _f4.tii -rw-r--r-- 1 tomcat60 tomcat60 369203 Feb 28 18:00 _f4.tis -rw-r--r-- 1 tomcat60 tomcat60280 Feb 28 18:00 segments_7m -rw-r--r-- 1 tomcat60 tomcat60 20 Feb 28 18:00 segments.gen On Tue, Feb 28, 2012 at 6:13 PM, Guillaume Fenollar guillaume.fenol...@xwiki.com wrote: Hi, When the page stucks, it's often because of a write issue on xwiki work directory. Are you sure you put a static value of xwiki.work.dir in xwiki.cfg file after your upgrade? Your application server must have the right to write in there. 2012/2/28 Joshua Davis pgm...@gmail.com Interesting. Is there some way to tell the Lucene not to index the attachments? On Tue, Feb 28, 2012 at 4:39 PM, ricardo.julio.rodriguez.fernan...@sergas.es wrote: Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30
Re: [xwiki-users] Lucene re-index problem
I think I'm hitting a ulimit issue, I'm going to bump up ulimit for my tomcat user and restart. On Tue, Feb 28, 2012 at 6:24 PM, Joshua Davis pgm...@gmail.com wrote: In my case I am using the default value in xwiki.cfg: xwiki.work.dir=data This seems to be pointing to /home/tomcat user/data on my system. Looks like the Lucene index is in /home/tomcat user/data/lucene ... hmm. I deleted a different directory before. Anyway... yes, the user has access to this directory. -bash-3.2$ ll data total 8 drwxr-xr-x 3 tomcat60 tomcat60 4096 Feb 22 06:24 extension drwxr-xr-x 2 tomcat60 tomcat60 4096 Feb 28 18:00 lucene -bash-3.2$ ll data/lucene total 1548 -rw-r--r-- 1 tomcat60 tomcat60 638394 Feb 28 18:00 _f4.fdt -rw-r--r-- 1 tomcat60 tomcat60 3404 Feb 28 18:00 _f4.fdx -rw-r--r-- 1 tomcat60 tomcat60 11929 Feb 28 18:00 _f4.fnm -rw-r--r-- 1 tomcat60 tomcat60 125852 Feb 28 18:00 _f4.frq -rw-r--r-- 1 tomcat60 tomcat60 141104 Feb 28 18:00 _f4.nrm -rw-r--r-- 1 tomcat60 tomcat60 248013 Feb 28 18:00 _f4.prx -rw-r--r-- 1 tomcat60 tomcat60 5082 Feb 28 18:00 _f4.tii -rw-r--r-- 1 tomcat60 tomcat60 369203 Feb 28 18:00 _f4.tis -rw-r--r-- 1 tomcat60 tomcat60280 Feb 28 18:00 segments_7m -rw-r--r-- 1 tomcat60 tomcat60 20 Feb 28 18:00 segments.gen On Tue, Feb 28, 2012 at 6:13 PM, Guillaume Fenollar guillaume.fenol...@xwiki.com wrote: Hi, When the page stucks, it's often because of a write issue on xwiki work directory. Are you sure you put a static value of xwiki.work.dir in xwiki.cfg file after your upgrade? Your application server must have the right to write in there. 2012/2/28 Joshua Davis pgm...@gmail.com Interesting. Is there some way to tell the Lucene not to index the attachments? On Tue, Feb 28, 2012 at 4:39 PM, ricardo.julio.rodriguez.fernan...@sergas.es wrote: Hi Joshua, From: users-boun...@xwiki.org [users-boun...@xwiki.org] On Behalf Of Joshua Davis [pgm...@gmail.com] Sent: 28 February 2012 21:30 To: users@xwiki.org Subject: [xwiki-users] Lucene re-index problem After upgrading to XWiki 3.5 (from 2.6) I noticed that the Lucene search index was not working properly. On advice I got in the #xwiki IRC chat, I shut down the server (Tomcat), deleted the Lucene index directory, and restarted the server. Only about 300 of the thousands of pages were indexed at that point.I tried re-indexing using the Admin-Search page, but every time I do that the Admin-Search page gets 'stuck' (won't render). I saw no errors in the catalina.out log file, and I configured WEB-INF/classes/logback.xml to enable some more logging to a separate file... still nothing very informative. Any ideas what I can do to troubleshoot this? I'm guessing there's some page in my wiki database that is causing the re-index process to get stuck. Thanks in advance, - Josh Almost two years ago we face a similar problem while updating to XWiki Enterprise 2.4.30451. We have been not able to get Lucene working in this installation. The problem here seems related with Microsoft Office attachements. I don't remember exactly how do we reach this conclusion and it could be a kind of shot in the air, but I remember that PDF files seem to index fine, but xls and doc files causes the failure of the process. As in your case, AdminSearch won't render. We have never find a solution. This installation is still running but we are using database search instead. i keep following this thread! Cheers, Ricardo ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users Nota: A informaci?n contida nesta mensaxe e os seus posibles documentos adxuntos ? privada e confidencial e est? dirixida ?nicamente ? seu destinatario/a. Se vostede non ? o/a destinatario/a orixinal desta mensaxe, por favor elim?nea. A distribuci?n ou copia desta mensaxe non est? autorizada. Nota: La informaci?n contenida en este mensaje y sus posibles documentos adjuntos es privada y confidencial y est? dirigida ?nicamente a su destinatario/a. Si usted no es el/la destinatario/a original de este mensaje, por favor elim?nelo. La distribuci?n o copia de este mensaje no est? autorizada. See more languages: http://www.sergas.es/aviso_confidencialidad.htm ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users ___ users mailing list users@xwiki.org http://lists.xwiki.org/mailman/listinfo/users -- Guillaume Fenollar XWiki SysAdmin Tel : +33 (0)1.83.62.66.15 ___ users mailing list users@xwiki.org
