[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229537#comment-17229537
]
FP commented on JSPWIKI-1131:
-
Hi [~juanpablo],
I just repeated the tests described in
https://issues.apache.org/jira/browse/JSPWIKI-1131?focusedCommentId=17228500=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17228500
with commit 58d0820bf0185744e645c00d74e6ae246c8c91ca and it works as expected!
Thank a lot!
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229500#comment-17229500
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi [~uo5waiBil9],
thanks for your time and patience pursuing this one; based on your info I was
able to finally reproduce the bug and just pushed the fix, it should be
available with 2.11.0-M8-git-04.
The code responsible of orchestrating the page save + indexing was retrieving
the page from the Context, performing the save and the indexing the page,
without taking into account the new page version. This was fine with the
FileSystemProvider, but not with the VersioningFileProvider. Now this piece of
code ensures that the search indexing always indexes the page last version +
there are a couple more unit tests to ensure this behaviour.
Would you mind giving a try to latest master and confirming that the issue is
fixed?
thanks + best regards
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17229481#comment-17229481
]
ASF subversion and git services commented on JSPWIKI-1131:
--
Commit b5dda3813c66820fdd29e43a593ea178ee837f03 in jspwiki's branch
refs/heads/master from juanpablo
[ https://gitbox.apache.org/repos/asf?p=jspwiki.git;h=b5dda38 ]
JSPWIKI-1131: Ensure latest version of page gets indexed, so changes using
VersioningFileProvider get returned by searches
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17228500#comment-17228500
]
FP commented on JSPWIKI-1131:
-
Hi [~juanpablo],
I built commit 9406f01b338a643e37ea124ef8d5e794d709859b and was able to
pinpoint the issue to the use of `VersioningFileProvider` as page provider.
* The steps below were all executed on the same page: `TestLucene`
* The data directory only contained the wikipages-de at the start of the test
* Lucene index was rebuilt before starting the test
||Version before||Action||Version after||Searches||
|–|Create page and add "correct"|1|"correct": ✔|
|1|Add "horse"|2|"correct" ✔ "horse" ✘|
|2|Add "battery"|3|"correct" ✔ "horse" ✔ "battery" ✘|
|3|Add "staple"|4|"correct" ✔ "horse" ✔ "battery" ✔ "staple" ✘|
* It seems that the indexer does not pick up the latest version of the file
but the version before. The file TestLucene.txt in JSPWiki's pages directory
always contains the correct (=latest) version of the page and the corresponding
files in `OLD/TestLucene` seem to be correct, too.
* The following debug log entries are recorded when after adding "horse" (=
row2):
{code:java}
DEBUG org.apache.wiki.WatchDog - JSPWiki Lucene Indexer: Entering state
Emptying index queue, expected completion in 60 s
DEBUG org.apache.wiki.search.LuceneSearchProvider - Updating Lucene index for
page 'TestLucene'...
DEBUG org.apache.wiki.search.LuceneSearchProvider - Indexing TestLucene...
DEBUG org.apache.wiki.providers.CachingAttachmentProvider - Listing attachments
for WikiPage [Wiki:TestLucene,ver=1,mod=Mon Nov 09 11:04:08 CET 2020]
DEBUG org.apache.wiki.providers.CachingAttachmentProvider - LIST from cache,
TestLucene, size=0
DEBUG org.apache.wiki.search.LuceneSearchProvider - Done updating Lucene index
for page 'TestLucene'.
DEBUG org.apache.wiki.WatchDog - JSPWiki Lucene Indexer: Exiting state{code}
Some additional notes:
* Search works as expected, once I switch from `VersioningFileProvider` to
`FileSystemProvider`
* The setting `jspwiki.usePageCache` (true in may case) does not change the
behavior.
* The setting `jspwiki.lucene.indexdelay` does not change the behavior.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17227877#comment-17227877
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi [~uo5waiBil9],
the code indexing a page save is the same that indexes all the pages at the
startup, so I don't know where else to look to reproduce your issue; latest
push includes functional tests for search actions. Other than maven and a jdk,
these tests only require to have Chrome executable available on $PATH
The fastest way to verify their behaviour against a prepared JSPWiki war is to
{code:java}
mvn clean install -pl jspwiki-it-tests/jspwiki-it-test-custom -am
-Pintegration-tests
{code}
You may also run the functional tests individually from your IDE, against your
running instance. To do so, locate the class
{{org.apache.wiki.its.environment.Env}} on the jspwiki-selenide-tests and
modify the {{TESTS_BASE_URL}} variable, so it points to your running instance.
After that, you may run the {{org.apache.wiki.its.SearchIT}} as a regular test
from within your IDE. Would you mind verifying if these two approaches
(functional tests against prepared war vs functional test against you running
instance) yield different results?
Also, a couple more points that may help pinpoint your issue
* latest push updates the version of Lucene to latest, 8.7.0
* the prepared war makes use of two properties so that indexing takes place as
soon as possible
** {{jspwiki.lucene.initialdelay}} = 0 # this is the amount of seconds JSPWiki
will wait before starting the global indexing at startup (defaults to 60)
** {{jspwiki.lucene.indexdelay}} = 1 # this is the amount of seconds the
lucene indexer thread waits before indexing changes (defaults to 5)
** maybe these properties are worth playing with?
HTH
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214569#comment-17214569
]
FP commented on JSPWIKI-1131:
-
Hi [~juanpablo] ,
I built commit 1042544144954d9895ae8d2d2c49c0ca52ed47e7 and retested it with my
big repository and wikipages-de as built with the mentioned commit. The results
are identical in both setups. The steps are:
* Stop tomcat and purge lucene's index
* Start JSPWiki and wait until the index is generated
* Login
* Search for an existing word: found
* Edit any existing page and add a word to it (e.g. _kinship_)
* Search for _kinship_: not found
* Create a new page and add _kinship_ to it, save.
* Search for _kinship_ again: the only search result is the new page
* Stop tomcat and purge lucene's index
* Start JSPWiki and wait until the index is generated
* Login
* Search again for _kinship_: found on both pages
The timestamps and files in the index directory are updated with each relevant
step. The wikipages seem to be sufficient to trigger my issue.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214122#comment-17214122
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi again [~uo5waiBil9],
as page indexing seems to be having taking place, but not being applied, I'm
guessing/suspecting that may be there's a concurrency problem taking place,
which may be showing only on your installation due to the big page repository
size. In order to try to pinpoint this condition, 2.11.0-M8-git-02 brings a
couple of changes on LuceneSearchProvider :
* it is using now NIOFSDirectory instead of (deprecated) SimpleFSDirectory
* all Lucene's index writes are synchronized, whereas reads are not; previously
only 1 out of 3 writes were synchronized
would you mind testing latest master? also, would you mind trying to see if you
get this behaviour with a smaller page repository? f.ex. [this
one|https://search.maven.org/artifact/org.apache.jspwiki.wikipages/jspwiki-wikipages-en/2.11.0.M7/jar].
Let's see if this helps to identify your issue..
best regards,
juan pablo
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Fix For: 2.11.0-M8
>
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17214114#comment-17214114
]
ASF subversion and git services commented on JSPWIKI-1131:
--
Commit 1caf85ac50871b0eef48556b148702d7e5ec4e04 in jspwiki's branch
refs/heads/master from juanpablo
[ https://gitbox.apache.org/repos/asf?p=jspwiki.git;h=1caf85a ]
LuceneSearchProvider using now NIOFSDirectory instead of (deprecated)
SimpleFSDirectory
Also, all Lucene's index writes are synchronized, whereas reads are not. This
should help with issues noted at JSPWIKI-1131
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt, illegal-page-name.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209511#comment-17209511
]
FP commented on JSPWIKI-1131:
-
Hi [~juanpablo] ,
thanks a lot for your patience, I really appreciate it!
I used luke to inspect the lucene index and I still see the odd behavior
regarding existing/new pages. Here are the steps I took:
* Recorded the timestamp of the lucene directory.
* Edited a page that has a single sentence in it. No ACL rules or any markup.
There, I added the word 'kinship'.
* Recorded the timestamp of the lucene directory and compared them with the
previous timestamps. They changed and new files were added.
* Opened the index in luke (ro) and searched for 'contents:kinship' - No
matches in luke (nor jspwiki search). (number of documents: 13424; number of
terms: 229559)
* Recorded the timestamp of the lucene directory
* Created a new page `TestLucene9` and added the word 'kinship'.
* Compared the timestamps again; they changed once again.
* Opened the index in luke (ro) and searched for 'contents:kinship' - 1 match
on the page 'TestLucene9'. Same results in jspwiki search. (number of
documents: 13425; number of terms: 229562).
The debug log entries report success for index updating: "Done updating Lucene
index for page '$page'."
Just to make sure that 'kinship' is a word that gets picked up by lucene:
* Stop tomcat
* Remove the lucene directory
* Start tomcat and waited until the lucene index was rebuilt
* Opened the index in luke (ro) and searched for 'contents:kinship' - 2
matches on the pages 'TestLucene9' the existing one. Same results in jspwiki
search. (number of documents: 13425; number of terms: 229516).
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17209067#comment-17209067
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi [~uo5waiBil9],
been trying to reproduce the error this weekend, without success. As per your
log entries, it seems that the page reindex gets done, so I'm thinking that
you're issue may be related to the page manager; As soon as the search is
performed, we check if the user requesting the search has enough permissions to
see the page. That may be triggering the
{{org.apache.wiki.api.exceptions.ProviderException: Illegal page name}}
exception, so if you could share the associated stack trace may be it could
help to see if it's related or not to this issue.
Also, to double check that the reindex is behaving properly, would you mind
inspecting the Lucene's index files with
[Luke|https://medium.com/@mocobeta/luke-become-an-apache-lucene-module-as-of-lucene-8-1-7d139c998b2]
after saving a page? The query should look like something similar to
{{content:your_term}}, f.ex., {{content:staple}}
Last, as for the {{JSPWikiMarkupParser}} log message, it can safely be
discarded wrt this issue, some of your pages seem to reference the
BubbleChartPlugin, which hasn't been installed on the development instance
best regards,
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206150#comment-17206150
]
FP commented on JSPWIKI-1131:
-
These are the steps to setup the development snapshot:
* Build 43a15c85600d33054eec8fea8c7b622dd582d493 via `mvn clean package`
* Setup a fresh Debian 10 host.
* Copy the data directory from the existing installation (~ 21 GB of wiki
pages and attachments)
* Setup my build. There are no old classes or lucene indices lingering around.
Everything is setup afresh.
* Start tomcat
The lucene index is created from scratch as the log reports:
{code:java}
[JSPWiki Lucene Indexer] INFO org.apache.wiki.search.LuceneSearchProvider -
Starting Lucene reindexing, this can take a couple of minutes...
[...]
[JSPWiki Lucene Indexer] INFO org.apache.wiki.search.LuceneSearchProvider -
Full Lucene index finished in 101851 milliseconds{code}
I repeated the exact same steps as mentioned in my previous comment. The
behavior is identical:
* Words on new pages are found immediately.
* Words on existing pages are not recognized by the search. A restart of
tomcat does not find those new words either. The search only recognizes those
words after deleting the lucene indices and let JSPWiki rebuild them from
scratch.
During testing, I noticed the following messages/exceptions. I don't know if
those effect the search (but I suppose no, as I edited pages that are OK):
*
{code:java}
[main] INFO org.apache.wiki.parser.JSPWikiMarkupParser - Wiki : Foo - Failed to
insert plugin: Could not find plugin BubbleChartPlugin{code}
*
{code:java}
org.apache.wiki.api.exceptions.ProviderException: Illegal page name{code}
The "Illegal page name" page name exception points towards
[https://github.com/apache/jspwiki/blob/f47298624be7bfc277d12e12437d68cfbc793d0c/jspwiki-main/src/main/java/org/apache/wiki/pages/DefaultPageManager.java#L500].
Until now, I was unable to find the culprit.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205823#comment-17205823
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi [~uo5waiBil9],
thanks for your patience and your time testing. I've proceeded with the actions
noted on a fresh 2.11.0-M8-git-01 installation, and the search is behaving as
expected, that is, founds all pages for a given a term. I'm unsure of what may
be happening as I can't reproduce the issue. As the version of Lucene has
changed between 2.11.0.M7 and 2.11.0-M8-git-01, would you mind repeating the
testing but removing the lucene index files first (they should be under
"${jspwiki.workDir}/lucene")? I'm also thinking about emptying
$TOMCAT_HOME/work/.../JSPWiki files too, to discard older precompiled classes
interfering
best regards,
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205444#comment-17205444
]
FP commented on JSPWIKI-1131:
-
I built 43a15c85600d33054eec8fea8c7b622dd582d493 via `mvn clean package` and
updated our installation. The exception when a user saves the page is now gone.
But I noticed the following:
* New pages are indexed immediately: created a new page »TestLucene1« and
provided the following content: »correct horse battery staple«. The search for
each of the words above presented the »TestLucene1« page.
* Edited 3 randomly selected pages and also added the content »correct horse
battery staple« to them. The search for each of the words above found the page
»TestLucene1« as before but none of the randomly edited pages.
One of the 3 edited pages is called »Changelogs«. With debugging enabled, I can
see these lines in the log:
{code:java}
[...]
[JSPWiki Lucene Indexer] DEBUG org.apache.wiki.WatchDog - JSPWiki Lucene
Indexer: Entering state Emptying index queue, expected completion in 60 s
[JSPWiki Lucene Indexer] DEBUG org.apache.wiki.search.LuceneSearchProvider -
Updating Lucene index for page 'Changelogs'...
[JSPWiki Lucene Indexer] DEBUG org.apache.wiki.search.LuceneSearchProvider -
Indexing Changelogs...
[JSPWiki Lucene Indexer] DEBUG
org.apache.wiki.providers.CachingAttachmentProvider - Listing attachments for
WikiPage [Wiki:Changelogs,ver=137,mod=Thu Oct 01 13:58:29 CEST 2020]
[JSPWiki Lucene Indexer] DEBUG
org.apache.wiki.providers.CachingAttachmentProvider - LIST from cache,
Changelogs, size=8
[JSPWiki Lucene Indexer] DEBUG org.apache.wiki.search.LuceneSearchProvider -
Done updating Lucene index for page 'Changelogs'.
[JSPWiki Lucene Indexer] DEBUG org.apache.wiki.WatchDog - JSPWiki Lucene
Indexer: Exiting state Emptying index queue
[...]{code}
In my opinion, the search results should pop up as soon as indexing is complete
as depicted in the logs. This is not the case.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205359#comment-17205359
]
FP commented on JSPWIKI-1131:
-
Yes, will do so soonish.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
Hi,
this should be fixed on 2.11.0-M8-git-01. It wasn't as straightforward as I
thought, but should be fixed on latest master.
As of the page index not being done, it seems it was simply not done. I
suspect page indexing was kicked indirectly, and that was probably lost on
one of the refactorings that took place on 2.11.0.M7.
Next thing I'm planning to do is to add a couple of functional tests so
edits and page searches can be automatically tested.
best regards,
juan pablo
On Sun, Sep 27, 2020 at 7:41 PM Murray Altheim wrote:
> Hi Harry,
>
> Yes, I think I've gotten so used to this I forgot it would be considered
> a bug. This occurs on my older wikis as well as my latest wiki for
> robots.org.nz's wiki at https://service.robots.org.nz/wiki/
>
> I'm trying to think back as to what triggers a Lucene update, probably
> the EventListener/EventManager? Sorry I can't be more helpful on this.
>
> Cheers,
>
> Murray
>
> On 28/09/20 1:10 am, Harry Metske (Jira) wrote:
> >
> > [
> https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202816#comment-17202816
> ]
> >
> > Harry Metske commented on JSPWIKI-1131:
> > ---
> >
> > This is really an annoying one, I experience the same on my own wiki.
> >
> >> Lucene Index not updated on edits/new page
> >> --
> >>
> >> Key: JSPWIKI-1131
> >> URL:
> https://issues.apache.org/jira/browse/JSPWIKI-1131
> >> Project: JSPWiki
> >> Issue Type: Bug
> >> Components: Search
> >> Affects Versions: 2.11.0-M7
> >> Environment: * Debian 10
> >> * Tomcat 9: 9.0.31-1~deb10u2
> >> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> >> * JSPWiki 2.11.0-M7
> >> Reporter: FP
> >> Priority: Critical
> >> Attachments: exception.txt
> >>
> >>
> >> It seems that the lucene search index is not updated when a user edits
> an existing page or creates a new page. In both cases, the page is not
> found when someone searches for it.
> >> Search works as expected when:
> >> * Tomcat is stopped
> >> * The files in the {{lucene}} directory are removed
> >> * Tomcat is started
> >> * Wait until the Lucene Indexer thread finishes reindexing
> >> The following exception occurs when a page is edited (full stack trace
> attached):
> >> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
> >
> >
> >
> > --
> > This message was sent by Atlassian Jira
> > (v8.3.4#803005)
> >
>
> --
>
> ...
> Murray Altheim= = ===
> http://www.altheim.com/murray/ ===
> ===
> = =
> ===
> In the evening
> The rice leaves in the garden
> Rustle in the autumn wind
> That blows through my reed hut.
> -- Minamoto no Tsunenobu
>
>
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205015#comment-17205015
]
ASF subversion and git services commented on JSPWIKI-1131:
--
Commit f582ecd5fdae98924de659b35b12faf8b51b6ec7 in jspwiki's branch
refs/heads/master from juanpablo
[ https://gitbox.apache.org/repos/asf?p=jspwiki.git;h=f582ecd ]
JSPWIKI-1131: tasks don't store contexts anymore they receive them through the
execute(..) method
Context must be carried around through workflow, tasks and classes using them
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205016#comment-17205016
]
ASF subversion and git services commented on JSPWIKI-1131:
--
Commit a20613b41432b2ea62c0548ce8dcfa3ade09d98c in jspwiki's branch
refs/heads/master from juanpablo
[ https://gitbox.apache.org/repos/asf?p=jspwiki.git;h=a20613b ]
JSPWIKI-1131: provide context to workflow methods so it can be passed around
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17205014#comment-17205014
]
ASF subversion and git services commented on JSPWIKI-1131:
--
Commit 0676327ccbd86a1ef62b29321db8ed7eb81e09e5 in jspwiki's branch
refs/heads/master from juanpablo
[ https://gitbox.apache.org/repos/asf?p=jspwiki.git;h=0676327 ]
JSPWIKI-1131: allow events to carry args - objects related to event's src
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: [jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
Hi Harry,
Yes, I think I've gotten so used to this I forgot it would be considered
a bug. This occurs on my older wikis as well as my latest wiki for
robots.org.nz's wiki at https://service.robots.org.nz/wiki/
I'm trying to think back as to what triggers a Lucene update, probably
the EventListener/EventManager? Sorry I can't be more helpful on this.
Cheers,
Murray
On 28/09/20 1:10 am, Harry Metske (Jira) wrote:
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202816#comment-17202816
]
Harry Metske commented on JSPWIKI-1131:
---
This is really an annoying one, I experience the same on my own wiki.
Lucene Index not updated on edits/new page
--
Key: JSPWIKI-1131
URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
Project: JSPWiki
Issue Type: Bug
Components: Search
Affects Versions: 2.11.0-M7
Environment: * Debian 10
* Tomcat 9: 9.0.31-1~deb10u2
* OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
* JSPWiki 2.11.0-M7
Reporter: FP
Priority: Critical
Attachments: exception.txt
It seems that the lucene search index is not updated when a user edits an
existing page or creates a new page. In both cases, the page is not found when
someone searches for it.
Search works as expected when:
* Tomcat is stopped
* The files in the {{lucene}} directory are removed
* Tomcat is started
* Wait until the Lucene Indexer thread finishes reindexing
The following exception occurs when a page is edited (full stack trace
attached):
{{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
--
...
Murray Altheim= = ===
http://www.altheim.com/murray/ === ===
= = ===
In the evening
The rice leaves in the garden
Rustle in the autumn wind
That blows through my reed hut.
-- Minamoto no Tsunenobu
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202816#comment-17202816
]
Harry Metske commented on JSPWIKI-1131:
---
This is really an annoying one, I experience the same on my own wiki.
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Critical
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (JSPWIKI-1131) Lucene Index not updated on edits/new page
[
https://issues.apache.org/jira/browse/JSPWIKI-1131?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17202341#comment-17202341
]
Juan Pablo Santos Rodríguez commented on JSPWIKI-1131:
--
Hi, been able to reproduce the bug, seems that there are two separate issues
here:
* the attached exception: seems that workflows are not entirely serializable -
they have associated tasks, which are a type of steps, which store Contexts,
which aren't serializable. Workflow with attached tasks are serialized when
writting the completed workflows list. Hence the exception. This isn't catched
by current tests, IIRC, they start a workflow, don't finish it and invoke the
(de)serialization stuff.
* an exception serializing workflows should not affect search indexing or any
other thing, if there is an exception when (de)serialization is taking place,
it is logged and swallowed, with JSPWiki resuming its normal activity. Seems to
me that search indexing event is not being fired at all, not being related to
the exception noted above. Have to take a closer look at it, though.
As of how to fix first point, I was thinking to refactor tasks, so they don't
store a context; Tasks should extract needed information from Context and store
it, instead of directly storing Context. This is easy to do, however, on the
execute method, the tasks also make use of some engine's managers. Having a
reference to the Context is easy to grab the Engine, but if we are going to not
store the Context (or the Engine or whatever manager needed as none of them are
Serializable), then we have to pass the Engine as a parameter to the execute()
method. This involves some refactoring on the workflow code. It is not
specially complicated, but it involves a number of classes on the workflow
package, so I've to take a better look at it.
As for the second point, it may be as simple as firing the search index event,
but as said, I'd prefer to have a better look at it in order to see what's
happening to properly fix it. Will increase the severity of the issue because
of this second point.
(and apologies on the late response, not having too much time to spend on
JSPWiki these past months)
> Lucene Index not updated on edits/new page
> --
>
> Key: JSPWIKI-1131
> URL: https://issues.apache.org/jira/browse/JSPWIKI-1131
> Project: JSPWiki
> Issue Type: Bug
> Components: Search
>Affects Versions: 2.11.0-M7
> Environment: * Debian 10
> * Tomcat 9: 9.0.31-1~deb10u2
> * OpenJDK-JRE 11: 11.0.7+10-3~deb10u1
> * JSPWiki 2.11.0-M7
>Reporter: FP
>Priority: Major
> Attachments: exception.txt
>
>
> It seems that the lucene search index is not updated when a user edits an
> existing page or creates a new page. In both cases, the page is not found
> when someone searches for it.
> Search works as expected when:
> * Tomcat is stopped
> * The files in the {{lucene}} directory are removed
> * Tomcat is started
> * Wait until the Lucene Indexer thread finishes reindexing
> The following exception occurs when a page is edited (full stack trace
> attached):
> {{java.io.NotSerializableException: org.apache.wiki.WikiContext}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
