I bet you can overwrite Spring XML bindings and define your own. I have no idea how to do that, and it seems dangerous.
Regards, Benoit On 05/11/2019 22:03, Jerry Malcolm wrote: > No problem. Is there a quick-fix / hack that perhaps I can hardcode to > bypass insertion until the actual fix is available? > > Just speculating here... we've got an option (lazyIndex) that rebuilds > the index every time whether it is necessary or not. And we have an > option (luceneIndex) that won't build a full index automatically even > when no index exists for the folder. Seems to me a much better option > would be a 'build it once if it doesn't exist' option. That would be > one less thing for an overwhelmed administrator (me) to remember to do > when migrating James. Ideas? > > Jerry > > On 11/5/2019 1:04 AM, Tellier Benoit wrote: >> My bad, there seems to be Spring injection issues: >> >> mailbox-tools.xml defines: >> >> <bean id ="fake-reindexer" >> class="org.apache.mailbox.tools.indexer.ThrowsReIndexer" >> lazy-init="true"/> >> >> As the indexer is configurable, we are not able to inject the >> ListeningMessageSearchIndex for doing that. >> >> We may need aditional logic upon the indexer choice for providing the >> correct reIndexer binding: >> - keep the throwing one when lucene is not selected >> - Use ReIndexerImpl with luceneSearchIndex injected in it when lucene >> is configured. >> >> I will write the issue, contributions are welcomed here! >> >> See https://issues.apache.org/jira/browse/JAMES-2965 >> >> Benoit >> >> On 05/11/2019 13:14, Jerry Malcolm wrote: >>> I had already changed back to luceneIndex from lazyIndex and rebooted. >>> I also removed the lazyIndex's version of the lucene folder. But I'm >>> still getting the same exception. >>> >>> >>> On 11/5/2019 12:07 AM, Tellier Benoit wrote: >>>> ReIndexing is not implemented on top of lazyIndex. >>>> >>>> You need to configure lucene in index.xml (meaning for you, downtime in >>>> search) >>>> >>>> On 05/11/2019 12:44, Jerry Malcolm wrote: >>>>> Benoit, >>>>> >>>>> This all makes sense. I tried the CLI to ReindexAll and got an >>>>> exception. I figured I was just doing something wrong on the CLI with >>>>> JMX, etc. But every other command I tried worked fine. Just the >>>>> ReindexAll and ReindexMailbox failed. (BTW.... the web page shows >>>>> the >>>>> command: |{cli} Reindex #private u...@domain.tld INBOX but the actual >>>>> command is ReindexMailbox for a specific mailbox.)| >>>>> >>>>> |Before I start digging into what is happening, does this exception >>>>> tell >>>>> you anything? Do I need to configure something in the james-cli.sh >>>>> file >>>>> telling it that I'm using JPA instead of file-based mailboxes or >>>>> anything like that? >>>>> | >>>>> >>>>> |Exception: >>>>> | >>>>> >>>>> [root@ip-172-31-32-236 james-3.4.0]# ./bin/james-cli.sh ReindexAll >>>>> ERROR 23:22:54,105 | org.apache.james.cli.ServerCmd | Error while >>>>> playing command >>>>> org.apache.james.mailbox.exception.MailboxException: Not implemented >>>>> at >>>>> org.apache.mailbox.tools.indexer.ThrowsReIndexer.reIndex(ThrowsReIndexer.java:43) >>>>> >>>>> >>>>> >>>>> at >>>>> org.apache.james.adapter.mailbox.ReIndexerManagement.reIndex(ReIndexerManagement.java:67) >>>>> >>>>> >>>>> >>>>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native >>>>> Method) >>>>> at >>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) >>>>> >>>>> >>>>> >>>>> at >>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) >>>>> >>>>> >>>>> >>>>> at java.lang.reflect.Method.invoke(Method.java:498) >>>>> >>>>> On 11/4/2019 8:48 PM, Tellier Benoit wrote: >>>>>> Hi Jerry, >>>>>> >>>>>> Thanks for the updated details, it now makes perfect sens! >>>>>> >>>>>> Long story short: Your Lucene index needs to be migrated as well. I >>>>>> suspect you migrated only your database (full copy), hence issues. >>>>>> >>>>>> To do migration, at Linagora, we prefer using external protocols, and >>>>>> leaving the inner data structures untouched (it's just too easy to >>>>>> make >>>>>> mistakes...). >>>>>> >>>>>> What happens is that then you had the data in SQL database but >>>>>> none in >>>>>> the Lucene index, hence the unaccurate search results harming the IOS >>>>>> experience. >>>>>> >>>>>> From that point: >>>>>> - You can fix that from re-indexing data from the SQL database >>>>>> and >>>>>> rebuild the Lucene index from it. There's webadmin(guice only) [1] >>>>>> and >>>>>> CLI [2] support for that (though the CLI experience, the only >>>>>> available >>>>>> option for Spring is more primitive. >>>>>> >>>>>> lazyIndex means full scan on every search request. Expect the AWS >>>>>> bill >>>>>> to be expansive. Expect your clients to complain about latencies. I >>>>>> would be you, I'd rebuild the Lucene index at the first occasion. >>>>>> >>>>>> Now, I also believe cascading user flags issues might be caused by >>>>>> some >>>>>> unappropriate table structure, there was some bugs reported regarding >>>>>> this. I'm unable to retrieve corresponding information. >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-webadmin.md#reindexing-all-mails >>>>>> >>>>>> >>>>>> >>>>>> [2] >>>>>> https://github.com/apache/james-project/blob/master/src/site/markdown/server/manage-cli.md#re-indexing >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> Cheers, >>>>>> >>>>>> Benoit >>>>>> >>>>>> On 05/11/2019 07:47, Jerry Malcolm wrote: >>>>>>> I'm starting a new thread for this specific issue. One external >>>>>>> symptom >>>>>>> that has now been isolated to this problem is documented in my >>>>>>> long-running (and hopefully completed) thread about iPhones >>>>>>> losing all >>>>>>> of their mail. >>>>>>> >>>>>>> Here's what I know about this so far: >>>>>>> >>>>>>> 1) I had no clue what 'Lucene' was or what it did. So I've never >>>>>>> touched anything in any config file related to it. Whatever >>>>>>> happened >>>>>>> came right out of the box as far as lucene config is concern. (ver >>>>>>> 3.4+... I extracted the master branch a day or so after 3.4 was >>>>>>> released). >>>>>>> >>>>>>> 2) I have been in the process of migrating 17+ years of mail in a >>>>>>> James >>>>>>> SQL db from v3b5 on a dedicated server to ver 3.4 in Amazon Web >>>>>>> Services >>>>>>> environment. >>>>>>> >>>>>>> 3) On 21Oct, I threw the switch and went live on my AWS environment. >>>>>>> >>>>>>> 4) Immediately some of my clients with iPhones complained that >>>>>>> all of >>>>>>> their mail disappeared from their phone. >>>>>>> >>>>>>> 5) Being an iPhone user myself, I verified the problem. However, I >>>>>>> noticed that mail arriving to the new James instance was showing >>>>>>> up in >>>>>>> the iPhone. Just no mail pre 21Oct. >>>>>>> >>>>>>> 6) Thunderbird was showing all of the old mail in the folders. But >>>>>>> I've >>>>>>> since determined that TBird uses a different imap command that goes >>>>>>> straight to the JPA database to populate folders, bypassing the >>>>>>> Lucene >>>>>>> index. >>>>>>> >>>>>>> 7) After two weeks of educating myself on the entire imap component >>>>>>> and >>>>>>> doing a whole lot of adding log statements and trial & error debug, >>>>>>> the >>>>>>> Lucene index became the prime suspect >>>>>>> >>>>>>> 8) Today, I removed the var/store/lucene folder and rebooted. The >>>>>>> iPhones now wiped out the last two weeks of email and either showed >>>>>>> "No >>>>>>> Mail" or mail that arrived in the few minutes after wiping the >>>>>>> lucene >>>>>>> folder. Note that it STILL didn't reindex old mail, even with an >>>>>>> index >>>>>>> 'restart'. >>>>>>> >>>>>>> 9) Finally on a whim, I changed indexer.xml to use "lazyIndex" >>>>>>> instead >>>>>>> of "luceneIndex". Finally, the indexes rebuilt successfully, and >>>>>>> the >>>>>>> iPhones now show all of the mail. Success... >>>>>>> >>>>>>> Summary: lazyIndex seems to have no problems. Not sure what's >>>>>>> going >>>>>>> wrong when luceneIndex is selected. But it isn't right. >>>>>>> >>>>>>> Do I have any mis-assumptions? Is there something else I need to >>>>>>> do in >>>>>>> order to get Lucene to index old mail? >>>>>>> >>>>>>> This has been a tough one... I think I'll take a nap now...:-) >>>>>>> >>>>>>> Jerry >>>>>>> >>>>>>> >>>>>>> --------------------------------------------------------------------- >>>>>>> >>>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>>> >>>>>> --------------------------------------------------------------------- >>>>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>>> For additional commands, e-mail: server-dev-h...@james.apache.org >>>> >>> --------------------------------------------------------------------- >>> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >>> For additional commands, e-mail: server-dev-h...@james.apache.org >>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org >> For additional commands, e-mail: server-dev-h...@james.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org > For additional commands, e-mail: server-dev-h...@james.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: server-dev-unsubscr...@james.apache.org For additional commands, e-mail: server-dev-h...@james.apache.org
