Hi, On Wed, Nov 28, 2012 at 5:35 PM, Ludovic Dubost <[email protected]> wrote:
> 2012/11/28 Eduard Moraru <[email protected]> > > > Hi Ludovic, > > > > Thanks for the reply. Please read below... > > > > On Tue, Nov 27, 2012 at 5:44 PM, Ludovic Dubost <[email protected]> > wrote: > > > > > Hi Edy, > > > > > > I'm not a huge fan of the title_fr title_en title_morelanguages > approach > > as > > > indeed it seems to be quite complex at the query level. I was more > > leaning > > > towards multiple indexes if we can query them globally but I understand > > > this is complex too. > > > > > > Now let's see the use cases that are hugely important: > > > > > > 1/ Make sure that if you decide your wiki is monolingual: > > > - the indexing uses the specific language analyzer > > > - make sure the query uses the specific language analyzer > > > - make sure the search looks in all content even if the language > > setting > > > of the document is wrongly set (consider all documents being of the > > > specific language) > > > > > > > You mean that, if the wiki is monolingual, we should ignore the language > > filter and hardcode it to "All languages", right? > > > > However, what would be the advantage of this? Why would be want to > pollute > > the results with irrelevant documents (caused by a probable recent > > configuration change that went from multi-lingual to mono-lignual)? > Wasn't > > that the whole reason why the admin switched to mono-lingual? > > > > > If the user is monolingual, we can safely ignore the language setting of > each document and only the "main" document will be shown anyway to the > user. > So we should make sure we search on ALL documents that are available to the > user. > > The user might have to "reindex" to make sure this is properly taken into > account by the search engine. > > What is important here, is that even if the wiki is set to "fr", then even > if documents have "en" for the main language they will still show up in the > search. The opposite would be bad. > Why/How can you end up with documents of other languages than the current default one if the wiki is mono-lingual? > > > > > > > > 2/ Allow is a wiki is multi-lingual: > > > - search in the language you decide (maybe the UI should display a > > > language choice for the query) > > > > > > > We already support this, by using the "Filtered Search" option and > > selecting the language. > > > > Sure. I was just repeating what is important. > > > > > > - search in content that is analyzed in the proper language when the > > > content is declared in this language > > > - allow to specify if you want to restrict your search to documents > > > declared in the language of your query, versus search more widely in > all > > > documents accross languages. If you search in only the language of the > > > query only one document can show up but it should point to the right > > > translation that matches, if you search in multiple languages then you > > can > > > show individual translations. > > > > > > > I think this one is the same as the first bullet above. > > > > No I think it's different. The user can decide to make his search in > "French" but look in the "French" + "English" dataset. For me if both > documents with the same page name match, both should come out separately. > > It's the same as in the UI allows you to specify which language to search in, but it also allows you to specify that you want to search in "All languages", that is what I meant. > > > > > - allow technical users to search for all documents across all > > languages > > > (where the language analysis does not really matter) > > > > > > > Do you mean as an API? > > > > Not specifically as an API > > > > > > What exactly do you mean by "language analysis does not really matter"? > Any > > Example? > > > > I mean here that as a technical user your objective is to make sure your > search spans ALL content in the wiki. In this case you don't care about > stemming and such. > The standard UI could take this into account by choosing "any language > search" and with data set "all languages". We just need to make sure that > this won't exclude any content of the search > > (Here is an example of a case where such exclusion might occur. Suppose you > have done only "French" and "English" indexing and there is "German" > content also in your wiki but since you have not asked for German search in > your wiki you don't have title_de and content_de fields or don't have a > german specific solr index (in the other method), then your german content > would not be indexed at all ?) > > I understand now what you mean. Well, to avoid this, we can use only one notion and that is the one of "supported languages". All the supported languages should be indexed in order for the Search feature to work as expected (and not have the case where you have a document that is not indexed because of its language). I am not sure if there are many examples where admins would choose to index just some of the supported languages and not index all of them. WDYT? > > > > > > > > > > > > From an admin point of view it makes good sense for the admin to be > able > > to > > > specify in a multilingual wiki which language analysis should be > > activated, > > > and then have this transmitted to SOLR to properly configure the > engine. > > > Reindexing is ok when changing the configuration. > > > > > > I believe in the end wether you use multiple fields with _fr _en or > > > multiple SOLR cores, as long as you can query accross SOLR cores is a > bit > > > the same. If you cannot run a query merging multiple indexes then the > > first > > > solution is kind of absolutely necessary as it would be the only one > > > allowing to search across all languages. > > > > > > Maybe a solution would be to create one index per language and index > ALL > > > content regardless of it's language using the language analyzer of that > > > index. This would allow to have better results even though the users > have > > > badly tagged the language of a document, and it's only the job of the > UI > > to > > > limit the search to only the language of the query, or all documents. > > > > > > > So you could have a configuration in the admin that says: > > > > > > 1/ Create an English Index > > > 2/ Create an additional French index > > > > > > The UI would allow to search in English and French, + would add a > > language > > > restriction for the documents. > > > > > > > Applying the language specific analyzers (for Chinese, for example) to > all > > the documents will just create a mess for all the documents that do not > > match the analyzer's language. I`m not sure the results for the > > badly-indexed languages will make any sense to users. > > > > > I undestand the issue here, but in most cases the user will say "french > search" on "french content", he will only expand to non french content if > he was not satisfied by his search. What is just allowed here is to also > search in "french" inside all content. That would cover content that would > have the bad language setting as well as any other content. The results > might be a bit noisy but I don't think it's a big issue. > What I think you are actually asking is that all languages are queried, but that results in the requested language get boosted/elevated and thus are displayed before all other language results. Perhaps this can be achieved by using the lang field as a query field and not as a query filter. This means that it will influence the score. Couple that with a higher boost for the lang field and results in the requested language should be scored better than all the rest (because there will always be a hit for the lang field resulting in a better score). There might still be some results from other languages that could have a better score than results from the requested language (because they just have more hits on diferent fields), but it should work in most cases. However, I`m not sure we want this. Users might results from other languages as a bug. Problems might also be harder to debug. Instead, we could do something like Google and if we see a small number of results, we could suggest that the user searches in all languages. In any case, I`m not sure this is such a big issue right now, since it's basically an optimisation that we could choose to do, or let the user do it himself, since he has the tools (UI). > > > Also, this is very similar to the multi-core approach (one core per > > language), just that you also add documents that are indexed with the > wrong > > analyzers. We have the same problem regarding merging relevance scores > > across indexes (cores) that is a big turn-off for the original multi-core > > approach. > > > > This is a more serious issue. If it's hard to merge the results spanning > over multiple cores this could be a showstopper. However the solution of > having only one Lucene document for all languages is not so cool either as > it would make it difficult to know which ones of the languages has matched > and present them separately with separate scores. > I agree. The one-document-for-all-languages solution was proposed by Paul, but I don`t think it's the way to go. We are currently only considering the one-document-per-language direction. > It's really the core issue to decide on. What are the benefits and > drawbacks of the different solutions. For each solution is there something > in the UI that you cannot do ? > So far I've heard: > > 1/ Presenting different scores for documents in different languages with > the same doc name if the title_fr,content_fr method is used > Since we are not considering the one-document-for-all-languages solution, this is not an issue. 2/ Merging scores accross indexes in multicore approach > > Other ? Can we list them in a wiki page ? > > > > > > > > In the future if we are able to "detect" the language of the documents > we > > > could add a lucene field with the "detected" language instead of the > > > "provided" language of the documents, therefore increasing the quality > of > > > searches only on documents of a specific language. > > > > > > > In the previous discussions (on the GSoC thread) we agreed that language > in > > XWiki is known before-hard, so no recognition is required, at least not > at > > document level. > > > > Let's forget this > > > > > > > > > > This later solution would be the only one that would really work on > file > > > attachements as we have no information about the specific language of > > file > > > attachements (or even XWiki objects) which are attached to the main > > > document and not to the translated document. > > > > > > > Yes, this is a problem right now. AFAIU, the plan [1] is to support > > translated objects and maybe attachments as well. Until then, we could > > either: > > 1) Use the original document's language to index the attachment's content > > > > This is not a good solution. If I understand correctly we could not end up > not searching in a french attachment because the original document is in > marked "en". > I'm for Paul's solution to index objects and attachments in each > translation (if we have separate entities for translated documents). Yep, we already do this for objects, just need to do it for attachments as well. > I > understand that in the title_fr,content_fr approach this problem does not > happen. > This problem is not related to how we handle multiple translations (separate indexes or not), as long as each translation is a separate entity. Basically this last part is the problem. If the French translation is a separate entity from the original document (i.e. that is in English), any object/attachment *index field* of the English original version will need to be duplicated into the French translation as well, or the French translation risks not to get a hit on the object/attachment. > > > 2) Use a language detection library to try to detect the attachment > > content's language and index it accordingly. > > > > Not sure we can for now > > > > The above could also be applied for objects and their properties. > > ---------- > > [1] http://jira.xwiki.org/browse/XWIKI-69 > > > > > > > > > > This later issues shows that a search on "only french content" should > > still > > > include the attachements because we have no idea if the attachments are > > > "french" or "english". > > > > > > > (The paragraphs below discuss on what currently exists and what could be > > done, ignoring the possible language detection mentioned above) > > > > Right now a document also indexes the object's properties in a field > called > > "objcontent". I do this for all translations, thus duplicating the > field's > > value in all translations. I can do the same for attachments. The purpose > > is, indeed, to be able to find document translations based on hits in > their > > objects/attachments. If a language filter is used and there is a hit in > an > > object, only one document is returned. If there are no language filters, > > all translations will be returned. > > > > It seems we have to do this for now > > > > > > However, if we search for the object/property/attachment itself, it will > > only be assigned to one language: the language of the original document. > > This means that if we search for all languages, the object itself will be > > found too (there is no language filter used). If we add a language filter > > that is different from the object/property/attachment's original document > > language, the object/property/attachment will not be found. > > > > Maybe we can come up with some processing of the query in the search > > application, that applies the language filter only for documents: > > > > ((-type:"OBJECT" OR -type:"OBJECT_PROPERTY" OR -type:"ATTACHMENT") OR > > lang:"<userSelectedLanguage>") -- writing it like this because the > default > > operand is AND in the query filter clause that we use in the Search > > application. > > > > The problem with this is that that, when a language filter is used, the > > object/property/attachments that are now included in the results might > not > > have the specified language and will pollute the results. > > > > I'm not sure I understand. We have an "objcontent" field for each > translation that has the full content of objects and properties, but > individual object fields we don't have in each translations ? > "objcontent" stores the properties (format: "propName:propValue") of each object inside the original document (multi-valued field). Besides XWikiDocuments, we also index Objects, Properties and Attachments as Lucene/Solr first-class documents. Each of these entries has the wiki, space, name and lang fields set from the document to which they belong to. This is what I meant above with "if we search for the object/property/attachment itself" So to reiterate, the idea was that if you want to search for an indexed Object (type:"OBJECT"), you will *have* to avoid setting a language filter, or you might not find the object you are looking for, since it is indexed under the language of original document. --- While writing this down, I just thought of an elegant idea to fix this. The lang field could be set to multi-valued. This means that, when we index objects, properties and attachments, we could set in the lang field all the values corresponding to all the existing translations of the owning document. Example: An Object: id: xwiki:Main.Page^XWiki.Class[0] <-- (object reference) class: XWiki.Class wiki: xwiki space: Main name: Page lang: en <-- proposal to make it multi-valued would look like this, stored like a list of values. fr de type: OBJECT Note that this solution would also affect document languages as well (since they share the same schema), but we will just put one value and it will not affect queries. Queries will still be written like: "lang:en" If we apply this solution, even if a language filter exists in the user's query, it will still hit the object because the lang field of the object contains the value. > The more I see all the issues, the more I lean towards a separate index per > language solution. Again, these issues are not related to this, just to the fact that Document translations are first-class Lucene documents. > The reason I do is that the main need is for a non > English user to have very relevant results in his own language. Therefore > we need to make sure that all content that the users have published has the > chance to be analyzed using the non English language analyzer. So indexing > all objects and attachments with the relevant language analyzer is the > solution. This is also why I proposed to index all content in this specific > index regardless of the language declared, which would only be used in the > UI to limit searches to the specific language. > > In this view: > > 0/ There would be a language specific index per language with the objects > and attachments indexed only in the language of the index > 1/ The user chooses the language in which he searches > 2/ Automatically that sets the index to be used to be the "french" index > 3/ Automatically that presets to limit the span of the search to declared > "french" documents > 4/ The user can decide to go for non french documents at his own risks > knowing that the results might be weird because of wrong analysis (this is > what happens today with english analysis over french documents) > > The benefit here is that you don't have a merging score over multiple index > issue, since you would never have to do a search across multiple indexes. > Searches are still simple to write. By default results are quite relevant > since you limit the search the french declared documents (this would be the > same as limiting your search to title_fr, content_fr) and still cover what > needs to be covered (objects and attachments). > Another benefit is that this falls back gracefully to monolingual as you > just have to have one index in the language declared for the monolingual > wiki. > The drawback is that the indexing is more costly and there is duplicated > content in the index. Howerver it is the Admin that say which languages he > wants available and he takes responsibility of the ressources this needs. > > Could this solution work ? > Unfortunately I am still not convinced by this approach. Besides the complexities of managing multiple cores (each with it's own schema and config files), the user is exposed to a lot of unnecessary badly indexed data that will make him stay away from the search feature, as it has happened with Lucene so far. I believe this discussion thread has come up with some nice solutions to most of our problems and that the multiple-fields direction is one that can give us relevant results, properly indexed content for all languages (even when searching in all languages) and a good solution for objects, attachments and properties (though, again, this is not related to this specific choice). I will try to make a summary of the ideas from this thread and put them into a wiki page that documents the design/progress of the multi-lingual related work. Thank you very much for now and, of course, this does not mean that the discussion is over in any way :) -Eduard > > Ludovic > > > > > > > > Thanks, > > Eduard > > > > > > > Ludovic > > > > > > > > > > > > 2012/11/26 Eduard Moraru <[email protected]> > > > > > > > Hi devs, > > > > > > > > Any other input on this matter? > > > > > > > > To summarize a bit, if we go with the multiple fields for each > > language, > > > we > > > > end up with an index like: > > > > > > > > English version: > > > > id: xwiki:Main.SomeDocument_en > > > > language: en > > > > space: Main > > > > title_en: XWiki document > > > > doccontent_en: This is some content > > > > > > > > French version: > > > > id: xwiki:Main.SomeDocument_fr > > > > language: fr > > > > space: Main > > > > title_fr: XWiki document > > > > doccontent_fr: This is some content > > > > > > > > The Solr configuration is generated by some XWiki UI that returns a > zip > > > > that the admin has to unpack in his (remote) Solr instance. This > could > > be > > > > automated for the embedded instance. This operation is to be > performed > > > each > > > > time an admin changes the indexed languages (rarely or even only > once). > > > > > > > > Querying such a schema is a bit tricky when you are interested in > more > > > than > > > > one language, because you have to add all the clauses (title_en, > > > title_fr, > > > > etc.) specific to the languages you are interested in. > > > > > > > > Some extra fields might also be added like title_ws (for whitespace > > > > tokenization only) that have various approaches to the indexing > > > operation, > > > > with the aim of improving the relevancy. > > > > > > > > One solution to simplify the query for API clients would be to use > > fields > > > > like "title" and "doccontent" and to put as values very lightly (or > not > > > at > > > > all) analyzed content, as Paul suggested. This would allow > applications > > > to > > > > write simple (and backwards compatible maybe) queries that will still > > > work, > > > > but will not catch some of the nuances of specific languages. As far > as > > > > I`ve seen until now, applications are not very interested in nuances, > > but > > > > rather in filtering the results, a task for which this solution might > > be > > > > well suited. Of course, nothing stops applications from using the > *new* > > > and > > > > more expressive fields that are properly analized. > > > > > > > > Thus, the search application will be the major beneficiary of these > > > > analyzed fields (title_en, title_fr, etc.), while still allowing > > > > applications to get their job done (trough generic, but less/not > > analized > > > > fields like "title", "doccontent", etc.). > > > > > > > > WDYT? > > > > > > > > Thanks, > > > > Eduard > > > > > > > > > > > > > > > > > > > > On Wed, Nov 21, 2012 at 10:49 PM, Eduard Moraru < > [email protected] > > > > >wrote: > > > > > > > > > Hi Paul, > > > > > > > > > > I was counting on your feedback :) > > > > > > > > > > On Wed, Nov 21, 2012 at 3:04 PM, Paul Libbrecht <[email protected] > > > > > > wrote: > > > > > > > > > >> > > > > >> Hello Eduard, > > > > >> > > > > >> it's nice of you to see you take this further. > > > > >> > > > > >> > This issue has already been previously [1] discussed during the > > GSoC > > > > >> > project, but I am not particularly happy with the chosen > approach. > > > > >> > When handling multiple languages, there are generally[2][3] 3 > > > > different > > > > >> > approaches: > > > > >> > > > > > >> > 1) Indexing the content in a single field (like title, > doccontent, > > > > etc.) > > > > >> > - This has the advantage that queries are clear and fast > > > > >> > - The disadvantage is that you can not run very well tuned > > analyzers > > > > on > > > > >> the > > > > >> > fields, having to resort to (at best) basic tokenization and > > > > >> lowercasing. > > > > >> > > > > > >> > 2) Indexing the content in multiple fields, one field for each > > > > language > > > > >> > (like title_en, title_fr, doccontent_en, doccontent_fr, etc.) > > > > >> > - This has the advantage that you can easily specify (as dynamic > > > > fields) > > > > >> > that *_en fields are of type text_en (and analyzed by an > > > > >> english-centered > > > > >> > chain of analyzers); *_fr of type text_fr (focused on french, > > etc.), > > > > >> thus > > > > >> > making the results much better. > > > > >> > > > > >> I would add one more field here: title_ws and text_ws where the > full > > > > text > > > > >> is analyzed just as words (using the whitespace-tokenizer?). > > > > >> A match there would even be preferred to a match in the below > > > > text-fields. > > > > >> > > > > >> (maybe that would be called title and text?) > > > > >> > > > > >> > - The disadvantage is that querying such a schema is a pain. If > > you > > > > want > > > > >> > all the results in all languages, you end up with a big and > > > expensive > > > > >> > query. > > > > >> > > > > >> Why is this an issue? > > > > >> Dismax does it for you for free (thanks to the "form" parameter > that > > > > >> gives weight to each of the fields). > > > > >> This is an issue only if you start to have more than 100 languages > > or > > > > >> so... > > > > >> Lucene, the underlying engine of solr, handles thousands of > clauses > > > in a > > > > >> query without an issue (this is how prefix-queries are handled... > > they > > > > are > > > > >> expanded to a query for any of the term that matches the prefix, a > > > > setting > > > > >> deep somewhere, which is about 2000 avoids this to explode). > > > > >> > > > > > > > > > > Sure, Solr is great when you want to do simple queries like "XWiki > > Open > > > > > Source", however, since in XWiki we also expose the Solr/Lucene > query > > > > APIs > > > > > to the platform, there will be (as as it is currently with Lucene) > a > > > lot > > > > of > > > > > extensions wanting to do search using this API. These extensions > > (like > > > > the > > > > > search suggest for example, rest search, etc) want to do something > > like > > > > > "title:'Open Source' AND type:document AND doccontent:XWiki". > Because > > > > > option 2) is so messy in it's fields, it would mean that the > > extension > > > > > would have to come up with a query like "title_en:'Open Source' AND > > > > > type:document AND doccontent_en:XWiki" (assuming that it is only > > > limited > > > > to > > > > > the current -- english or whatever -- language; what happens if it > > > wants > > > > to > > > > > do that no matter what language? It will have to specify each > > > combination > > > > > possible because we can't use generic field names). > > > > > > > > > > Solr's approach works for using it in your web application's search > > > > input, > > > > > in a specific usecase, where you have precisely specified the > default > > > > > search fields and their boosts inside your schema.xml. However, as > a > > > > search > > > > > API, using option 2) you are making the life of anyone else wanting > > to > > > > use > > > > > the Solr search API really hard. Also, your search application will > > > work > > > > > nicely when the user enters a simple query in the input field, but > an > > > > > advanced user will suffer the same fate when trying to write an > > > advanced > > > > > query, thus not relying on the default query (computed by solr > based > > on > > > > > schema.xml). > > > > > > > > > > Also, based on your note above regarding improvements like title_ws > > and > > > > > such, again, all of these are very well suited for the search > > > application > > > > > use case, together with the default query that you configure in > > > > schema.xml, > > > > > making the search results perform really well. However, what does > all > > > > these > > > > > fields mean to another extension wanting to do search? Will it have > > to > > > > > handle all these implementation details to query for title, content > > and > > > > > such? I`m not sure how well this would work in practice. > > > > > > > > > > Unrealistic idea(?): perhaps we should come up with an abstract > > search > > > > > language (Solr/Lucene clone) that parses the searched fields > andhides > > > the > > > > > complexities of all the indexed fields, allowing to write simple > > > queries > > > > > like "title:XWiki", while this gets translated to "title_en:XWiki > OR > > > > > title_fr:XWiki OR title_de:XWiki..." :) > > > > > > > > > > Am I approaching this wrong by trying to have both a > > tweakable/tweaked > > > > > search application AND a search API? Are the two not compatible? Do > > we > > > > have > > > > > to sacrifice search result performance (no language-specific stuff) > > to > > > be > > > > > able to have a usable API? > > > > > > > > > > > > > > >> > If you want just some language, you have to read the right > fields > > > > >> > (ex title_en) instead of just getting a clear field name > (title). > > > > >> > > > > >> You have to be careful, this is really only if you want to be > > > specific. > > > > >> In this case, it is likely that you also do not want so much > > stemming. > > > > >> My experience, which was before dismax on curriki.org, has made > it > > so > > > > >> that any query that is a bit specific is likely to not desire > > > stemming. > > > > >> > > > > > > > > > > Can you please elaborate on this? I`m not sure I understand the > > > problem. > > > > > > > > > > > > > > >> > > > > >> > -- Also, the schema.xml definition is a static one in this > > concern, > > > > >> > requiring you to know beforehand which languages you want to > > support > > > > >> (for > > > > >> > example when defining the default fields to search for). Adding > a > > > new > > > > >> > language requires you to start editing the xml files by hand. > > > > >> > > > > >> True but the available languages are almost all hand-coded. > > > > >> You could generate the schema.xml based on the available languages > > if > > > > not > > > > >> hand-generated? > > > > >> > > > > > > > > > > Basically I would have to output a zip with schema.xml, > > solrconfig.xml > > > > and > > > > > then all the resources specific to all the selected languages > > > (stopwords, > > > > > synonims, etc) for the languages that we can provide out of the > box. > > > For > > > > > other languages, the admin would have to get dirty with the xmls. > > > > > > > > > > > > > > >> > > > > >> There's one catch with this approach which is new to me but seems > to > > > be > > > > >> quite important to implement this approach: the idf should be > > > modified, > > > > the > > > > >> Similarity class should be, so that the total number of documents > is > > > the > > > > >> total number of documents having that language. > > > > >> See: > > > > >> > > > > >> > > > > > > > > > > http://mail-archives.apache.org/mod_mbox/lucene-solr-user/201211.mbox/%[email protected]%3E > > > > >> The solution sketched there sounds easy but I have not tried it. > > > > >> > > > > >> > 3) Indexing the content in different Solr cores (indexes), one > for > > > > each > > > > >> > language. Each core requires it's on directory and configuration > > > > files. > > > > >> > - The advantage is that queries are clean to write (like option > 1) > > > and > > > > >> that > > > > >> > you have a nice separation > > > > >> > - The disadvantage is that it's difficult to get it right > > > > >> (administrative > > > > >> > issues) and then you also have the (considerable) problem of > > having > > > to > > > > >> fix > > > > >> > the relevancy score of a query result that has entries from > > > different > > > > >> > cores; each core has it's own relevancy computed and does not > > > consider > > > > >> the > > > > >> > others. > > > > >> > - To make it even worst, it seems that you can not [5] also push > > to > > > a > > > > >> > remote Solr instance the configuration files when creating a new > > > core > > > > >> > programatically. However, if we are running an embedded Solr > > > instance, > > > > >> we > > > > >> > could provide a way to generate the config files and write them > to > > > the > > > > >> data > > > > >> > directory. > > > > >> > > > > >> Post-processing results is very very very dangerous as performance > > is > > > at > > > > >> risk (e.g. if a core does not answer)... I would tend to avoid > that > > as > > > > much > > > > >> as possible. > > > > >> > > > > > > > > > > Not really related, but this reminds me about the post processing > > that > > > I > > > > > do for checking view rights over the returned result, but that's > > > another > > > > > discussion that we will probably need to have :) > > > > > > > > > > > > > > >> > > > > >> > Currently I have implemented option 1) in our existing Solr > > > > integration, > > > > >> > which is also more or less compatible with our existing Lucene > > > > queries, > > > > >> but > > > > >> > I would like to find a better solution that actually analyses > the > > > > >> content. > > > > >> > > > > > >> > During GSoC, option 2) was preferred but the implementation did > > not > > > > >> > consider practical reasons like the ones described above (query > > > > >> complexity, > > > > >> > user configuration, etc.) > > > > >> > > > > >> True, Savitha surfed the possibility of having different solr > > > documents > > > > >> per language. > > > > >> I still could not be sure that this was not showing the document > > match > > > > >> single in one language. > > > > >> > > > > >> However, indicating which language it is matched into is probably > > > > >> useful... > > > > >> > > > > > > > > > > Already doing that. > > > > > > > > > > > > > > >> Funnily, cross-language-retrieval is a mature research field but > > > > >> retrieval for multilanguage user is not so! > > > > >> > > > > >> > On a related note, I have also watched an interesting > presentation > > > [3] > > > > >> > about how Drupal handles its Solr integration and, > particularly, a > > > > >> plugin > > > > >> > [4] that handles the multilingual aspect. > > > > >> > The idea seen there is that you have this UI that helps you > > generate > > > > >> > configuration files, depending you your needs. For instance, you > > > > (admin) > > > > >> > check that you need search for language English, French and > German > > > and > > > > >> the > > > > >> > ui/extension gives you a zip with the configuration you need to > > use > > > in > > > > >> your > > > > >> > (remote or embedded) solr instance. The configuration for each > > > > language > > > > >> > comes preset with the analyzers you should use for it and the > > > > additional > > > > >> > resources (stopwords.txt, synonims.txt, etc.). > > > > >> > This approach helps with avoiding the need for admins to be > forced > > > to > > > > >> edit > > > > >> > xml files and could also still be useful for other cases, not > only > > > > >> option > > > > >> > 2). > > > > >> > > > > >> Generating sounds like an easy approach to me. > > > > >> > > > > > > > > > > Yes, however I don`t like the fact that we can not do everything > from > > > the > > > > > webapp and the admin needs to access the filesystem to install the > > > given > > > > > configuration on the embedded/remote solr directory. Lucene does > not > > > have > > > > > this problem now. It just works with XWiki and everything is done > > from > > > > > XWiki UI. I feel that losing this commodity will not be very well > > > > received > > > > > by users that now have some new install steps to get XWiki running. > > > > > > > > > > Well, of course, for the embedded solr version, we could handle it > > like > > > > we > > > > > do now and push the files directly from the webapp to the > filesystem. > > > > Since > > > > > embedded will be default, it should be OK and avoid the extra > install > > > > step. > > > > > Users with a remote solr machine should have the option to get the > > zip > > > > > instead. > > > > > > > > > > Not sure if we can apply the new configuration without a restart, > but > > > > I`ll > > > > > have to look more into it. I know the multi-core architecture > > supports > > > > > something like this but will have to see the details. > > > > > > > > > > > > > > >> > > > > >> > All these problems basically come from the fact that there is no > > way > > > > to > > > > >> > specify in the schema.xml that, based on the value of a field > > (like > > > > the > > > > >> > field "lang" that stores the document language), you want to run > > > this > > > > or > > > > >> > that group of analyzers. > > > > >> > > > > >> Well, this is possible with ThreadLocal but is not necessarily a > > good > > > > >> idea. > > > > >> Also, it is very common that users formulate queries without > > > formulating > > > > >> their language and thus you need to "or" the user's queries > through > > > > >> multiple languages (e.g. given by the browser). > > > > >> > > > > >> > Perhaps a solution would be a custom kind of > "AggregatorAnalyzer" > > > that > > > > >> > would call other analyzers at runtime, based on the value of the > > > lang > > > > >> > field. However, this solution could only be applied at index > time, > > > > when > > > > >> you > > > > >> > have the lang information (in the solrDocument to be indexed), > but > > > > when > > > > >> you > > > > >> > perform the query, you can not analyze the query text since you > do > > > not > > > > >> know > > > > >> > the language of the field you're querying (it was determined at > > > > runtime > > > > >> - > > > > >> > at index time) and thus do not know what operations to apply to > > the > > > > >> query > > > > >> > (to reduce it to the same form as the indexed values). > > > > >> > > > > >> How would that look at query time? > > > > >> > > > > > > > > > > That's what I was saying, that at query time, the searched term > will > > > not > > > > > get analyzed by the right chain. When you search for a single > > language, > > > > you > > > > > could add that language as a query filter and then you could apply > > the > > > > > right chain, but when searching in 2 or more (or no, meaning all) > > > > languages > > > > > you are stuck. > > > > > > > > > >> > > > > >> > I have also read another interesting analysis [6] on this > problem > > > that > > > > >> > elaborates on the complexities and limitations of each options. > > > > (Ignore > > > > >> the > > > > >> > Rosette stuff mentioned there) > > > > >> > > > > > >> > I have been thinking about this for some time now, but the > > solution > > > is > > > > >> > probably somewhere in between, finding an option that is > > acceptable > > > > >> while > > > > >> > not restrictive. I will probably also send a mail on the Solr > list > > > to > > > > >> get > > > > >> > some more input from there, but I get the feeling that whatever > > > > >> solution we > > > > >> > choose, it will most likely require the users to at least copy > (or > > > > even > > > > >> > edit) some files into some directories (configurations and/or > > jars), > > > > >> since > > > > >> > it does not seem to be easy/possible to do everything > on-the-fly, > > > > >> > programatically. > > > > >> > > > > >> The only hard step is when changing the supported languages, I > > think. > > > > >> In this case, when automatically generating the index, you need to > > > warn > > > > >> the user. > > > > >> The admin UI should have a checkbox "use generated schema" or a > > > textarea > > > > >> for the schema. > > > > >> > > > > > > > > > > Please see above regarding configuration generation. Basically, > since > > > we > > > > > are going to support both embedded and remote solr instances, we > > could > > > > > support things like editing the schema from XWiki only for the > > embedded > > > > > instance, but not for the remote one. We might end up having > separate > > > UIs > > > > > for each case, since we might want to exploit the flexibility of > the > > > > > embedded one as much as possible. > > > > > > > > > > > > > > >> > > > > >> Those that want particular fields and tunings need to write their > > own > > > > >> schema. > > > > >> > > > > >> The same UI could also include whether to include a phonetic track > > or > > > > not > > > > >> (then require reindexing). > > > > > > > > > > > > > > >> hope it helps. > > > > >> > > > > > > > > > > Yes, very helpful so far. I`m counting on your expertise with > > > Lucene/Solr > > > > > on the details. My current approach is a practical one without > > previous > > > > > experience on the topic, so I`m still doing mostly guesswork in > some > > > > areas. > > > > > > > > > > Thanks, > > > > > Eduard > > > > > > > > > > > > > > >> paul > > > > >> _______________________________________________ > > > > >> devs mailing list > > > > >> [email protected] > > > > >> http://lists.xwiki.org/mailman/listinfo/devs > > > > >> > > > > > > > > > > > > > > _______________________________________________ > > > > devs mailing list > > > > [email protected] > > > > http://lists.xwiki.org/mailman/listinfo/devs > > > > > > > > > > > > > > > > -- > > > Ludovic Dubost > > > Founder and CEO > > > Blog: http://blog.ludovic.org/ > > > XWiki: http://www.xwiki.com > > > Skype: ldubost GTalk: ldubost > > > _______________________________________________ > > > devs mailing list > > > [email protected] > > > http://lists.xwiki.org/mailman/listinfo/devs > > > > > _______________________________________________ > > devs mailing list > > [email protected] > > http://lists.xwiki.org/mailman/listinfo/devs > > > > > > -- > Ludovic Dubost > Founder and CEO > Blog: http://blog.ludovic.org/ > XWiki: http://www.xwiki.com > Skype: ldubost GTalk: ldubost > _______________________________________________ > devs mailing list > [email protected] > http://lists.xwiki.org/mailman/listinfo/devs > _______________________________________________ devs mailing list [email protected] http://lists.xwiki.org/mailman/listinfo/devs
