RE: PreAnalyzed FieldType, and simultaneously importing JSON

[Markus Jelsma]

Hi David!

Many thanks, this looks much better!

Regards,
Markus
 
-Original message-
> From:David Smiley <david.w.smi...@gmail.com>
> Sent: Monday 2nd April 2018 21:27
> To: solr-user@lucene.apache.org
> Subject: Re: PreAnalyzed FieldType, and simultaneously importing JSON
> 
> Hello Markus,
> 
> It appears you are not familiar with PreAnalyzedUpdateProcessor?  Using
> that is much more flexible -- you could have different URP chains for your
> use-cases. IMO PreAnalyzedField ought to go away.  I argued for the URP
> version and thus it's superiority to the FieldType here:
> https://issues.apache.org/jira/browse/SOLR-4619?focusedCommentId=13611191=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611191
> Sadly, the FieldType is the one that is documented in the ref guide, but
> not the URP :-(
> 
> ~ David
> 
> On Thu, Mar 29, 2018 at 5:06 PM Markus Jelsma <markus.jel...@openindex.io>
> wrote:
> 
> > Hello,
> >
> > We want to move to PreAnalyzed FieldType to offload our very heavy
> > analysis chain away from the search cluster, so we have to configure our
> > fields to accept pre-analyzed tokens in production.
> >
> > But we use the same schema in development environments too, and that is
> > where we use JSON files, or stream (export/import) data directly from
> > production servers into a development environment, again via JSON. And in
> > case of disaster recovery, we can import the daily exported JSON bzipped
> > files back into our production servers.
> >
> > But this JSON loading does not work with PreAnalyzed FieldType. So to load
> > JSON we must reset all fields back to their respective language specific
> > FieldTypes on-the-fly, we could automate, but it is a hassle we like to
> > avoid.
> >
> > Have i overlooked any configuration parameters that can help? Must we
> > automate the on-the-fly schema reconfiguration and reset to PreAnalyzed
> > after JSON loading is finished?
> >
> > Many thanks!
> > Markus
> >
> -- 
> Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
> LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
> http://www.solrenterprisesearchserver.com
> 

Re: PreAnalyzed FieldType, and simultaneously importing JSON

[David Smiley]

Hello Markus,

It appears you are not familiar with PreAnalyzedUpdateProcessor?  Using
that is much more flexible -- you could have different URP chains for your
use-cases. IMO PreAnalyzedField ought to go away.  I argued for the URP
version and thus it's superiority to the FieldType here:
https://issues.apache.org/jira/browse/SOLR-4619?focusedCommentId=13611191=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13611191
Sadly, the FieldType is the one that is documented in the ref guide, but
not the URP :-(

~ David

On Thu, Mar 29, 2018 at 5:06 PM Markus Jelsma 
wrote:

> Hello,
>
> We want to move to PreAnalyzed FieldType to offload our very heavy
> analysis chain away from the search cluster, so we have to configure our
> fields to accept pre-analyzed tokens in production.
>
> But we use the same schema in development environments too, and that is
> where we use JSON files, or stream (export/import) data directly from
> production servers into a development environment, again via JSON. And in
> case of disaster recovery, we can import the daily exported JSON bzipped
> files back into our production servers.
>
> But this JSON loading does not work with PreAnalyzed FieldType. So to load
> JSON we must reset all fields back to their respective language specific
> FieldTypes on-the-fly, we could automate, but it is a hassle we like to
> avoid.
>
> Have i overlooked any configuration parameters that can help? Must we
> automate the on-the-fly schema reconfiguration and reset to PreAnalyzed
> after JSON loading is finished?
>
> Many thanks!
> Markus
>
-- 
Lucene/Solr Search Committer, Consultant, Developer, Author, Speaker
LinkedIn: http://linkedin.com/in/davidwsmiley | Book:
http://www.solrenterprisesearchserver.com

PreAnalyzed FieldType, and simultaneously importing JSON

[Markus Jelsma]

Hello,

We want to move to PreAnalyzed FieldType to offload our very heavy analysis 
chain away from the search cluster, so we have to configure our fields to 
accept pre-analyzed tokens in production.

But we use the same schema in development environments too, and that is where 
we use JSON files, or stream (export/import) data directly from production 
servers into a development environment, again via JSON. And in case of disaster 
recovery, we can import the daily exported JSON bzipped files back into our 
production servers.

But this JSON loading does not work with PreAnalyzed FieldType. So to load JSON 
we must reset all fields back to their respective language specific FieldTypes 
on-the-fly, we could automate, but it is a hassle we like to avoid.

Have i overlooked any configuration parameters that can help? Must we automate 
the on-the-fly schema reconfiguration and reset to PreAnalyzed after JSON 
loading is finished?

Many thanks!
Markus