Re: Update schema.xml without restarting Solr?

[Alexandre Rafalovitch]

I would say that all this points at existence of middle-ware in front
of Solr. Therefore, the next action would be to identify the
middle-ware and ask this question on _their_ mailing list.

Regards,
   Alex.

http://www.solr-start.com/ - Resources for Solr users, new and experienced


On 21 April 2017 at 08:38, Shawn Heisey  wrote:
>  On 4/21/2017 2:20 AM, Lingeshm wrote:
>> I can’t change the schema name of an existing index.
>>
>> I want to change the schema name from “schemaV1" to “schemaV2 for one of the
>> existing index
>>
>> curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type:
>> application/json" -d '{"schema":"schemaV2"}'
>>
>> the funny part is curl is not returning anything such either error or
>> success. I expect the next GET to return {"schema":" schemaV2"} but it still
>> returns {"schema":" schemaV1"}
>>
>> I am using solr 4.10.4 version.
>
> I have never seen a URL path like "/search/index/my_idx-H" for Solr.
> Typically any URL path for Solr will start with "/solr".  With a 4.x
> version, you might have changed the context path in the container, in
> newer versions the admin UI will break if that is changed.  If you are
> attempting to use the Schema API, then your URL path would contain
> "/schema" somewhere, which is missing.
>
> Can you point to a location in the Solr documentation that describes
> what you are trying to do?
>
> I have never seen anything like the JSON that you are sending.  The
> Schema API documentation for version 6.6 (not yet released) doesn't
> mention any ability to change the name.  Such a change would cause no
> difference in how the schema works even if it were possible -- it's
> cosmetic, and something somebody who uses Solr will never see.
>
> Thanks,
> Shawn
>

Re: Update schema.xml without restarting Solr?

[Shawn Heisey]

 On 4/21/2017 2:20 AM, Lingeshm wrote:
> I can’t change the schema name of an existing index.
>
> I want to change the schema name from “schemaV1" to “schemaV2 for one of the
> existing index
>
> curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type:
> application/json" -d '{"schema":"schemaV2"}'
>
> the funny part is curl is not returning anything such either error or
> success. I expect the next GET to return {"schema":" schemaV2"} but it still
> returns {"schema":" schemaV1"}
>
> I am using solr 4.10.4 version.

I have never seen a URL path like "/search/index/my_idx-H" for Solr. 
Typically any URL path for Solr will start with "/solr".  With a 4.x
version, you might have changed the context path in the container, in
newer versions the admin UI will break if that is changed.  If you are
attempting to use the Schema API, then your URL path would contain
"/schema" somewhere, which is missing.

Can you point to a location in the Solr documentation that describes
what you are trying to do?

I have never seen anything like the JSON that you are sending.  The
Schema API documentation for version 6.6 (not yet released) doesn't
mention any ability to change the name.  Such a change would cause no
difference in how the schema works even if it were possible -- it's
cosmetic, and something somebody who uses Solr will never see.

Thanks,
Shawn

Re: Update schema.xml without restarting Solr?

[Mikhail Khludnev]

> the funny part is curl is not returning anything such either error or
success

can you add -v or so to curl, to see http status code at least?

On Fri, Apr 21, 2017 at 11:20 AM, Lingeshm  wrote:

> Hello Team ,
>
> I can’t change the schema name of an existing index.
>
> I want to change the schema name from “schemaV1" to “schemaV2 for one of
> the
> existing index
>
> curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type:
> application/json" -d '{"schema":"schemaV2"}'
>
> the funny part is curl is not returning anything such either error or
> success. I expect the next GET to return {"schema":" schemaV2"} but it
> still
> returns {"schema":" schemaV1"}
>
> I am using solr 4.10.4 version.
>
> Have you faced any such issue ??
>
> Regards
> Lingesh M
>
>
>
> --
> View this message in context: http://lucene.472066.n3.
> nabble.com/Update-schema-xml-without-restarting-Solr-tp484263p4331225.html
> Sent from the Solr - User mailing list archive at Nabble.com.
>



-- 
Sincerely yours
Mikhail Khludnev

Re: Update schema.xml without restarting Solr?

[Lingeshm]

Hello Team ,

I can’t change the schema name of an existing index.

I want to change the schema name from “schemaV1" to “schemaV2 for one of the
existing index

curl -XPUT http://localhost:8098/search/index/my_idx-H "Content-Type:
application/json" -d '{"schema":"schemaV2"}'

the funny part is curl is not returning anything such either error or
success. I expect the next GET to return {"schema":" schemaV2"} but it still
returns {"schema":" schemaV1"}

I am using solr 4.10.4 version.

Have you faced any such issue ??

Regards
Lingesh M



--
View this message in context: 
http://lucene.472066.n3.nabble.com/Update-schema-xml-without-restarting-Solr-tp484263p4331225.html
Sent from the Solr - User mailing list archive at Nabble.com.

Re: Update schema.xml without restarting Solr?

[solr]

Quoting Ryan McKinley [EMAIL PROTECTED]:


In general, you need to be very careful when you change the schema
without reindexing.  Many changes will break all search, some may be
just fine.

for example, if you change sint to slong anything already indexed as an
sint will be incompatible with the current settings.


This example, with changing from sint to slong, was just an example of  
bad  design in my

opinion, not an example of what we need to do without reindexing.

Actually, as long as a reindexing doesn't take too long time I don't  
have any problem with reindexing per say. But I would like it to  
happen without having the search functionality disabled in the mean  
time. I guess the MultiCore stuff is the option here.


Thanks for your input, Ryan.

/Jimi

Re: Update schema.xml without restarting Solr?

[Jeryl Cook]

Top often requested feature:
1. Make the option on using the RAMDirectory to hook in Terracotta(
billion(s) of items in an index anyone?..it would be possible using
this.)
2. Make the schema.xml configurable at runtime, not really sure the
best way to address this, because changing the schema would require
re-indexing the documents.


Terracotta:
http://www.terracotta.org/

On Tue, Mar 25, 2008 at 11:27 AM,  [EMAIL PROTECTED] wrote:
 Hi,

  The wiki for Solr talks about the schema.xml, and it seems that
  changes in this file requires a restart of Solr before they have effect.

  In the wiki it says:

  
  How can I rebuild my index from scratch if I change my schema?

  The most efficient/complete way is to...

 1. Stop your application server
 2. Change your schema.xml file
 3. Delete the index directory in your data directory
 4. Start your application server (Solr will detect that there is
  no existing index and make a new one)
 5. Re-Index your data

  If the permission scheme of your server does not allow you to manually
  delete the index directory an alternate technique is...

 1. Stop your application server
 2. Change your schema.xml file
 3. Start your application server
 4. Use the match all docs query in a delete by query command:
  deletequery*:*/query/delete
 5. Send an optimize/ command.
 6. Re-Index your data
  

  Is this really the case? I find that quite strange that you need to
  restart solr for a change in the schema.xml. The way we plan to use
  Solr together with a Content Management System is that the
  authors/editors can create new article/document types when needed,
  without any need to restart anything. The CMS itself has full support
  for this. But we need Solr to also support this. Is that possible?
  Like a simple realoadSchemaXml/ command, maybe, that would trigger
  Solr to re-read it's schema.xml file.

  If this is not possible to do, is it really necessary to restart the
  entire application server for a change in schema.xml to have effect?
  Or only the solr webapp?

  Regards
  /Jimi




-- 
Jeryl Cook
/^\ Pharaoh /^\
http://pharaohofkush.blogspot.com/
..Act your age, and not your shoe size.. -Prince(1986)

Re: Update schema.xml without restarting Solr?

[solr]

Quoting Jeryl Cook [EMAIL PROTECTED]:


2. Make the schema.xml configurable at runtime, not really sure the
best way to address this, because changing the schema would require
re-indexing the documents.


Isn't the best way to address this just to leave it to the persons  
that integrate solr into their system? I mean, if a change in the  
schema only effects 1% of all documents, then it's a bad idea to  
reindex them all (at least if the dataset is big).


/Jimi

Re: Update schema.xml without restarting Solr?

[solr]

Quoting Daniel Papasian [EMAIL PROTECTED]:


[EMAIL PROTECTED] wrote:

Quoting Jeryl Cook [EMAIL PROTECTED]:


2. Make the schema.xml configurable at runtime, not really sure the
best way to address this, because changing the schema would require
re-indexing the documents.


Isn't the best way to address this just to leave it to the persons that
integrate solr into their system? I mean, if a change in the schema only
effects 1% of all documents, then it's a bad idea to reindex them all
(at least if the dataset is big).


Or if you're adding a new field to the schema (perhaps the most common
need for editing schema.xml), you don't need to reindex any documents at
all, right?  Unless I'm missing something?


Well, it all depends on if that field (not solar/lucene field)  
exists on the already indexed material, but that particular field was  
never indexed. Lets say that we have a bunch of articles, that has a  
field author that someone decided  that it doesn't need to be in the  
index. But then later he changes his mind, and add the author field to  
the schema. In this case all articles that has a populated author  
field should now be reindexed.



I suppose if you add a new dynamic field specification that conflicts
with existing fields, reindexing is probably a good idea, but if you're
doing that... well, I probably don't want to know.


I must say that I'm abit confused by these dynamic fields. Can someone  
tell me if there is any reasonable use of dynamic fields without  
having the variable type (for example i for int/sint) in the name?


/Jimi

Re: Update schema.xml without restarting Solr?

[Ryan McKinley]

Jeryl Cook wrote:

Top often requested feature:
1. Make the option on using the RAMDirectory to hook in Terracotta(
billion(s) of items in an index anyone?..it would be possible using
this.)


This is noted in: https://issues.apache.org/jira/browse/SOLR-465

Out of cueriosity, any sense of performance with a terracotta index?  It 
seems like it would have to be *substantially* slower.  Also, if it is a 
RAM directly, does it persist?


If your looking to support billions of docs, perhaps consider:
http://wiki.apache.org/solr/DistributedSearch

ryan

Re: Update schema.xml without restarting Solr?

[Daniel Papasian]

[EMAIL PROTECTED] wrote:

Quoting Daniel Papasian [EMAIL PROTECTED]:

Or if you're adding a new field to the schema (perhaps the most common
need for editing schema.xml), you don't need to reindex any documents at
all, right?  Unless I'm missing something?


Well, it all depends on if that field (not solar/lucene field) exists 
on the already indexed material, but that particular field was never 
indexed. Lets say that we have a bunch of articles, that has a field 
author that someone decided  that it doesn't need to be in the index. 
But then later he changes his mind, and add the author field to the 
schema. In this case all articles that has a populated author field 
should now be reindexed.


Yeah, I guess the use case I was thinking of was someone who had 
multiple different types of content in their index (say, articles, 
events, organizations) and when they added a new content type (book 
review) if they found the need to add a new field for that content type 
(say, publisher) that would only be relevant for that type -- as you're 
adding it before any data that would have it was indexed, I believe 
you'd be fine making that schema change without reindexing anything.



I suppose if you add a new dynamic field specification that conflicts
with existing fields, reindexing is probably a good idea, but if you're
doing that... well, I probably don't want to know.


I must say that I'm abit confused by these dynamic fields. Can someone 
tell me if there is any reasonable use of dynamic fields without having 
the variable type (for example i for int/sint) in the name?


Well, perhaps this is fulfilling your requirement on a technicality, but 
there's always higher order types...  Offhand, I can think of things 
where you might want to define a dynamic field like *_propername or 
*_cost and then you'd be able to use fields like author_propername or 
editor_propername, or book_cost or volume_cost or what have you.


Daniel

Re: Update schema.xml without restarting Solr?

[Otis Gospodnetic]

Hey Ryan, why do you say a Lucene/Solr index served via Terracotta would be 
substantially slower?
I often wanted to try Terracotta + Lucene, but... time.

Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

- Original Message 
From: Ryan McKinley [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Wednesday, March 26, 2008 10:52:45 AM
Subject: Re: Update schema.xml without restarting Solr?

Jeryl Cook wrote:
 Top often requested feature:
 1. Make the option on using the RAMDirectory to hook in Terracotta(
 billion(s) of items in an index anyone?..it would be possible using
 this.)

This is noted in: https://issues.apache.org/jira/browse/SOLR-465

Out of cueriosity, any sense of performance with a terracotta index?  It 
seems like it would have to be *substantially* slower.  Also, if it is a 
RAM directly, does it persist?

If your looking to support billions of docs, perhaps consider:
http://wiki.apache.org/solr/DistributedSearch

ryan

Re: Update schema.xml without restarting Solr?

[Ryan McKinley]

just intuition - haven't tried it, so i'd love to be proved wrong. 
Instrumenting Objects and magically passing them around seems like it 
would be slower then a tuned approach used in SOLR-303.


It looks like they have a lucene example:
http://www.terracotta.org/confluence/display/integrations/Lucene

Also, i don't understand how terracotta could get lucene past the 
Integer.MAX_VALUE limit because it does not change the API, it works 
within it.


ryan


Otis Gospodnetic wrote:

Hey Ryan, why do you say a Lucene/Solr index served via Terracotta would be 
substantially slower?
I often wanted to try Terracotta + Lucene, but... time.

Thanks,
Otis
--
Sematext -- http://sematext.com/ -- Lucene - Solr - Nutch

- Original Message 
From: Ryan McKinley [EMAIL PROTECTED]
To: solr-user@lucene.apache.org
Sent: Wednesday, March 26, 2008 10:52:45 AM
Subject: Re: Update schema.xml without restarting Solr?

Jeryl Cook wrote:

Top often requested feature:
1. Make the option on using the RAMDirectory to hook in Terracotta(
billion(s) of items in an index anyone?..it would be possible using
this.)


This is noted in: https://issues.apache.org/jira/browse/SOLR-465

Out of cueriosity, any sense of performance with a terracotta index?  It 
seems like it would have to be *substantially* slower.  Also, if it is a 
RAM directly, does it persist?


If your looking to support billions of docs, perhaps consider:
http://wiki.apache.org/solr/DistributedSearch

ryan

Re: Update schema.xml without restarting Solr?

[Yonik Seeley]

On Wed, Mar 26, 2008 at 4:41 PM, Ryan McKinley [EMAIL PROTECTED] wrote:
 just intuition - haven't tried it, so i'd love to be proved wrong.
  Instrumenting Objects and magically passing them around seems like it
  would be slower then a tuned approach used in SOLR-303.

Yep, that's my sense too.  No magic solutions when it comes to scalability.

-Yonik

Re: Update schema.xml without restarting Solr?

[Jeryl Cook]

i wouldn't call Terracotta approach magic(smile)..., it's being used
quite a bit in many scalable high performing projects...

i personally used Terracotta and Lucene, and it worked but did not try
to cluster it with multiple terracotta(workers) across nodes , and
the Terracotta(master)..just a single box with two tomcat instances...

However talk is cheap, if I have the time over the next few weeks
ill make a bench mark test based on the Terracotta and Lucene, with
maybe 3 nodes?and a 1 million documents..
maybe some others can do the same :)..

FYI: 
http://www.terracotta.org/confluence/display/tcforge/Proposal+-+Terracotta+for+Lucene

Jeryl Cook

On Wed, Mar 26, 2008 at 5:16 PM, Yonik Seeley [EMAIL PROTECTED] wrote:
 On Wed, Mar 26, 2008 at 4:41 PM, Ryan McKinley [EMAIL PROTECTED] wrote:
   just intuition - haven't tried it, so i'd love to be proved wrong.
Instrumenting Objects and magically passing them around seems like it
would be slower then a tuned approach used in SOLR-303.

  Yep, that's my sense too.  No magic solutions when it comes to scalability.

  -Yonik




-- 
Jeryl Cook
/^\ Pharaoh /^\
http://pharaohofkush.blogspot.com/
..Act your age, and not your shoe size.. -Prince(1986)

Re: Update schema.xml without restarting Solr?

[Chris Hostetter]

:  Top often requested feature:
:  1. Make the option on using the RAMDirectory to hook in Terracotta(
:  billion(s) of items in an index anyone?..it would be possible using
:  this.)
: 
: This is noted in: https://issues.apache.org/jira/browse/SOLR-465

...and if people posted comments in the issue saying they tried the 
patch and it worked well for them (or didn't work well and described 
their use case and what they think a better API would be) that would help 
raise the visibility of the visibility of hte issue and increase the 
likelyhod of geting it commited.

API changes, particularly new plugin hooks have to be made carefully 
because once they are released they have to be supported indefinitley.  so 
we need people to really help us test them out while they are still just 
patches.


-Hoss

Re: Update schema.xml without restarting Solr?

[Ryan McKinley]

The way we plan to use Solr 
together with a Content Management System is that the authors/editors 
can create new article/document types when needed, without any need to 
restart anything. 


Perhaps consider using dynamic fields if you need new fields:
http://wiki.apache.org/solr/SchemaXml#head-82dba16404c8e3318021320638b669b3a6d780d0

- - - -

Do you really need to change the schema?  Your CMS will transform from 
blog software to invoice tracking to forum without restarting?  With 
clever use of dynamic fields this should be possible too.  Consider 
changing the schema the same way you consider changing on for SQL -- you 
will likely change it and test for a while before deploying.


That said, the new MultiCore stuff in trunk allows you to reload a core 
at runtime:

http://wiki.apache.org/solr/MultiCore#head-429a06cb83e1ce7b06857fd03c38d1200c4bcfc1




If this is not possible to do, is it really necessary to restart the 
entire application server for a change in schema.xml to have effect? Or 
only the solr webapp?


correct, you only need to restart the solr webapp for all settings to 
get reloaded.


ryan

Re: Update schema.xml without restarting Solr?

[solr]

Quoting Ryan McKinley [EMAIL PROTECTED]:



The way we plan to use Solr together with a Content Management   
System is that the authors/editors can create new article/document   
types when needed, without any need to restart anything.


Do you really need to change the schema?  Your CMS will transform from
blog software to invoice tracking to forum without restarting?


Well, it will not transform in that way :)
But new article/document types gets created every once in a while. And  
the CMS in itself has no restart-requirements before they can be used.  
The only limitation we have is that new or updated presentation jsp  
pages needs to be transfered to the live server, and this is done by  
the build (and that involves a tomcat restart). But sometimes we have  
no need to add/update jsp files, but we still want to input content  
based on new or updated article types, and index them, without  
restarting. And we also want to be able to modify existing article  
types, so that the fields gets indexed in a different way maybe, and  
reindex all articles/documents while people still can do searches on  
the site.



With
clever use of dynamic fields this should be possible too.


Is that true also for changes made on how a specific field should be  
indexed? The wiki example is:

 dynamicField name=*_i  type=integer  indexed=true  stored=true/

What if we want to change this from stored=true to stored=false or  
something similar, without renaming the field?


But I must say, that in general I consider type information in the  
name to be ugly design. What if we have a number field myNumber that  
we first define as an int (or sint, to use solr semantics), and we  
call that myNumber_i in solr? If we later feel that we should change  
this to be a slong, then we need to change the name of the field to  
nyNumber_f.



Consider
changing the schema the same way you consider changing on for SQL --
you will likely change it and test for a while before deploying.

That said, the new MultiCore stuff in trunk allows you to reload a core
at runtime:
http://wiki.apache.org/solr/MultiCore#head-429a06cb83e1ce7b06857fd03c38d1200c4bcfc1


Ok, I'll look into that. Thanks.


correct, you only need to restart the solr webapp for all settings to
get reloaded.


Ok, thanks. The wiki was unclear on this one.

/Jimi

Re: Update schema.xml without restarting Solr?

[Ryan McKinley]

[EMAIL PROTECTED] wrote:

Quoting Ryan McKinley [EMAIL PROTECTED]:



The way we plan to use Solr together with a Content Management  
System is that the authors/editors can create new article/document  
types when needed, without any need to restart anything.


Do you really need to change the schema?  Your CMS will transform from
blog software to invoice tracking to forum without restarting?


Well, it will not transform in that way :)
But new article/document types gets created every once in a while. And 
the CMS in itself has no restart-requirements before they can be used. 
The only limitation we have is that new or updated presentation jsp 
pages needs to be transfered to the live server, and this is done by the 
build (and that involves a tomcat restart). But sometimes we have no 
need to add/update jsp files, but we still want to input content based 
on new or updated article types, and index them, without restarting. And 
we also want to be able to modify existing article types, so that the 
fields gets indexed in a different way maybe, and reindex all 
articles/documents while people still can do searches on the site.



With
clever use of dynamic fields this should be possible too.


Is that true also for changes made on how a specific field should be 
indexed? The wiki example is:

 dynamicField name=*_i  type=integer  indexed=true  stored=true/

What if we want to change this from stored=true to stored=false or 
something similar, without renaming the field?


But I must say, that in general I consider type information in the name 
to be ugly design. What if we have a number field myNumber that we first 
define as an int (or sint, to use solr semantics), and we call that 
myNumber_i in solr? If we later feel that we should change this to be a 
slong, then we need to change the name of the field to nyNumber_f.




In general, you need to be very careful when you change the schema 
without reindexing.  Many changes will break all search, some may be 
just fine.


for example, if you change sint to slong anything already indexed as an 
sint will be incompatible with the current settings.  Likewise, if you 
change the stored settings, it will only change thing for the incoming 
documents.


I'm not saying its impossible -- it is just something you need to be 
very careful with because a seemingly simple change can break a lot of 
things (until you reinidex)


That said -- with the MultiCore stuff and SOLR-139, it is possible to 
build a system that can be dynamically updated and managed.


ryan