Re: Problem with uploading Large synonym files in cloud mode

[Jörn Franke]

You can use the configset API:
https://lucene.apache.org/solr/guide/7_7/configsets-api.html

I don’t recommend to use Schema.xml , but managed Schemas:

https://lucene.apache.org/solr/guide/6_6/schema-api.html

For people new to Solr I generally recommend to read a recent book about Solr 
from beginning to end - that will bring you up to speed much faster than trying 
to find all the information via Internet and will prepare you to deliver 
results much faster in better quality.
Then it is also much easier to understand and use the reference guide

> Am 02.08.2019 um 08:30 schrieb Salmaan Rashid Syed 
> :
> 
> Hi Bernd,
> 
> Yet, another noob question.
> 
> Consider that my conf directory for creating a collection is _default. Suppose
> now I made changes to managed-schema and conf.xml, How do I upload it to
> external zookeeper at 2181 port?
> 
> Can you please give me the command that uploads altered config.xml and
> managed-schema to zookeeper?
> 
> Thanks.
> 
> 
> On Fri, Aug 2, 2019 at 11:53 AM Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de> wrote:
> 
>> 
>> to 1) yes, because -Djute.maxbuffer is going to JAVA as a start parameter.
>> 
>> to 2) I don't know because i never use internal zookeeper
>> 
>> to 3) the configs are located at solr/server/solr/configsets/
>>   - choose one configset, make your changes and upload it to zookeeper
>>   - when creating a new collection choose your uploaded config
>>   - whenever you change something at your config you have to upload
>> it to zookeeper
>> 
>> I don't know which Solr version you are using, but a good starting point
>> with solr cloud is
>> http://lucene.apache.org/solr/guide/6_6/solrcloud.html
>> 
>> Regards
>> Bernd
>> 
>> 
>> 
>>> Am 02.08.19 um 07:59 schrieb Salmaan Rashid Syed:
>>> Hi Bernd,
>>> 
>>> Sorry for noob questions.
>>> 
>>> 1) What do you mean by restart? Do you mean that I shoud issue ./bin/solr
>>> stop -all?
>>> 
>>> And then issue these commands,
>>> 
>>> bin/solr restart -cloud -s example/cloud/node1/solr -p 8983
>>> 
>>> bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr
>>> 
>>> 
>>> 2) Where can I find solr internal Zookeeper folder for issuing this
>> command
>>> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"?
>>> 
>>> 
>>> 3) Where can I find schema.xml and config.xmo files for Solr Cloud Cores
>> to
>>> make changes in schema and configuration? Or do I have to make chages in
>>> the directory that contains managed-schema and config.xml files with
>> which
>>> I initialized and created collections? And then the solr will pick them
>> up
>>> from there when it restarts?
>>> 
>>> 
>>> Regards,
>>> 
>>> Salmaan
>>> 
>>> 
>>> 
>>> On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling <
>> bernd.fehl...@uni-bielefeld.de>
>>> wrote:
>>> 
 
 
> Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:
> After I make the -Djute.maxbuffer changes to Solr, deployed in
 production,
> Do I need to restart the solr to be able to add synonyms >1MB?
 
 Yes, you have to restart Solr.
 
 
> 
> Or, Was this supposed to be done before putting Solr to production
>> ever?
> Can we make chages when the Solr is running in production?
 
 It depends on your system. In my cloud with 5 shards and 3 replicas I
>> can
 take one by one offline, stop, modify and start again without problems.
 
 
> 
> Thanks.
> 
> Regards,
> Salmaan
> 
> 
> 
> On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de> wrote:
> 
>> You have to increase the -Djute.maxbuffer for large configs.
>> 
>> In Solr bin/solr/solr.in.sh use e.g.
>> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
>> This will increase maxbuffer for zookeeper on solr side to 10MB.
>> 
>> In Zookeeper zookeeper/conf/zookeeper-env.sh
>> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
>> 
>> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works
>> perfect.
>> 
>> Regards
>> 
>> 
>>> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
>>> Hi Solr Users,
>>> 
>>> I have a very big synonym file (>5MB). I am unable to start Solr in
 cloud
>>> mode as it throws an error message stating that the synonmys file is
>>> too large. I figured out that the zookeeper doesn't take a file
>> greater
>>> than 1MB size.
>>> 
>>> I tried to break down my synonyms file to smaller chunks less than
>> 1MB
>>> each. But, I am not sure about how to include all the filenames into
 the
>>> Solr schema.
>>> 
>>> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
>>> __2_synonyms.txt, __3synonyms.txt"
>>> 
>>> Or is there a better way of doing that? Will the bigger file when
 broken
>>> down to smaller chunks will be uploaded to zookeeper as well.
>>> 
>>> Please help or please guide 

Re: Problem with uploading Large synonym files in cloud mode

[Bernd Fehling]

http://lucene.apache.org/solr/guide/6_6/command-line-utilities.html
"Upload a configuration directory"

Take my advise and read the SolrCloud section of Solr Ref Guide.
It will answer most of your questions and is a good start.



Am 02.08.19 um 08:30 schrieb Salmaan Rashid Syed:

Hi Bernd,

Yet, another noob question.

Consider that my conf directory for creating a collection is _default. Suppose
now I made changes to managed-schema and conf.xml, How do I upload it to
external zookeeper at 2181 port?

Can you please give me the command that uploads altered config.xml and
managed-schema to zookeeper?

Thanks.


On Fri, Aug 2, 2019 at 11:53 AM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:



to 1) yes, because -Djute.maxbuffer is going to JAVA as a start parameter.

to 2) I don't know because i never use internal zookeeper

to 3) the configs are located at solr/server/solr/configsets/
- choose one configset, make your changes and upload it to zookeeper
- when creating a new collection choose your uploaded config
- whenever you change something at your config you have to upload
it to zookeeper

I don't know which Solr version you are using, but a good starting point
with solr cloud is
http://lucene.apache.org/solr/guide/6_6/solrcloud.html

Regards
Bernd



Am 02.08.19 um 07:59 schrieb Salmaan Rashid Syed:

Hi Bernd,

Sorry for noob questions.

1) What do you mean by restart? Do you mean that I shoud issue ./bin/solr
stop -all?

And then issue these commands,

bin/solr restart -cloud -s example/cloud/node1/solr -p 8983

bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr


2) Where can I find solr internal Zookeeper folder for issuing this

command

SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"?


3) Where can I find schema.xml and config.xmo files for Solr Cloud Cores

to

make changes in schema and configuration? Or do I have to make chages in
the directory that contains managed-schema and config.xml files with

which

I initialized and created collections? And then the solr will pick them

up

from there when it restarts?


Regards,

Salmaan



On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling <

bernd.fehl...@uni-bielefeld.de>

wrote:




Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:

After I make the -Djute.maxbuffer changes to Solr, deployed in

production,

Do I need to restart the solr to be able to add synonyms >1MB?


Yes, you have to restart Solr.




Or, Was this supposed to be done before putting Solr to production

ever?

Can we make chages when the Solr is running in production?


It depends on your system. In my cloud with 5 shards and 3 replicas I

can

take one by one offline, stop, modify and start again without problems.




Thanks.

Regards,
Salmaan



On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:


You have to increase the -Djute.maxbuffer for large configs.

In Solr bin/solr/solr.in.sh use e.g.
SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
This will increase maxbuffer for zookeeper on solr side to 10MB.

In Zookeeper zookeeper/conf/zookeeper-env.sh
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"

I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works

perfect.


Regards


Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:

Hi Solr Users,

I have a very big synonym file (>5MB). I am unable to start Solr in

cloud

mode as it throws an error message stating that the synonmys file is
too large. I figured out that the zookeeper doesn't take a file

greater

than 1MB size.

I tried to break down my synonyms file to smaller chunks less than

1MB

each. But, I am not sure about how to include all the filenames into

the

Solr schema.

Should it be seperated by commas like synonyms = "__1_synonyms.txt,
__2_synonyms.txt, __3synonyms.txt"

Or is there a better way of doing that? Will the bigger file when

broken

down to smaller chunks will be uploaded to zookeeper as well.

Please help or please guide me to relevant documentation regarding

this.


Thank you.

Regards.
Salmaan.

Re: Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

Hi Bernd,

Yet, another noob question.

Consider that my conf directory for creating a collection is _default. Suppose
now I made changes to managed-schema and conf.xml, How do I upload it to
external zookeeper at 2181 port?

Can you please give me the command that uploads altered config.xml and
managed-schema to zookeeper?

Thanks.


On Fri, Aug 2, 2019 at 11:53 AM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

>
> to 1) yes, because -Djute.maxbuffer is going to JAVA as a start parameter.
>
> to 2) I don't know because i never use internal zookeeper
>
> to 3) the configs are located at solr/server/solr/configsets/
>- choose one configset, make your changes and upload it to zookeeper
>- when creating a new collection choose your uploaded config
>- whenever you change something at your config you have to upload
> it to zookeeper
>
> I don't know which Solr version you are using, but a good starting point
> with solr cloud is
> http://lucene.apache.org/solr/guide/6_6/solrcloud.html
>
> Regards
> Bernd
>
>
>
> Am 02.08.19 um 07:59 schrieb Salmaan Rashid Syed:
> > Hi Bernd,
> >
> > Sorry for noob questions.
> >
> > 1) What do you mean by restart? Do you mean that I shoud issue ./bin/solr
> > stop -all?
> >
> > And then issue these commands,
> >
> > bin/solr restart -cloud -s example/cloud/node1/solr -p 8983
> >
> > bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr
> >
> >
> > 2) Where can I find solr internal Zookeeper folder for issuing this
> command
> > SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"?
> >
> >
> > 3) Where can I find schema.xml and config.xmo files for Solr Cloud Cores
> to
> > make changes in schema and configuration? Or do I have to make chages in
> > the directory that contains managed-schema and config.xml files with
> which
> > I initialized and created collections? And then the solr will pick them
> up
> > from there when it restarts?
> >
> >
> > Regards,
> >
> > Salmaan
> >
> >
> >
> > On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de>
> > wrote:
> >
> >>
> >>
> >> Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:
> >>> After I make the -Djute.maxbuffer changes to Solr, deployed in
> >> production,
> >>> Do I need to restart the solr to be able to add synonyms >1MB?
> >>
> >> Yes, you have to restart Solr.
> >>
> >>
> >>>
> >>> Or, Was this supposed to be done before putting Solr to production
> ever?
> >>> Can we make chages when the Solr is running in production?
> >>
> >> It depends on your system. In my cloud with 5 shards and 3 replicas I
> can
> >> take one by one offline, stop, modify and start again without problems.
> >>
> >>
> >>>
> >>> Thanks.
> >>>
> >>> Regards,
> >>> Salmaan
> >>>
> >>>
> >>>
> >>> On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> >>> bernd.fehl...@uni-bielefeld.de> wrote:
> >>>
>  You have to increase the -Djute.maxbuffer for large configs.
> 
>  In Solr bin/solr/solr.in.sh use e.g.
>  SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
>  This will increase maxbuffer for zookeeper on solr side to 10MB.
> 
>  In Zookeeper zookeeper/conf/zookeeper-env.sh
>  SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
> 
>  I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works
> perfect.
> 
>  Regards
> 
> 
>  Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
> > Hi Solr Users,
> >
> > I have a very big synonym file (>5MB). I am unable to start Solr in
> >> cloud
> > mode as it throws an error message stating that the synonmys file is
> > too large. I figured out that the zookeeper doesn't take a file
> greater
> > than 1MB size.
> >
> > I tried to break down my synonyms file to smaller chunks less than
> 1MB
> > each. But, I am not sure about how to include all the filenames into
> >> the
> > Solr schema.
> >
> > Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> > __2_synonyms.txt, __3synonyms.txt"
> >
> > Or is there a better way of doing that? Will the bigger file when
> >> broken
> > down to smaller chunks will be uploaded to zookeeper as well.
> >
> > Please help or please guide me to relevant documentation regarding
> >> this.
> >
> > Thank you.
> >
> > Regards.
> > Salmaan.
> >
> 
> >>>
> >>
> >
>

Re: Problem with uploading Large synonym files in cloud mode

[Bernd Fehling]

to 1) yes, because -Djute.maxbuffer is going to JAVA as a start parameter.

to 2) I don't know because i never use internal zookeeper

to 3) the configs are located at solr/server/solr/configsets/
  - choose one configset, make your changes and upload it to zookeeper
  - when creating a new collection choose your uploaded config
  - whenever you change something at your config you have to upload it to 
zookeeper

I don't know which Solr version you are using, but a good starting point with 
solr cloud is
http://lucene.apache.org/solr/guide/6_6/solrcloud.html

Regards
Bernd



Am 02.08.19 um 07:59 schrieb Salmaan Rashid Syed:

Hi Bernd,

Sorry for noob questions.

1) What do you mean by restart? Do you mean that I shoud issue ./bin/solr
stop -all?

And then issue these commands,

bin/solr restart -cloud -s example/cloud/node1/solr -p 8983

bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr


2) Where can I find solr internal Zookeeper folder for issuing this command
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"?


3) Where can I find schema.xml and config.xmo files for Solr Cloud Cores to
make changes in schema and configuration? Or do I have to make chages in
the directory that contains managed-schema and config.xml files with which
I initialized and created collections? And then the solr will pick them up
from there when it restarts?


Regards,

Salmaan



On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling 
wrote:




Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:

After I make the -Djute.maxbuffer changes to Solr, deployed in

production,

Do I need to restart the solr to be able to add synonyms >1MB?


Yes, you have to restart Solr.




Or, Was this supposed to be done before putting Solr to production ever?
Can we make chages when the Solr is running in production?


It depends on your system. In my cloud with 5 shards and 3 replicas I can
take one by one offline, stop, modify and start again without problems.




Thanks.

Regards,
Salmaan



On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:


You have to increase the -Djute.maxbuffer for large configs.

In Solr bin/solr/solr.in.sh use e.g.
SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
This will increase maxbuffer for zookeeper on solr side to 10MB.

In Zookeeper zookeeper/conf/zookeeper-env.sh
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"

I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.

Regards


Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:

Hi Solr Users,

I have a very big synonym file (>5MB). I am unable to start Solr in

cloud

mode as it throws an error message stating that the synonmys file is
too large. I figured out that the zookeeper doesn't take a file greater
than 1MB size.

I tried to break down my synonyms file to smaller chunks less than 1MB
each. But, I am not sure about how to include all the filenames into

the

Solr schema.

Should it be seperated by commas like synonyms = "__1_synonyms.txt,
__2_synonyms.txt, __3synonyms.txt"

Or is there a better way of doing that? Will the bigger file when

broken

down to smaller chunks will be uploaded to zookeeper as well.

Please help or please guide me to relevant documentation regarding

this.


Thank you.

Regards.
Salmaan.

Re: Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

Hi Bernd,

Sorry for noob questions.

1) What do you mean by restart? Do you mean that I shoud issue ./bin/solr
stop -all?

And then issue these commands,

bin/solr restart -cloud -s example/cloud/node1/solr -p 8983

bin/solr restart -c -p 7574 -z localhost:9983 -s example/cloud/node2/solr


2) Where can I find solr internal Zookeeper folder for issuing this command
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"?


3) Where can I find schema.xml and config.xmo files for Solr Cloud Cores to
make changes in schema and configuration? Or do I have to make chages in
the directory that contains managed-schema and config.xml files with which
I initialized and created collections? And then the solr will pick them up
from there when it restarts?


Regards,

Salmaan



On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling 
wrote:

>
>
> Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:
> > After I make the -Djute.maxbuffer changes to Solr, deployed in
> production,
> > Do I need to restart the solr to be able to add synonyms >1MB?
>
> Yes, you have to restart Solr.
>
>
> >
> > Or, Was this supposed to be done before putting Solr to production ever?
> > Can we make chages when the Solr is running in production?
>
> It depends on your system. In my cloud with 5 shards and 3 replicas I can
> take one by one offline, stop, modify and start again without problems.
>
>
> >
> > Thanks.
> >
> > Regards,
> > Salmaan
> >
> >
> >
> > On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> > bernd.fehl...@uni-bielefeld.de> wrote:
> >
> >> You have to increase the -Djute.maxbuffer for large configs.
> >>
> >> In Solr bin/solr/solr.in.sh use e.g.
> >> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
> >> This will increase maxbuffer for zookeeper on solr side to 10MB.
> >>
> >> In Zookeeper zookeeper/conf/zookeeper-env.sh
> >> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
> >>
> >> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
> >>
> >> Regards
> >>
> >>
> >> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
> >>> Hi Solr Users,
> >>>
> >>> I have a very big synonym file (>5MB). I am unable to start Solr in
> cloud
> >>> mode as it throws an error message stating that the synonmys file is
> >>> too large. I figured out that the zookeeper doesn't take a file greater
> >>> than 1MB size.
> >>>
> >>> I tried to break down my synonyms file to smaller chunks less than 1MB
> >>> each. But, I am not sure about how to include all the filenames into
> the
> >>> Solr schema.
> >>>
> >>> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> >>> __2_synonyms.txt, __3synonyms.txt"
> >>>
> >>> Or is there a better way of doing that? Will the bigger file when
> broken
> >>> down to smaller chunks will be uploaded to zookeeper as well.
> >>>
> >>> Please help or please guide me to relevant documentation regarding
> this.
> >>>
> >>> Thank you.
> >>>
> >>> Regards.
> >>> Salmaan.
> >>>
> >>
> >
>

Re: Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

Thank you, I will definitely try it.

Regards,
Salmaan



On Thu, Aug 1, 2019 at 5:40 PM Bernd Fehling 
wrote:

>
>
> Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:
> > After I make the -Djute.maxbuffer changes to Solr, deployed in
> production,
> > Do I need to restart the solr to be able to add synonyms >1MB?
>
> Yes, you have to restart Solr.
>
>
> >
> > Or, Was this supposed to be done before putting Solr to production ever?
> > Can we make chages when the Solr is running in production?
>
> It depends on your system. In my cloud with 5 shards and 3 replicas I can
> take one by one offline, stop, modify and start again without problems.
>
>
> >
> > Thanks.
> >
> > Regards,
> > Salmaan
> >
> >
> >
> > On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> > bernd.fehl...@uni-bielefeld.de> wrote:
> >
> >> You have to increase the -Djute.maxbuffer for large configs.
> >>
> >> In Solr bin/solr/solr.in.sh use e.g.
> >> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
> >> This will increase maxbuffer for zookeeper on solr side to 10MB.
> >>
> >> In Zookeeper zookeeper/conf/zookeeper-env.sh
> >> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
> >>
> >> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
> >>
> >> Regards
> >>
> >>
> >> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
> >>> Hi Solr Users,
> >>>
> >>> I have a very big synonym file (>5MB). I am unable to start Solr in
> cloud
> >>> mode as it throws an error message stating that the synonmys file is
> >>> too large. I figured out that the zookeeper doesn't take a file greater
> >>> than 1MB size.
> >>>
> >>> I tried to break down my synonyms file to smaller chunks less than 1MB
> >>> each. But, I am not sure about how to include all the filenames into
> the
> >>> Solr schema.
> >>>
> >>> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> >>> __2_synonyms.txt, __3synonyms.txt"
> >>>
> >>> Or is there a better way of doing that? Will the bigger file when
> broken
> >>> down to smaller chunks will be uploaded to zookeeper as well.
> >>>
> >>> Please help or please guide me to relevant documentation regarding
> this.
> >>>
> >>> Thank you.
> >>>
> >>> Regards.
> >>> Salmaan.
> >>>
> >>
> >
>

Re: Problem with uploading Large synonym files in cloud mode

[Bernd Fehling]

Am 01.08.19 um 13:57 schrieb Salmaan Rashid Syed:

After I make the -Djute.maxbuffer changes to Solr, deployed in production,
Do I need to restart the solr to be able to add synonyms >1MB?


Yes, you have to restart Solr.




Or, Was this supposed to be done before putting Solr to production ever?
Can we make chages when the Solr is running in production?


It depends on your system. In my cloud with 5 shards and 3 replicas I can
take one by one offline, stop, modify and start again without problems.




Thanks.

Regards,
Salmaan



On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:


You have to increase the -Djute.maxbuffer for large configs.

In Solr bin/solr/solr.in.sh use e.g.
SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
This will increase maxbuffer for zookeeper on solr side to 10MB.

In Zookeeper zookeeper/conf/zookeeper-env.sh
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"

I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.

Regards


Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:

Hi Solr Users,

I have a very big synonym file (>5MB). I am unable to start Solr in cloud
mode as it throws an error message stating that the synonmys file is
too large. I figured out that the zookeeper doesn't take a file greater
than 1MB size.

I tried to break down my synonyms file to smaller chunks less than 1MB
each. But, I am not sure about how to include all the filenames into the
Solr schema.

Should it be seperated by commas like synonyms = "__1_synonyms.txt,
__2_synonyms.txt, __3synonyms.txt"

Or is there a better way of doing that? Will the bigger file when broken
down to smaller chunks will be uploaded to zookeeper as well.

Please help or please guide me to relevant documentation regarding this.

Thank you.

Regards.
Salmaan.

Re: Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

After I make the -Djute.maxbuffer changes to Solr, deployed in production,
Do I need to restart the solr to be able to add synonyms >1MB?

Or, Was this supposed to be done before putting Solr to production ever?
Can we make chages when the Solr is running in production?

Thanks.

Regards,
Salmaan



On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

> You have to increase the -Djute.maxbuffer for large configs.
>
> In Solr bin/solr/solr.in.sh use e.g.
> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
> This will increase maxbuffer for zookeeper on solr side to 10MB.
>
> In Zookeeper zookeeper/conf/zookeeper-env.sh
> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
>
> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
>
> Regards
>
>
> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
> > Hi Solr Users,
> >
> > I have a very big synonym file (>5MB). I am unable to start Solr in cloud
> > mode as it throws an error message stating that the synonmys file is
> > too large. I figured out that the zookeeper doesn't take a file greater
> > than 1MB size.
> >
> > I tried to break down my synonyms file to smaller chunks less than 1MB
> > each. But, I am not sure about how to include all the filenames into the
> > Solr schema.
> >
> > Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> > __2_synonyms.txt, __3synonyms.txt"
> >
> > Or is there a better way of doing that? Will the bigger file when broken
> > down to smaller chunks will be uploaded to zookeeper as well.
> >
> > Please help or please guide me to relevant documentation regarding this.
> >
> > Thank you.
> >
> > Regards.
> > Salmaan.
> >
>

Re: Problem with uploading Large synonym files in cloud mode

[Jörn Franke]

The idea of using an external program could be good. 

> Am 31.07.2019 um 08:06 schrieb Salmaan Rashid Syed 
> :
> 
> Hi all,
> 
> Thanks for your invaluable and helpful answers.
> 
> I currently don't have an external zookeeper loaded. I am working as per
> the documentation for solr cloud without external zookeeper. I will later
> add the external zookeeper once the changes works as expected.
> 
> *1) Will I still need to make changes to zookeeper-env.sh? Or the changes
> to solr.in.sh  will suffice?*
> 
> I have an additional query that is slightly off topic but related to
> synonyms.
> My synonyms file will be updated with new words with time. What is the
> procedure to update the synonyms file without shutting down the solr in
> production?
> 
> What I am thinking is to replace all the similar words in a documents using
> an external program before I index them to Solr. This way I don't have to
> worry about the synonyms file size and updation.
> 
> *2) Do you think this is better way forward?*
> 
> Thanks for all you help.
> 
> Regards,
> Salmaan
> 
> 
> 
> 
> On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de> wrote:
> 
>> You have to increase the -Djute.maxbuffer for large configs.
>> 
>> In Solr bin/solr/solr.in.sh use e.g.
>> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
>> This will increase maxbuffer for zookeeper on solr side to 10MB.
>> 
>> In Zookeeper zookeeper/conf/zookeeper-env.sh
>> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
>> 
>> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
>> 
>> Regards
>> 
>> 
>>> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
>>> Hi Solr Users,
>>> 
>>> I have a very big synonym file (>5MB). I am unable to start Solr in cloud
>>> mode as it throws an error message stating that the synonmys file is
>>> too large. I figured out that the zookeeper doesn't take a file greater
>>> than 1MB size.
>>> 
>>> I tried to break down my synonyms file to smaller chunks less than 1MB
>>> each. But, I am not sure about how to include all the filenames into the
>>> Solr schema.
>>> 
>>> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
>>> __2_synonyms.txt, __3synonyms.txt"
>>> 
>>> Or is there a better way of doing that? Will the bigger file when broken
>>> down to smaller chunks will be uploaded to zookeeper as well.
>>> 
>>> Please help or please guide me to relevant documentation regarding this.
>>> 
>>> Thank you.
>>> 
>>> Regards.
>>> Salmaan.
>>> 
>> 

Re: Problem with uploading Large synonym files in cloud mode

[Jörn Franke]

Ad 1) it needs to be configured in Zookeeper server and Solr and all other ZK 
clients

Ad 2) you never need to shut it down in production for updating Synonym files.
Use the config set API to reupload the full configuration included updated 
synonyms:
https://lucene.apache.org/solr/guide/7_4/configsets-api.html
Then: reload the collection and optionally reindex (if you use synonyms at the 
index level)
Alternatively used managed synonyms 

https://lucene.apache.org/solr/guide/6_6/managed-resources.html

And optionally reindex

> Am 31.07.2019 um 08:06 schrieb Salmaan Rashid Syed 
> :
> 
> Hi all,
> 
> Thanks for your invaluable and helpful answers.
> 
> I currently don't have an external zookeeper loaded. I am working as per
> the documentation for solr cloud without external zookeeper. I will later
> add the external zookeeper once the changes works as expected.
> 
> *1) Will I still need to make changes to zookeeper-env.sh? Or the changes
> to solr.in.sh  will suffice?*
> 
> I have an additional query that is slightly off topic but related to
> synonyms.
> My synonyms file will be updated with new words with time. What is the
> procedure to update the synonyms file without shutting down the solr in
> production?
> 
> What I am thinking is to replace all the similar words in a documents using
> an external program before I index them to Solr. This way I don't have to
> worry about the synonyms file size and updation.
> 
> *2) Do you think this is better way forward?*
> 
> Thanks for all you help.
> 
> Regards,
> Salmaan
> 
> 
> 
> 
> On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
> bernd.fehl...@uni-bielefeld.de> wrote:
> 
>> You have to increase the -Djute.maxbuffer for large configs.
>> 
>> In Solr bin/solr/solr.in.sh use e.g.
>> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
>> This will increase maxbuffer for zookeeper on solr side to 10MB.
>> 
>> In Zookeeper zookeeper/conf/zookeeper-env.sh
>> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
>> 
>> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
>> 
>> Regards
>> 
>> 
>>> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
>>> Hi Solr Users,
>>> 
>>> I have a very big synonym file (>5MB). I am unable to start Solr in cloud
>>> mode as it throws an error message stating that the synonmys file is
>>> too large. I figured out that the zookeeper doesn't take a file greater
>>> than 1MB size.
>>> 
>>> I tried to break down my synonyms file to smaller chunks less than 1MB
>>> each. But, I am not sure about how to include all the filenames into the
>>> Solr schema.
>>> 
>>> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
>>> __2_synonyms.txt, __3synonyms.txt"
>>> 
>>> Or is there a better way of doing that? Will the bigger file when broken
>>> down to smaller chunks will be uploaded to zookeeper as well.
>>> 
>>> Please help or please guide me to relevant documentation regarding this.
>>> 
>>> Thank you.
>>> 
>>> Regards.
>>> Salmaan.
>>> 
>> 

Re: Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

Hi all,

Thanks for your invaluable and helpful answers.

I currently don't have an external zookeeper loaded. I am working as per
the documentation for solr cloud without external zookeeper. I will later
add the external zookeeper once the changes works as expected.

*1) Will I still need to make changes to zookeeper-env.sh? Or the changes
to solr.in.sh  will suffice?*

I have an additional query that is slightly off topic but related to
synonyms.
My synonyms file will be updated with new words with time. What is the
procedure to update the synonyms file without shutting down the solr in
production?

What I am thinking is to replace all the similar words in a documents using
an external program before I index them to Solr. This way I don't have to
worry about the synonyms file size and updation.

*2) Do you think this is better way forward?*

Thanks for all you help.

Regards,
Salmaan




On Tue, Jul 30, 2019 at 4:53 PM Bernd Fehling <
bernd.fehl...@uni-bielefeld.de> wrote:

> You have to increase the -Djute.maxbuffer for large configs.
>
> In Solr bin/solr/solr.in.sh use e.g.
> SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
> This will increase maxbuffer for zookeeper on solr side to 10MB.
>
> In Zookeeper zookeeper/conf/zookeeper-env.sh
> SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"
>
> I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.
>
> Regards
>
>
> Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:
> > Hi Solr Users,
> >
> > I have a very big synonym file (>5MB). I am unable to start Solr in cloud
> > mode as it throws an error message stating that the synonmys file is
> > too large. I figured out that the zookeeper doesn't take a file greater
> > than 1MB size.
> >
> > I tried to break down my synonyms file to smaller chunks less than 1MB
> > each. But, I am not sure about how to include all the filenames into the
> > Solr schema.
> >
> > Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> > __2_synonyms.txt, __3synonyms.txt"
> >
> > Or is there a better way of doing that? Will the bigger file when broken
> > down to smaller chunks will be uploaded to zookeeper as well.
> >
> > Please help or please guide me to relevant documentation regarding this.
> >
> > Thank you.
> >
> > Regards.
> > Salmaan.
> >
>

Re: Problem with uploading Large synonym files in cloud mode

[Bernd Fehling]

You have to increase the -Djute.maxbuffer for large configs.

In Solr bin/solr/solr.in.sh use e.g.
SOLR_OPTS="$SOLR_OPTS -Djute.maxbuffer=1000"
This will increase maxbuffer for zookeeper on solr side to 10MB.

In Zookeeper zookeeper/conf/zookeeper-env.sh
SERVER_JVMFLAGS="$SERVER_JVMFLAGS -Djute.maxbuffer=1000"

I have a >10MB Thesaurus and use 30MB for jute.maxbuffer, works perfect.

Regards


Am 30.07.19 um 13:09 schrieb Salmaan Rashid Syed:

Hi Solr Users,

I have a very big synonym file (>5MB). I am unable to start Solr in cloud
mode as it throws an error message stating that the synonmys file is
too large. I figured out that the zookeeper doesn't take a file greater
than 1MB size.

I tried to break down my synonyms file to smaller chunks less than 1MB
each. But, I am not sure about how to include all the filenames into the
Solr schema.

Should it be seperated by commas like synonyms = "__1_synonyms.txt,
__2_synonyms.txt, __3synonyms.txt"

Or is there a better way of doing that? Will the bigger file when broken
down to smaller chunks will be uploaded to zookeeper as well.

Please help or please guide me to relevant documentation regarding this.

Thank you.

Regards.
Salmaan.

Re: Problem with uploading Large synonym files in cloud mode

[Jörn Franke]

Aside that a 5 MB synonym file is rather strange (what is the use case for such 
a large synonym file?) and that it will have impact on index size and/or query 
time:

You can configure zookeeper server and the Solr client to allow larger files 
using the jute.maxbuffer option.

> Am 30.07.2019 um 13:09 schrieb Salmaan Rashid Syed 
> :
> 
> Hi Solr Users,
> 
> I have a very big synonym file (>5MB). I am unable to start Solr in cloud
> mode as it throws an error message stating that the synonmys file is
> too large. I figured out that the zookeeper doesn't take a file greater
> than 1MB size.
> 
> I tried to break down my synonyms file to smaller chunks less than 1MB
> each. But, I am not sure about how to include all the filenames into the
> Solr schema.
> 
> Should it be seperated by commas like synonyms = "__1_synonyms.txt,
> __2_synonyms.txt, __3synonyms.txt"
> 
> Or is there a better way of doing that? Will the bigger file when broken
> down to smaller chunks will be uploaded to zookeeper as well.
> 
> Please help or please guide me to relevant documentation regarding this.
> 
> Thank you.
> 
> Regards.
> Salmaan.

Problem with uploading Large synonym files in cloud mode

[Salmaan Rashid Syed]

Hi Solr Users,

I have a very big synonym file (>5MB). I am unable to start Solr in cloud
mode as it throws an error message stating that the synonmys file is
too large. I figured out that the zookeeper doesn't take a file greater
than 1MB size.

I tried to break down my synonyms file to smaller chunks less than 1MB
each. But, I am not sure about how to include all the filenames into the
Solr schema.

Should it be seperated by commas like synonyms = "__1_synonyms.txt,
__2_synonyms.txt, __3synonyms.txt"

Or is there a better way of doing that? Will the bigger file when broken
down to smaller chunks will be uploaded to zookeeper as well.

Please help or please guide me to relevant documentation regarding this.

Thank you.

Regards.
Salmaan.