[jira] [Comment Edited] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445295#comment-16445295
]
Steve Rowe edited comment on SOLR-4793 at 4/20/18 4:53 AM:
---
I've attached a patch that adds {{jute.maxbuffer}} documentation to the ref
guide page on setting up an external ZooKeeper cluster that [~ctargett]
improved over on SOLR-12163.
I've included links to the new section:
# from the large model section on the LTR page, as an alternative to
SOLR-11250's {{DefaultWrapperModel}}; inspired by the patch on SOLR-11049 - cc
[~cpoerschke];
# from the OpenNLP NER mention on the URP page (LUCENE-2899); and
# from the OpenNLP NER URP javadocs.
Feedback is welcome.
was (Author: steve_rowe):
I've attached a patch that adds {{jute.maxbuffer}} documentation to the ref
guide page on setting up an external ZooKeeper cluster that [~ctargett]
improved over on SOLR-12163.
I've included a links to the new section:
# from the large model section on the LTR page, as an alternative to
SOLR-11250's {{DefaultWrapperModel}}; inspired by the patch on SOLR-11049 - cc
[~cpoerschke];
# from the OpenNLP NER mention on the URP page (LUCENE-2899); and
# from the OpenNLP NER URP javadocs.
Feedback is welcome.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>Priority: Major
> Attachments: SOLR-4793.patch
>
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445295#comment-16445295
]
Steve Rowe edited comment on SOLR-4793 at 4/20/18 4:50 AM:
---
I've attached a patch that adds {{jute.maxbuffer}} documentation to the ref
guide page on setting up an external ZooKeeper cluster that [~ctargett]
improved over on SOLR-12163.
I've included a links to the new section:
# from the large model section on the LTR page, as an alternative to
SOLR-11250's {{DefaultWrapperModel}}; inspired by the patch on SOLR-11049 - cc
[~cpoerschke];
# from the OpenNLP NER mention on the URP page (LUCENE-2899); and
# from the OpenNLP NER URP javadocs.
Feedback is welcome.
was (Author: steve_rowe):
I've attached a patch that adds {{jute.maxbuffer}} documentation to the ref
guide page on setting up an external ZooKeeper cluster that [~ctargett]
improved over on SOLR-12163.
I've included a links to the new section:
# from the large model section on the LTR page, as an alternative to
SOLR-11250's {{DefaultWrapperModel}}; inspired by the patch on SOLR-11049 - cc
[~cpoerschke];
# from the OpenNLP NER mention on the URP page; and
# from the OpenNLP NER URP javadocs.
Feedback is welcome.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>Priority: Major
> Attachments: SOLR-4793.patch
>
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Comment Edited] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651066#comment-13651066
]
Yago Riveiro edited comment on SOLR-4793 at 5/7/13 5:07 PM:
A workaround for this is set the param -Djute.maxbuffer with a value greater
than 1M.
>From zookeeper
>[doc|http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_configuration]
{quote}
jute.maxbuffer:
(Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper
prefix on it. It specifies the maximum size of the data that can be stored in a
znode. The default is 0xf, or just under 1M. If this option is changed, the
system property must be set on all servers and clients otherwise problems will
arise. This is really a sanity check. ZooKeeper is designed to store data on
the order of kilobytes in size.
{quote}
Notice that to use this configuration param, is necessary set the param in
solrcloud and zookeeper init script.
was (Author: yriveiro):
A workaround for this is set the param -Djute.maxbuffer with a value
greater than 1M.
>From zookeeper
>[doc|http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_configuration]
{quote}
jute.maxbuffer:
(Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper
prefix on it. It specifies the maximum size of the data that can be stored in a
znode. The default is 0xf, or just under 1M. If this option is changed, the
system property must be set on all servers and clients otherwise problems will
arise. This is really a sanity check. ZooKeeper is designed to store data on
the order of kilobytes in size.
{quote}
Notice that to use this configuration param, is necessary set this param in
solrcloud and zookeeper init script.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
