[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506493#comment-16506493 ] ASF subversion and git services commented on SOLR-4793: --- Commit c35edc8fc394f4b88185fe24b83f748b7793dde9 in lucene-solr's branch refs/heads/branch_7x from [~ctargett] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=c35edc8 ] SOLR-4793: Ref Guide: shorten the section heading & fix refs > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Assignee: Steve Rowe >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506495#comment-16506495 ] ASF subversion and git services commented on SOLR-4793: --- Commit 4031935f3ba008a6beb8216b94e67548bfbf9ec2 in lucene-solr's branch refs/heads/branch_7_4 from [~ctargett] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4031935 ] SOLR-4793: Ref Guide: shorten the section heading & fix refs > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Assignee: Steve Rowe >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506491#comment-16506491 ] ASF subversion and git services commented on SOLR-4793: --- Commit 4c7b7c0063b2fd194d2037fd769c2b0e5fcf in lucene-solr's branch refs/heads/master from [~ctargett] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=4c7b7c0 ] SOLR-4793: Ref Guide: shorten the section heading & fix refs > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Assignee: Steve Rowe >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446404#comment-16446404 ] Jan Høydahl commented on SOLR-4793: --- Yes, better start simple and choose one location for config sets. Blob, HDFS, FileSystem... Looks like we have a cleanup job with the SolrResourceLoader hierarchy, there is a ton of zkSolrResourceLoader hardcoding, and we should probably generalise some of the zkRL features up into SolrResourceLoader, or a new SolrResourceLoaderBase. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Assignee: Steve Rowe >Priority: Major > Fix For: 7.4, master (8.0) > > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446294#comment-16446294 ] ASF subversion and git services commented on SOLR-4793: --- Commit 22c4b9c36f5dfdf0578bacea2e83740714512765 in lucene-solr's branch refs/heads/master from [~steve_rowe] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=22c4b9c ] SOLR-4793: Document usage of ZooKeeper's jute.maxbuffer sysprop for increasing the file size limit above 1MB > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446293#comment-16446293 ] ASF subversion and git services commented on SOLR-4793: --- Commit 9592221193971732b8d2c4b2c2994417bd7a3072 in lucene-solr's branch refs/heads/branch_7x from [~steve_rowe] [ https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=9592221 ] SOLR-4793: Document usage of ZooKeeper's jute.maxbuffer sysprop for increasing the file size limit above 1MB > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446257#comment-16446257 ] Christine Poerschke commented on SOLR-4793: --- bq. I think the long term solution could be to implement something like a BlobStoreResourceLoader, and a configset (as a whole or in parts) could be loaded from ZK or blob store. I like the idea of (where appropriate) getting parts of a configset from 'elsewhere' and that could be from a blob store or from somewhere else. ticket cross-reference: * SOLR-9887 is about stopwords and synonyms from a JDBC source - cc [~kaessmann] [~tboeghk] + [~dsmiley] re: the pulling from a streaming expression mention > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16446036#comment-16446036 ] Shawn Heisey commented on SOLR-4793: bq. Hmm, I don't think we should be starting with this kind of magic If there are still things to work out with the blob store (this is the .system collection, right?) then I agree, we should let those things bake for a while before we implement automagic redirection from ZK. I do like Jan's idea, though. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445754#comment-16445754
]
Steve Rowe commented on SOLR-4793:
--
bq. I think the long term solution could be to implement something like a
BlobStoreResourceLoader
Agreed - see SOLR-8751 and SOLR-9175.
bq. and a configset (as a whole or in parts) could be loaded from ZK or blob
store.
I'm not sure how useful it would be to store whole configsets in the blob
store. In any case, that won't be the first step here.
bq. Can we keep all but large files in zk. When zkClient is asked to upload a
large file it will upload it to blob instead and create a shadow file with same
name in Zk, but with a body telling it is a blob file with a pointer to blob
store ID. Then when zk resource loader gets a file it will detect such files
and serve them from blob transparently.
Hmm, I don't think we should be *starting* with this kind of magic - I'm much
more comfortable with separate blob store upload (already implemented) and
schema reference steps (SOLR-8751/SOLR-9175)
bq. This probably means that backup/restore feature also needs to backup the
blob store?
Yes, but this is already true right now: {{solrconfig.xml}} can load handler
and component classes from blobs in the blob store.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>Priority: Major
> Attachments: SOLR-4793.patch
>
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445424#comment-16445424 ] Jan Høydahl commented on SOLR-4793: --- This probably means that backup/restore feature also needs to backup the blob store? > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445422#comment-16445422 ] Jan Høydahl commented on SOLR-4793: --- Can we keep all but large files in zk. When zkClient is asked to upload a large file it will upload it to blob instead and create a shadow file with same name in Zk, but with a body telling it is a blob file with a pointer to blob store ID. Then when zk resource loader gets a file it will detect such files and serve them from blob transparently. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445350#comment-16445350 ] Ishan Chattopadhyaya commented on SOLR-4793: I think the long term solution could be to implement something like a BlobStoreResourceLoader, and a configset (as a whole or in parts) could be loaded from ZK or blob store. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen >Priority: Major > Attachments: SOLR-4793.patch > > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445311#comment-16445311
]
Steve Rowe commented on SOLR-4793:
--
FYI I considered adding info on configuring ZooKeeper clients other than those
invoked by {{bin/solr}}, e.g. ZK's {{zkCli.sh}} and Solr's cloud script
{{zkcli.sh}}, but neither of those are covered elsewhere in the ref guide, and
I *think* {{bin/solr zk}} commands cover most users needs, so I didn't end up
including info on configuring those clients. (ZK's {{zkCli.sh}} reads
{{zookeeper-env.sh}}, so a user employing that approach for configuring ZK
nodes will get that client configured for free.)
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>Priority: Major
> Attachments: SOLR-4793.patch
>
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16445295#comment-16445295
]
Steve Rowe commented on SOLR-4793:
--
I've attached a patch that adds {{jute.maxbuffer}} documentation to the ref
guide page on setting up an external ZooKeeper cluster that [~ctargett]
improved over on SOLR-12163.
I've included a links to the new section:
# from the large model section on the LTR page, as an alternative to
SOLR-11250's {{DefaultWrapperModel}}; inspired by the patch on SOLR-11049 - cc
[~cpoerschke];
# from the OpenNLP NER mention on the URP page; and
# from the OpenNLP NER URP javadocs.
Feedback is welcome.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>Priority: Major
> Attachments: SOLR-4793.patch
>
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039198#comment-14039198 ] Yago Riveiro commented on SOLR-4793: it's probably that I was tweaked the zkServer file a bit ... :P > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039111#comment-14039111 ] Elaine Cario commented on SOLR-4793: Not sure either, but I couldn't find where ZOO_ENV was actually referenced in any of the zookeeper scripts, so I changed the zookeeper-env.sh to this: JVMFLAGS="$JVMFLAGS -Djute.maxbuffer=5000" And then it all came together. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038970#comment-14038970 ] Yago Riveiro commented on SOLR-4793: Elaine now is easier to do the debug, you know where the "problem" is :). Note: I'm using the 3.4.5 version of zookeeper, I don't know if the zkServer.sh was changed > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038948#comment-14038948 ] Elaine Cario commented on SOLR-4793: Yago - we have partial success! I added the file as you did, but for some reason the parameter was still not being added to the zookeeper startup (checked PID using ps -ww -fp PID, and didn't see it). So, for the moment I modified the zkServer.sh to just add the param directly to the java startup, and that seemed to work - everything started up with the big files. Now I just need to figure out what's going wrong with the scripting, but that's just some investigation in my part. I'm sure I did something wrong there. Thanks so much for your help!! This has been bugging me for weeks. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038918#comment-14038918 ] Yago Riveiro commented on SOLR-4793: Indeed, after dive into the zkEnv file, I realised that if the zookeeper-env.sh exists, the zookeeper append the configurations to the init command. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038913#comment-14038913 ] Elaine Cario commented on SOLR-4793: Yago, I was going to ask how that gets loaded, but I just found an old post of yours with more info: let me try that (reposting it here): In my case I have: > > - Zookeeper: a file into conf folder named zookeeper-env.sh with: > > #!/usr/bin/env bash > > ZOO_ENV="-Djute.maxbuffer=10485761" > > The zookeeper-env.sh is loaded automatically by the zkEnv.sh if the file > exists. > > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038898#comment-14038898
]
Yago Riveiro commented on SOLR-4793:
About tocamt's configuration I have the same configuration.
In the case of Zookeeper I have all custom configurations into a file named
zookeeper-env.sh located into bin/conf folder with this content:
{code}
#!/usr/bin/env bash
ZOO_ENV="-Djute.maxbuffer= 5000"
{code}
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038889#comment-14038889
]
Elaine Cario commented on SOLR-4793:
Here is the SOLR_OPTS for Tomcat:
SOLR_OPTS="-Dhost.port=8080 -Dhost.context=solr -Dsolr.install.dir=${SOLR_DIST}
-Dsolr.solr.home=${SOLR_HOME} -Dsolr.base.data.dir=$SOLR_DATA
-Dbootstrap_conf=true -Dsolr.log=${SOLR_LOGDIR}
-Dcollection.configName=wkcontent -DnumShards=2 -DzkHost=10.208.152.230:2181
-Djute.maxbuffer=5000
-Dlog4j.configuration=file:///apps/solr/collections/log4j.properties"
Here is zoo.cfg
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/apps/zookeeper/zk_datadir
dataLogDir=/apps/zookeeper/zk_logs
clientPort=2181
server.1=zookeep01:2881:3881
jute.maxbuffer=5000
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message was sent by Atlassian JIRA
(v6.2#6252)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038874#comment-14038874 ] Yago Riveiro commented on SOLR-4793: Elaine can you paste the configuration for tomcat and zookeeper that you have for the jute.maxbuffer? > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038868#comment-14038868 ] Elaine Cario commented on SOLR-4793: Absolute paths didn't work either - it seems to turn it into a relaitve path, and still uses the ZKSolrResourceLoader to find it. I may have to either split the files, or dig into the FSTSynonymFilterFactory to figure out some other workaround. ERROR - 2014-06-20 10:45:13.522; org.apache.solr.core.CoreContainer; Unable to create core: dictindex_uk org.apache.solr.common.SolrException: Could not load core configuration for core dictindex_uk at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:66) at org.apache.solr.core.CoreContainer.create(CoreContainer.java:554) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:261) at org.apache.solr.core.CoreContainer$1.call(CoreContainer.java:253) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) Caused by: java.lang.RuntimeException: java.io.IOException: Error opening /configs/dictindex_uk//apps/solr/collections/en_lemmatization.txt at org.apache.solr.schema.IndexSchema.(IndexSchema.java:169) at org.apache.solr.schema.IndexSchemaFactory.create(IndexSchemaFactory.java:55) at org.apache.solr.schema.IndexSchemaFactory.buildIndexSchema(IndexSchemaFactory.java:69) at org.apache.solr.core.ConfigSetService.createIndexSchema(ConfigSetService.java:89) at org.apache.solr.core.ConfigSetService.getConfig(ConfigSetService.java:62) ... 9 more Caused by: java.io.IOException: Error opening /configs/dictindex_uk//apps/solr/collections/en_lemmatization.txt at org.apache.solr.cloud.ZkSolrResourceLoader.openResource(ZkSolrResourceLoader.java:83) at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.loadSynonyms(FSTSynonymFilterFactory.java:137) at org.apache.lucene.analysis.synonym.FSTSynonymFilterFactory.inform(FSTSynonymFilterFactory.java:112) at org.apache.lucene.analysis.synonym.SynonymFilterFactory.inform(SynonymFilterFactory.java:90) at org.apache.solr.core.SolrResourceLoader.inform(SolrResourceLoader.java:675) at org.apache.solr.schema.IndexSchema.(IndexSchema.java:167) ... 13 more > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038845#comment-14038845 ] Elaine Cario commented on SOLR-4793: Scratch that, we're already at zookeeper 3.4.6. Thanks Nicole - that was going to be my very next step. Updates aren't an issue, the files are under version control and our deployment scripts can take care of moving them wherever we need to make it work :-). Will report back... > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038842#comment-14038842 ] Nicole Lacoste commented on SOLR-4793: -- I also had trouble with relative paths, try full paths. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038835#comment-14038835 ] Elaine Cario commented on SOLR-4793: I got this exception in the logs, they always happen on the lemmatization files, which run anywhere from 2MB to 20MB in size: ERROR - 2014-06-20 09:29:59.082; org.apache.solr.core.ZkContainer; org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /configs/dictindex_uk/en_lemmatization.txt at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783) at org.apache.solr.common.cloud.SolrZkClient$10.execute(SolrZkClient.java:432) at org.apache.solr.common.cloud.ZkCmdExecutor.retryOperation(ZkCmdExecutor.java:73) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:429) at org.apache.solr.common.cloud.SolrZkClient.makePath(SolrZkClient.java:339) at org.apache.solr.cloud.ZkController.uploadToZK(ZkController.java:1318) at org.apache.solr.cloud.ZkController.uploadConfigDir(ZkController.java:1355) at org.apache.solr.cloud.ZkController.bootstrapConf(ZkController.java:1565) at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:188) at org.apache.solr.core.ZkContainer.initZooKeeper(ZkContainer.java:67) at org.apache.solr.core.CoreContainer.load(CoreContainer.java:216) at org.apache.solr.servlet.SolrDispatchFilter.createCoreContainer(SolrDispatchFilter.java:189) at org.apache.solr.servlet.SolrDispatchFilter.init(SolrDispatchFilter.java:136) at org.apache.catalina.core.ApplicationFilterConfig.initFilter(ApplicationFilterConfig.java:279) at org.apache.catalina.core.ApplicationFilterConfig.getFilter(ApplicationFilterConfig.java:260) at org.apache.catalina.core.ApplicationFilterConfig.(ApplicationFilterConfig.java:105) at org.apache.catalina.core.StandardContext.filterStart(StandardContext.java:4809) at org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5485) at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:150) at org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:901) at org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:877) at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:632) at org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1073) at org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:1857) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:744) ERROR - 2014-06-20 09:29:59.087; org.apache.solr.servlet.SolrDispatchFilter; Could not start Solr. Check solr/home property and the logs We were working around it by splitting the files up in < 1MB chunks, but that sometimes worked and sometimes didn't (we'd still get errors), and for the larger files it's, well, a pain to list > 20 files in the schema. We tried moving the files, but then it complains that relative paths aren't allowed in the specification. This is a fresh install of 4.8 (technically it was an upgrade, but we removed the index and re-fed all our content). But we didn't upgrade zookeeper, so I will try that and report back. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: de
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038834#comment-14038834 ] Nicole Lacoste commented on SOLR-4793: -- Elaine, We got stuck at the same thing and if I remember right we put the synonyms files in a matching folder on each of the machines and put the full path in the schema. It means if there is an update you have to take care of it yourself. I am not sure if the REST api for the synonyms work in this case you'd have to test that. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038810#comment-14038810 ] Yago Riveiro commented on SOLR-4793: I think that version 4.8 updates zookeeper version to 3.4.6 If the workaround doesn't work then is serious issue if you have a large number of collections and replicas because all metadata about the cluster is into clusterstate.json file. [~ecario], How you notice it that the workaround doesn't work? Have you any logs or something? and last question, do you upgrade Solr from 4.7 to 4.8 or is a fresh install? > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[ https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038797#comment-14038797 ] Elaine Cario commented on SOLR-4793: I'm finding that setting jute.maxbuffer both in Solr (-D option at tomcat startup) and zookeeper (zoo.cfg) doesn't seem to work (at least with solr 4.8). This is really becoming a blocker for us, as we are using index time synonym replacement as a "poor man's lemmatization" and these files get quite large. Would be nice to have some option to have these files managed outside of zookeeper. > Solr Cloud can't upload large config files ( > 1MB) to Zookeeper > - > > Key: SOLR-4793 > URL: https://issues.apache.org/jira/browse/SOLR-4793 > Project: Solr > Issue Type: Improvement >Reporter: Son Nguyen > > Zookeeper set znode size limit to 1MB by default. So we can't start Solr > Cloud with some large config files, like synonyms.txt. > Jan Høydahl has a good idea: > "SolrCloud is designed with an assumption that you should be able to upload > your whole disk-based conf folder into ZK, and that you should be able to add > an empty Solr node to a cluster and it would download all config from ZK. So > immediately a splitting strategy automatically handled by ZkSolresourceLoader > for large files could be one way forward, i.e. store synonyms.txt as e.g. > __001_synonyms.txt __002_synonyms.txt" -- This message was sent by Atlassian JIRA (v6.2#6252) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-4793) Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
[
https://issues.apache.org/jira/browse/SOLR-4793?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13651066#comment-13651066
]
Yago Riveiro commented on SOLR-4793:
A workaround for this is set the param -Djute.maxbuffer with a value greater
than 1M.
>From zookeeper
>[doc|http://zookeeper.apache.org/doc/r3.4.5/zookeeperAdmin.html#sc_configuration]
{quote}
jute.maxbuffer:
(Java system property: jute.maxbuffer)
This option can only be set as a Java system property. There is no zookeeper
prefix on it. It specifies the maximum size of the data that can be stored in a
znode. The default is 0xf, or just under 1M. If this option is changed, the
system property must be set on all servers and clients otherwise problems will
arise. This is really a sanity check. ZooKeeper is designed to store data on
the order of kilobytes in size.
{quote}
Notice that to use this configuration param, is necessary set this param in
solrcloud and zookeeper init script.
> Solr Cloud can't upload large config files ( > 1MB) to Zookeeper
> -
>
> Key: SOLR-4793
> URL: https://issues.apache.org/jira/browse/SOLR-4793
> Project: Solr
> Issue Type: Improvement
>Reporter: Son Nguyen
>
> Zookeeper set znode size limit to 1MB by default. So we can't start Solr
> Cloud with some large config files, like synonyms.txt.
> Jan Høydahl has a good idea:
> "SolrCloud is designed with an assumption that you should be able to upload
> your whole disk-based conf folder into ZK, and that you should be able to add
> an empty Solr node to a cluster and it would download all config from ZK. So
> immediately a splitting strategy automatically handled by ZkSolresourceLoader
> for large files could be one way forward, i.e. store synonyms.txt as e.g.
> __001_synonyms.txt __002_synonyms.txt"
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
