Hello, I have gone through a good amount of research and effort trying to get a large amount of databases replicating, with a good amount of help from this list. I have hit a road block and am unable to get more then 2000+ databases replicating.
Something tells me [1] might be part of my issue. I have downloaded and installed couchdb-0.11.1, I made sure that patch was present in the source. I included the limits in the INI file but they do not seem to be respected, or I am hitting a different kind of limit. Here [2] is the errors that couch starts throwing when I get to roughly 2048~ (the default number of allowed connections). These exist in my couchdb init script: ulimit -n 1000000 export ERL_MAX_PORTS=100000 export ERL_MAX_ETS_TABLES=10000 This is the init script start command: daemon --user couchdb "$exec -b -p /<dir>/couchdb-0.11.1-test/couch.pid -a /<dir>/couchdb-0.11.1-test/local.ini +P 65536" I am certain the following local.ini is being parsed, as I run on a non-standard port, in addition I am certain the path is correct: [couchdb] os_process_timeout=60000 max_dbs_open = 100000 [httpd] max_connections = 9000 port = 59840 bind_address = 0.0.0.0 At this point I am looking for some expertise from other people who have dealt with large-scale replication. Kind Regards, -Chris [1] https://issues.apache.org/jira/browse/COUCHDB-705 [2] --- [Thu, 15 Jul 2010 23:03:53 GMT] [error] [emulator] ** Too many db tables ** [Thu, 15 Jul 2010 23:03:53 GMT] [error] [<0.21959.3>] {error_report,<0.28.0>, {<0.21959.3>,crash_report, [[{pid,<0.21959.3>}, {registered_name,[]}, {error_info, {exit, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}, [{gen_server,init_it,6},{proc_lib,init_p_do_apply,3}]}}, {initial_call,{couch_rep,init,['Argument__1']}}, {ancestors, [couch_rep_sup,couch_primary_services,couch_server_sup,<0.29.0>]}, {messages,[]}, {links,[<0.22081.3>,<0.22182.3>,<0.22268.3>,<0.22166.3>,<0.78.0>]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,2584}, {stack_size,23}, {reductions,2097}], [{neighbour, [{pid,<0.22268.3>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{erlang,apply,2}}, {ancestors,[]}, {messages,[]}, {links,[<0.21959.3>]}, {dictionary,[]}, {trap_exit,false}, {status,runnable}, {heap_size,377}, {stack_size,0}, {reductions,0}]}]]}} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] ** Generic server <0.22166.3> terminating ** Last message in was {'EXIT',<0.21959.3>, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}} ** When Server state == {state,<0.22167.3>,nil, {db,<0.21978.3>,<0.22062.3>,nil, <<"1279235033964957">>,<0.21967.3>, <0.22063.3>, {db_header,5,0,0,nil,nil,nil,0,nil,nil,1000}, 0, {btree,<0.21967.3>,nil, #Fun<couch_db_updater.7.82129660>, #Fun<couch_db_updater.8.42953822>, #Fun<couch_btree.5.124754102>, #Fun<couch_db_updater.9.115326703>}, {btree,<0.21967.3>,nil, #Fun<couch_db_updater.10.103072508>, #Fun<couch_db_updater.11.104248294>, #Fun<couch_btree.5.124754102>, #Fun<couch_db_updater.12.125559248>}, {btree,<0.21967.3>,nil, #Fun<couch_btree.0.83553141>, #Fun<couch_btree.1.30790806>, #Fun<couch_btree.2.124754102>,nil}, 0,<<"db_2213">>, "/var/lib/mysql_data/couchdb-0.11.1-test/var/lib/couchdb/db_2213.couch", [],[],nil, {user_ctx,<<"network">>, [<<"_admin">>], <<"{couch_httpd_auth, default_authentication_handler}">>}, nil,1000, [before_header,after_header,on_file_open]}, <0.21959.3>,false,0, {<0.22184.3>,#Ref<0.0.2.116318>}, {[],[]}, 0,0,0} ** Reason for termination == ** {system_limit,[{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22166.3>] {error_report,<0.28.0>, {<0.22166.3>,crash_report, [[{pid,<0.22166.3>}, {registered_name,[]}, {error_info, {exit, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}, [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, {initial_call,{couch_rep_missing_revs,init,['Argument__1']}}, {ancestors, [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup, <0.29.0>]}, {messages,[]}, {links,[<0.22167.3>]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,233}, {stack_size,23}, {reductions,144}], []]}} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] ** Generic server <0.22182.3> terminating ** Last message in was {'EXIT',<0.21959.3>, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}} ** When Server state == {state,<0.21959.3>, {http_db, "http://network:[email protected]:59845/db_462/";, [],[], [{"User-Agent","CouchDB/0.11.1"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, <0.22166.3>,<0.22184.3>,[],0, {[],[]}, nil,false,0,nil,[],[]} ** Reason for termination == ** {system_limit,[{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22182.3>] {error_report,<0.28.0>, {<0.22182.3>,crash_report, [[{pid,<0.22182.3>}, {registered_name,[]}, {error_info, {exit, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}, [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, {initial_call,{couch_rep_reader,init,['Argument__1']}}, {ancestors, [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup, <0.29.0>]}, {messages,[]}, {links,[<0.22184.3>]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,987}, {stack_size,23}, {reductions,282}], [{neighbour, [{pid,<0.22184.3>}, {registered_name,[]}, {initial_call,{erlang,apply,2}}, {current_function,{gen,wait_resp_mon,3}}, {ancestors,[]}, {messages, [{'DOWN',#Ref<0.0.2.116318>,process,<0.22166.3>, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}}]}, {links,[<0.22182.3>]}, {dictionary,[]}, {trap_exit,false}, {status,runnable}, {heap_size,377}, {stack_size,15}, {reductions,14}]}]]}} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.31207.0>] Uncaught error in HTTP request: {error, {case_clause, {error, {{system_limit, [{ets,new, [replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}, {child,undefined, "a0bd902c5803e5fb8e59918c8805d776+continuous+create_target", {gen_server,start_link, [couch_rep, ["a0bd902c5803e5fb8e59918c8805d776", {[{<<"source">>, <<"http://network:[email protected]:59845/db_462";>>}, {<<"target">>,<<"db_2213">>}, {<<"continuous">>,true}, {<<"create_target">>,true}, {<<"cancel">>,false}]}, {user_ctx,<<"network">>, [<<"_admin">>], <<"{couch_httpd_auth, default_authentication_handler}">>}], []]}, temporary,1,worker, [couch_rep]}}}}} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] ** Generic server <0.22081.3> terminating ** Last message in was {'EXIT',<0.21959.3>, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}} ** When Server state == {state,nil,nil, [<0.21959.3>, {http_db, "http://network:[email protected]:59845/db_462/";, [],[], [{"User-Agent","CouchDB/0.11.1"}, {"Accept","application/json"}, {"Accept-Encoding","gzip"}], [],get,nil, [{response_format,binary}, {inactivity_timeout,30000}], 10,500,nil}, 0, [{<<"source">>, <<"http://network:[email protected]:59845/db_462";>>}, {<<"target">>,<<"db_2213">>}, {<<"continuous">>,true}, {<<"create_target">>,true}, {<<"cancel">>,false}]], 0,<0.22165.3>, {1279,235033,989017}, false,0,<<>>, {<0.22167.3>,#Ref<0.0.2.116315>}, {[],[]}} ** Reason for termination == ** {system_limit,[{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]} [Thu, 15 Jul 2010 23:03:54 GMT] [error] [<0.22081.3>] {error_report,<0.28.0>, {<0.22081.3>,crash_report, [[{pid,<0.22081.3>}, {registered_name,[]}, {error_info, {exit, {system_limit, [{ets,new,[replication_stats,[set,private]]}, {couch_rep,do_init,1}, {couch_rep,init,1}, {gen_server,init_it,6}, {proc_lib,init_p_do_apply,3}]}, [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}}, {initial_call,{couch_rep_changes_feed,init,['Argument__1']}}, {ancestors, [<0.21959.3>,couch_rep_sup,couch_primary_services,couch_server_sup, <0.29.0>]}, {messages, [{ibrowse_async_response, {1279,235033,989017}, {error,closing_on_request}}, {'EXIT',<0.22165.3>,normal}]}, {links,[]}, {dictionary,[]}, {trap_exit,true}, {status,running}, {heap_size,987}, {stack_size,23}, {reductions,903}], []]}} [Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] Stacktrace: [{couch_rep,start_replication_server,1}, {couch_rep,replicate,2}, {couch_httpd_misc_handlers,handle_replicate_req,1}, {couch_httpd,handle_request_int,5}, {mochiweb_http,headers,5}, {proc_lib,init_p_do_apply,3}] [Thu, 15 Jul 2010 23:03:54 GMT] [info] [<0.31207.0>] 172.19.68.122 - - 'POST' /_replicate 500 [Thu, 15 Jul 2010 23:03:54 GMT] [debug] [<0.31207.0>] httpd 500 error response: {"error":"case_clause","reason":"{error,\n {{system_limit,\n [{ets,new,[replication_stats,[set,private]]},\n {couch_rep,do_init,1},\n {couch_rep,init,1},\n {gen_server,init_it,6},\n {proc_lib,init_p_do_apply,3}]},\n {child,undefined,\n \"a0bd902c5803e5fb8e59918c8805d776+continuous+create_target\",\n {gen_server,start_link,\n [couch_rep,\n [\"a0bd902c5803e5fb8e59918c8805d776\",\n {[{<<\"source\">>,\n <<\"http://network:[email protected]:59845/db_462\";>>},\n {<<\"target\">>,<<\"db_2213\">>},\n {<<\"continuous\">>,true},\n {<<\"create_target\">>,true},\n {<<\"cancel\">>,false}]},\n {user_ctx,<<\"network\">>,\n [<<\"_admin\">>],\n <<\"{couch_httpd_auth, default_authentication_handler}\">>}],\n []]},\n temporary,1,worker,\n [couch_rep]}}}"} ---
