[jira] [Commented] (COUCHDB-1153) Database and view index compaction daemon
[ https://issues.apache.org/jira/browse/COUCHDB-1153?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090078#comment-13090078 ] Jan Lehnardt commented on COUCHDB-1153: --- Paul, this came across wrong, sorry for the added confusion and feelings hurt. My post was aimed at let's get back to work, not disregarding your efforts. Just to close the loop, I was going through this ticket and dev@ looking for open issues and then categorised them into this code needs work and this code needs work and thus should not be in trunk at this point (with the obvious goal of removing all of the latter ones, so we can retroactively have the patch in trunk okayed). I used the catch-all bikedshed for the former category and hence totally disregarded your comments. I was meaning to say that these aren't blockers (in my opinion) that would warrant a revert of the patch and I didn't mean to imply that this is some opinion you pulled out of thin air. Sorry, again. I hope we can put this to rest now. Database and view index compaction daemon - Key: COUCHDB-1153 URL: https://issues.apache.org/jira/browse/COUCHDB-1153 Project: CouchDB Issue Type: New Feature Environment: trunk Reporter: Filipe Manana Assignee: Filipe Manana Priority: Minor Labels: compaction I've recently written an Erlang process to automatically compact databases and they're views based on some configurable parameters. These parameters can be global or per database and are: minimum database fragmentation, minimum view fragmentation, allowed period and strict_window (whether an ongoing compaction should be canceled if it doesn't finish within the allowed period). These fragmentation values are based on the recently added data_size parameter to the database and view group information URIs (COUCHDB-1132). I've documented the .ini configuration, as a comment in default.ini, which I paste here: [compaction_daemon] ; The delay, in seconds, between each check for which database and view indexes ; need to be compacted. check_interval = 60 ; If a database or view index file is smaller then this value (in bytes), ; compaction will not happen. Very small files always have a very high ; fragmentation therefore it's not worth to compact them. min_file_size = 131072 [compactions] ; List of compaction rules for the compaction daemon. ; The daemon compacts databases and they're respective view groups when all the ; condition parameters are satisfied. Configuration can be per database or ; global, and it has the following format: ; ; database_name = parameter=value [, parameter=value]* ; _default = parameter=value [, parameter=value]* ; ; Possible parameters: ; ; * db_fragmentation - If the ratio (as an integer percentage), of the amount ; of old data (and its supporting metadata) over the database ; file size is equal to or greater then this value, this ; database compaction condition is satisfied. ; This value is computed as: ; ; (file_size - data_size) / file_size * 100 ; ; The data_size and file_size values can be obtained when ; querying a database's information URI (GET /dbname/). ; ; * view_fragmentation - If the ratio (as an integer percentage), of the amount ;of old data (and its supporting metadata) over the view ;index (view group) file size is equal to or greater then ;this value, then this view index compaction condition is ;satisfied. This value is computed as: ; ;(file_size - data_size) / file_size * 100 ; ;The data_size and file_size values can be obtained when ;querying a view group's information URI ;(GET /dbname/_design/groupname/_info). ; ; * period - The period for which a database (and its view groups) compaction ;is allowed. This value must obey the following format: ; ;HH:MM - HH:MM (HH in [0..23], MM in [0..59]) ; ; * strict_window - If a compaction is still running after the end of the allowed ; period, it will be canceled if this parameter is set to yes. ; It defaults to no and it's meaningful only if the *period* ; parameter is also specified. ; ; * parallel_view_compaction - If set to yes, the database and its views are ; compacted in parallel. This is only useful on ; certain setups, like for example when the database ;
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090083#comment-13090083
]
James Cohen commented on COUCHDB-1258:
--
Overnight I ran a dump/load of the database with the Python tools couchdb-dump
and couchdb-load, I now have
doc_count:593274,doc_del_count:0,update_seq:593274
The views are generated with minimal memory usage (far better than in either
case before) and the problem doesn't seem to be present.
emitting null instead of the doc results in high memory usage by beam.smp and
ultimately the same OOM error
eheap_alloc OOM errors when attempting to build selected views
--
Key: COUCHDB-1258
URL: https://issues.apache.org/jira/browse/COUCHDB-1258
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.1
Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
Couch database exhibiting this behaviour:
{db_name:activity_new,doc_count:593274,doc_del_count:4743352,update_seq:10559287,purge_seq:0,compact_running:false,disk_size:3366396013,instance_start_time:1314097423985726,disk_format_version:5,committed_update_seq:10559287}
Reporter: James Cohen
Priority: Critical
We spotted OOM errors crashing our CouchDb instance when attempting to
rebuild selected views. CouchDb was dying with the following messages (worth
noting that the type reported varies between heap/old_heap
eheap_alloc: Cannot allocate 478288480 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 747325720 bytes of memory (of type old_heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type old_heap).
By modifying the view I was able to find a view that could consistently crash
the server and another that ran fine. They are as follows:
Runs out of memory v.quickly
{
_id: _design/cleanup,
_rev: 5-e004fbab278355e9d08763877e5a8295,
views: {
byDate: {
map: function(doc) { if (! doc.action) emit([doc.date], doc); }
}
}
}
Runs fine with minimal memory usage (returns 88128 docs in the view)
{
_id: _design/cleanup,
_rev: 6-3823be6b72ca2441e235addfece6900c,
views: {
byDate: {
map: function(doc) { if (doc.action) emit([doc.date], doc); }
}
}
}
The only difference between the two is the negation of the if conditional.
memory usage was monitored with top on the machine while the view was being
built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
server's memory. With the view that causes problems that memory usage
increased until the RAM/swap on the server was exhausted (as you can see from
the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: wrong case in Last-Modified
HTTP headers are required to be treated case-insensitively (http://www.w3.org/Protocols/rfc2616/rfc2616-sec4.html#sec4.2). CouchDB supports http 1.0 and 1.1 but returns a 1.0 response to a 1.0 request. Using wget on my machine gives a 1.1 response, but perhaps your wget is older (it used to only support 1.0, iirc). B. On 24 August 2011 17:28, Andrey Somov trophyb...@googlemail.com wrote: Hi all, I have noticed that the HTTP header contains lower case value for 'Last-Modified' (http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.29) ~$ wget -q -O - $@http://localhost:5984/favicon.ico --proxy=off --server-response HTTP/1.0 200 OK Server: CouchDB/1.2.0a-1151390 (Erlang OTP/R13B03) last-modified: Tue, 26 Jul 2011 04:19:47 GMT Expires: Fri, 24 Aug 2012 15:02:33 GMT Date: Wed, 24 Aug 2011 15:02:33 GMT Content-Type: image/x-icon Content-Length: 9326 Cache-Control: public, max-age=31536000 By the way, the protocol in the reply is HTTP 1.0. Does it mean that CouchDB does not support HTTP 1.1 ? - Andrey
Re: wrong case in Last-Modified
Thank you. (but I would not mind if someone would change it to look perfect :)
Re: wrong case in Last-Modified
On 24 Aug 2011, at 17:49, Andrey Somov wrote: (but I would not mind if someone would change it to look perfect :) Patches welcome!
Re: wrong case in Last-Modified
(but I would not mind if someone would change it to look perfect :) Patches welcome! I am sorry. I looked in the source and the header is generated by mochiweb.
Re: wrong case in Last-Modified
I bet they welcome patches too! :P On 24 Aug 2011, at 18:16, Andrey Somov wrote: (but I would not mind if someone would change it to look perfect :) Patches welcome! I am sorry. I looked in the source and the header is generated by mochiweb.
[jira] [Assigned] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Chesneau reassigned COUCHDB-431: --- Assignee: Benoit Chesneau (was: Randall Leeds) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[
https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoit Chesneau updated COUCHDB-431:
Attachment: test_cors2.tgz
0001-cors-support.-should-fix-COUCHDB-431.patch
This patch add support for CORS based on http://www.w3.org/TR/cors/ spec +
tests.
This ia a different implentation from the one proposed by Jason . It uses the
process registry to pass custom headers so we can manage authentication easily.
Supported :
- simple preflight requests.
- possibility to forbid access per db by settings origins in secob. (see the
erlang tests) :
{
...
origins: [http://someorigin;]
}
- CORS headers are only available if origin header is present in the request
(like the spec say).
test_cors2.tgz is also attached to test it. To use it untar the file, push the
couchapp using couchapp or erica to a database named couch then open cors.html
file under another server or from the file system.
Support cross domain XMLHttpRequest (XHR) calls by implementing Access
Control spec
---
Key: COUCHDB-431
URL: https://issues.apache.org/jira/browse/COUCHDB-431
Project: CouchDB
Issue Type: New Feature
Components: HTTP Interface
Affects Versions: 0.9
Reporter: James Burke
Assignee: Benoit Chesneau
Priority: Minor
Fix For: 1.2
Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch,
A_0001-Generalize-computing-the-appropriate-headers-for-any.patch,
A_0002-Send-server-headers-for-externals-responses.patch,
A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch,
A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html,
test_cors2.tgz
Historically, browsers have been restricted to making XMLHttpRequests (XHRs)
to the same origin (domain) as the web page making the request. However, the
latest browsers now support cross-domain requests by implementing the Access
Control spec from the W3C:
http://dev.w3.org/2006/waf/access-control/
In order to keep older servers safe that assume browsers only do same-domain
requests, the Access Control spec requires the server to opt-in to allow
cross domain requests by the use of special HTTP headers and supporting some
pre-flight HTTP calls.
Why should CouchDB support this: in larger, high traffic site, it is common
to serve the static UI files from a separate, differently scaled server
complex than the data access/API server layer. Also, there are some API
services that are meant to be centrally hosted, but allow API consumers to
use the API from different domains. In these cases, the UI in the browser
would need to do cross domain requests to access CouchDB servers that act as
the API/data access server layer.
JSONP is not enough in these cases since it is limited to GET requests, so no
POSTing or PUTing of documents.
Some information from Firefox's perspective (functionality available as of
Firefox 3.5):
https://developer.mozilla.org/en/HTTP_access_control
And information on Safari/Webkit (functionality in latest WebKit and Safari
4):
http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html
IE 8 also uses the Access Control spec, but the requests have to go through
their XDomainRequest object (XDR):
http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx
and I thought IE8 only allowed GET or POST requests through their XDR.
But as far as CouchDB is concerned, implementing the Access Control headers
should be enough, and hopefully IE 9 will allow normal xdomain requests via
XHR.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090382#comment-13090382 ] Robert Newson commented on COUCHDB-431: --- also '?LOG_INFO(la, [])' probably shouldn't be in here, right? :) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, test_cors2.tgz Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090381#comment-13090381 ] Robert Newson commented on COUCHDB-431: --- The new patch combines indentation changes with code changes, can that be separated? Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, test_cors2.tgz Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090384#comment-13090384 ] Benoit Chesneau commented on COUCHDB-431: - mmm by indentation change do you mean the new lines added with new options ? I can probably put them back on one line if it's better. Or did I miss other things (which is totally possible I'm a little blind tonight)? New patch remove this spurious log is coming. Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, test_cors2.tgz Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoit Chesneau updated COUCHDB-431: Attachment: 0001-cors-support.-should-fix-COUCHDB-431.patch new version of the patch. removing spurious log checkout back run.tpl . Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, 0001-cors-support.-should-fix-COUCHDB-431.patch, A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, test_cors2.tgz Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-431) Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec
[ https://issues.apache.org/jira/browse/COUCHDB-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090391#comment-13090391 ] Robert Newson commented on COUCHDB-431: --- Don't get me wrong, the lines in this module are far too long anyway, but reindenting makes it harder to see the substantive change you're making. The first one where you added OPTIONS was the one I was referring to, but I had failed to notice that the before/after were different. Given the alternative is to make these overlong lines even longer, I guess there's no need to change it, but I'd refactor this into two commits, one which simply wraps these long lines better and a second that then adds the extra bits. Not a deal breaker for me if you don't do it. Support cross domain XMLHttpRequest (XHR) calls by implementing Access Control spec --- Key: COUCHDB-431 URL: https://issues.apache.org/jira/browse/COUCHDB-431 Project: CouchDB Issue Type: New Feature Components: HTTP Interface Affects Versions: 0.9 Reporter: James Burke Assignee: Benoit Chesneau Priority: Minor Fix For: 1.2 Attachments: 0001-cors-support.-should-fix-COUCHDB-431.patch, 0001-cors-support.-should-fix-COUCHDB-431.patch, A_0001-Generalize-computing-the-appropriate-headers-for-any.patch, A_0002-Send-server-headers-for-externals-responses.patch, A_0003-Usably-correct-w3c-CORS-headers-for-valid-requests.patch, A_0004-Respond-to-CORS-preflight-checks-HTTP-OPTIONS.patch, cors.html, test_cors2.tgz Historically, browsers have been restricted to making XMLHttpRequests (XHRs) to the same origin (domain) as the web page making the request. However, the latest browsers now support cross-domain requests by implementing the Access Control spec from the W3C: http://dev.w3.org/2006/waf/access-control/ In order to keep older servers safe that assume browsers only do same-domain requests, the Access Control spec requires the server to opt-in to allow cross domain requests by the use of special HTTP headers and supporting some pre-flight HTTP calls. Why should CouchDB support this: in larger, high traffic site, it is common to serve the static UI files from a separate, differently scaled server complex than the data access/API server layer. Also, there are some API services that are meant to be centrally hosted, but allow API consumers to use the API from different domains. In these cases, the UI in the browser would need to do cross domain requests to access CouchDB servers that act as the API/data access server layer. JSONP is not enough in these cases since it is limited to GET requests, so no POSTing or PUTing of documents. Some information from Firefox's perspective (functionality available as of Firefox 3.5): https://developer.mozilla.org/en/HTTP_access_control And information on Safari/Webkit (functionality in latest WebKit and Safari 4): http://developer.apple.com/safari/library/documentation/AppleApplications/Conceptual/SafariJSProgTopics/Articles/XHR.html IE 8 also uses the Access Control spec, but the requests have to go through their XDomainRequest object (XDR): http://msdn.microsoft.com/en-us/library/cc288060%28VS.85%29.aspx and I thought IE8 only allowed GET or POST requests through their XDR. But as far as CouchDB is concerned, implementing the Access Control headers should be enough, and hopefully IE 9 will allow normal xdomain requests via XHR. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1259) Replication ID is not stable if local server has a dynamic port number
[
https://issues.apache.org/jira/browse/COUCHDB-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090396#comment-13090396
]
Jens Alfke commented on COUCHDB-1259:
-
@Jason — I see where you're coming from with bad web citizen but it refers
only to a specific usage of CouchDB as a traditional web server. We have
equally valid uses of it that do not act as traditional servers nor have stable
URLs, for which we absolutely need this fix. I don't think arguments about
whether URLs change or not are productive here in a specific bug report.
Replication ID is not stable if local server has a dynamic port number
--
Key: COUCHDB-1259
URL: https://issues.apache.org/jira/browse/COUCHDB-1259
Project: CouchDB
Issue Type: Bug
Components: Replication
Affects Versions: 1.1
Reporter: Jens Alfke
I noticed that when Couchbase Mobile running on iOS replicates to/from a
remote server (on iriscouch in this case), the replication has to fetch the
full _changes feed every time it starts. Filipe helped me track down the
problem -- the replication ID is coming out different every time. The reason
for this is that the local port number, which is one of the inputs to the
hash that generates the replication ID, is randomly assigned by the OS. (I.e.
it uses a port number of 0 when opening its listener socket.) This is because
there could be multiple apps using Couchbase Mobile running on the same
device and we can't have their ports colliding.
The underlying problem is that CouchDB is attempting to generate a unique ID
for a particular pair of {source, destination} databases, but it's basing it
on attributes that aren't fundamental to the database and can change, like
the hostname or port number.
One solution, proposed by Filipe and me, is to assign each database (or each
server?) a random UUID when it's created, and use that to generate
replication IDs.
Another solution, proposed by Damien, is to have CouchDB let the client work
out the replication ID on its own, and set it as a property in the
replication document (or the JSON body of a _replicate request.) This is even
more flexible and will handle tricky scenarios like full P2P replication
where there may be no low-level way to uniquely identify the remote database
being synced with.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1238) CouchDB uses _users db for storing oauth credentials
[ https://issues.apache.org/jira/browse/COUCHDB-1238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090431#comment-13090431 ] Pete Vander Giessen commented on COUCHDB-1238: -- @Filipe (just realized that I misspelled your name twice in my last comment; fixed now -- sorry): you are the expert. Any changes that you deem necessary are fine by me :-) (And reading through the code, it looks like all the changes you made make things better.) Let me know if I need to do anything else on this one -- happy to help in any way I can. CouchDB uses _users db for storing oauth credentials Key: COUCHDB-1238 URL: https://issues.apache.org/jira/browse/COUCHDB-1238 Project: CouchDB Issue Type: New Feature Components: Database Core Affects Versions: 1.1 Reporter: Pete Vander Giessen Assignee: Filipe Manana Fix For: 1.2 Attachments: git_commits_as_patch.zip, oauth_users_db_patch.zip We want to store oauth credentials in the _users db, rather than in the .ini. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Our views keep failing with timeouts until a couchdb restart
We have about 3000 active databases on our server, about 200 of those
users (most active ones) keep randomly failing to load views. I am
having difficulty trying to decipher this stack trace to make any
sense of it, does anyone have any suggestions about what is happening
here? Are we just reaching some sort of capacity limit? The machines
are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
are fine on disk space, and requests per second to be honest is around
1 with plenty of idle time between little spurts.
-Chris
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] 'GET'
/db_946/_design/views/_view/getByType?skip=0key=[model,M4e551a78a9b3d]limit=25
{1,
1} from WITHHELD
Headers: [{'Accept',*/*},
{'Authorization',Basic WITHHELD},
{'Content-Type',application/json; charset=utf-8},
{'Host',WITHHELD:5984},
{'User-Agent',PECL::HTTP/1.7.0 (PHP/5.3.4)}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] OAuth Params:
[{skip,0},
{key,
[\model\,\M4e551a78a9b3d\]},
{limit,25}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] request_group
{Pid, Seq} {0.25836.0,1077}
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.82.0] New task status for
db_946 _design/views: Processed 0 of 25 changes (0%)
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.82.0] New task status for
db_946 _design/views: Finishing.
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [0.31520.215] 'GET'
/_all_dbs {1,1} from WITHHELD
Headers: [{'Accept',*/*},
{'Authorization',Basic WITHHELD},
{'Content-Type',application/json; charset=utf-8},
{'Host',WITHHELD:5984},
{'User-Agent',PECL::HTTP/1.7.0 (PHP/5.3.4)}]
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [0.31520.215] OAuth Params: []
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [0.25836.0] Exit from linked
pid: {0.27488.217,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,
javascript}]}}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [0.25836.0] ** Generic
server 0.25836.0 terminating
** Last message in was {'EXIT',0.27488.217,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,javascript}]}}}
** When Server state == {group_state,undefined,db_946,
{/WITHHELD/5984/views,db_946,
{group,
88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203,
nil,nil,_design/views,javascript,[],
[{view,0,0,0,
[getByType],
function(doc) {\n \n var sdoc =
eval(uneval(doc));\n if((typeof doc.model) !== \undefined\) {\n
sdoc.data.rev = doc._rev;\nemit([doc.type, doc.model],
sdoc.data);\n } else if((typeof doc.type)
!== \undefined\ doc.type === \couchimport\) {\n
emit([doc.type, doc.time], null);\n }\n},
nil,[],[]}],
{[]},
nil,0,0,nil,nil}},
{group,
88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203,
{db,0.6703.0,0.6717.0,nil,
1313769124135037,0.24135.71,0.24137.71,
{db_header,5,1077,0,
{1011747,{410,120}},
{1014120,530},
{1022521,[]},
0,nil,nil,1000},
1077,
{btree,0.24135.71,
{1011747,{410,120}},
#Funcouch_db_updater.10.118480628,
#Funcouch_db_updater.11.70256084,
#Funcouch_btree.5.49826377,
#Funcouch_db_updater.12.32819211},
{btree,0.24135.71,
{1014120,530},
#Funcouch_db_updater.13.25773661,
#Funcouch_db_updater.14.39338119,
#Funcouch_btree.5.49826377,
#Funcouch_db_updater.15.41342280},
{btree,0.24135.71,
{1022521,[]},
#Funcouch_btree.0.67022066,
#Funcouch_btree.1.72205677,
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090466#comment-13090466
]
Paul Joseph Davis commented on COUCHDB-1258:
Sounds reasonable. A map function that emits no keys coupled with a lot of
deleted documents means that the partial_update frequency will be quite large.
couch_work_queue's behave quite poorly when the dequeue side is faster than the
queue side and end up basically not bulking updates together. A quick test
would be to guard the partial_update message to only fire at most every 1000
writes. A quick hack to store calls in the process dict in do_writes should
give a quick idea if it's the partial updates.
A should_flush for every partial update seems a bit extreme. Although I can't
think of a better solution that doesn't just make an equivalent to should_flush
that's more specific to this case.
eheap_alloc OOM errors when attempting to build selected views
--
Key: COUCHDB-1258
URL: https://issues.apache.org/jira/browse/COUCHDB-1258
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.1
Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
Couch database exhibiting this behaviour:
{db_name:activity_new,doc_count:593274,doc_del_count:4743352,update_seq:10559287,purge_seq:0,compact_running:false,disk_size:3366396013,instance_start_time:1314097423985726,disk_format_version:5,committed_update_seq:10559287}
Reporter: James Cohen
Priority: Critical
We spotted OOM errors crashing our CouchDb instance when attempting to
rebuild selected views. CouchDb was dying with the following messages (worth
noting that the type reported varies between heap/old_heap
eheap_alloc: Cannot allocate 478288480 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 747325720 bytes of memory (of type old_heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type old_heap).
By modifying the view I was able to find a view that could consistently crash
the server and another that ran fine. They are as follows:
Runs out of memory v.quickly
{
_id: _design/cleanup,
_rev: 5-e004fbab278355e9d08763877e5a8295,
views: {
byDate: {
map: function(doc) { if (! doc.action) emit([doc.date], doc); }
}
}
}
Runs fine with minimal memory usage (returns 88128 docs in the view)
{
_id: _design/cleanup,
_rev: 6-3823be6b72ca2441e235addfece6900c,
views: {
byDate: {
map: function(doc) { if (doc.action) emit([doc.date], doc); }
}
}
}
The only difference between the two is the negation of the if conditional.
memory usage was monitored with top on the machine while the view was being
built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
server's memory. With the view that causes problems that memory usage
increased until the RAM/swap on the server was exhausted (as you can see from
the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
I bet you're hitting a bug we just recently fixed on trunk. Basically,
there was a possibility that errors in some of the JS functions would
end up causing a couchjs process to be come unusable without removing
it from the list. Eventually there wouldn't be any spots left and
clients would get timeouts like you see.
Patch is at [1]. If it doesn't apply cleanly, you really only need the
bits from couch_os_process.erl and couch_query_servers.erl. The rest
is just test code.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
On Wed, Aug 24, 2011 at 3:42 PM, Chris Stockton
chrisstockto...@gmail.com wrote:
We have about 3000 active databases on our server, about 200 of those
users (most active ones) keep randomly failing to load views. I am
having difficulty trying to decipher this stack trace to make any
sense of it, does anyone have any suggestions about what is happening
here? Are we just reaching some sort of capacity limit? The machines
are Intel(R) Xeon(R) CPU L5520 @ 2.27GHz x16 core with 50mb ram, we
are fine on disk space, and requests per second to be honest is around
1 with plenty of idle time between little spurts.
-Chris
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] 'GET'
/db_946/_design/views/_view/getByType?skip=0key=[model,M4e551a78a9b3d]limit=25
{1,
1} from WITHHELD
Headers: [{'Accept',*/*},
{'Authorization',Basic WITHHELD},
{'Content-Type',application/json; charset=utf-8},
{'Host',WITHHELD:5984},
{'User-Agent',PECL::HTTP/1.7.0 (PHP/5.3.4)}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] OAuth Params:
[{skip,0},
{key,
[\model\,\M4e551a78a9b3d\]},
{limit,25}]
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.14283.217] request_group
{Pid, Seq} {0.25836.0,1077}
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.82.0] New task status for
db_946 _design/views: Processed 0 of 25 changes (0%)
[Wed, 24 Aug 2011 15:36:31 GMT] [debug] [0.82.0] New task status for
db_946 _design/views: Finishing.
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [0.31520.215] 'GET'
/_all_dbs {1,1} from WITHHELD
Headers: [{'Accept',*/*},
{'Authorization',Basic WITHHELD},
{'Content-Type',application/json; charset=utf-8},
{'Host',WITHHELD:5984},
{'User-Agent',PECL::HTTP/1.7.0 (PHP/5.3.4)}]
[Wed, 24 Aug 2011 15:36:35 GMT] [debug] [0.31520.215] OAuth Params: []
[Wed, 24 Aug 2011 15:36:36 GMT] [debug] [0.25836.0] Exit from linked
pid: {0.27488.217,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,
javascript}]}}}
[Wed, 24 Aug 2011 15:36:36 GMT] [error] [0.25836.0] ** Generic
server 0.25836.0 terminating
** Last message in was {'EXIT',0.27488.217,
{timeout,
{gen_server,call,
[couch_query_servers,
{get_proc,javascript}]}}}
** When Server state == {group_state,undefined,db_946,
{/WITHHELD/5984/views,db_946,
{group,
88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203,
nil,nil,_design/views,javascript,[],
[{view,0,0,0,
[getByType],
function(doc) {\n \n var sdoc =
eval(uneval(doc));\n if((typeof doc.model) !== \undefined\) {\n
sdoc.data.rev = doc._rev;\n emit([doc.type, doc.model],
sdoc.data);\n } else if((typeof doc.type)
!== \undefined\ doc.type === \couchimport\) {\n
emit([doc.type, doc.time], null);\n }\n},
nil,[],[]}],
{[]},
nil,0,0,nil,nil}},
{group,
88,143,236,54,199,129,241,187,65,118,56,84,121,73,
170,203,
{db,0.6703.0,0.6717.0,nil,
1313769124135037,0.24135.71,0.24137.71,
{db_header,5,1077,0,
{1011747,{410,120}},
{1014120,530},
{1022521,[]},
0,nil,nil,1000},
1077,
{btree,0.24135.71,
{1011747,{410,120}},
#Funcouch_db_updater.10.118480628,
#Funcouch_db_updater.11.70256084,
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090478#comment-13090478
]
Robert Newson commented on COUCHDB-1258:
you could try this to see if it helps with memory usage. Verified locally that
it radically reduces the number of checkpoints.
diff --git a/src/couchdb/couch_view_group.erl b/src/couchdb/couch_view_group.erl
index cde949b..7002d07 100644
--- a/src/couchdb/couch_view_group.erl
+++ b/src/couchdb/couch_view_group.erl
@@ -243,7 +243,7 @@ handle_cast({partial_update, Pid, NewGroup},
#group_state{updater_pid=Pid}
group = Group
} = State,
NewSeq = NewGroup#group.current_seq,
-case NewSeq Group#group.current_seq of
+case NewSeq 1000 + Group#group.current_seq of
true -
?LOG_INFO(checkpointing view update at seq ~p for ~s ~s, [NewSeq,
DbName, NewGroup#group.name]),
eheap_alloc OOM errors when attempting to build selected views
--
Key: COUCHDB-1258
URL: https://issues.apache.org/jira/browse/COUCHDB-1258
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.1
Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
Couch database exhibiting this behaviour:
{db_name:activity_new,doc_count:593274,doc_del_count:4743352,update_seq:10559287,purge_seq:0,compact_running:false,disk_size:3366396013,instance_start_time:1314097423985726,disk_format_version:5,committed_update_seq:10559287}
Reporter: James Cohen
Priority: Critical
We spotted OOM errors crashing our CouchDb instance when attempting to
rebuild selected views. CouchDb was dying with the following messages (worth
noting that the type reported varies between heap/old_heap
eheap_alloc: Cannot allocate 478288480 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 747325720 bytes of memory (of type old_heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type old_heap).
By modifying the view I was able to find a view that could consistently crash
the server and another that ran fine. They are as follows:
Runs out of memory v.quickly
{
_id: _design/cleanup,
_rev: 5-e004fbab278355e9d08763877e5a8295,
views: {
byDate: {
map: function(doc) { if (! doc.action) emit([doc.date], doc); }
}
}
}
Runs fine with minimal memory usage (returns 88128 docs in the view)
{
_id: _design/cleanup,
_rev: 6-3823be6b72ca2441e235addfece6900c,
views: {
byDate: {
map: function(doc) { if (doc.action) emit([doc.date], doc); }
}
}
}
The only difference between the two is the negation of the if conditional.
memory usage was monitored with top on the machine while the view was being
built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
server's memory. With the view that causes problems that memory usage
increased until the RAM/swap on the server was exhausted (as you can see from
the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090484#comment-13090484
]
Chris Stockton commented on COUCHDB-1257:
-
Adam it appears this error is a bit more serious then original understood.
While this has been happening we were having a similar error with views no
longer returning results and just timing out. It seems each reduce over flow
error results in a couchjs process remaining open forever.
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
14
induce error 10 times
[root@p3nldbnowcdb01 ~]# ps auxww|grep -i main.js|wc -l
24
This ultimately leads to couch becoming unstable.
Reduction detection in share/server/views.js runReduce to primitive for some
use cases and overall reduce architecture
--
Key: COUCHDB-1257
URL: https://issues.apache.org/jira/browse/COUCHDB-1257
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 1.1
Environment: Architecture independent
Reporter: Chris Stockton
Priority: Minor
Labels: javascript, patch
There is two parts to this report, one is my question I have in regards to
the protocol in which couchdb actually talks to the view server
(server/main.js), the second part as discussed here [1] in short is the
growth detection [2] done in share/server/views.js.
First issue:
While troubleshooting and tracking down this issue, I saw that when running a
reduce function for every doc called, a separate read line is delegated a
reset call. I do not know the internals of couchdb but it seems like it could
be a performance improvement if on a rereduce call, as it appears in the
server Loop function every time reduce or rereduce is called the state is
first reset with reset, to make a minor re-factor and reset the state
within the Views dispatcher itself. This saves some likely blocking io back
and forth.
Reduction detection:
The portion of this issue that actually has affected us is the detection
done, I think that it should be changed to require a sampling of data, the
easiest fix is to simply change (the very arbitrary) [2] into a large length,
such as * 10 or something, but the upper bounds will always be reached for
some data set. A new algorithm could take in consideration the total reduce
calls thus far, as well as the initial size of the very first reduce call, to
see if a large aggregation of some sort is being created.
Suggested change:
Change the signature of rereduce call to main.js to include the total count
of rereduce calls thus far, or give some kind of handle where this logic
could be reliably kept track of inside of main.js, as it stands now your only
hope would be to use the reduce function itself as a lookup, which is not
safe under all edge cases. With this information you can keep the initial
rereduce calls result size, and the total number of calls, to make sure that
aside from initial aggregation or structure initialization the structure is
remaining constant.
There would be several other routes to go to fix this, but I think it is
valid to be considered. I could volunteer my time to write a patch if a
consensus could be reached on behavior, if not denied all together.
-Chris
[1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
[2] if (State.query_config State.query_config.reduce_limit
[2] reduce_length 200 ((reduce_length * 2)
State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
Hello, On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis paul.joseph.da...@gmail.com wrote: I bet you're hitting a bug we just recently fixed on trunk. Basically, there was a possibility that errors in some of the JS functions would end up causing a couchjs process to be come unusable without removing it from the list. Eventually there wouldn't be any spots left and clients would get timeouts like you see. Patch is at [1]. If it doesn't apply cleanly, you really only need the bits from couch_os_process.erl and couch_query_servers.erl. The rest is just test code. https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9 Right after sending this I finally saw what the issue was, our bug report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484 as a side effect was leaving lingering processes ultimately leading to instability of couch. We are working to patch our reduce issue, and will look at applying that commit perhaps once it hits mainline couch? Thanks for the quick response, -Chris
[jira] [Assigned] (COUCHDB-1012) Utility to help plugin developers manage paths
[ https://issues.apache.org/jira/browse/COUCHDB-1012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds reassigned COUCHDB-1012: -- Assignee: Benoit Chesneau (was: Randall Leeds) Utility to help plugin developers manage paths -- Key: COUCHDB-1012 URL: https://issues.apache.org/jira/browse/COUCHDB-1012 Project: CouchDB Issue Type: New Feature Components: Build System Reporter: Randall Leeds Assignee: Benoit Chesneau Fix For: 1.2 Attachments: 0001-add-couch-config-file-used-to-ease-the-build-of-plug.patch, 0001-add-couch-config-file-used-to-ease-the-build-of-plug.patch, 0001-support-pkg-config-for-plugins-COUCHDB-1012.patch Developers may want to write plugins (like GeoCouch) for CouchDB. Many hooks in the configuration system allow loading arbitrary Erlang modules to handle various internal tasks, but currently there is no straightforward and portable way for developers of these plugins to discover the location of the CouchDB library files. Two options that have been proposed are to use pkg-config or install a separate script that could be invoked (e.g. as couch-config --erl-libs) to discover important CouchDB installation paths. As far as I know the loudest argument against pkg-config is lack of support for Windows. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-752) InternalError: script stack space quota is exhausted
[
https://issues.apache.org/jira/browse/COUCHDB-752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randall Leeds reassigned COUCHDB-752:
-
Assignee: Randall Leeds
InternalError: script stack space quota is exhausted
Key: COUCHDB-752
URL: https://issues.apache.org/jira/browse/COUCHDB-752
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 0.11
Reporter: Benoit Chesneau
Assignee: Randall Leeds
Priority: Critical
Fix For: 1.2
Attachments: ddoc.patch
Couchdb freeze and use full CPU after accessing to a 93MB designdoc:
debug] [0.91.0] Spawning new group server for view group _design/aimpl in
database aimpl.
[debug] [0.94.0] request_group {Pid, Seq} {0.116.0,61}
[debug] [0.81.0] New task status for aimpl _design/aimpl: Finishing.
[debug] [0.126.0] OS Process Start :: #Port0.2040
[debug] [0.87.0] Teach ddoc to new proc {proc,0.126.0,javascript,[],
{couch_os_process,prompt},
{couch_os_process,set_timeout},
{couch_os_process,stop}} with DDocKey:
{_design/aimpl,
30-2ab623a62b83b0d9f2616ee209f62659}
InternalError: script stack space quota is exhausted
Most of data are in attachments :
enlil:couchapp benoitc$ du -sh aimpl
93M aimpl
enlil:couchapp benoitc$ du -sh aimpl/_attachments/
93M aimpl/_attachments/
about 100k in views/shows/...
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (COUCHDB-1232) trunk does not build with spidermonkey 1.7
[ https://issues.apache.org/jira/browse/COUCHDB-1232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Randall Leeds reassigned COUCHDB-1232: -- Assignee: Paul Joseph Davis (was: Randall Leeds) Paul said he's working on this right now. trunk does not build with spidermonkey 1.7 -- Key: COUCHDB-1232 URL: https://issues.apache.org/jira/browse/COUCHDB-1232 Project: CouchDB Issue Type: Improvement Components: Build System Affects Versions: 1.2 Environment: Gentoo Linux Reporter: James Marca Assignee: Paul Joseph Davis Priority: Minor Fix For: 1.2 Original Estimate: 4h Remaining Estimate: 4h Original bug report: configure does not require spidermonkey 1.8.5, but trunk (1.2) failed to build without it. Trunk should fall back to a slightly older spidermonkey and configure definitely fails if you don't have either. However, by popular demand I'm told I should make trunk compile with 1.7 again, so I will. I'm changing this ticket to track that. Thanks. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090548#comment-13090548
]
Paul Joseph Davis commented on COUCHDB-1257:
We just fixed this in trunk. It's a small patch to fix the issue if you're
interested. You only need the couch_os_file.erl and couch_query_servers.erl
diffs to make it work.
https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9
Reduction detection in share/server/views.js runReduce to primitive for some
use cases and overall reduce architecture
--
Key: COUCHDB-1257
URL: https://issues.apache.org/jira/browse/COUCHDB-1257
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 1.1
Environment: Architecture independent
Reporter: Chris Stockton
Priority: Minor
Labels: javascript, patch
There is two parts to this report, one is my question I have in regards to
the protocol in which couchdb actually talks to the view server
(server/main.js), the second part as discussed here [1] in short is the
growth detection [2] done in share/server/views.js.
First issue:
While troubleshooting and tracking down this issue, I saw that when running a
reduce function for every doc called, a separate read line is delegated a
reset call. I do not know the internals of couchdb but it seems like it could
be a performance improvement if on a rereduce call, as it appears in the
server Loop function every time reduce or rereduce is called the state is
first reset with reset, to make a minor re-factor and reset the state
within the Views dispatcher itself. This saves some likely blocking io back
and forth.
Reduction detection:
The portion of this issue that actually has affected us is the detection
done, I think that it should be changed to require a sampling of data, the
easiest fix is to simply change (the very arbitrary) [2] into a large length,
such as * 10 or something, but the upper bounds will always be reached for
some data set. A new algorithm could take in consideration the total reduce
calls thus far, as well as the initial size of the very first reduce call, to
see if a large aggregation of some sort is being created.
Suggested change:
Change the signature of rereduce call to main.js to include the total count
of rereduce calls thus far, or give some kind of handle where this logic
could be reliably kept track of inside of main.js, as it stands now your only
hope would be to use the reduce function itself as a lookup, which is not
safe under all edge cases. With this information you can keep the initial
rereduce calls result size, and the total number of calls, to make sure that
aside from initial aggregation or structure initialization the structure is
remaining constant.
There would be several other routes to go to fix this, but I think it is
valid to be considered. I could volunteer my time to write a patch if a
consensus could be reached on behavior, if not denied all together.
-Chris
[1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
[2] if (State.query_config State.query_config.reduce_limit
[2] reduce_length 200 ((reduce_length * 2)
State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Our views keep failing with timeouts until a couchdb restart
That link is to trunk. We should backport it to 1.1.x so it's in 1.1.1 as well. Perhaps even a 1.0.4. On Wed, Aug 24, 2011 at 4:05 PM, Chris Stockton chrisstockto...@gmail.com wrote: Hello, On Wed, Aug 24, 2011 at 1:53 PM, Paul Davis paul.joseph.da...@gmail.com wrote: I bet you're hitting a bug we just recently fixed on trunk. Basically, there was a possibility that errors in some of the JS functions would end up causing a couchjs process to be come unusable without removing it from the list. Eventually there wouldn't be any spots left and clients would get timeouts like you see. Patch is at [1]. If it doesn't apply cleanly, you really only need the bits from couch_os_process.erl and couch_query_servers.erl. The rest is just test code. https://github.com/apache/couchdb/commit/95da6f6f4246d2e8e86a3cf92ddf6487e46c10e9 Right after sending this I finally saw what the issue was, our bug report here: https://issues.apache.org/jira/browse/COUCHDB-1257?focusedCommentId=13090484#comment-13090484 as a side effect was leaving lingering processes ultimately leading to instability of couch. We are working to patch our reduce issue, and will look at applying that commit perhaps once it hits mainline couch? Thanks for the quick response, -Chris
[jira] [Commented] (COUCHDB-1257) Reduction detection in share/server/views.js runReduce to primitive for some use cases and overall reduce architecture
[
https://issues.apache.org/jira/browse/COUCHDB-1257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090551#comment-13090551
]
Paul Joseph Davis commented on COUCHDB-1257:
Totally just realized this is the same issue as the mailing list. Either way,
yeah...
Reduction detection in share/server/views.js runReduce to primitive for some
use cases and overall reduce architecture
--
Key: COUCHDB-1257
URL: https://issues.apache.org/jira/browse/COUCHDB-1257
Project: CouchDB
Issue Type: Bug
Components: JavaScript View Server
Affects Versions: 1.1
Environment: Architecture independent
Reporter: Chris Stockton
Priority: Minor
Labels: javascript, patch
There is two parts to this report, one is my question I have in regards to
the protocol in which couchdb actually talks to the view server
(server/main.js), the second part as discussed here [1] in short is the
growth detection [2] done in share/server/views.js.
First issue:
While troubleshooting and tracking down this issue, I saw that when running a
reduce function for every doc called, a separate read line is delegated a
reset call. I do not know the internals of couchdb but it seems like it could
be a performance improvement if on a rereduce call, as it appears in the
server Loop function every time reduce or rereduce is called the state is
first reset with reset, to make a minor re-factor and reset the state
within the Views dispatcher itself. This saves some likely blocking io back
and forth.
Reduction detection:
The portion of this issue that actually has affected us is the detection
done, I think that it should be changed to require a sampling of data, the
easiest fix is to simply change (the very arbitrary) [2] into a large length,
such as * 10 or something, but the upper bounds will always be reached for
some data set. A new algorithm could take in consideration the total reduce
calls thus far, as well as the initial size of the very first reduce call, to
see if a large aggregation of some sort is being created.
Suggested change:
Change the signature of rereduce call to main.js to include the total count
of rereduce calls thus far, or give some kind of handle where this logic
could be reliably kept track of inside of main.js, as it stands now your only
hope would be to use the reduce function itself as a lookup, which is not
safe under all edge cases. With this information you can keep the initial
rereduce calls result size, and the total number of calls, to make sure that
aside from initial aggregation or structure initialization the structure is
remaining constant.
There would be several other routes to go to fix this, but I think it is
valid to be considered. I could volunteer my time to write a patch if a
consensus could be reached on behavior, if not denied all together.
-Chris
[1] http://www.mail-archive.com/user@couchdb.apache.org/msg15250.html
[2] if (State.query_config State.query_config.reduce_limit
[2] reduce_length 200 ((reduce_length * 2)
State.line_length)) {
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1258) eheap_alloc OOM errors when attempting to build selected views
[
https://issues.apache.org/jira/browse/COUCHDB-1258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090592#comment-13090592
]
Filipe Manana commented on COUCHDB-1258:
Robert,
That solution works perfectly fine for my case. Memory consumption doesn't go
beyond ~40Mb (for the whole Erlang VM). Perhaps a lower limit (to match the
work queue max items) could be generally better, I haven't tested it however.
eheap_alloc OOM errors when attempting to build selected views
--
Key: COUCHDB-1258
URL: https://issues.apache.org/jira/browse/COUCHDB-1258
Project: CouchDB
Issue Type: Bug
Affects Versions: 1.1
Environment: CentOS 5.6, 1GB RAM (approx 800MB free)
CouchDb 1.1.0, Erlang R14B-03, js lib 1.7.0-8 EPEL RPM build
Couch database exhibiting this behaviour:
{db_name:activity_new,doc_count:593274,doc_del_count:4743352,update_seq:10559287,purge_seq:0,compact_running:false,disk_size:3366396013,instance_start_time:1314097423985726,disk_format_version:5,committed_update_seq:10559287}
Reporter: James Cohen
Priority: Critical
We spotted OOM errors crashing our CouchDb instance when attempting to
rebuild selected views. CouchDb was dying with the following messages (worth
noting that the type reported varies between heap/old_heap
eheap_alloc: Cannot allocate 478288480 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type heap).
eheap_alloc: Cannot allocate 747325720 bytes of memory (of type old_heap).
eheap_alloc: Cannot allocate 597860600 bytes of memory (of type old_heap).
By modifying the view I was able to find a view that could consistently crash
the server and another that ran fine. They are as follows:
Runs out of memory v.quickly
{
_id: _design/cleanup,
_rev: 5-e004fbab278355e9d08763877e5a8295,
views: {
byDate: {
map: function(doc) { if (! doc.action) emit([doc.date], doc); }
}
}
}
Runs fine with minimal memory usage (returns 88128 docs in the view)
{
_id: _design/cleanup,
_rev: 6-3823be6b72ca2441e235addfece6900c,
views: {
byDate: {
map: function(doc) { if (doc.action) emit([doc.date], doc); }
}
}
}
The only difference between the two is the negation of the if conditional.
memory usage was monitored with top on the machine while the view was being
built. Under correct behaviour I could see beam.smp using just 3 or 4% of the
server's memory. With the view that causes problems that memory usage
increased until the RAM/swap on the server was exhausted (as you can see from
the error messages around 500/700MB)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (COUCHDB-1259) Replication ID is not stable if local server has a dynamic port number
[
https://issues.apache.org/jira/browse/COUCHDB-1259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13090680#comment-13090680
]
Jason Smith commented on COUCHDB-1259:
--
Hi, Jens. Yes, I take your point to stick to the specific topic. Thanks.
To summarize my long rants: A Couch UUID already exists. A database UUID won't
work. Allowing the client to provide the replication ID would be great!
Replication ID is not stable if local server has a dynamic port number
--
Key: COUCHDB-1259
URL: https://issues.apache.org/jira/browse/COUCHDB-1259
Project: CouchDB
Issue Type: Bug
Components: Replication
Affects Versions: 1.1
Reporter: Jens Alfke
I noticed that when Couchbase Mobile running on iOS replicates to/from a
remote server (on iriscouch in this case), the replication has to fetch the
full _changes feed every time it starts. Filipe helped me track down the
problem -- the replication ID is coming out different every time. The reason
for this is that the local port number, which is one of the inputs to the
hash that generates the replication ID, is randomly assigned by the OS. (I.e.
it uses a port number of 0 when opening its listener socket.) This is because
there could be multiple apps using Couchbase Mobile running on the same
device and we can't have their ports colliding.
The underlying problem is that CouchDB is attempting to generate a unique ID
for a particular pair of {source, destination} databases, but it's basing it
on attributes that aren't fundamental to the database and can change, like
the hostname or port number.
One solution, proposed by Filipe and me, is to assign each database (or each
server?) a random UUID when it's created, and use that to generate
replication IDs.
Another solution, proposed by Damien, is to have CouchDB let the client work
out the replication ID on its own, and set it as a property in the
replication document (or the JSON body of a _replicate request.) This is even
more flexible and will handle tricky scenarios like full P2P replication
where there may be no low-level way to uniquely identify the remote database
being synced with.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
