Hi All, Since this about couchDB cluster as per requested by Robert Kowalski I just had an IRC chat with Robert Newson who is an expert on Clusters in CouchDB. I am sharing this in order to other new comes (GSOC applicants) also benefit from this
I was pointed to the following resources. http://docs.cloudant.com/guides.html#cap-theorem https://cloudant.com/blog/dynamo-and-couchdb-clusters/ http://bigcouch.cloudant.com/api https://github.com/cloudant/bigcouch Also I was pointed to 'GET /_membership' That's the endpoint that tells you how many nodes are in a cluster. 10,000 nodes is far, far higher than any cluster we've ever seen. hundreds would be a more reasonable maximum expectation and dozens much more common Bigcouch or couchdb are the 2 variants for cluster and stand-alone systems respectively. couchDB perview 2.0 is the later version of bigcouch and cloudant was also highlighted in the chat which refers to the same system. Robert Newson, If I have missed any important point please feel free to share. On Wed, Mar 25, 2015 at 6:13 PM, madhuka udantha <[email protected]> wrote: > looping Robert Newson > > According to Robert Kowalski's suggestion I'm sharing my mockup UI's > regarding to CouchDB Cluster. > > Your feedback will be mostly welcomed. > > Thanks in Advance. > > > > > > > > On Wed, Mar 25, 2015 at 12:20 AM, madhuka udantha < > [email protected]> wrote: > >> Hi, >> >> After communicating with couchdb community got to know that there can be >> 100~ 10000 nodes in cluster. So there will 2 layouts in cluster link >> depending on node count in cluster. >> >> My first UI is for simple count of node in cluster (4 nodes in cluster) >> >> Here is mock UIs where it hit more than 5 to 100000 nodes, I am using >> matrix visual. >> Here is initial look of the page. [Page1v1-initial-page] >> https://app.box.com/s/sn84u6wuzya58trssw11ba6fwtmn862f >> Features: >> 1. User can select the attribute eg: Disk usage, http status, data usage, >> alerts. etc. >> 2. Chart / matrix will be draw with legend >> >> [Page2v1-searching-node] >> https://app.box.com/s/i5uomdhirn96co6acbbrvvviifyw95pb >> Who can we search node in there? >> You will have search in top of it. (it allow to search the node), It will >> show it with red color box. >> Or every node have tool tip to verify node, what is you look for. >> >> >> [Page3v1-selecting-node] >> https://app.box.com/s/rsd5ysld597gvra2nkcn7027bpok37jc >> When user click on node, node panel will be open RHS. >> It have all the details of the node and operation that you can perform in >> node leve >> >> [Page4v1-searching-db_in_node] >> https://app.box.com/s/jk3ojb0khj192ldnyw0eqs1p5t7l485r >> User can search Databases in nodes even >> This searching have advance search query support >> >> [Page5v1-db_view] >> https://app.box.com/s/20hjgtcb7c9ikgfdn7xy8xli4uhorbni >> User can select the database and it will show database details. >> >> >> feed backs for UIs are welcome. >> >> >> On Tue, Mar 24, 2015 at 4:35 PM, madhuka udantha < >> [email protected]> wrote: >> >>> Hi James, >>> >>> Thanks for you feedback. It will help me to improve my proposal and to >>> get better understanding about couchdb cluster >>> >>> Since I am new to couchdb and I did not know much user case that you all >>> have come across. But when you share those ideas. It is help to understand >>> couchdb cluster. All you feedback is important. >>> >>> On Tue, Mar 24, 2015 at 12:07 PM, Mutton, James <[email protected]> >>> wrote: >>> >>>> I can share a few quick thoughts (which I admit is rare, but I happen >>>> to have a few minutes and this hit the top of my stack): >>>> >>>> * How do you envision this overflowing when you have hundreds of >>>> databases within a cluster, or maybe 64 nodes in a cluster. >>>> >>> Attached UIs can only handle very low level nodes (4 to 8) according to >>> layout (we can called it expanded layout). When I am drawing I did not >>> thought that level of count of nodes and databases. Therefore I will >>> revisit the mockup ui. Here is updated UI >>> >>> Here you can have 1 ~ 1000 node even and each node can contain 100 >>> databases. >>> Node are listed and user can pick one of it then it will show it details >>> and databases of it in list. >>> >>> UI for Listing node when nodes count is above 8 in cluster (This is how >>> it look as it have 64 nodes in cluster) >>> https://app.box.com/s/kqg2kadf0xa1zoym9sme22q5m4wbg8i5 >>> [Page10-list-nodes-more-than-8] >>> >>> *Features of the UI:* >>> >>> - Table header support sorting (ascending and descending) by just >>> click on it >>> - User have searching option to search (node/databases by name) >>> - It support advance search query (popup form ui) >>> - Notification listed with icon and warning/alert level is shown by >>> color of icon >>> - User can perform node operation by clicking hyperlink in it's raw >>> >>> >>> Node contains 100 databases (answer) >>> https://app.box.com/s/ggbzr00orkwacsi4cflpvn9d60q7h1st >>> [Page11-list-dbs-in-selected-node] >>> >>> >>> >>> *Features of the UI* >>> >>> - In Database panel contain search that scope to databases of the >>> selected node >>> - When user select database it will show database summary details >>> and operations that can be perform for that database >>> >>> >>> >>>> ** On a related note, drag and drop is cool, but level of difficulty >>>> involved in adding a shard from node-1 increases in direct proportion to >>>> the number of nodes there are. >>> >>> Yes, we can add button ( in database/node level >>> >>> I think the “Add Database” button on a new server should be labeled >>>> “Add Shard” and should present the option of selecting what shards to add. >>>> The add button is also not present on the nodes that have shards already so >>>> it would be good to keep it around somehow. >>>> >>> Yes, It will be label as “Add Shard”. It is possible to show only that >>> is not shard so far. >>> >>> >>> * Speaking then of shards… They don’t appear to be captured here. How >>>> does one know which shard-set each server has? Each database has a number >>>> of shards that all have multiple shard copies and it is not a requirement >>>> that all servers have all shards. None of that is really represented in >>>> these mockups. When looking at a shard, it would be good to know if it is >>>> 00000000-15555554 or aaaaaaa8-bffffffc, or even both. Knowing this and >>>> being able to use it to scale/move hotspots in a cluster is pretty >>>> important for operations, once databases hit a significant size. >>>> >>> Thanks for pointing out. shards will add for visualization. >>> >>> Can you share resources/documentations which that help to understand the >>> couchdb clustering? >>> I like to know what are most important operations regard couch >>> clustering? (that you think good to added in here. since I am new for >>> couchdb and some time I have miss them) >>> >>> >>> >>>> * I don’t think doc-count and deleted-count is useful information at >>>> the overview page >>> >>> Thanks, We can remove doc count details from overview page. If user need >>> to read such details he/she can click 'more' so there it will show them. >>> >>>> >>> >>> and runs the risk of requiring a lot of requests to fulfill (see point 1 >>>> on hundreds of databases). >>>> >>> Sorry, In here I did not get your point >>> >>> * What does compact: false mean? That the database is not small? or >>>> that it’s not actively compacting? And if the latter, wouldn’t that be on >>>> a db-shard basis, not a whole server. >>>> >>> compact running is mapping to 'compact_running'[2] >>> >>> [1] https://wiki.apache.org/couchdb/Compaction >>> >>> >>>> * It would be nice if cluster usage graphing page would show http-stats >>>> as well. Would give people a snapshot indication of where their load is >>>> hot. >>>> >>> Sure, Different status codes will be notify on cluster usage graph. >>> (Mockups UI will shared tonight) >>> >>> Thanks JamesM for your feed backs >>> >>>> >>>> </JamesM> >>>> >>>> >>>> On Mar 23, 2015, at 20:32, madhuka udantha <[email protected]> >>>> wrote: >>>> >>>> > Hi All, >>>> > >>>> > are there any ideas and feedback for above mockup UIs? >>>> > >>>> > Community feedback will be useful in here. :) >>>> > >>>> > On Sun, Mar 22, 2015 at 11:51 PM, madhuka udantha < >>>> [email protected]> >>>> > wrote: >>>> > >>>> >> Hi, >>>> >> >>>> >> Here are the mockup UIs for Visualizing the CouchDB Clusters. It is >>>> just >>>> >> the initial UIs. >>>> >> >>>> >> * Page 1-cluster.png* [ >>>> >> https://app.box.com/s/isjbfop69j2cmz0kg0qar7l5nmupjnfl >>>> >> >>>> >> This Page visualize nodes in the clusters with summary information >>>> for >>>> >> each node. (Name, Memory usage..etc..) with a nice web UI interface. >>>> It is >>>> >> interactive (drag and drop supporting over nodes) >>>> >> >>>> >> *Page 2-cluster-notification-tip.png [* >>>> >> https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb >>>> >> <https://app.box.com/s/euc5megmx5lwseqib4511309ocoypryb>] >>>> >> >>>> >> Top of each panel of the nodes it contains a notification icon if >>>> there is >>>> >> notification. >>>> >> eg: Memory/disk space availability, Error in node etc... >>>> >> Notification will have a simple drop down structure as it is showing. >>>> >> >>>> >> *Page 3-cluster-warning.png* [ >>>> >> https://app.box.com/s/f2ljwxvz2w01f8uepsj4zyarhwbpiz8b] >>>> >> >>>> >> Critical level alert is showen. (Alert level can be set in config and >>>> >> notification center) >>>> >> >>>> >> *Page 4-adding-new-node.png* [ >>>> >> https://app.box.com/s/noygbeyze792bh4t93iaoxywdciau057] >>>> >> >>>> >> By clicking "add new node" you can add new nodes for clusters. UI >>>> >> Consistency is maintained >>>> >> >>>> >> *Page 5-adding-new-node-by-drag-drop.png*[ >>>> >> https://app.box.com/s/gbfsbwb6zt1jdkijve72mnullplecpj7] >>>> >> >>>> >> User interface is supported for drag-drop of databases in nodes. >>>> >> (replication operation in the BE) >>>> >> >>>> >> *Page 6-selected-db-in-node.png*[ >>>> >> https://app.box.com/s/6ojrvpxuvmo9k18dlnf8i4nmibzdx5s9] >>>> >> Each node shows the databases in it. By Clicking on database you can >>>> see >>>> >> detailed view of a particular database. This allows users to >>>> understand >>>> >> memory/data/doc distribution over DBs in particular nodes. >>>> >> >>>> >> *Page 7-custer-mgt-center.png* [ >>>> >> https://app.box.com/s/t9s42g3gwvv6bocbdsxrgjimdrh2ty0z] >>>> >> Each node is shown with disk usage under icon and summary text of >>>> the node >>>> >> is also given. It is easy to identify the disk space - free >>>> >> availability. (In here we need some BE services help) >>>> >> >>>> >> >>>> >> *Page 8-custer-mgt-center-infor.png* [ >>>> >> https://app.box.com/s/vvo4v9ji79l8japo3nm9f4lfbjhdamkc] >>>> >> User can customize notifications for each nodes by level (warning, >>>> infor, >>>> >> etc..) >>>> >> >>>> >> *Page 9-charting-disk-usages.png*[ >>>> >> https://app.box.com/s/fk7eyoyrtg8d3b29xoo0mdho9rr8cmyz] >>>> >> Cluster usage of the system is shown as a bar chart and the >>>> attributes to >>>> >> be shown can be selected according to user's preferences. >>>> >> >>>> >> You ideas are welcome in here. >>>> >> Thanks >>>> >> >>>> >> On Sun, Mar 22, 2015 at 7:34 PM, madhuka udantha < >>>> [email protected] >>>> >>> wrote: >>>> >> >>>> >>> Hi All, >>>> >>> >>>> >>> As I'm drafting my proposal for GSoC 2015 I went through some of the >>>> >>> existing Cluster Management Systems with Visualization. I >>>> recognized some >>>> >>> of the features regarding to Cluster Management HTTP API. (Some >>>> times not >>>> >>> fully documented yet) >>>> >>> Such us partitioning, re-balance..etc. >>>> >>> I was unable to find them on CouchDB API Documentation(only at the >>>> >>> implemented level but not documented yet? ) but I was able to find >>>> some of >>>> >>> the features such as adding nodes to clusters through dev-mailing >>>> list. >>>> >>> Are some of these features regarding cluster management present on >>>> the >>>> >>> API. If so where can I find the relevant documentations/resources? >>>> Even >>>> >>> the source code is fine to me. >>>> >>> >>>> >>> >>>> >>> Since I was able to clearly identify those missing features can I >>>> start a >>>> >>> discussion on a thread on the best feature list for cluster >>>> management on >>>> >>> CouchDB to be added to it on a future release. >>>> >>> My current project could also include these features that are >>>> supported >>>> >>> by the API else I could help on including them. >>>> >>> >>>> >>> [1] http://guide.couchdb.org/draft/clustering.html >>>> >>> Thanks. >>>> >>> BR, >>>> >>> >>>> >>> On Sat, Mar 21, 2015 at 11:40 AM, madhuka udantha < >>>> >>> [email protected]> wrote: >>>> >>> >>>> >>>> >>>> >>>> >>>> >>>> On Sat, Mar 21, 2015 at 12:17 AM, Alexander Shorin < >>>> [email protected]> >>>> >>>> wrote: >>>> >>>> >>>> >>>>> On Fri, Mar 20, 2015 at 9:28 PM, madhuka udantha >>>> >>>>> <[email protected]> wrote: >>>> >>>>>> I like to know what is *different of '_nodes' and 'node'*? >>>> >>>>>> >>>> >>>>>> http://localhost:15986/nodes >>>> -->{"db_name":"nodes","doc_count":3..} >>>> >>>>>> http://localhost:25986/nodes >>>> -->{"db_name":"nodes","doc_count":1..} >>>> >>>>>> http://localhost:35986/nodes >>>> >>>>> -->{"error":"not_found","reason":"no_db_file"} >>>> >>>>> >>>> >>>>> Where did you get these nodes? >>>> >>>>> >>>> >>>> By '~/test/1/couchdb $ dev/run -n 4' I can added 4 nodes. (default >>>> it is >>>> >>>> 3 nodes) >>>> >>>> >>>> >>>> Here is my console print, Yes now I have four node running. >>>> >>>> >>>> >>>> *max@max-VirtualBox ~/test/1/couchdb $ dev/run -n 4* >>>> >>>> *Setup environment... ok* >>>> >>>> *Ensure CouchDB is built... ok* >>>> >>>> *Prepare configuration files... ok* >>>> >>>> *Start node node1... ok* >>>> >>>> *Start node node2... ok* >>>> >>>> *Start node node3... ok* >>>> >>>> *Start node node4... ok* >>>> >>>> *Ensure all nodes are run... ok* >>>> >>>> *Join nodes into cluster... ok* >>>> >>>> *Developers cluster is set up at http://127.0.0.1:15984 >>>> >>>> <http://127.0.0.1:15984>. Time to hack!... * >>>> >>>> >>>> >>>> Each above nodes are running in localhost in port 15986, 25986, >>>> 35986 >>>> >>>> and 45986 >>>> >>>> >>>> >>>> It seems docs are mainly in 15986 (node1) shown by 'doc_count' >>>> value, >>>> >>>> is there any reason for that or is it happening randomly? >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>>> >>>> >>>>> -- >>>> >>>>> ,,,^..^,,, >>>> >>>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> -- >>>> >>>> Cheers, >>>> >>>> Madhuka Udantha >>>> >>>> http://madhukaudantha.blogspot.com >>>> >>>> >>>> >>> >>>> >>> >>>> >>> >>>> >>> -- >>>> >>> Cheers, >>>> >>> Madhuka Udantha >>>> >>> http://madhukaudantha.blogspot.com >>>> >>> >>>> >> >>>> >> >>>> >> >>>> >> -- >>>> >> Cheers, >>>> >> Madhuka Udantha >>>> >> http://madhukaudantha.blogspot.com >>>> >> >>>> > >>>> > >>>> > >>>> > -- >>>> > Cheers, >>>> > Madhuka Udantha >>>> > http://madhukaudantha.blogspot.com >>>> >>>> >>> >>> >>> -- >>> Cheers, >>> Madhuka Udantha >>> http://madhukaudantha.blogspot.com >>> >> >> >> >> -- >> Cheers, >> Madhuka Udantha >> http://madhukaudantha.blogspot.com >> > > > > -- > Cheers, > Madhuka Udantha > http://madhukaudantha.blogspot.com > -- Cheers, Madhuka Udantha http://madhukaudantha.blogspot.com
