Re: Atomicity of if_not_modified?
Justin Sheehy wrote Yes, I confirmed this earlier in this thread: http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-January/010672.html -Justin Sorry for the double-post, that was not intended. What about setting DW=DR=2? In this case only at most one partition can succeed at writing, and the partition that fails to durably write the value would be interpreted as 'failed', and one could simply retry a bit later. What do you think? Kaspar -- View this message in context: http://riak-users.197444.n3.nabble.com/Atomicity-of-if-not-modified-tp4026430p4026482.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: Stop the node to rename with `riak stop` Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». Rename the node in vm.args. Delete the ring directory. Start the node with `riak start`. It will come up as a single instance which you can verify with `riak-admin member-status`. Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» Plan the changes with `riak-admin cluster plan` Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) 3. Use cluster replace a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Same MR query, different results every run...........
Hi,
If I am correct, adding a reduce function should return the same number
of items.
I'm a riak noob but I faced some similar issue while testing with a map
function only. Adding a reduce fixed it.
I believe as the map fans out to multiple nodes, whichever node returns
data first is written to output and not collected by a reduce stage.
Please correct me if I'm wrong.
Thanks
--
Mridul
On Sunday 06 January 2013 11:07 AM, David Montgomery wrote:
Hi,
Here is my my mapper...
query.map('''
function(value, keyData, arg) {
if(value.length == 0){
return [];
}else{
var data = Riak.mapValuesJson(value)[0];
var obj = {};
if(data['campaign_id']=='%s'){
try{
var alt_key = data['ckid'] + '||' + data['gid'] +
'||' + data['ts_hms'];
}
catch(err){
var alt_key = 'error';
}
obj[alt_key] = 1;
return [ obj ];
}else{
return [];
}
}
}''' % campaign_id)
When I run the the query repeatedly, over and over, about every 2
seconds I get the below. A few times I get 14 rows and a few times I
get 13 then back to 14 etc. So.why? There should be no
variation. I have a three node cluster, two cores, 4 gigs or ram on
ubuntu 12.06 using the latest riak.
TOTAL: 14
3dc3f58f-faea-4751-94b5-8a9a076d4b3f||CAESEGYMM1Q34DV8Ev0i12IVKdY||2012-12-31
08:36:21 1
b4d82fa0-5cd4-4813-a150-554ebca30f1f||CAESEM98NHldIIyAzY0CIUnKudw||2013-01-04
06:18:37 1
8743af22-a664-4b60-ac59-b79d52c12e9e||CAESEH2PIdEYXvk3Dsg2_vF6Qcc||2013-01-04
09:13:30 1
cef36621-527c-4b7a-be6f-5842e13a1350||CAESEHsyPPSizUsT-j31I-nCLzQ||2013-01-05
12:50:22 1
663fb22d-c60d-46b7-8b5b-c9be103c2084||CAESEDtHYmtttm7DBCRpCSU9zYE||2013-01-04
08:55:06 1
e2b6afda-b838-48d5-a449-7b568b9f6b04||CAESEBciJaIqccs2584wIgdsOqc||2013-01-04
04:02:13 1
66aa05fe-9c55-43b2-93ae-c8cb19d097d7||CAESEBuVyK-X_iNGaiiLhPsT0TE||2013-01-02
01:29:38 1
0969a7ca-4324-4118-9038-b6fc11f08a36||CAESENwCD1bw1VvtIamGBCUl_zk||2013-01-02
00:55:01 1
f78b77f6-a08c-4f07-b982-7b2cdcefba4f||CAESEJiWNlcbRN7Sx9o2FB7fbaU||2012-12-29
05:22:46 1
8050e5a7-1583-459a-983f-55feaf0e2a6c||CAESED2NyW9XDEbiKb1UD4sTzvI||2013-01-05
12:18:59 1
58b84566-ad3a-4a3f-91bd-1c61986fbadb||CAESELQcGkigDvXrtRDgOlw9rX0||2013-01-04
16:19:25 1
0db77e8d-ed94-43cf-8860-b4e43dfa24aa||CAESECbwN7VY6o8om79mZ905GIA||2013-01-02
16:15:34 1
67e79552-7e06-44bd-9e95-87f7cb634de3||CAESEFA6fd_C1PBslKgOj6_BI28||2012-12-29
05:23:11 1
ffc3c6ae-beee-4dfe-b41d-ec3a72bddf67||CAESEN_MAXs55jCPIwuyvfTZIZc||2012-12-28
07:56:03 1
TOTAL: 13
b4d82fa0-5cd4-4813-a150-554ebca30f1f||CAESEM98NHldIIyAzY0CIUnKudw||2013-01-04
06:18:37 1
8743af22-a664-4b60-ac59-b79d52c12e9e||CAESEH2PIdEYXvk3Dsg2_vF6Qcc||2013-01-04
09:13:30 1
cef36621-527c-4b7a-be6f-5842e13a1350||CAESEHsyPPSizUsT-j31I-nCLzQ||2013-01-05
12:50:22 1
663fb22d-c60d-46b7-8b5b-c9be103c2084||CAESEDtHYmtttm7DBCRpCSU9zYE||2013-01-04
08:55:06 1
e2b6afda-b838-48d5-a449-7b568b9f6b04||CAESEBciJaIqccs2584wIgdsOqc||2013-01-04
04:02:13 1
66aa05fe-9c55-43b2-93ae-c8cb19d097d7||CAESEBuVyK-X_iNGaiiLhPsT0TE||2013-01-02
01:29:38 1
0969a7ca-4324-4118-9038-b6fc11f08a36||CAESENwCD1bw1VvtIamGBCUl_zk||2013-01-02
00:55:01 1
f78b77f6-a08c-4f07-b982-7b2cdcefba4f||CAESEJiWNlcbRN7Sx9o2FB7fbaU||2012-12-29
05:22:46 1
8050e5a7-1583-459a-983f-55feaf0e2a6c||CAESED2NyW9XDEbiKb1UD4sTzvI||2013-01-05
12:18:59 1
58b84566-ad3a-4a3f-91bd-1c61986fbadb||CAESELQcGkigDvXrtRDgOlw9rX0||2013-01-04
16:19:25 1
3dc3f58f-faea-4751-94b5-8a9a076d4b3f||CAESEGYMM1Q34DV8Ev0i12IVKdY||2012-12-31
08:36:21 1
67e79552-7e06-44bd-9e95-87f7cb634de3||CAESEFA6fd_C1PBslKgOj6_BI28||2012-12-29
05:23:11 1
0db77e8d-ed94-43cf-8860-b4e43dfa24aa||CAESECbwN7VY6o8om79mZ905GIA||2013-01-02
16:15:34 1
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Benchmark test for bitcask
Hi All, Anyone of you have did the benchmark testing on riak with bitcask as backend ? If yes then can you guys share those information with me. -- Thanks and Regards, Vijay Gupta ( ??) Engineering Group. Hostway Corporation NOTICE: This email and any file transmitted are confidential and/or legally privileged and intended only for the person(s) directly addressed. If you are not the intended recipient, any use, copying, transmission, distribution, or other forms of dissemination is strictly prohibited. If you have received this email in error, please notify the sender immediately and permanently delete the email and files, if any. smime.p7s Description: S/MIME Cryptographic Signature ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Benchmark test for bitcask
Hi Vijay, I would recommend performing your own benchmarks over taking stock in others'. There is a benchmarking utility called basho_bench[0] that you could use to benchmark Bitcask and configure[1] it to your own specifications. [0] https://github.com/basho/basho_bench [1] http://docs.basho.com/riak/latest/cookbooks/Benchmarking/ Tom On Mon, Jan 7, 2013 at 4:37 AM, Vijay Gupta vija...@affinity.com wrote: Hi All, Anyone of you have did the benchmark testing on riak with bitcask as backend ? If yes then can you guys share those information with me. -- Thanks and Regards, Vijay Gupta (વિજય ગુપ્તા) Engineering Group. Hostway Corporation NOTICE: This email and any file transmitted are confidential and/or legally privileged and intended only for the person(s) directly addressed. If you are not the intended recipient, any use, copying, transmission, distribution, or other forms of dissemination is strictly prohibited. If you have received this email in error, please notify the sender immediately and permanently delete the email and files, if any. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: store performance
Could you provide an example of using Multi-threading on the client? -- View this message in context: http://riak-users.197444.n3.nabble.com/store-performance-tp4026462p4026488.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: - Stop the node to rename with `riak stop` - Mark it 'down' *from another node in the cluster *using `riak-admin down «old nodename». - Rename the node in vm.args. - Delete the ring directory. - Start the node with `riak start`. - It will come up as a single instance which you can verify with `riak-admin member-status`. - Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` - Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» - Plan the changes with `riak-admin cluster plan` - Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? Do let me know your thoughts. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
riak java client causing OOMs on high latency
We're seeing instances of a JVM app which talks to riak run out of memory when riak operations rise in latency or riak becomes otherwise unresponsive. A heap dump of the JVM at the time of the OOM show that 91% of the 1G (active) heap is consumed by large byte[] instances. In our case 3 of those byte[]s are in the 200MB range with size dropping off after that. The byte[] instances cannot be traced back to a specific variable as their references appear to be stack-allocated local method variables. But, based on the name of the thread, we can tell that the thread is doing a store operation against riak@localhost. Inspection of the data in one of these byte[]s shows what looks like an r_object response with headers and footer boilerplate around our object payload. This 200+MB byte[] is filled with 0s after the 338th element which is really confusing and indicates that far too much space is being allocated to read the protobuf payload. Here's a dump of one of these instances: https://gist.github.com/40ef9b2ff561e973a72c It's also worth mentioning that, according to /stats, get_fsm_objsize_100 is consistently under 1MB so there is no reason to think that our objects are actually this large. At this point I'm suspicious of the following code creating too large a byte[] from possibly too large a return from dis.readInt() https://github.com/basho/riak-java-client/blob/master/src/main/java/com/basho/riak/pbc/RiakConnection.java#L110 Unsure if that indicates a problem in the driver or the server-side erlang protobuf server. Suspicious that requests pile up and many of these byte[]s are hanging out--enough to cause an OOM. It's possible that they are always very large, but are short-lived enough as to not cause a problem until latencies rise increasing their numbers briefly. Thoughts? Thanks, D ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
Matt: You would need to use (or implement your own) DNS service that you could programmatically access--Route 53 has an API that you could use to create DNS entries that point to the internal addresses of your nodes. In very carefully re-reading the thread Deepak mentions, one problem that will occur is that each node needs to be able to resolve the other nodes by name also. The only way for this to occur reasonably, would be to register the internal addresses with a single point that they share. Some examples of free services that you might use for this are DynDns[1], DNSDynamic[2], or DNS-O-Matic[3]. I have also seen some projects floating around the web that might enable you to create a self-hosted dynamic DNS like opendyn[4] and GnuDIP[5]; however, I have had no occasion to use something like this in my own environment. Some additional discussion about creating your own Dynamic DNS server is also at http://unix.stackexchange.com/questions/29049/how-to-create-a-custom-dynamic-dns-solution Hope this helps! Charlie [1] http://www.dyn.com [2] http://www.dnsdynamic.org [3] http://www.dnsomatic.com [4] http://code.google.com/p/opendyn/ [5] http://gnudip2.sourceforge.net/ On Jan 7, 2013, at 5:00 PM, Matt Black matt.bl...@jbadigital.com wrote: Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: Stop the node to rename with `riak stop` Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». Rename the node in vm.args. Delete the ring directory. Start the node with `riak start`. It will come up as a single instance which you can verify with `riak-admin member-status`. Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» Plan the changes with `riak-admin cluster plan` Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) 3. Use cluster replace a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? b . What happens if 2 nodes go down and one was joined with the other. They
Re: Riak and host names
Hi Matt, Just to add to Charlie's suggestion, you could take a look at EC2 elastic IP addresses which would allow you to permanently map a public and private address to an EC2 instance, assignDNS hostnames and not have them change on reboot[1] [1] http://aws.amazon.com/articles/1346 Regards Richard On 7 Jan 2013, at 23:03, Charlie Voiselle cvoise...@basho.com wrote: Matt: You would need to use (or implement your own) DNS service that you could programmatically access--Route 53 has an API that you could use to create DNS entries that point to the internal addresses of your nodes. In very carefully re-reading the thread Deepak mentions, one problem that will occur is that each node needs to be able to resolve the other nodes by name also. The only way for this to occur reasonably, would be to register the internal addresses with a single point that they share. Some examples of free services that you might use for this are DynDns[1], DNSDynamic[2], or DNS-O-Matic[3]. I have also seen some projects floating around the web that might enable you to create a self-hosted dynamic DNS like opendyn[4] and GnuDIP[5]; however, I have had no occasion to use something like this in my own environment. Some additional discussion about creating your own Dynamic DNS server is also at http://unix.stackexchange.com/questions/29049/how-to-create-a-custom-dynamic-dns-solution Hope this helps! Charlie [1] http://www.dyn.com [2] http://www.dnsdynamic.org [3] http://www.dnsomatic.com [4] http://code.google.com/p/opendyn/ [5] http://gnudip2.sourceforge.net/ On Jan 7, 2013, at 5:00 PM, Matt Black matt.bl...@jbadigital.com wrote: Thanks for this Charlie. I'm running a production Riak cluster on AWS which runs constantly, and I've been wondering how I might be able to easliy stop and start AWS nodes for a testing and benchmarking cluster (to save on cost). By using the 'riaknode1.priv' hostname method you describe, would I be able to stop and then restart a whole cluster of nodes at once? (As described by Deepak, AWS assigns new IPs when a VM starts). Thanks Matt On 8 January 2013 01:31, Charlie Voiselle cvoise...@basho.com wrote: Deepak: When you name a node in app.config with -name it has to have a '.' in it, like r...@hostname.net As you have surmised, you can get around that if you use the -sname argument instead. They have to be done consistently. In your example, had you used the -sname argument, `riak@riaknode1` would work. Making a host entry `riaknode1.priv` that points to the local address would work with the -name argument. The inportant thing about -name and -sname is that they can't mix within a cluster. Cluster replace is designed to replace a node with a new one and transfer all the partitions. You can cheat and use it to rename a node though. The process to do this would look like the following: • Stop the node to rename with `riak stop` • Mark it 'down' from another node in the cluster using `riak-admin down «old nodename». • Rename the node in vm.args. • Delete the ring directory. • Start the node with `riak start`. • It will come up as a single instance which you can verify with `riak-admin member-status`. • Join the node to the cluster with `riak-admin cluster join «cluster nodename» ` • Set it to replace the old instance of itself with `riak-admin cluster replace «old nodename» «new nodename» • Plan the changes with `riak-admin cluster plan` • Commit the changes with `riak-admin cluster commit` As you can see, this is a very large effort, so best to use hostnames that aren't moving around. Apologies for you getting this twice, Deepak. I failed to reply to the list as well. Hope this makes sense... Charlie On Jan 1, 2013, at 2:43 PM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. 1. Preventive measures Someone on this thread dated May 2011 suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? 2. Use -sname Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of
Re: Same MR query, different results every run...........
Hi,
i do have a reduce phase
On Tue, Jan 8, 2013 at 12:08 AM, Mridul Kashatria mri...@readwhere.comwrote:
Hi,
If I am correct, adding a reduce function should return the same number of
items.
I'm a riak noob but I faced some similar issue while testing with a map
function only. Adding a reduce fixed it.
I believe as the map fans out to multiple nodes, whichever node returns
data first is written to output and not collected by a reduce stage.
Please correct me if I'm wrong.
Thanks
--
Mridul
On Sunday 06 January 2013 11:07 AM, David Montgomery wrote:
Hi,
Here is my my mapper...
query.map('''
function(value, keyData, arg) {
if(value.length == 0){
return [];
}else{
var data = Riak.mapValuesJson(value)[0];
var obj = {};
if(data['campaign_id']=='%s'){
try{
var alt_key = data['ckid'] + '||' + data['gid'] + '||'
+ data['ts_hms'];
}
catch(err){
var alt_key = 'error';
}
obj[alt_key] = 1;
return [ obj ];
}else{
return [];
}
}
}''' % campaign_id)
When I run the the query repeatedly, over and over, about every 2
seconds I get the below. A few times I get 14 rows and a few times I get
13 then back to 14 etc. So.why? There should be no variation. I have
a three node cluster, two cores, 4 gigs or ram on ubuntu 12.06 using the
latest riak.
TOTAL: 14
3dc3f58f-faea-4751-94b5-8a9a076d4b3f||CAESEGYMM1Q34DV8Ev0i12IVKdY||2012-12-31
08:36:21 1
b4d82fa0-5cd4-4813-a150-554ebca30f1f||CAESEM98NHldIIyAzY0CIUnKudw||2013-01-04
06:18:37 1
8743af22-a664-4b60-ac59-b79d52c12e9e||CAESEH2PIdEYXvk3Dsg2_vF6Qcc||2013-01-04
09:13:30 1
cef36621-527c-4b7a-be6f-5842e13a1350||CAESEHsyPPSizUsT-j31I-nCLzQ||2013-01-05
12:50:22 1
663fb22d-c60d-46b7-8b5b-c9be103c2084||CAESEDtHYmtttm7DBCRpCSU9zYE||2013-01-04
08:55:06 1
e2b6afda-b838-48d5-a449-7b568b9f6b04||CAESEBciJaIqccs2584wIgdsOqc||2013-01-04
04:02:13 1
66aa05fe-9c55-43b2-93ae-c8cb19d097d7||CAESEBuVyK-X_iNGaiiLhPsT0TE||2013-01-02
01:29:38 1
0969a7ca-4324-4118-9038-b6fc11f08a36||CAESENwCD1bw1VvtIamGBCUl_zk||2013-01-02
00:55:01 1
f78b77f6-a08c-4f07-b982-7b2cdcefba4f||CAESEJiWNlcbRN7Sx9o2FB7fbaU||2012-12-29
05:22:46 1
8050e5a7-1583-459a-983f-55feaf0e2a6c||CAESED2NyW9XDEbiKb1UD4sTzvI||2013-01-05
12:18:59 1
58b84566-ad3a-4a3f-91bd-1c61986fbadb||CAESELQcGkigDvXrtRDgOlw9rX0||2013-01-04
16:19:25 1
0db77e8d-ed94-43cf-8860-b4e43dfa24aa||CAESECbwN7VY6o8om79mZ905GIA||2013-01-02
16:15:34 1
67e79552-7e06-44bd-9e95-87f7cb634de3||CAESEFA6fd_C1PBslKgOj6_BI28||2012-12-29
05:23:11 1
ffc3c6ae-beee-4dfe-b41d-ec3a72bddf67||CAESEN_MAXs55jCPIwuyvfTZIZc||2012-12-28
07:56:03 1
TOTAL: 13
b4d82fa0-5cd4-4813-a150-554ebca30f1f||CAESEM98NHldIIyAzY0CIUnKudw||2013-01-04
06:18:37 1
8743af22-a664-4b60-ac59-b79d52c12e9e||CAESEH2PIdEYXvk3Dsg2_vF6Qcc||2013-01-04
09:13:30 1
cef36621-527c-4b7a-be6f-5842e13a1350||CAESEHsyPPSizUsT-j31I-nCLzQ||2013-01-05
12:50:22 1
663fb22d-c60d-46b7-8b5b-c9be103c2084||CAESEDtHYmtttm7DBCRpCSU9zYE||2013-01-04
08:55:06 1
e2b6afda-b838-48d5-a449-7b568b9f6b04||CAESEBciJaIqccs2584wIgdsOqc||2013-01-04
04:02:13 1
66aa05fe-9c55-43b2-93ae-c8cb19d097d7||CAESEBuVyK-X_iNGaiiLhPsT0TE||2013-01-02
01:29:38 1
0969a7ca-4324-4118-9038-b6fc11f08a36||CAESENwCD1bw1VvtIamGBCUl_zk||2013-01-02
00:55:01 1
f78b77f6-a08c-4f07-b982-7b2cdcefba4f||CAESEJiWNlcbRN7Sx9o2FB7fbaU||2012-12-29
05:22:46 1
8050e5a7-1583-459a-983f-55feaf0e2a6c||CAESED2NyW9XDEbiKb1UD4sTzvI||2013-01-05
12:18:59 1
58b84566-ad3a-4a3f-91bd-1c61986fbadb||CAESELQcGkigDvXrtRDgOlw9rX0||2013-01-04
16:19:25 1
3dc3f58f-faea-4751-94b5-8a9a076d4b3f||CAESEGYMM1Q34DV8Ev0i12IVKdY||2012-12-31
08:36:21 1
67e79552-7e06-44bd-9e95-87f7cb634de3||CAESEFA6fd_C1PBslKgOj6_BI28||2012-12-29
05:23:11 1
0db77e8d-ed94-43cf-8860-b4e43dfa24aa||CAESECbwN7VY6o8om79mZ905GIA||2013-01-02
16:15:34 1
___
riak-users mailing
listriak-users@lists.basho.comhttp://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
___
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Riak and host names
I'm not sure how much this will help, but I will throw my two cents into the hat. Given that I don't know the details of your requirements, and I am fairly new to riak and aws, please take everything with a grain of salt. On Tue, Jan 1, 2013 at 11:43 AM, Deepak Balasubramanyam deepak.b...@gmail.com wrote: I took the AWS EC2 riak image for a spin today. I have a query regarding riak nodes and how they behave when the machine reboots. When an EC2 instance reboots, the internal ip / internal DNS / external DNS change. This renders the app.config and -name argument on vm.args incorrect. I was exploring solutions to deal with this problem. *1. Preventive measures* Someone on this thread dated May 2011http://riak-users.197444.n3.nabble.com/EC2-and-node-names-td2892047.html suggested using host file entries that point to the local internal IP address. That does not seem to work. Riak fails with the following error when I add a new entry to /etc/hosts and configure vm.args with -name riak@riaknode1 If you add something that looks like a legitimate FQDN (like riaknode1.mydomaind.com) into /etc/hosts and utilize that instead of just riaknode1, that should work just fine. The caveat is that all nodes you your cluster need to have entries in /etc/hosts for all the other nodes. This is where you may have some management headaches. It seems silly to me that erlang requires you to use a different command line option depending on what the host name looks like, but then there are many things that elrang does that seem silly to me. There my be a good technical explanation, but I can't come up with one. Hostname riaknode1 is illegal I confirmed that riaknode1 pings correctly before starting riak. I guess erlang tries to match the hostname of the system resulting in this failure ? Can anyone throw some light on this ? *2. Use -sname* Is starting the erlang VM with the sname flag an option if it will help prevent the 'illegal hostname' error ? Disclaimer: My knowledge of erlang is close to zilch, so sorry if that option sounded like something you could dismiss easily :) As far as erlang is concerned, you absolutely could do this and it should work fine for riak. But the caveate is that you likely will have to rework the riak init scripts or simply write your own. RabbitMQ is another erlang application I have used and it's init script requires that you use the short name with no dots in it and passes that to erlang with -sname. That bugs me a bit as I perfer to use the FQDN, but the amount of work to manage and maintain my own init scripts was just not work it for something that is seemingly just asthetics. *3. Use cluster replace * a. I understand that the IPs in app.config and vm.args can be replaced with the correct IP on a restart and using a subsequent 'cluster replace' command will do. Will executing the 'cluster plan' and 'cluster commit' commands now produce network chatter ? Rather than micro managing the ips in app.config, I've opted to just have riak listing on any IP by specifying 0.0.0.0 and then restrict access using IP tables to deny access base on the interface name which is generally consistent across all nodes. This means the configs are much more generic and easy to maintain for me. This might be a suitable option for you as well. b . What happens if 2 nodes go down and one was joined with the other. They both have 2 different IP addresses on restart. How will 'cluster replace' work now ? I've not played around with cluster replace much, but given riak likes to do everything by hostname (which it then resolved to an IP), it shouldn't matter if the ip changes as long was the node names don't change and those names resolve to the correct IP addresses. But then that means each time a node comes up, you have to update DNS or /etc/hosts as appropriate which will be a pain. Do let me know your thoughts. Ultimately, I wouldn't want to be building a riak cluster where the IP address of the nodes could/would change any time the node was rebooted. Surely there is a way to make sure the private IP of the nodes say the same across reboots? That will make your life a lot easier. Thanks -Deepak ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
LevelDB compaction and timeouts
I've had a few situations arise where one or two nodes (all it needs is one node) will begin a heavy compaction cycle (determined by using gstat + looking at leveldb LOG files) and ALL queries put through the cluster (it doesn't matter which node) return a timeout. I can fix this situation by killing the node(s) in question then marking them as down; but the minute I bring them up the take everything down again until they complete. Is there a process by which I can keep queries being issued to certain nodes while having them live and dealing with their compaction? It seems highly inconvenient that I have to pick some time at night to bring them up so my users experience as few of the timeout errors are possible... ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: store performance
The client is thread-safe. You can create an instance of it, pass it to multiple threads, and perform multiple store operations in parallel. Thanks, - Roach On Mon, Jan 7, 2013 at 12:46 PM, catchme lhariha...@hotmail.com wrote: Could you provide an example of using Multi-threading on the client? -- View this message in context: http://riak-users.197444.n3.nabble.com/store-performance-tp4026462p4026488.html Sent from the Riak Users mailing list archive at Nabble.com. ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Ripple file attachments
Hi Ariel, On Tue, Dec 18, 2012 at 7:23 PM, Ariel Valentin ar...@arielvalentin.com wrote: Does Ripple provide file attachement functionality similar to paperclip or attachement_fu? I'd like to be able to attach binary data to a Ripple Document object that is being stored in a different bucket. It looks like you can add any property to Ripple::Document [0]. Though it doesn't seem to be documented (as far as I can tell), this could certainly be a binary blob. (Just make sure said blob isn't much bigger than 4MB or so.) Also, since you mentioned it, someone released a Paperclip module for Riak [1] about a month back. Hope that helps. Mark twitter.com/pharkmillups [0] http://basho.github.com/ripple/#Document_Model_Examples [1] https://github.com/xinuc/paperclip-riak Ariel Valentin e-mail: ar...@arielvalentin.com website: http://blog.arielvalentin.com skype: ariel.s.valentin twitter: arielvalentin linkedin: http://www.linkedin.com/profile/view?id=8996534 --- *simplicity *communication *feedback *courage *respect ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
Re: Ripple file attachments
Thanks for the reply. I recently learned about carrierwave, which does not have a dependency on activerecord. I think am going to give that a try. I don't like the idea of having embedded documents in this case but rather a reference to a value in a separate bucket. I will RTFM when I get a chance but is 4MB a magic number? What happens when an entry exceeds that size? Thanks, Ariel --- Sent from my mobile device. Please excuse any errors. On Jan 7, 2013, at 10:04 PM, Mark Phillips m...@basho.com wrote: Hi Ariel, On Tue, Dec 18, 2012 at 7:23 PM, Ariel Valentin ar...@arielvalentin.com wrote: Does Ripple provide file attachement functionality similar to paperclip or attachement_fu? I'd like to be able to attach binary data to a Ripple Document object that is being stored in a different bucket. It looks like you can add any property to Ripple::Document [0]. Though it doesn't seem to be documented (as far as I can tell), this could certainly be a binary blob. (Just make sure said blob isn't much bigger than 4MB or so.) Also, since you mentioned it, someone released a Paperclip module for Riak [1] about a month back. Hope that helps. Mark twitter.com/pharkmillups [0] http://basho.github.com/ripple/#Document_Model_Examples [1] https://github.com/xinuc/paperclip-riak Ariel Valentin e-mail: ar...@arielvalentin.com website: http://blog.arielvalentin.com skype: ariel.s.valentin twitter: arielvalentin linkedin: http://www.linkedin.com/profile/view?id=8996534 --- *simplicity *communication *feedback *courage *respect ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com ___ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
