When starting the erlang console you need to add the deps to your code paths:
erl -pa $PATH_TO_RIAKC/ebin $PATH_TO_RIAKC/deps/*/ebin
This will resolve the module could not be loaded issue.
You my want to try port 8087 instead of 8081. Port 8081 is only valid when
working against devrel.
The 'Client' variable is only valid when using the native erlang client instead
of the PBC. You should call 'riakc_pb_socket:mapred' instead.
Dan
Sent from my iPhone
On Sep 18, 2010, at 8:00 AM, Nils Petersohn <p...@berlin.de> wrote:
> i am trying to execute map reduce via erlang but i have several problems:
>
> nilsens-MacBook-Pro:riakc nils$ erl -pa $PATH_TO_RIAKC/ebin
> Erlang R14B (erts-5.8.1) [source] [64-bit] [smp:2:2] [rq:2] [async-threads:0]
> [hipe] [kernel-poll:false]
>
> Eshell V5.8.1 (abort with ^G)
> 1> code:which(riakc_pb_socket).
> "/Users/nils/dev/riaktip/deps/riakc/ebin/riakc_pb_socket.beam"
> 4> {ok, Pid} = riakc_pb_socket:start_link("192.168.0.100", 8091).
> {ok,<0.42.0>}
> 5> Object = riakc_obj:new(<<"groceries">>, <<"mine">>, <<"eggs & bacon">>).
> {riakc_obj,<<"groceries">>,<<"mine">>,undefined,[],
> undefined,<<"eggs & bacon">>}
> 6> riakc_pb_socket:put(Pid, Object).
>
> =ERROR REPORT==== 18-Sep-2010::13:42:42 ===
> ** Generic server <0.42.0> terminating
> ** Last message in was {req,
> {rpbputreq,<<"groceries">>,<<"mine">>,undefined,
> {rpbcontent,<<"eggs & bacon">>,undefined,
> undefined,undefined,undefined,undefined,
> undefined,undefined,undefined},
> undefined,undefined,undefined},
> 60000}
> ** When Server state == {state,"192.168.0.100",8091,false,false,#Port<0.694>,
> undefined,
> {[],[]},
> 1,[],100}
> ** Reason for termination ==
> ** {'module could not be loaded',
> [{protobuffs,encode,[1,<<"groceries">>,bytes]},
> {riakclient_pb,iolist,2},
> {riakc_pb,encode,1},
> {riakc_pb_socket,send_request,2},
> {riakc_pb_socket,handle_call,3},
> {gen_server,handle_msg,5},
> {proc_lib,init_p_do_apply,3}]}
> ** exception exit: undef
> in function protobuffs:encode/3
> called as protobuffs:encode(1,<<"groceries">>,bytes)
> in call from riakclient_pb:iolist/2
> in call from riakc_pb:encode/1
> in call from riakc_pb_socket:send_request/2
> in call from riakc_pb_socket:handle_call/3
> in call from gen_server:handle_msg/5
> in call from proc_lib:init_p_do_apply/3
> 7> riakc_pb_socket:stream_list_keys(Pid, <<"groceries">>).
> ** exception exit: {noproc,{gen_server,call,
> [<0.42.0>,
> {req,{rpblistkeysreq,<<"groceries">>},
> 60000,
> {14345399,<0.45.0>}}]}}
> in function gen_server:call/2
> 8>
>
> -----------------------------------------------------------------------------------------------------
>
> which module could not be loaded ? even with the erlang version 13B the same
> error.
>
>
>
> -----------------------------------------------------------------------------------------------------
> when i want to send a ping via:
> riakc_pb_socket:ping(Pid).
> than there is no output and the erlang shell is not responding anymore ...
> (probably a timeout)
> ------------------------------------------------------------------------------------------------------
> when i want to use:
> {ok, [R]} = Client:mapred([{<<"groceries">>, .......
>
> than the "Client" is unbound... how can i make an instance of the client?
>
> is there a step-by-step tutorial with the erlang mapreduce query somewhere?
>
> ----------------------------------------------------------------------------------------------------------------------------
> i wanted to "make all" again with the new erlang version (14B) but i got this
> error now:
> ....
> Compiled src/riak_core_handoff_manager.erl
> Compiled src/riak_core_handoff_receiver.erl
> Compiled src/merkerl.erl
> Compiled src/riak_core_claim.erl
> Compiled src/riak_core_node_watcher_events.erl
> Compiled src/riak_core_handoff_sender.erl
> Compiled src/riak_core_ring_events.erl
> Compiled src/riak_core_ring_handler.erl
> Compiled src/riak_core_node_watcher.erl
> Compiled src/riak_core_sup.erl
> Compiled src/riak_core_ring_manager.erl
> src/riak_core_util.erl:130: Warning: ambiguous call of overridden
> auto-imported BIF integer_to_list/2
> - use erlang:integer_to_list/2 or
> "-compile({no_auto_import,[integer_to_list/2]})." to resolve name clash
> make: *** [compile] Error 1
>
> best regards
> nils
>
>
> On Sep 17, 2010, at 5:22 PM, Grant Schofield wrote:
>
>> VMWare is probably contributing to a certain amount of the latency, but the
>> overhead in using the Javascript VMs is most likely where most of the
>> latency is coming from. You can increase the amount of Javascript VMs you
>> have, but you may be hitting a wall with CPU usage (especially because some
>> nodes are under VMWare). Erlang would be much faster but still may have some
>> of the same problems with CPU usage.
>>
>> Grant Schofield
>> Developer Advocate
>> Basho Technologies, Inc.
>>
>> On Sep 17, 2010, at 7:02 AM, Nils Petersohn wrote:
>>
>>> same setup with the tip, did not speedup the map phase, i even generated a
>>> 2Mb post request file:
>>>
>>> {"inputs":[["actionbucket","10000"],["actionbucket","10001"],["actionbucket","10002"],["actionbucket","10003"],["actionbucket","10004"],["actionbucket","10005"],["actionbucket","10006"],["actionbucket","10
>>>
>>> 007"],["actionbucket","10008"],["actionbucket","10009"],["actionbucket","10010"],["actionbucket","10011"],["actionbucket","10012"],["actionbucket","10013"],["actionbucket","10014"],["actionbucket","1001...
>>>
>>> but this did not speedup the process too...
>>>
>>> my computer setup is the following:
>>> Processor Name: Intel Core 2 Duo
>>> Processor Speed: 2.53 GHz
>>> Number Of Processors: 1
>>> Total Number Of Cores: 2
>>> L2 Cache: 3 MB
>>> Memory: 4 GB
>>>
>>> i use win7 on a intel quadcore 6600 cpu with 4gb ram.
>>> the two vmware virtual machines have 2 cpus and ~ 1400mb ram configured +
>>> fedora 12.
>>> the firewalls are turned off completly
>>>
>>> everything is connected with ethernet through a small switch and a static
>>> ips.
>>>
>>> all the nine riak instances have this config (only the ip and port are
>>> changing):
>>> ------------------------------------------------------------------------------------------------------------------------------------------
>>> %% -*- tab-width: 4;erlang-indent-level: 4;indent-tabs-mode: nil -*-
>>> %% ex: ts=4 sw=4 et
>>>
>>> %%
>>> %% etc/app.config
>>> %%
>>> {ring_state_dir, "data/ring"}.
>>> {web_ip, "192.168.0.100"}.
>>> {web_port, 8091}.
>>> {handoff_port, 8101}.
>>> {pb_ip, "192.168.0.100"}.
>>> {pb_port, 8081}.
>>> {bitcask_data_root, "data/bitcask"}.
>>> {sasl_error_log, "log/sasl-error.log"}.
>>> {sasl_log_dir, "log/sasl"}.
>>>
>>> %%
>>> %% etc/vm.args
>>> %%
>>> {node, "d...@192.168.0.100"}.
>>>
>>> %%
>>> %% bin/riak
>>> %%
>>> {runner_script_dir, "$(cd ${0%/*} && pwd)"}.
>>> {runner_base_dir, "${RUNNER_SCRIPT_DIR%/*}"}.
>>> {runner_etc_dir, "$RUNNER_BASE_DIR/etc"}.
>>> {runner_log_dir, "$RUNNER_BASE_DIR/log"}.
>>> {pipe_dir, "/tmp/$RUNNER_BASE_DIR/"}.
>>> {runner_user, ""}.
>>> ------------------------------------------------------------------------------------------------------------------------------------------
>>>
>>> is vmware the bottleneck? should i use erlang to do the mr job?
>>>
>>> best regards
>>> nils
>>>
>>>
>>>
>>> On Sep 17, 2010, at 12:40 AM, Grant Schofield wrote:
>>>
>>>> I think the slowness is coming from the older list keys implementation in
>>>> 0.12.1, list keys has been changed in the tip version of Riak and is quite
>>>> a bit faster now. In addition there have been a lot of improvements to the
>>>> Javascript map reduce implementation that should help the speed of your
>>>> query. For the time being you will need to run Riak tip to get access to
>>>> these enhancements.
>>>>
>>>> Grant Schofield
>>>> Developer Advocate
>>>> Basho Technologies, Inc.
>>>>
>>>>
>>>> On Sep 16, 2010, at 5:17 PM, Nils Petersohn wrote:
>>>>
>>>>> ok, my ring seems ok now.
>>>>> what i did was to change the rel/vars/dev[1,2,3]_vars.config file.
>>>>> in there i was just replacing the ips...
>>>>> this reip thing did not really work out ...
>>>>>
>>>>> here is my riak ring now:
>>>>> (d...@192.168.0.100)1> riak_core_ring_manager:get_my_ring().
>>>>> {ok,{chstate,'d...@192.168.0.100',
>>>>> [{'d...@192.168.0.107',{65,63451889794}},
>>>>> {'d...@192.168.0.105',{13,63451889512}},
>>>>> {'d...@192.168.0.100',{104,63451889512}},
>>>>> {'d...@192.168.0.105',{49,63451889512}},
>>>>> {'d...@192.168.0.100',{32,63451889009}},
>>>>> {'d...@192.168.0.105',{94,63451889253}},
>>>>> {'d...@192.168.0.107',{9,63451889769}},
>>>>> {'d...@192.168.0.100',{97,63451889494}}],
>>>>> {64,
>>>>> [{0,'d...@192.168.0.100'},
>>>>> {22835963083295358096932575511191922182123945984,
>>>>> 'd...@192.168.0.105'},
>>>>> {45671926166590716193865151022383844364247891968,
>>>>> 'd...@192.168.0.107'},
>>>>> {68507889249886074290797726533575766546371837952,
>>>>> 'd...@192.168.0.100'},
>>>>> {91343852333181432387730302044767688728495783936,
>>>>> 'd...@192.168.0.105'},
>>>>> {114179815416476790484662877555959610910619729920,
>>>>> 'd...@192.168.0.107'},
>>>>> {137015778499772148581595453067151533092743675904,
>>>>> 'd...@192.168.0.100'},
>>>>> {159851741583067506678528028578343455274867621888,
>>>>> 'd...@192.168.0.105'},
>>>>> {182687704666362864775460604089535377456991567872,
>>>>> 'd...@192.168.0.100'},
>>>>> {205523667749658222872393179600727299639115513856,
>>>>> 'd...@192.168.0.105'},
>>>>> {228359630832953580969325755111919221821239459840,
>>>>> 'd...@192.168.0.107'},
>>>>> {251195593916248939066258330623111144003363405824,
>>>>> 'd...@192.168.0.100'},
>>>>> {274031556999544297163190906134303066185487351808,
>>>>> 'd...@192.168.0.105'},
>>>>> {296867520082839655260123481645494988367611297792,
>>>>> 'd...@192.168.0.107'},
>>>>> {319703483166135013357056057156686910549735243776,
>>>>> 'd...@192.168.0.100'},
>>>>> {342539446249430371453988632667878832731859189760,
>>>>> 'd...@192.168.0.105'},
>>>>> {365375409332725729550921208179070754913983135744,
>>>>> 'd...@192.168.0.100'},
>>>>> {388211372416021087647853783690262677096107081728,
>>>>> 'd...@192.168.0.105'},
>>>>> {411047335499316445744786359201454599278231027712,
>>>>> 'd...@192.168.0.107'},
>>>>> {433883298582611803841718934712646521460354973696,...},
>>>>> {...}|...]},
>>>>> {dict,0,16,16,8,80,48,
>>>>> {[],[],[],[],[],[],[],[],[],[],[],[],[],[],...},
>>>>> {{[],[],[],[],[],[],[],[],[],[],[],[],...}}}}}
>>>>> (d...@192.168.0.100)2>
>>>>>
>>>>> i am using 0.12.1 on my mac and 0.12 on both vms. i have now a set of
>>>>> 100.000 entrys like this (just for testing):
>>>>> {"id":"42164", "actionTime":"2007-05-11 17:08:55", "action":"some
>>>>> action", "res":"7024", "user":"5", "client":"2787"}
>>>>>
>>>>>
>>>>> and my mr job looks like this (just for testing):
>>>>> {"inputs":"actionbucket",
>>>>> "query":[
>>>>> {"map":{"language":"javascript", "source":
>>>>> "function(values, keyData, arg) {
>>>>>
>>>>> var value = Riak.mapValuesJson(values)[0];
>>>>> if(value.reservation == '4084'){
>>>>> return [value];
>>>>> }
>>>>> return [];
>>>>> }","keep":true}}
>>>>> ],"timeout": 900000
>>>>> }
>>>>>
>>>>>
>>>>> the beam instances are all showing on "top" now, and there is some
>>>>> traffic going back and forth. (~200kb / s)
>>>>>
>>>>> but this job takes like 1:30 min.
>>>>>
>>>>> i know that this is not really comparable with a mysql query because you
>>>>> can do more calculations in the mr job to produce much more special
>>>>> results and the mr job has a ~linear "worktime"... but ~1:30 min is still
>>>>> pretty bad ....
>>>>>
>>>>> is there any way to do much better ?
>>>>>
>>>>> best regards
>>>>> nils
>>>>>
>>>>> On Sep 16, 2010, at 7:08 PM, Grant Schofield wrote:
>>>>>
>>>>>>
>>>>>> On Sep 15, 2010, at 2:40 PM, Nils Petersohn wrote:
>>>>>>
>>>>>>> hello,
>>>>>>>
>>>>>>> i was setting up 9 riak instances:
>>>>>>>
>>>>>>> three on my mac with the appropriate app config
>>>>>>> and six with two virtual machines on a different computer.
>>>>>>>
>>>>>>> all 8 joined the d...@192.168.1.20
>>>>>>> and the join request was sent.
>>>>>>>
>>>>>>> after setting this up:
>>>>>>> i wanted to put data with the java client on d...@192.168.1.20 than i
>>>>>>> got a timeout ?!?
>>>>>>>
>>>>>>
>>>>>> I am curious if you started this node and then changed its name in the
>>>>>> config file? Errors like this can happen if you don't riak-admin reip
>>>>>> the node, also the ring file would be wrong and this could lead to some
>>>>>> of the other errors you saw below. One thing you may want to look at is
>>>>>> the state of your ring from the Riak console using
>>>>>> riak_core_ring_manager:get_my_ring(). That might show any problems with
>>>>>> the ring, feel free to send that along so we can take a look at it.
>>>>>>
>>>>>>> when i put data on one of the other machines than only this machine was
>>>>>>> using cpu time and none of the other ...
>>>>>>> if consistent hashing works like expected, than all the machines should
>>>>>>> show up on "top"
>>>>>>>
>>>>>>> when i did a mapreduce job than only this machine was using cpu time
>>>>>>> and none of the other ...
>>>>>>>
>>>>>>> i had "top" running on all of them.
>>>>>>>
>>>>>>> -------------------------------------------------------
>>>>>>> the other problem is:
>>>>>>>
>>>>>>> when i have 1/2 mio. entrys in one bucket with less than 100 chars for
>>>>>>> each entry
>>>>>>> and i do a really simple mapreduce job, than it takes forever (15
>>>>>>> minutes ...)
>>>>>>> while sql uses .005 secons....
>>>>>>>
>>>>>>> i know that doing a mr on a complete bucket, than it takes very long if
>>>>>>> i don't secify keys in the bucket. but how should i know which keys to
>>>>>>> use ...
>>>>>>
>>>>>> What version of Riak are you using? There has been a fair amount of
>>>>>> improvement to the map reduce system as well as list keys. Are the map
>>>>>> reduce jobs you are running javascript?
>>>>>>
>>>>>>> ------------------------------------------------------
>>>>>>>
>>>>>>> if i put stuff in one bucket and add a machine with the join request,
>>>>>>> how can i rebalance the bucket???? so that the other machine is taking
>>>>>>> some values too.
>>>>>>
>>>>>> This happens automatically. When the new node joins the cluster you
>>>>>> should see handoff messages in the erlang.log.X log file. Rebalancing
>>>>>> is handled by the cluster and shouldn't be done manually.
>>>>>>
>>>>>> Grant Schofield
>>>>>> Developer Advocate
>>>>>> Basho Technologies, Inc.
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> ------------------------------------------------------
>>>>>>>
>>>>>>> i don't understand these issues/behaviors (timeout, 15min. etc.,
>>>>>>> rebalancing), maybe i was setting the one of the three params incorrect
>>>>>>> ? i left everything to the default settings.
>>>>>>>
>>>>>>> thx in advance for any hints...
>>>>>>>
>>>>>>> nils
>>>>>>> _______________________________________________
>>>>>>> riak-users mailing list
>>>>>>> riak-users@lists.basho.com
>>>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>>>
>>>>>
>>>>> Nils M. Petersohn
>>>>> xing.com/profile/Nils_Petersohn
>>>>> blog.srvme.de
>>>>> twitter.com/snackycracky
>>>>> facebook.com/nils.petersohn
>>>>> myspace.com/electrash
>>>>>
>>>>> p...@berlin.de
>>>>> 0049 (0)151 40 511 351
>>>>> skype: nilz_berlin
>>>>>
>>>>> Ebertystr. 47
>>>>> 10249 Berlin
>>>>>
>>>>
>>>
>>> Nils M. Petersohn
>>> xing.com/profile/Nils_Petersohn
>>> blog.srvme.de
>>> twitter.com/snackycracky
>>> facebook.com/nils.petersohn
>>> myspace.com/electrash
>>>
>>> p...@berlin.de
>>> 0049 (0)151 40 511 351
>>> skype: nilz_berlin
>>>
>>> Ebertystr. 47
>>> 10249 Berlin
>>>
>>
>
> Nils M. Petersohn
> xing.com/profile/Nils_Petersohn
> blog.srvme.de
> twitter.com/snackycracky
> facebook.com/nils.petersohn
> myspace.com/electrash
>
> p...@berlin.de
> 0049 (0)151 40 511 351
> skype: nilz_berlin
>
> Ebertystr. 47
> 10249 Berlin
>
>
> _______________________________________________
> riak-users mailing list
> riak-users@lists.basho.com
> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
_______________________________________________
riak-users mailing list
riak-users@lists.basho.com
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
