Hi Chris, thanks for this report. So, next steps: 1) What Erlang VM version are you using? Does it have native code capabilities? Pasting the banner from an invocation of `erl` would suffice.
2) Two common compiler optimizations are +native and +inline. The first one causes the compiler to generate native machine code instead of BEAM bytecode for a particular module. You need to have a VM that supports it (hence my question above). The second one inlines functions, as you might have guessed. json_bin_is_safe/2 is recursive, so I'm not sure +inline makes a lot of sense in this case. You can activate these by setting the ERLC_FLAGS environment variable when you build. json_bin_is_safe is an old optimization; the VM has improved dramatically in the past 2 years, and its handling of binaries has changed in subtle ways. It could be that the assumptions that went into that code are no longer true. I'd have to do some experimentation myself to say for certain. In the longer term, we've done some work on a fast interface to the excellent Yajl JSON library. JSON processing in C has a different performance profile that may be better suited to Couch's use case. But again, we'll need to measure. Hmm, so that was a bit of a brain dump :/ I'd say try out the native code compilation for mochijson2 if your Erlang build supports it (you should see a [hipe] in the erl banner). Cheers, Adam On Apr 20, 2010, at 7:48 PM, Chris Stockton wrote: > Hello, > > After some profiling we found some pretty big slowdowns as you > suspected in the json encoding. I am very surprised by this, it seems > that there is a lot of room for improvement in the json encoding side > of things. I do not know the inner workings of erlang, but seeing that > this call: mochijson2:json_bin_is_safe/ is 12% of the work of the > response, but is a very simple string check perhaps it can be inlined > into the callers? That would be a significant speedup in most > languages. > > -Chris > > ---------------------------------- > > 24> eprof:total_analyse(). > FUNCTION CALLS TIME > user_drv:io_request/3 266 19 % > mochijson2:json_bin_is_safe/1 59095133 12 % > erlang:port_command/2 2445617 3 % > gen:wait_resp_mon/3 2449385 3 % > mochijson2:json_encode_string/2 6111299 3 % > couch_util:to_hex/1 3114803 3 % > couch_file:read_raw_iolist/3 2272537 3 % > mochijson2:'-json_encode_proplist/2-fun-0-'/3 3317537 2 % > gen_server:loop/6 2281063 2 % > prim_file:drv_command/4 2272768 2 % > lists:foldl/3 4435469 2 % > gen_server:handle_msg/5 2281060 2 % > gen:do_call/4 2449386 2 % > couch_btree:find_first_gteq/5 3914715 2 % > prim_file:drv_get_response/1 2272768 2 % > prim_file:pread/3 2272542 1 % > couch_btree:lookup_kpnode/5 1380410 1 % > couch_file:pread_iolist/2 1049994 1 % > couch_file:handle_call/3 2272598 1 % > gen_server:call/3 2272615 1 % > prim_file:reverse/2 2272542 1 % > prim_file:drv_command/2 2272767 1 % > prim_file:translate_response/2 2272768 1 % > couch_file:pread_term/2 1049994 1 % > couch_btree:less/3 4088828 1 % > couch_file:pread_binary/2 1049994 1 % > couch_util:to_digit/1 5537536 1 % > couch_db_updater:'-init_db/4-fun-0-'/2 4088460 1 % > prim_file:transform_ldata/4 2272542 1 % > mochijson2:json_encode/2 3490228 1 % > erlang:binary_to_term/1 1049995 1 % > erlang:monitor/2 2449861 1 % > couch_btree:lookup/3 862756 1 % > file:pread/3 2272542 1 % > gen_server:reply/2 2276495 1 % > couch_db_updater:less_docid/2 4088764 1 % > lists:splitwith/3 1380530 1 % > prim_file:transform_ldata/1 2272542 1 % > couch_file:remove_block_prefixes/2 2618126 1 % > mochijson2:json_string_is_safe/1 2934384 1 % > gen:call/4 2449386 1 % > erlang:demonitor/1 2449849 1 % > lists:reverse/2 3485939 1 % > lists:reverse/1 2249330 0 % > couch_doc:to_json_obj/2 172539 0 % > prim_file:drv_command/3 2272767 0 % > erlang:iolist_to_binary/1 2273617 0 % > couch_file:calculate_total_read_len/2 2272684 0 % > couch_view:less_json/2 517614 0 % > couch_db:open_doc_int/3 345101 0 % > gen_server:decode_msg/8 2281060 0 % > couch_btree:get_node/2 877445 0 % > couch_db:doc_meta_info/3 172549 0 % > erlang:md5_trap/2 0 0 % > couch_key_tree:get_full_key_paths/4 517831 0 % > prim_file:drv_get_response/2 2272768 0 % > erlang:bump_reductions/1 2272768 0 % > couch_btree:lookup_kvnode/5 345102 0 % > couch_btree:'-lookup_kpnode/5-fun-0-'/3 690205 0 % > erlang:'++'/2 2074206 0 % > couch_doc:to_doc_info_path/1 172853 0 % > mochijson2:json_encode_proplist/2 519798 0 % > couch_httpd_view:send_json_view_row/5 172508 0 % > dict:on_bucket/3 172906 0 % > couch_key_tree:map_simple/3 518815 0 % > gen_event:fetch_msg/5 173052 0 % > gen_event:server_call/4 172890 0 % > couch_key_tree:map/2 345706 0 % > couch_httpd_view:view_row_with_doc/3 172485 0 % > couch_btree:lookup/2 172552 0 % > couch_doc:to_json_rev/2 172539 0 % > dict:get_slot/2 346014 0 % > prim_inet:send/2 172843 0 % > lists:map/2 652977 0 % > couch_httpd_view:'-make_view_fold_fun/6-fun-0 172544 0 %
