Fisher,
Yes you will get poorer performance using the synchronous api, but the tradeoff
is you can detect failures as they happen. Alternatively you could use the
asynchronous api and verify the file size matches what it should be at the end
and if not reload the file. The best choice depends on the size of files you're
dealing with, the reliability of the network, and the needs of your
application. However, if I understand correctly, your concern about more open
files with the synchronous api is not warranted. You're not actually opening
more files when you use the synchronous api versus the asynchronous api, you're
just creating more erlang references to the luwak file objects. Hopefully I
interpreted your concern correctly and that helps shed some light on it. Cheers.
Kelly
On Dec 8, 2011, at 11:21 PM, vuleetu wrote:
> Hi, Kelly
>
> Is there any efficiency problem if i use synchronous api, i.e.
>
> luwak_io:put_range(Riak::riak(), File::luwak_file(), Start::int(),
> Data::binary()) -> {ok, WrittenBlocks, NewFile}
>
> Because in general the stream way is better for the continuous
> write operation, it reduce the amount of open file operation.
>
>
> Thanks
> Fisher
>
> On Fri, Dec 9, 2011 at 2:13 PM, Kelly McLaughlin <[email protected]> wrote:
>> Fisher,
>>
>> Currently you're using the asynchronous api so you won't be able to
>> determine if the send calls actually succeed or not. The synchronous api is
>> probably going to be what you want to use. There are some examples in the
>> luwak readme file that should help you get started with it.
>>
>> Kelly
>>
>>
>> On Dec 8, 2011, at 9:57 PM, vuleetu wrote:
>>
>>> Hi, Kelly
>>>
>>> Actually, i did it as you said yesterday. it still return the
>>> unexpected size. If i remove flush() call, it will work. So right now
>>> this is a big problem now. the problem is the portion data of file is
>>> lost. I does not happen every time.
>>>
>>> Is there any way to let me know if the write to riak via luwak is
>>> success or not? because the connection to RiakNode might be down
>>> during the write operation using luwak_put_stream:send().
>>>
>>>
>>> RiakNode = '[email protected]'.
>>> {ok, Riak} = riak:client_connect(RiakNode).
>>>
>>> luwak_put_stream:send()
>>>
>>> %% assume node is down at this point
>>>
>>> %% continue write
>>> luwak_put_stream:send()
>>> luwak_put_stream:send()
>>>
>>> luwak_put_stream:close().
>>>
>>>
>>> I don't know if the write is successful or not, so it may cause the
>>> data lost sometime, is that right?
>>>
>>>
>>> Thanks
>>> Fisher
>>>
>>> On Fri, Dec 9, 2011 at 12:51 AM, Kelly McLaughlin <[email protected]> wrote:
>>>> Fisher,
>>>>
>>>> Yeah don't use flush. Make the calls to send to put the data and then call
>>>> close and you should get the expected behavior.
>>>>
>>>> Kelly
>>>>
>>>> On Dec 7, 2011, at 11:13 PM, vuleetu wrote:
>>>>
>>>>> On Thu, Dec 8, 2011 at 1:57 PM, Kelly McLaughlin <[email protected]> wrote:
>>>>>> Fisher,
>>>>>>
>>>>>> You need to call luwak_put_stream:close(Ps) to force the flush. That
>>>>>> should
>>>>>> get it for you. Cheers.
>>>>>>
>>>>>> Kelly
>>>>>
>>>>> Hi, Kelly
>>>>>
>>>>> RiakNode = '[email protected]'.
>>>>> {ok, Riak} = riak:client_connect(RiakNode).
>>>>> {ok, RiakFile} = luwak_file:create(Riak, <<"testabc22yeah3">>,
>>>>> dict:new()).
>>>>> Ps = luwak_put_stream:start(Riak, RiakFile, 0, 1000000).
>>>>> luwak_put_stream:send(Ps, <<"1234">>).
>>>>> luwak_put_stream:flush(Ps).
>>>>> {ok, RiakFile2} = luwak_file:get(Riak, <<"testabc22yeah3">>).
>>>>> luwak_file:length(Riak, RiakFile2). %% Length is 4
>>>>> luwak_put_stream:send(Ps, <<"56789">>).
>>>>> luwak_put_stream:close(Ps).
>>>>> {ok, RiakFile3} = luwak_file:get(Riak, <<"testabc22yeah3">>).
>>>>> luwak_file:length(Riak, RiakFile3). %% Length still 13
>>>>>
>>>>>
>>>>> After i call close(), I still get the unexpected size(13bytes), is
>>>>> this the expected behavior? Can I not call flush() manually on a
>>>>> stream?
>>>>>
>>>>> this is the version info:
>>>>> luwak-1.1.0
>>>>> riak_core-1.0.0 riak_kv-1.0.0 riak_pipe-1.0.0
>>>>> riak_search-1.0.0 riak_sysmon-1.0.0 riakc-1.2.0
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Fisher
>>>>>
>>>>>
>>>>>>
>>>>>> On Dec 7, 2011, at 10:05 PM, vuleetu wrote:
>>>>>>
>>>>>> luwak_put_stream:send(Ps, <<"56789">>).
>>>>>> luwak_put_stream:flush(Ps).
>>>>>> {ok, RiakFile3} = luwak_file:get(Riak, <<"testabc22yeah3">>).
>>>>>> luwak_file:length(Riak, RiakFile3).
>>>>>>
>>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> riak-users mailing list
>>>>> [email protected]
>>>>> http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
>>>>
>>
_______________________________________________
riak-users mailing list
[email protected]
http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
