On Wed, Dec 7, 2011 at 11:56 PM, Kelly McLaughlin <[email protected]> wrote: > > Fisher, > > In the cases where you see file size discrepancies, are you able to retrieve > those files intact from riak? Having a node in the cluster with an unstable > network connection should not cause data loss because replicas should be > saved to other nodes in your cluster, but it may cause some issues with being > able to properly access those files via luwak. The chances of data being lost > when a riak node goes down is dependent on what you mean by the node going > down. In most cases, the replicas will take care of it, but in the case that > multiple machines in your cluster fail, you should be familiar with the sync > strategies for the backend you are using. For example, you can read about the > bitcask sync strategies here: http://wiki.basho.com/Bitcask.html. > > Kelly
Kelly, We are only able to retrieve the data in Riak that it reports it has (from luwak_file:length()), but it does not match what we attempted to put into Riak. I brought up the network questions only as a theory, otherwise I don't know any reason why we are having this problem. It is affecting about 0.0012% of the files we are putting into Riak (via luwak), and without any way of determining if a write operation was successful we don't know how to resolve this problem. We tried calling luwak_put_stream:flush() before to "guarantee" that our data would write because in the docs for luwak_put_stream:send() it's a little misleading about what it does, but on buffer block sizes < 1000000 the flush() does not do what we expected at all: RiakNode = '[email protected]'. {ok, Riak} = riak:client_connect(RiakNode). {ok, RiakFile} = luwak_file:create(Riak, <<"testabc22yeah3">>, dict:new()). Ps = luwak_put_stream:start(Riak, RiakFile, 0, 1000000). luwak_put_stream:send(Ps, <<"1234">>). luwak_put_stream:flush(Ps). {ok, RiakFile2} = luwak_file:get(Riak, <<"testabc22yeah3">>). luwak_file:length(Riak, RiakFile2). %% Length is 4 luwak_put_stream:send(Ps, <<"56789">>). luwak_put_stream:flush(Ps). {ok, RiakFile3} = luwak_file:get(Riak, <<"testabc22yeah3">>). luwak_file:length(Riak, RiakFile3). %% Length is 13, but we would have expected 9 luwak_io:get_range(Riak, RiakFile3, 0, 13). %% [<<"1234">>,<<"123456789">>] >From the documentation though it says: "If you want guarantees that your write is actually written then you need to call flush/1 on the put stream." So besides that, is there any way to actually tell if the node received the write request? Thanks, Fisher > > > > On Dec 5, 2011, at 12:56 AM, vuleetu wrote: > > Hi, all > > I am working on a storage system which receives data from the client and > stores it to luwak. The server also records the file size of that file which > is uploaded successfully to the database, but i found it is not the same size > as the data in luwak even after several days, so that is not a consistency > problem. > > The following is the main process: > > 1) check if file exists already > luwak_file:exists() > > 2) if it doesn't exist, will use luwak_file:create to create one, or else > will use luwak_file:get to get that file handle, > > 3) use the last offset which is recorded in database, and the create put > stream > luwak_put_stream:start(Riak, RiakFile, Offset, ?STREAM_PUT_TTL) > > 4) receive data from client, and write to put stream created above > gen_tcp: recv() > luwak_put_stream:send(PutStream, Binary) > > Is there any possibility that data is lost because one of the riak node's > network is not stable? Because i found when i call luwak_put_stream:put(), it > just send message contain the data to one process, so i dont know if data is > lost or sent successfully. Even if I call flush, it is still a request > message to that process. Is there any function to tell me if the write was > successful or not? What about if that riak node goes down in the middle of > writing data to luwak, what would happen, will it cause the data to be lost? > > > Thanks, > Fisher > _______________________________________________ > riak-users mailing list > [email protected] > http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com > > _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
