Re: Is it possible to read/write a ledger concurrently
I thought we had agreed at some point that the application should do it in the case it needs this feature. That is, every so often the app writer either writes to ZooKeeper its last confirmed write or it sends directly to the reader. Knowing a confirmed write x enables the reader to read up to x.-Flavio On Oct 22, 2010, at 7:39 AM, Benjamin Reed wrote: currently program1 can read and write to an open ledger, but program2 must wait for the ledger to be closed before doing the read. the problem is that program2 needs to know the last valid entry in the ledger. (there may be entries that may not yet be valid.) for performance reasons, only program1 knows the end. so you need a way to propagate that information.we have talked about a way to push the last entry into the bookkeeper handle. flavio was working on it, but i don't think it has been implemented.benOn 10/21/2010 10:22 PM, amit jaiswal wrote:Hi,In BookKeeper documentation, the sample program creates a ledger, writes someentries and then *closes* the ledger. Then a client program opens the ledger,and reads the entries from it.Is it possible for program1 to write to a ledger, and program2 to read from theledger at the same time. In BookKeeper code, if a client tries to read from aledger which is not being closed (as per its metadata in zk), then a recoveryprocess is started to check for consistency.Waiting for ledger to get closed can introduce lot of latency at the clientside. Can somebody explain this functionality?-regardsAmit flaviojunqueira research scientist f...@yahoo-inc.comdirect +34 93-183-8828 avinguda diagonal 177, 8th floor, barcelona, 08018, esphone (408) 349 3300fax (408) 349 3301
Re: Is it possible to read/write a ledger concurrently
in hedwig one hub does both the publish and subscribe for a given topic and therefore is the only processes reading and writing from/to a ledger, so there isn't an issue. The ReadAheadCache does read-ahead :) it is so that we can minimize latency when doing sequential reads. ben On 10/21/2010 11:30 PM, amit jaiswal wrote: Hi, How does Hedwig handles this scenario? Since only one of the hubs have the ownership of a topic, the same hub is able to serve both publish and subscribe requests concurrently. Is my understanding correct ? Also, what is the purpose of ReadAheadCache class in Hedwig? Is it used somewhere for this concurrent read/write problem? -regards Amit - Original Message From: Benjamin Reed To: zookeeper-user@hadoop.apache.org Sent: Fri, 22 October, 2010 11:09:07 AM Subject: Re: Is it possible to read/write a ledger concurrently currently program1 can read and write to an open ledger, but program2 must wait for the ledger to be closed before doing the read. the problem is that program2 needs to know the last valid entry in the ledger. (there may be entries that may not yet be valid.) for performance reasons, only program1 knows the end. so you need a way to propagate that information. we have talked about a way to push the last entry into the bookkeeper handle. flavio was working on it, but i don't think it has been implemented. ben On 10/21/2010 10:22 PM, amit jaiswal wrote: Hi, In BookKeeper documentation, the sample program creates a ledger, writes some entries and then *closes* the ledger. Then a client program opens the ledger, and reads the entries from it. Is it possible for program1 to write to a ledger, and program2 to read from the ledger at the same time. In BookKeeper code, if a client tries to read from a ledger which is not being closed (as per its metadata in zk), then a recovery process is started to check for consistency. Waiting for ledger to get closed can introduce lot of latency at the client side. Can somebody explain this functionality? -regards Amit
Recovery from a complete node failure in BookKeeper
Hi, There is a class BookKeeperTools that has methods for complete recovery of a node. The recovery of dead bookie involves updating zk first with the replacement bookie and then replicating the necessary ledger entries. So, if the recovery process / target bookie dies before the actual entries could get copied, then there can be data inconsistency issues. Data copy can take time and thus increases the window during a which a node can potentially fail. Is this an issue that needs to be addressed? Also, this tool needs to be triggered manually for doing node recovery. Any plans for automatic node recovery (similar to Hadoop HDFS) in which if a machine goes down, then some background process replicates data to maintain the replication factor (quorum). -regards Amit
