[
https://issues.apache.org/jira/browse/BOOKKEEPER-220?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13293585#comment-13293585
]
Ivan Kelly commented on BOOKKEEPER-220:
---------------------------------------
{quote}
Yes, there is actually no enforcement that makes sure the a managed ledger is
running only on one box and it would be definitely better to have some kind of
protection. I wasn't sure on how to implement it (my first thought
was to have a lock on zk), and the other reason is that from our side we are
already protecting this using a zk lock (a topic maps to one managed ledger and
one topic is served by one broker).
Regarding versioned writes, what should be the behavior when a versioned write
fails (meaning someone else is messing up with the managed ledger) ? Versioned
writes would make the state consistent but there would still be problems in
running the managed ledgers from different machines (consumer not getting
messages,
And, how can I verify that a ledger is closed for writing (I think I'd need to
check only the last ledger in the list) ?
{quote}
One point of reference for you could be the BookKeeperJournalManager in HDFS
[1]. This has a lot of crossover with the functionallity of managed ledger.
You can use the managed ledger znode which contains a list of all ledgers in
the managed ledger for locking. The steps for added a new ledger to the managed
ledger should be.
1. Read list of ledgers and version for managed ledger znode.
2. Ensure that all ledgers in list of ledgers are closed.
3. Create new ledger and add to list of ledgers.
4. setData with the new list of ledgers, with the saved version from 1. as the
version.
This should ensure that more than one client can't add data.
[1]
https://github.com/apache/hadoop-common/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal/src/main/java/org/apache/hadoop/contrib/bkjournal/BookKeeperJournalManager.java
{quote}
I'm all for making (at least) asyncAddEntry truly async and unifying the
sync/async path. My only concern is how to deal when having to call more than
one
BK async method. For example, asyncAddEntry can require to create a new ledger
if the current one is full. That would mean to call bk.asyncCreateLedger() and
in the callback use the newly created ledger with ledger.asyncAddEntry(). Is
this feasable? I think (maybe I'm wrong) that it's not safe to call BK api
methods from inside a BK callback.
Current implementation is cumbersome, but it's truly async in the most common
case of addEntry, when not crossing ledger boundaries.
{quote}
This is why you should use an actor pattern or something like Hedwig's
TopicOpQueue. It allows you to serialize, while still maintaining asynchrony.
{quote}
Yes, I initially made the readEntries() that returns a list because I thought
it was easier to get and easier to use for batch sending and let the client to
pick up the batch size, but I'm open to change it. What I would'n want is to
have both readEntries and the iterator style because that would be confusing.
{quote}
I agree about there being only one style. However, I think an iterator approach
is better, as it would allow us to control better how many entries we actually
read from the server. (with readEntries, the client specifies the minimum we
need to read, which could be a very large number).
{quote}
Ok, I agree with that. What would be more appropriate, to use BKException or
some other one (MLException, ...) ?
{quote}
I think you should define your own, so MLException
> Managed Ledger proposal
> -----------------------
>
> Key: BOOKKEEPER-220
> URL: https://issues.apache.org/jira/browse/BOOKKEEPER-220
> Project: Bookkeeper
> Issue Type: New Feature
> Components: bookkeeper-client
> Reporter: Matteo Merli
> Assignee: Matteo Merli
> Fix For: 4.2.0
>
> Attachments: 0001-BOOKKEEPER-220-Managed-Ledger-proposal.patch,
> 0001-BOOKKEEPER-220-Managed-Ledger-proposal.patch
>
>
> The ManagedLedger design is based on our need to manage a set of ledgers,
> with a single writer (at any point in time) and a set on consumers that read
> entries from it.
> The ManagedLedger also takes care of periodically closing ledgers to have a
> "reasonable" sized sets of ledgers that can individually deleted when no more
> needed.
> I've put on github the interface proposal (along with an early WIP
> implementation)
> http://github.com/merlimat/managed-ledger
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
