Il lun 2 ott 2017, 04:06 Sijie Guo <guosi...@gmail.com> ha scritto:
> On Sep 28, 2017 7:10 AM, "Enrico Olivelli" <eolive...@gmail.com> wrote:
>
> 2017-09-28 15:34 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
>
> > - these interfaces are more an admin interface. you can get an bookkeeper
> > admin object and then access ledger manager to fetch metadata, does that
> > work for you?
> >
>
> Yes as a workaround I am taking exactly this approch...listLedgers ->
> getLedgerMetadata (using BK 4.5)
>
> As the project I am working on will use 4.6 and the new
> org.apache.bookkeeper.client.api as I need volatily durability ledgers it
> is a pity to me that I will be stuck to use BookKeeperAdmin and
> LedgerMetadata
> only for listing ledgers
>
>
> We can propose the interface for bk admin. But I don't see a reason that
> listing ledgers being as part of bookkeeper.
>
Ok
>
>
> > - metadata filter is just a wrapper over ledger iterator or async process
> > ledger of ledger manager. most likely you will be the logic implementer.
> I
> > don't see any strong reason we need to expose this filter as well. I
> would
> > suggesting deferring this kind of requirements when we really need it.
> >
>
> we could at least add listLedgers + getLedgerMetadata API in the new API so
> that new clients won't need BookKeeperAdmin, which takes a direct ZK client
>
>
> -1 for any direct zk access. It should be hidden behind a ledger manager.
>
Sorry maybe I was not clear, I did not propose to make zk more visibile.
I wrote that I would like that new clients do not use legacy BookKeeper
Admin class which has explicit deps on zk
>
> or a legacy o.a.b.client.BookKeper object
>
> I have a good use case, to access the list of ledgers which satisfy a given
> condition on custom metadata, we introduced custom metadata in order to
> mark ledgers and to meta operations, classifications....
>
>
> If metadata store doesn't support predicate pushdown, you still iterate all
> the ledgers and do the filter at your side. So I don't see a reason to
> support filter.
>
In case of a ledgermanager which lives inside the bookie and in presence of
a big number of ledgers, say 100000, we will save a lot of network by
returning only a sublist of ledgers to the client.
In the new scenario of thin client this makes sense to me
>
> And if you are filtering custom metadata, this logic sounds tight to
> application logic, no?
>
Yes but we can give basic support for simple filtering. For instance I am
usibg metadata in order to:
- define the application which created the ledger
- define a group if semantically bound ledgers
- add explicit references to aplication ids
We introduced custom metada as key value pairs and I see it very simple to
create a basic but effective API to use them
>
>
> It is not blocker but if you are ok I will create an issue, but before
> creating an issue I would like to draft a starting idea which would work
> for the community as usual
>
>
> I don't see a strong value having a metadata filter, given it doesn't buy
> you any benefits at current metadata store. So I am -0 to this idea.
>
As I wrote above, I will see the benefits in new implementations, like the
http API and the thin client
Enrico
>
>
> Thanks
> Enrico
>
>
>
>
>
> >
> >
> >
> > On Thu, Sep 28, 2017 at 9:05 AM Enrico Olivelli <eolive...@gmail.com>
> > wrote:
> >
> > > 2017-09-28 14:55 GMT+02:00 Sijie Guo <guosi...@gmail.com>:
> > >
> > > > On Sep 28, 2017 6:20 AM, "Enrico Olivelli" <eolive...@gmail.com>
> > wrote:
> > > >
> > > > Hi,
> > > > I am looking for an API to read ledger metadata without actually
> > opening
> > > a
> > > > ledger.
> > > >
> > > > Currently opening a ledger sets a watch on ZK and this is really
> > > expensive
> > > > and in any case the desired action is not to "open a ledger" but to
> > > access
> > > > meta information about the ledger.
> > > >
> > > > My real usecase is that I want to list ledgers and filter the results
> > > using
> > > > custom metadata.
> > > >
> > > > If there is no available solution I would like to file and issue and
> a
> > > > proposal to have a new method in the new API like
> > > >
> > > > org.apache.bookkeeper.client.api.LedgerMetadata {
> > > > public long getId();
> > > > public Map<String, byte[]> getCustomMetadata();
> > > > public long getCtime();
> > > > }
> > > >
> > > >
> > > > I am fine with an interface for ledger metadata. It can help hide the
> > > > implementation details.
> > > >
> > > >
> > > > org.apache.bookkeeper.client.api.BookKeeper {
> > > >
> > > > LedgerMetadata getLedgerMetadata(long ledgerId);
> > > > void listLedgers(String query, Consumer<LedgerMetadata>
> consumer)
> > > >
> > > > }
> > > >
> > > >
> > > > I am not sure if we want this. Now, you can access the ledger manager
> > for
> > > > reading and listing metadata. And this is how it was used by bk
> shell.
> > I
> > > > would not suggest adding this to the public API until we really see a
> > > value
> > > > there.
> > > >
> > >
> > > BookKeeper object does not expose LedgerManager, I don't know if we
> want
> > to
> > > expose it in the new API
> > >
> > > we could add an API like:
> > >
> > > interface org.apache.bookkeeper.client.api.LedgerMetadataFilter {
> > > // marker interface
> > > }
> > >
> > > class org.apache.bookkeeper.client.api.CustomMetadataFilter implements
> > > LedgerMetadataFilter {
> > > // filter custom metadata field == value
> > > CustomMetadataFilter(String key, byte[] value);
> > > }
> > >
> > > class org.apache.bookkeeper.client.api.CompositeMetadataFilter
> > implements
> > > LedgerMetadataFilter {
> > > // apply a sequence of filters in AND
> > > CompositeMetadataFilter(LedgerMetadataFilter ... filters);
> > > }
> > >
> > >
> > > void listLedgers(LedgerFilter filter, Consumer<LedgerMetadata>
> consumer)
> > >
> > > Does this sound better to you ? it is more simpler but clear and the
> > > implemenation will be really easy
> > >
> > >
> > >
> > > >
> > > >
> > > > for the "query" we can define a simple "expression language"
> > > >
> > > > metadata.owner = 'xxx' and (metadata.type = 'yyyy' or metadata.type =
> > > > 'zzzz')
> > > >
> > > >
> > > > I am not a fan of having a query language for such purpose in bk. At
> > > least,
> > > > I don't see a lot of use cases that need this kind of query
> > capabilities.
> > > > If you can share more use cases about this, that would be good for
> > > > discussions.
> > > >
> > > >
> > > > we can provide a basic implementation of that language which actually
> > > works
> > > > on LedgerMetadata objects and internally we will loop over the
> ledgers
> > > and
> > > > apply the expression to every ledger
> > > > the basic implementation can leverage standard expression languages
> > like
> > > EL
> > > >
> > > > smarter implementations of LedgerManager will be able to narrow the
> > > search
> > > > and save resources
> > > >
> > > > Thoughts ?
> > > >
> > > > -- Enrico
> > > >
> > >
> >
>
--
-- Enrico Olivelli
