Sorry Vinoth for not being clear... If that is a work in progress would you
have a jira I could follow up and contribute to ? If not , what is the
module name you suggest me looking at?
Regards,
Mario.
On Fri, 5 Jun 2020, 02:12 Vinoth Chandar, wrote:
> Sorry did not understand the last part. :) are you suggesting we create a
> jira
>
> On Thu, Jun 4, 2020 at 1:08 AM Mario de Sá Vera
> wrote:
>
> > That sounds great ! Will check that and keep an eye on the long running
> > server approach... once it gets a ticket I could watch for just let me
> know
> > please.
> >
> > Thanks
> >
> >
> > On Thu, 4 Jun 2020, 05:34 Vinoth Chandar, wrote:
> >
> > > Hi Mario,
> > >
> > > We actually started with the idea of making the timeline server, a long
> > > running service. We have a module if you notice that builds our a
> bundle
> > > that you could deploy. May be you can play with it and see if that
> sounds
> > > interesting to you. It will definitely have some rough edges given it’s
> > not
> > > been widely used.
> > >
> > > Thanks
> > > Vinoth
> > >
> > > On Wed, Jun 3, 2020 at 2:33 AM Mario de Sá Vera
> > > wrote:
> > >
> > > > Hi Vinoth, thanks for your comments on this. I spent sometime
> thinking
> > > over
> > > > another possibility which would be externalising the Hudi timeline
> > > service
> > > > itself to an external server holding both operational (ie Hudi) and
> > > > business metadata.
> > > >
> > > > would you guys have any opinion on that ? would that be easy as I do
> > not
> > > > seem to see a way yet , except reading about RocksDB but that is
> still
> > > not
> > > > quite clear.
> > > >
> > > > best regards,
> > > >
> > > > Mario.
> > > >
> > > > Em seg., 1 de jun. de 2020 às 16:01, Vinoth Chandar <
> > > > mail.vinoth.chan...@gmail.com> escreveu:
> > > >
> > > > > Hi Mario,
> > > > >
> > > > > Thanks for the detailed explanation. Hudi already allows extra
> > metadata
> > > > to
> > > > > be written atomically with each commit i.e write operation. In
> fact,
> > > that
> > > > > is how we track checkpoints for our delta streamer tool.. It may
> not
> > > > solve
> > > > > the need for querying the data together with this information. but
> > > gives
> > > > > you ability to do some basic tagging.. if thats useful
> > > > >
> > > > > >>If we enable the timeline service metadata model to be extended
> we
> > > > could
> > > > > use the service instance itself to support specialised queries that
> > > > involve
> > > > > business qualifiers in order to return a proper set of metadata
> > > pointing
> > > > to
> > > > > the related commits
> > > > >
> > > > > This is a good idea actually.. There is another active discuss
> thread
> > > on
> > > > > making the metadata queryable.. there is also
> > > > > https://issues.apache.org/jira/browse/HUDI-309 which we paused for
> > > now..
> > > > > But that's more in line with what you are thinking IIUC
> > > > >
> > > > >
> > > > > Thanks
> > > > > vinoth
> > > > >
> > > > > On Mon, Jun 1, 2020 at 4:41 AM Mario de Sá Vera <
> desav...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > Hi Balaji,
> > > > > >
> > > > > > business metadata are all types of info related to the business
> > where
> > > > the
> > > > > > Hudi solution is being used... from a COB (ie close of business
> > date)
> > > > > > related to that commit to any qualifier related to that commit
> that
> > > > might
> > > > > > be useful to be associated with that commit id. If we enable the
> > > > timeline
> > > > > > service metadata model to be extended we could use the service
> > > instance
> > > > > > itself to support specialised queries that involve business
> > > qualifiers
> > > > in
> > > > > > order to return a proper set of metadata pointing to the related
> > > > commits
> > > > > > that answer a business query.
> > > > > >
> > > > > > if we do not have that flexibility we might end up creating a
> > > external
> > > > > > transaction log and then comes the hard task to make that service
> > in
> > > > sync
> > > > > > to the timeline service.
> > > > > >
> > > > > > let me know if that makes sense to you,
> > > > > >
> > > > > > Mario.
> > > > > >
> > > > > > Em seg., 1 de jun. de 2020 às 06:55, Balaji Varadarajan
> > > > > > escreveu:
> > > > > >
> > > > > > > Hi Mario,
> > > > > > > Timeline Server was designed to serve hudi metadata for Hudi
> > > writers
> > > > > and
> > > > > > > readers. it may not be suitable to serve arbitrary data. But,
> it
> > > is
> > > > an
> > > > > > > interesting thought. Can you elaborate more on what kind of
> > > business
> > > > > > > metadata are you looking. Is this something you are planning to
> > > store
> > > > > in
> > > > > > > commit files ?
> > > > > > > Balaji.V
> > > > > > >
> > > > > > > On Sunday, May 31, 2020, 04:22:27 PM PDT, Mario de Sá Vera
> <
> > > > > > > desav...@gmail.com> wrote:
> > > > > > >
> > > > > > > I see a need for extending the current timeline server schema
> so
> > > > that
> > > >
