Hi,
We can probably make a new JIRA. Not sure if there is an existing JIRA to
re-use.
The Following modules are good to look at.
hudi-timeline-service
packaging/hudi-timeline-server-bundle
Thanks
Vinoth
On Fri, Jun 5, 2020 at 12:56 AM Mario de Sá Vera wrote:
> Sorry Vinoth for not being clear... If that is a work in progress would you
> have a jira I could follow up and contribute to ? If not , what is the
> module name you suggest me looking at?
>
> Regards,
>
> Mario.
>
> On Fri, 5 Jun 2020, 02:12 Vinoth Chandar, wrote:
>
> > Sorry did not understand the last part. :) are you suggesting we create a
> > jira
> >
> > On Thu, Jun 4, 2020 at 1:08 AM Mario de Sá Vera
> > wrote:
> >
> > > That sounds great ! Will check that and keep an eye on the long running
> > > server approach... once it gets a ticket I could watch for just let me
> > know
> > > please.
> > >
> > > Thanks
> > >
> > >
> > > On Thu, 4 Jun 2020, 05:34 Vinoth Chandar, wrote:
> > >
> > > > Hi Mario,
> > > >
> > > > We actually started with the idea of making the timeline server, a
> long
> > > > running service. We have a module if you notice that builds our a
> > bundle
> > > > that you could deploy. May be you can play with it and see if that
> > sounds
> > > > interesting to you. It will definitely have some rough edges given
> it’s
> > > not
> > > > been widely used.
> > > >
> > > > Thanks
> > > > Vinoth
> > > >
> > > > On Wed, Jun 3, 2020 at 2:33 AM Mario de Sá Vera
> > > > wrote:
> > > >
> > > > > Hi Vinoth, thanks for your comments on this. I spent sometime
> > thinking
> > > > over
> > > > > another possibility which would be externalising the Hudi timeline
> > > > service
> > > > > itself to an external server holding both operational (ie Hudi) and
> > > > > business metadata.
> > > > >
> > > > > would you guys have any opinion on that ? would that be easy as I
> do
> > > not
> > > > > seem to see a way yet , except reading about RocksDB but that is
> > still
> > > > not
> > > > > quite clear.
> > > > >
> > > > > best regards,
> > > > >
> > > > > Mario.
> > > > >
> > > > > Em seg., 1 de jun. de 2020 às 16:01, Vinoth Chandar <
> > > > > mail.vinoth.chan...@gmail.com> escreveu:
> > > > >
> > > > > > Hi Mario,
> > > > > >
> > > > > > Thanks for the detailed explanation. Hudi already allows extra
> > > metadata
> > > > > to
> > > > > > be written atomically with each commit i.e write operation. In
> > fact,
> > > > that
> > > > > > is how we track checkpoints for our delta streamer tool.. It may
> > not
> > > > > solve
> > > > > > the need for querying the data together with this information.
> but
> > > > gives
> > > > > > you ability to do some basic tagging.. if thats useful
> > > > > >
> > > > > > >>If we enable the timeline service metadata model to be extended
> > we
> > > > > could
> > > > > > use the service instance itself to support specialised queries
> that
> > > > > involve
> > > > > > business qualifiers in order to return a proper set of metadata
> > > > pointing
> > > > > to
> > > > > > the related commits
> > > > > >
> > > > > > This is a good idea actually.. There is another active discuss
> > thread
> > > > on
> > > > > > making the metadata queryable.. there is also
> > > > > > https://issues.apache.org/jira/browse/HUDI-309 which we paused
> for
> > > > now..
> > > > > > But that's more in line with what you are thinking IIUC
> > > > > >
> > > > > >
> > > > > > Thanks
> > > > > > vinoth
> > > > > >
> > > > > > On Mon, Jun 1, 2020 at 4:41 AM Mario de Sá Vera <
> > desav...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Hi Balaji,
> > > > > > >
> > > > > > > business metadata are all types of info related to the business
> > > where
> > > > > the
> > > > > > > Hudi solution is being used... from a COB (ie close of business
> > > date)
> > > > > > > related to that commit to any qualifier related to that commit
> > that
> > > > > might
> > > > > > > be useful to be associated with that commit id. If we enable
> the
> > > > > timeline
> > > > > > > service metadata model to be extended we could use the service
> > > > instance
> > > > > > > itself to support specialised queries that involve business
> > > > qualifiers
> > > > > in
> > > > > > > order to return a proper set of metadata pointing to the
> related
> > > > > commits
> > > > > > > that answer a business query.
> > > > > > >
> > > > > > > if we do not have that flexibility we might end up creating a
> > > > external
> > > > > > > transaction log and then comes the hard task to make that
> service
> > > in
> > > > > sync
> > > > > > > to the timeline service.
> > > > > > >
> > > > > > > let me know if that makes sense to you,
> > > > > > >
> > > > > > > Mario.
> > > > > > >
> > > > > > > Em seg., 1 de jun. de 2020 às 06:55, Balaji Varadarajan
> > > > > > > escreveu:
> > > > > > >
> > > > > > > > Hi Mario,
> > > > > > > > Timeline Server was designed to serve hudi metadata for Hudi
> > > > writers
> > > > > > and
> > > > > > > > readers. it
