Re: [DISCUSS] RFC for Synchronous Metadata table for File listing

[Manoj Govindassamy]

+1 for the synchronous metadata updates. Looking forward to the RFC.


On Fri, Nov 12, 2021 at 4:46 PM Vinoth Chandar  wrote:

> +1 on this.
>
> On Fri, Nov 5, 2021 at 9:17 AM Sivabalan  wrote:
>
> > RFC-15
> > <
> >
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
> > >
> > made an attempt to boost performance of file listing by storing all file
> > information in metadata table. As we are looking to build more infra
> around
> > metadata table (RFC-27 for data skipping, etc), we felt having a
> > synchronous design will make it more tighter and will avoid some of the
> > corner cases with async approach.
> >
> > So, we will write up a new RFC for file listing based on metadata table
> > with synchronous updates.
> >
> > --
> > Regards,
> > -Sivabalan
> >
>

Re: [DISCUSS] RFC for Synchronous Metadata table for File listing

[Vinoth Chandar]

+1 on this.

On Fri, Nov 5, 2021 at 9:17 AM Sivabalan  wrote:

> RFC-15
> <
> https://cwiki.apache.org/confluence/display/HUDI/RFC+-+15%3A+HUDI+File+Listing+Improvements
> >
> made an attempt to boost performance of file listing by storing all file
> information in metadata table. As we are looking to build more infra around
> metadata table (RFC-27 for data skipping, etc), we felt having a
> synchronous design will make it more tighter and will avoid some of the
> corner cases with async approach.
>
> So, we will write up a new RFC for file listing based on metadata table
> with synchronous updates.
>
> --
> Regards,
> -Sivabalan
>

[DISCUSS] RFC for Synchronous Metadata table for File listing

[Sivabalan]

RFC-15

made an attempt to boost performance of file listing by storing all file
information in metadata table. As we are looking to build more infra around
metadata table (RFC-27 for data skipping, etc), we felt having a
synchronous design will make it more tighter and will avoid some of the
corner cases with async approach.

So, we will write up a new RFC for file listing based on metadata table
with synchronous updates.

-- 
Regards,
-Sivabalan