Hi Laurent,
I do agree that the generic table spec needs to be evolved to provide
richer standardization. However,
the current support of generic table does provides good amount of values:
1) Polaris can be a centralized catalog service for discovering all tables,
not just Iceberg tables. (Of course,
there is large room to improve, for Delta table, as far as the base
location is available, different engine will be
able to access them).
2) Engine specific plugin to help engine to interpret the spec. Spark is
the first one, which covers a large number
of use cases, and we do have users. We are also working on providing a
connector for trinno to unblock the
trinno use case.
As Polaris is currently Iceberg native, the strategy we are taking is to
start with simple and enough support to
unblock the Spark use cases. Since we own the spec, the spec should evolve
quickly to adapt for different use cases
and standardization across different formats.
In other words, the 1.0 generic table support is the first step for Polaris
to move towards non-iceberg tables support,
there is definitely long way till we can fully support all operations
across various table formats, and we would like to
evolve the spec quickly based on specific use cases.
The strategy and direction has been brought up to the community and there
is agreement. If we there is general
concerns about the direction of how generic table should evolve, i think we
can definitely open a different thread
and have a discussion there. This thread is only intended to discuss the
evolving the current Spec one step forward
to standardize one of the critical information for cross engine sharing,
and eventually help the support for credential
vending.
Best Regards,
Yun
On Wed, Jun 11, 2025 at 8:15 AM Laurent Goujon <laur...@dremio.com.invalid>
wrote:
> What I was trying to say is that i'm sure there's plenty of value for
> spark, but in it's current state the value is little from a Polaris point
> of view as an open catalog service?
>
> Of course we can follow-up on that but is the current spec still considered
> wip or when 1.0 will be released, we would have to keep supporting it even
> if we come up with something more comprehensive?
>
> On Wed, Jun 11, 2025, 00:22 Eric Maynard <eric.w.mayn...@gmail.com> wrote:
>
> > > I don't think there's a lot of value where the specification of a table
> > format is left to the client
> > Considering that you currently can use non-Iceberg tables in Polaris with
> > the Spark client and it works end-to-end, I'd have a hard time agreeing
> > that there is no value.
> >
> > But I think this discussion is maybe best moved to another thread. The
> > incremental change to add a location may make sense for the existing
> > generic table implementation, even if later we reach a consensus to rip
> it
> > out and replace it with something more "comprehensive".
> >
> > --EM
> >
>
