Hi Yufei
I left comments in the PR.
Thanks for that. That's an interesting approach but slightly different
to what I had in mind.
My proposal was:
1. The DAOs are Java records clearly descriptive, without "storage operations".
For instance, we can imagine:
public record CatalogDao(String id, String name, String location, ...) {
...
}
public record NamespaceDao(String id, String name, String parent, ...) {}
public record TableDao(String id, String name, ...) {}
etc
The advantages are:
- very simple and descriptive
- common to any backend implementation
2. The PolarisStore defines the DAO backend operations and mapping to
"internal" Polaris entities. Each persistence adapter implements the
PolarisStore.
For the persistence adapter implementer, he just has to implement the
DAO persistence (no need to understand other Polaris parts).
Each persistence adapter is in its own module (for isolation and dependencies).
During the Polaris Persistence Meeting last week, we already got
consensus on the approach proposed by Dennis and I. I propose to do a
new sync/review with Dennis.
Regards
JB
On Mon, Feb 17, 2025 at 9:40 AM Yufei Gu <flyrain...@gmail.com> wrote:
>
> Hi Folks:
>
> Here is a POC for persistence layer refactor. Please check it out and let
> me know what you think. Please note this is a POC, we still need a lot of
> effort to complete the refactor.
>
> PR: https://github.com/apache/polaris/pull/1011.
> Design doc:
> https://docs.google.com/document/d/1Vuhw5b9-6KAol2vU3HUs9FJwcgWtiVVXMYhLtGmz53s/edit?tab=t.0
>
> Experiment:
>
> - Added a DAO layer for the business entity namespace(except the read).
> - Integrated with existing DAO components (PolarisMetaStoreManager and
> PolarisMetaStoreSession).
> - All tests passed successfully, including a manual local run with Spark
> sql.
>
> Benefits:
>
> - Compatible with the existing backend(FDB), as we hide it behind the
> new DAO.
> - Adding new backends(Postgres/MongoDB) is much easier now, esp for
> Postgres, we could be able to use a similar model as Iceberg Jdbc catalog.
> - Allows gradual refactoring to remove old DAO dependencies
> (PolarisMetaStoreManager and PolarisMetaStoreSession).
> - Enables parallel development of new backend implementations.
>
> Next Steps:
>
> - Define business entities one by one to decouple them from FDB.
> - Create DAO interfaces for each entity to standardize operations (e.g.,
> CRUD, ID generation).
> - Expand DAO implementations to support additional backends over time.
>
>
>
>
> Yufei
