Hi JB,
DAO is often defined as an interface, used exactly for operations. Why do
we need another name for that? See the examples here
<https://www.geeksforgeeks.org/data-access-object-pattern/>, it can be with
JPA or non-JPA. In our case, it has to be compatible with non-JPA
implementation.
interface DeveloperDao {
public List<Developer> getAllDevelopers();
public Developer getDeveloper(int DeveloperId);
public void updateDeveloper(Developer Developer);
public void deleteDeveloper(Developer Developer);
}
public interface ProductDAO extends JpaRepository<Product, Long> {
// Additional custom queries or methods can be declared here
}
Yufei
On Mon, Feb 17, 2025 at 10:58 AM Jean-Baptiste Onofré <j...@nanthrax.net>
wrote:
> Hi Yufei
>
> I agree with you. My only concern with CatalogDAO and NamespaceDAO in the
> PoC is that it could not be obvious for implementer.
> Being very abstract force us to clearly decouple service and persistence.
> One interface or multiple interfaces is totally fine for me (for the
> storage operations). I just think that we have to be clear in the DAO.
>
> It’s what I proposed and discussed with Jack to also get his expertise for
> DynamoDB.
>
> So, in the PoC, I would:
> 1. Clearly define DAO, with enforcement on the data transport (like using
> Java records). Maybe it’s more a naming clarification. We can have
> CatalogRecord (Java records) and CatalogStoreOperations (instead of
> CatalogDAO and using CatalogRecord)
> 2. Split store operations for a backend in separate modules (one for FDB,
> one for Postgre)
>
> Thoughts ?
>
> Regards
> JB
>
> Le lun. 17 févr. 2025 à 18:36, Yufei Gu <flyrain...@gmail.com> a écrit :
>
> > Hi JB,
> >
> > The Java objects you mentioned should either belong to the
> service/business
> > layer or be unnecessary. Apart from the business entities in Polaris,
> each
> > backend will maintain its own set of persistence-layer entities. The
> > backend implementations (whether Postgres, FDB, or MongoDB) will be
> > responsible for mapping between these persistence-layer entities and the
> > business entities. Currently, the business entity set is tightly coupled
> > with FDB, which is not ideal for supporting multiple backends. While we
> can
> > refactor them, creating an entirely new set of business entities is not
> > required.
> >
> > Regarding the PolarisStore you mentioned, it functions similarly to the
> DAO
> > layer in the POC. It'd make more sense to split its responsibilities into
> > separate interfaces rather than combining them into a single one. A
> single
> > interface violates the Single Responsibility Principle and may lead to
> > maintenance challenges as we introduce new data access patterns. For
> > instance, we would be forced to continuously add new methods to
> accommodate
> > additional entities like Policy and Volume, or to support new query
> > patterns for existing entities.
> >
> > As for choosing between record or immutable classes, either is fine. But
> > I'd prefer to use any native Java feature if possible.
> >
> >
> >
> > Yufei
> >
> >
> > On Mon, Feb 17, 2025 at 9:10 AM Michael Collado <collado.m...@gmail.com>
> > wrote:
> >
> > > I prefer native Java constructs over third party libraries and compiler
> > > plugins, whenever possible. I’m a fan of Java records.
> > >
> > > Mike
> > >
> > > On Mon, Feb 17, 2025 at 6:55 AM Jean-Baptiste Onofré <j...@nanthrax.net>
> > > wrote:
> > >
> > > > Hi Robert,
> > > >
> > > > Yes, my intention about Java records (e.g.
> > > > https://openjdk.org/jeps/395) is to leverage:
> > > > - Devise an object-oriented construct that expresses a simple
> > > > aggregation of values.
> > > > - Focus on modeling immutable data rather than extensible behavior.
> > > > - Automatically implement data-driven methods such as equals and
> > > accessors.
> > > > - Preserve long-standing Java principles such as nominal typing and
> > > > migration compatibility.
> > > > - Provide inner builder, optionally with technical validation
> > > > (Objects.NotNull, etc), for instance:
> > > >
> > > > public record CatalogDAO(String id, String name, ...) {
> > > >
> > > > public static final class Builder {
> > > > String id;
> > > > String name;
> > > > public Builder() {}
> > > > public Builder id(String id) {
> > > > this.id = id;
> > > > return this;
> > > > }
> > > > public Builder name(String name) {
> > > > this.name = name;
> > > > return this;
> > > > }
> > > > public CatalogDAO build() {
> > > > return new CatalogDAO(id, name);
> > > > }
> > > > }
> > > >
> > > > }
> > > > NB: that's just a "raw" example :)
> > > >
> > > > That could help us for the "DAO" layer, with clean isolation and data
> > > > "transport". The "conversion" from Polaris Entity to DAO could be
> > > > performed by the intermediate logic layer (e.g.
> > > > PolarisMetaStoreManager), the pure persistence layer can deal with
> DAO
> > > > only (e.g. PolarisStore).
> > > >
> > > > Good point about immutable collections. In some projects, I "force"
> > > > the copy of a collection to ensure immutability, something like:
> > > >
> > > > record NamespaceDAO(Set<String> children) {
> > > > public NamespaceDAO {
> > > > children = Set.copyOf(children);
> > > > }
> > > > }
> > > >
> > > > OK, that's not super elegant :) but it does the trick ;)
> > > >
> > > > Regards
> > > > JB
> > > >
> > > > On Mon, Feb 17, 2025 at 5:28 PM Robert Stupp <sn...@snazy.de> wrote:
> > > > >
> > > > > Agree with JB, except I think "immutables" serves the need much
> > better.
> > > > > Java records are okay, but do not ensure that all fields are
> > immutable
> > > -
> > > > > especially collections. The other pro of immutables is that there
> are
> > > > > proper, descriptive builders - hence no need to constructors with a
> > > > > gazillion parameters.
> > > > >
> > > > > On 17.02.25 11:42, Jean-Baptiste Onofré wrote:
> > > > > > Hi Yufei
> > > > > >
> > > > > > I left comments in the PR.
> > > > > >
> > > > > > Thanks for that. That's an interesting approach but slightly
> > > different
> > > > > > to what I had in mind.
> > > > > >
> > > > > > My proposal was:
> > > > > >
> > > > > > 1. The DAOs are Java records clearly descriptive, without
> "storage
> > > > operations".
> > > > > > For instance, we can imagine:
> > > > > >
> > > > > > public record CatalogDao(String id, String name, String location,
> > > ...)
> > > > {
> > > > > > ...
> > > > > > }
> > > > > >
> > > > > > public record NamespaceDao(String id, String name, String parent,
> > > ...)
> > > > {}
> > > > > >
> > > > > > public record TableDao(String id, String name, ...) {}
> > > > > >
> > > > > > etc
> > > > > >
> > > > > > The advantages are:
> > > > > > - very simple and descriptive
> > > > > > - common to any backend implementation
> > > > > >
> > > > > > 2. The PolarisStore defines the DAO backend operations and
> mapping
> > to
> > > > > > "internal" Polaris entities. Each persistence adapter implements
> > the
> > > > > > PolarisStore.
> > > > > > For the persistence adapter implementer, he just has to implement
> > the
> > > > > > DAO persistence (no need to understand other Polaris parts).
> > > > > > Each persistence adapter is in its own module (for isolation and
> > > > dependencies).
> > > > > >
> > > > > > During the Polaris Persistence Meeting last week, we already got
> > > > > > consensus on the approach proposed by Dennis and I. I propose to
> > do a
> > > > > > new sync/review with Dennis.
> > > > > >
> > > > > > Regards
> > > > > > JB
> > > > > >
> > > > > > On Mon, Feb 17, 2025 at 9:40 AM Yufei Gu <flyrain...@gmail.com>
> > > wrote:
> > > > > >> Hi Folks:
> > > > > >>
> > > > > >> Here is a POC for persistence layer refactor. Please check it
> out
> > > and
> > > > let
> > > > > >> me know what you think. Please note this is a POC, we still
> need a
> > > > lot of
> > > > > >> effort to complete the refactor.
> > > > > >>
> > > > > >> PR: https://github.com/apache/polaris/pull/1011.
> > > > > >> Design doc:
> > > > > >>
> > > >
> > >
> >
> https://docs.google.com/document/d/1Vuhw5b9-6KAol2vU3HUs9FJwcgWtiVVXMYhLtGmz53s/edit?tab=t.0
> > > > > >>
> > > > > >> Experiment:
> > > > > >>
> > > > > >> - Added a DAO layer for the business entity namespace(except
> > the
> > > > read).
> > > > > >> - Integrated with existing DAO components
> > > > (PolarisMetaStoreManager and
> > > > > >> PolarisMetaStoreSession).
> > > > > >> - All tests passed successfully, including a manual local
> run
> > > > with Spark
> > > > > >> sql.
> > > > > >>
> > > > > >> Benefits:
> > > > > >>
> > > > > >> - Compatible with the existing backend(FDB), as we hide it
> > > behind
> > > > the
> > > > > >> new DAO.
> > > > > >> - Adding new backends(Postgres/MongoDB) is much easier now,
> > esp
> > > > for
> > > > > >> Postgres, we could be able to use a similar model as Iceberg
> > > Jdbc
> > > > catalog.
> > > > > >> - Allows gradual refactoring to remove old DAO dependencies
> > > > > >> (PolarisMetaStoreManager and PolarisMetaStoreSession).
> > > > > >> - Enables parallel development of new backend
> implementations.
> > > > > >>
> > > > > >> Next Steps:
> > > > > >>
> > > > > >> - Define business entities one by one to decouple them from
> > FDB.
> > > > > >> - Create DAO interfaces for each entity to standardize
> > > operations
> > > > (e.g.,
> > > > > >> CRUD, ID generation).
> > > > > >> - Expand DAO implementations to support additional backends
> > over
> > > > time.
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >>
> > > > > >> Yufei
> > > > >
> > > > > --
> > > > > Robert Stupp
> > > > > @snazy
> > > > >
> > > >
> > >
> >
>
