Yes, sorry I did not make it clear, I also agree it is not the right direction to invest a lot of community effort. I am more talking about casual use cases like importing a server for unit tests outside Iceberg, running some local debugging, etc. I think it would be valuable to provide a server in Iceberg for that purpose, and maybe vend it as test utils. Thoughts?
-Jack On Thu, Jan 25, 2024 at 11:35 AM Ryan Blue <b...@tabular.io> wrote: > > I know we have the RESTCatalogAdapter and RESTCatalogSevlet for unit > tests, and technically we have a very similar Jetty server implementation > in TestRESTCatalog. Should we think about making those components out of > the tests into an iceberg-rest-server module for this use case, and merge > with the implementation that Gravitino has? > > I think that this would take the Iceberg project in the wrong direction. > Iceberg has always been a library and I think it should continue to be. > Concerns about runtime should be left to other projects that need to fit > into existing infrastructure or skillsets of people maintaining them. The > question of whether to use Jetty or Tomcat or whatever else is a serious > consideration, as is how to monitor that application and send metrics. I > think it would slow down the core purpose of Iceberg if we got distracted > by these things. > > In fact, I think that this project shows that the library is getting the > balance right: it is using `CatalogHandlers` for their intended purpose. It > has opinions about how to run the actual HTTP service and people that agree > can use it. Other people could use `CatalogHandlers` to build on a > different foundation. > > On Thu, Jan 25, 2024 at 11:13 AM Jack Ye <yezhao...@gmail.com> wrote: > >> Really cool project! >> >> I browsed a bit of the codebase, and see this implementation of the REST >> service backend: >> - >> https://github.com/datastrato/gravitino/blob/main/catalogs/catalog-lakehouse-iceberg/src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/IcebergRESTService.java#L39 >> - >> https://github.com/datastrato/gravitino/blob/main/catalogs/catalog-lakehouse-iceberg/src/main/java/com/datastrato/gravitino/catalog/lakehouse/iceberg/ops/IcebergTableOps.java#L42-L51 >> >> Looks like it is initializing a Jetty server that uses CatalogHandlers >> to delegate the execution to a specific Java Catalog implementation. >> >> I think this is actually something that is lacking today in Iceberg, >> which is an easy way for users to start an actual REST HTTP server. >> >> I know we have the RESTCatalogAdapter and RESTCatalogSevlet for unit >> tests, and technically we have a very similar Jetty server implementation >> in TestRESTCatalog. Should we think about making those components out of >> the tests into an iceberg-rest-server module for this use case, and merge >> with the implementation that Gravitino has? >> >> Best, >> Jack Ye >> >> On Thu, Jan 25, 2024 at 10:47 AM Yufei Gu <flyrain...@gmail.com> wrote: >> >>> Thanks Justin for the sharing. >>> >>> It's pretty cool to see an open source REST catalog implementation in >>> action. Having dabbled a bit in the early development of Gravitino myself, >>> I'm really excited about its potential with the Iceberg REST catalog. >>> >>> The idea of Gravitino moving to an ASF project is promising. It’ll >>> surely boost its visibility and open up more doors for collaboration and >>> adoption. >>> >>> Looking forward to where this goes. Keep up the fantastic work! >>> >>> Yufei >>> >>> >>> On Thu, Jan 25, 2024 at 5:55 AM Jean-Baptiste Onofré <j...@nanthrax.net> >>> wrote: >>> >>>> Hi Justin, >>>> >>>> I talked with Junping a couple of months ago about Gravitino. Thanks >>>> for sharing ! >>>> >>>> Regards >>>> JB >>>> >>>> On Thu, Jan 25, 2024 at 12:15 AM Justin Mclean < >>>> jus...@classsoftware.com> wrote: >>>> > >>>> > Hi, >>>> > >>>> > We open-sourced a new project, Gravitino, in December and have been >>>> working on growing the community and adding new functionality. We plan to >>>> donate the project to the ASF this year. Gravitino is a unified metadata >>>> lake solution offering a unified approach to managing datasets from diverse >>>> sources and regions across multiple cloud platforms. Its core is an Iceberg >>>> REST catalog service implementation to manage Iceberg tables efficiently. >>>> > >>>> > If this sounds like something you would be interested in, then the >>>> following resources will help: >>>> > - Blog post: >>>> https://datastrato.ai/blog/gravitino-iceberg-rest-catalog-service/ >>>> > - Gravitino documentation: https://datastrato.ai/docs/0.3.1/ >>>> > - Iceberg REST service documentation: >>>> https://datastrato.ai/docs/0.3.1/iceberg-rest-service >>>> > >>>> > We welcome any feedback and suggestions you have, and as always, all >>>> contributions are welcome. You can find the source code at >>>> https://github.com/datastrato/gravitino. >>>> > >>>> > Kind Regards, >>>> > Justin >>>> >>> > > -- > Ryan Blue > Tabular >
