Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I see with option 1 is future extensibility. If we later decide to support another semantic model format that isn't JSON, such as OKF, option 2 would require a REST spec change, while option 1 would not. That's why I slightly prefer option 1, although I think either option is much better than modeling the semantic model structure directly in the REST API. What do you think?
Thanks, Yufei On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <[email protected]> wrote: > Hi Robert, > > Thanks for the thoughtful feedback. > > I agree that the current API is narrower than a complete semantic layer. > Today it's primarily about managing the lifecycle of semantic model > documents. Discovery, search, and client consumption are all important > topics, but I think they're broader than the Ossie semantic model itself. > We touched on some of these in the last community sync[1], and I'm happy to > continue that discussion in a separate thread. > > I also agree that we should be careful not to overstate what this initial > API provides. Calling it a beta semantic model registry or document hosting > API seems reasonable. > > That said, I think those questions are orthogonal to the payload > representation discussion. My original topic was simply whether the REST > API should treat the semantic model as a raw document, an opaque JSON > payload, or model the schema directly. Since the consumption story is still > evolving, I think keeping the REST contract loosely coupled to the > underlying semantic model specification gives us the most flexibility. > > [1] > https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing > > Thanks, > > Yufei > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <[email protected]> wrote: > >> Hi, >> >> I think the payload representation question depends on the client model we >> expect this API to support. >> >> I support Polaris hosting Apache Ossie semantic-model documents as a beta >> foundation. That seems like a useful first step while Ossie itself is >> still >> evolving. >> >> But I do not think we should describe the current API as enabling AI >> tools, >> BI tools, or human semantic-model discovery yet. >> >> The merged API is primarily namespace/name CRUD for a semantic-model >> document. >> That is opaque document storage and exact point-retrieval. >> >> It does not enable clients to discover the right semantic model from a >> query, >> nor to find semantic models by table, metric, domain, user, or capability. >> >> It also does not define a standard consuming API or tool contract, >> search/indexing contract, freshness model, or current/trusted/certified >> model >> semantics. >> >> That distinction matters because the REST API is the user-facing contract. >> >> If the beta API is intended only as opaque document hosting, I think the >> spec >> and docs should say that clearly, and users/clients should not infer >> broader >> discovery or interoperability semantics from the CRUD API. >> >> I am not asking to solve the full semantic-layer story immediately. >> >> I am asking that durable implementation work does not get ahead of the >> client-consumption story. >> >> The client model should drive the persistent data model, not the other way >> around. >> >> Once semantic models are stored as durable Polaris entities, choices >> around >> identity, versioning, validation, indexing, size limits, source-table >> references, and freshness become much harder to change. >> >> For now, I would be comfortable describing this as beta Apache Ossie >> document >> hosting / semantic-model registry work. >> >> I would not yet be comfortable describing it as enabling AI, BI, or human >> semantic workflows until the discovery and client-consumption story >> exists. >> >> Robert >> >> >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <[email protected]> >> wrote: >> >> > Hi Yufei, >> > >> > Thanks for starting this thread! >> > >> > I tend to think it is best to represent Ossie data as direct JSON >> without >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are >> > controlled by Ossie, not Polaris). I believe this corresponds to Option >> 2 >> > from your email. >> > >> > Polaris code that implements the new API will then interpret the Ossie >> > parts according to the declared version of the Ossie spec. >> > >> > With that in mind, the API should clearly state the format (Ossie or >> OKF) >> > and the specification version of the semantic data sub-object >> (apologies if >> > it has that already, I'm behind on the related PR updates). Obviously >> OKF >> > will have a different representation in the payload, but this should not >> > prevent Ossie from leveraging JSON synergies. >> > >> > Anand's work on solving a similar problem in the Metrics API [4115] may >> be >> > reusable here. >> > >> > [4115] https://github.com/apache/polaris/pull/4115 >> > >> > Cheers, >> > Dmitri. >> > >> > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <[email protected]> wrote: >> > >> > > Hi folks, >> > > >> > > Following JB's suggestion, I'd like to start a dedicated discussion on >> > the >> > > REST API payload representation for semantic models. >> > > >> > > I think there are three possible approaches: >> > > >> > > 1. >> > > >> > > Represent the semantic model as a raw string. >> > > 2. >> > > >> > > Represent the semantic model as an opaque JSON document. >> > > 3. >> > > >> > > Model the semantic model structure directly in the REST >> specification. >> > > >> > > I think it's helpful to separate the REST API from Polaris' internal >> > > representation. The REST API is the long-term contract with clients, >> > while >> > > the internal representation can evolve independently. >> > > >> > > I'm comfortable with either option 1 or option 2. Both avoid coupling >> the >> > > REST API to the Ossie schema and allow Polaris to validate the payload >> > > based on the semantic model type and version while preserving the >> > document >> > > through write and read operations. >> > > >> > > My concern is with option 3. Since the Ossie schema is versioned and >> > > expected to evolve, modeling the full semantic model structure >> directly >> > in >> > > the REST specification would tightly couple the Polaris REST API to >> Ossie >> > > versions. Every Ossie schema evolution could require changes to the >> REST >> > > specification, generated clients, and potentially client applications. >> > > >> > > Between options 1 and 2, I think there is an additional tradeoff. >> > > >> > > An opaque JSON document assumes that semantic models are always >> > represented >> > > as JSON. While that works well for Ossie today, Polaris may support >> other >> > > semantic model formats in the future. For example, OKF[1] is defined >> as >> > > Markdown rather than JSON. Using a raw string keeps the REST API >> > > independent of any particular document format, allowing Polaris to >> > support >> > > JSON, Markdown, or other representations without changing the API >> > contract. >> > > >> > > So my current view is: >> > > >> > > - >> > > >> > > Option 1 provides the greatest flexibility and is format agnostic. >> > > - >> > > >> > > Option 2 is a natural choice if we want to optimize specifically >> for >> > > JSON based semantic models. >> > > - >> > > >> > > Option 3 provides strong typing, but at the cost of coupling the >> REST >> > > API to Ossie schema evolution. >> > > >> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 for >> > the >> > > reasons above. >> > > >> > > Thoughts? >> > > >> > > 1. >> > > >> > > >> > >> https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing >> > > >> > > Thanks, >> > > >> > > Yufei >> > > >> > >> >
