Hi Yufei, I believe even with Option 2 the REST API response structure can be defined with an "envelope" around the semantic payload to allow both plain JSON and non-JSON formats to be represented. The latter will obviously need to be encoded inside the JSON response envelope.
Cheers, Dmitri. On Mon, Jul 13, 2026 at 2:13 PM Yufei Gu <[email protected]> wrote: > Hi Dmitri, Thanks! I think option 2 is a reasonable choice. The advantage I > see with option 1 is future extensibility. If we later decide to support > another semantic model format that isn't JSON, such as OKF, option 2 would > require a REST spec change, while option 1 would not. That's why I slightly > prefer option 1, although I think either option is much better than > modeling the semantic model structure directly in the REST API. What do you > think? > > Thanks, > Yufei > > > On Mon, Jul 13, 2026 at 11:10 AM Yufei Gu <[email protected]> wrote: > > > Hi Robert, > > > > Thanks for the thoughtful feedback. > > > > I agree that the current API is narrower than a complete semantic layer. > > Today it's primarily about managing the lifecycle of semantic model > > documents. Discovery, search, and client consumption are all important > > topics, but I think they're broader than the Ossie semantic model itself. > > We touched on some of these in the last community sync[1], and I'm happy > to > > continue that discussion in a separate thread. > > > > I also agree that we should be careful not to overstate what this initial > > API provides. Calling it a beta semantic model registry or document > hosting > > API seems reasonable. > > > > That said, I think those questions are orthogonal to the payload > > representation discussion. My original topic was simply whether the REST > > API should treat the semantic model as a raw document, an opaque JSON > > payload, or model the schema directly. Since the consumption story is > still > > evolving, I think keeping the REST contract loosely coupled to the > > underlying semantic model specification gives us the most flexibility. > > > > [1] > > > https://drive.google.com/file/d/1hxYkk2t-BcnFOk8eJG9NYCHfjkOXg3Iz/view?usp=sharing > > > > Thanks, > > > > Yufei > > > > On Mon, Jul 13, 2026 at 6:16 AM Robert Stupp <[email protected]> wrote: > > > >> Hi, > >> > >> I think the payload representation question depends on the client model > we > >> expect this API to support. > >> > >> I support Polaris hosting Apache Ossie semantic-model documents as a > beta > >> foundation. That seems like a useful first step while Ossie itself is > >> still > >> evolving. > >> > >> But I do not think we should describe the current API as enabling AI > >> tools, > >> BI tools, or human semantic-model discovery yet. > >> > >> The merged API is primarily namespace/name CRUD for a semantic-model > >> document. > >> That is opaque document storage and exact point-retrieval. > >> > >> It does not enable clients to discover the right semantic model from a > >> query, > >> nor to find semantic models by table, metric, domain, user, or > capability. > >> > >> It also does not define a standard consuming API or tool contract, > >> search/indexing contract, freshness model, or current/trusted/certified > >> model > >> semantics. > >> > >> That distinction matters because the REST API is the user-facing > contract. > >> > >> If the beta API is intended only as opaque document hosting, I think the > >> spec > >> and docs should say that clearly, and users/clients should not infer > >> broader > >> discovery or interoperability semantics from the CRUD API. > >> > >> I am not asking to solve the full semantic-layer story immediately. > >> > >> I am asking that durable implementation work does not get ahead of the > >> client-consumption story. > >> > >> The client model should drive the persistent data model, not the other > way > >> around. > >> > >> Once semantic models are stored as durable Polaris entities, choices > >> around > >> identity, versioning, validation, indexing, size limits, source-table > >> references, and freshness become much harder to change. > >> > >> For now, I would be comfortable describing this as beta Apache Ossie > >> document > >> hosting / semantic-model registry work. > >> > >> I would not yet be comfortable describing it as enabling AI, BI, or > human > >> semantic workflows until the discovery and client-consumption story > >> exists. > >> > >> Robert > >> > >> > >> On Fri, Jul 10, 2026 at 3:10 PM Dmitri Bourlatchkov <[email protected]> > >> wrote: > >> > >> > Hi Yufei, > >> > > >> > Thanks for starting this thread! > >> > > >> > I tend to think it is best to represent Ossie data as direct JSON > >> without > >> > defining its structure in the Polaris OpenAPI spec (Ossie schemas are > >> > controlled by Ossie, not Polaris). I believe this corresponds to > Option > >> 2 > >> > from your email. > >> > > >> > Polaris code that implements the new API will then interpret the Ossie > >> > parts according to the declared version of the Ossie spec. > >> > > >> > With that in mind, the API should clearly state the format (Ossie or > >> OKF) > >> > and the specification version of the semantic data sub-object > >> (apologies if > >> > it has that already, I'm behind on the related PR updates). Obviously > >> OKF > >> > will have a different representation in the payload, but this should > not > >> > prevent Ossie from leveraging JSON synergies. > >> > > >> > Anand's work on solving a similar problem in the Metrics API [4115] > may > >> be > >> > reusable here. > >> > > >> > [4115] https://github.com/apache/polaris/pull/4115 > >> > > >> > Cheers, > >> > Dmitri. > >> > > >> > > >> > On Wed, Jul 8, 2026 at 2:20 PM Yufei Gu <[email protected]> wrote: > >> > > >> > > Hi folks, > >> > > > >> > > Following JB's suggestion, I'd like to start a dedicated discussion > on > >> > the > >> > > REST API payload representation for semantic models. > >> > > > >> > > I think there are three possible approaches: > >> > > > >> > > 1. > >> > > > >> > > Represent the semantic model as a raw string. > >> > > 2. > >> > > > >> > > Represent the semantic model as an opaque JSON document. > >> > > 3. > >> > > > >> > > Model the semantic model structure directly in the REST > >> specification. > >> > > > >> > > I think it's helpful to separate the REST API from Polaris' internal > >> > > representation. The REST API is the long-term contract with clients, > >> > while > >> > > the internal representation can evolve independently. > >> > > > >> > > I'm comfortable with either option 1 or option 2. Both avoid > coupling > >> the > >> > > REST API to the Ossie schema and allow Polaris to validate the > payload > >> > > based on the semantic model type and version while preserving the > >> > document > >> > > through write and read operations. > >> > > > >> > > My concern is with option 3. Since the Ossie schema is versioned and > >> > > expected to evolve, modeling the full semantic model structure > >> directly > >> > in > >> > > the REST specification would tightly couple the Polaris REST API to > >> Ossie > >> > > versions. Every Ossie schema evolution could require changes to the > >> REST > >> > > specification, generated clients, and potentially client > applications. > >> > > > >> > > Between options 1 and 2, I think there is an additional tradeoff. > >> > > > >> > > An opaque JSON document assumes that semantic models are always > >> > represented > >> > > as JSON. While that works well for Ossie today, Polaris may support > >> other > >> > > semantic model formats in the future. For example, OKF[1] is defined > >> as > >> > > Markdown rather than JSON. Using a raw string keeps the REST API > >> > > independent of any particular document format, allowing Polaris to > >> > support > >> > > JSON, Markdown, or other representations without changing the API > >> > contract. > >> > > > >> > > So my current view is: > >> > > > >> > > - > >> > > > >> > > Option 1 provides the greatest flexibility and is format > agnostic. > >> > > - > >> > > > >> > > Option 2 is a natural choice if we want to optimize specifically > >> for > >> > > JSON based semantic models. > >> > > - > >> > > > >> > > Option 3 provides strong typing, but at the cost of coupling the > >> REST > >> > > API to Ossie schema evolution. > >> > > > >> > > I'm happy with either option 1 or option 2, but I'd avoid option 3 > for > >> > the > >> > > reasons above. > >> > > > >> > > Thoughts? > >> > > > >> > > 1. > >> > > > >> > > > >> > > >> > https://cloud.google.com/blog/products/data-analytics/how-the-open-knowledge-format-can-improve-data-sharing > >> > > > >> > > Thanks, > >> > > > >> > > Yufei > >> > > > >> > > >> > > >
