Hi folks, I am a bit confused by the current direction of this thread.
As far as I recall, the initial goal for OpenLineage support in Polaris was to act as a "gateway" to an OpenLineage backend (such as Marquez), with the potential to enrich OpenLineage events. It feels like the scope and purpose have shifted. Could we take a step back and return to the original problem statement? What exactly are we trying to achieve with OpenLineage support in Polaris, and what are the primary use cases? Clarifying these points should help us reach a consensus and move forward with the implementation details. Thanks! Regards, JB On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <[email protected]> wrote: > > Hi Adnan, > > Just to clarify: My questions about data retention are not a blocker for PR > 4705. > > However, these questions are specific to the Open Lineage feature. The > proposal talks about removing edge information and has references to data > retention in the gdoc and its comments. > > If edges are removed, the resulting inter-dataset graph will represent > "jammed" relationships for "all time", if old data is not removed. > Therefore, understanding how OL-specific data is aggregated and retained is > critical to this feature. > > By comparison, metrics and events form a time-series data. Even without > retention, the processing of this data is pretty clear. The lack of > retention affects only storage (database) size. It does not affect > processing semantics. > > Whether OL data queries are supposed to contain time-based clauses will > affect the query API, which is in fact added in PR 4705 > (cf. LineageQueryRequest). > > If queries are not supposed to contain time clauses (current state of PR > 4705), then OL data retention becomes critical to understanding how the > system will work end-to-end. It is not a simple matter of growing storage > size, but a matter of query semantics. I believe it is essential to clarify > this aspect. > > That said, assuming the "beta" status of this feature and openness to > follow-up changes, PR 4705 can probably be merged "as is", but these > questions still need to be addressed before opening the feature to end > users, I think. > > Cheers, > Dmitri. > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev <[email protected]> > wrote: > > > Hi folks, > > > > I agree that #4705 is mainly just scaffolding and doesn't direct much about > > the persistence contract. I have a few implementation-related comments, but > > once those are resolved, I think we can merge. > > > > > I also think the local-store path needs explicit usability and > > operational criteria before schema work merges > > > > We can surely discuss these topics, but then we must be very clear that we > > are applying similar questions to all features proposed. Having to answer > > these questions for features has not been the case in the past - and I am > > failing to understand why this bar is being attached specifically to the > > OpenLineage feature. > > > > WRT Dmitri's retention concerns: We **should** build retention in the > > future - not just for Lineage events, but also for Event and Metrics > > Persistence. Basically it should apply to any entity that grows linearly > > with user queries. IMO retention is an orthogonal effort - I don't think it > > is fair to strongly attach it only to OpenLineage. We should surely discuss > > this in a different thread, ideally with someone willing to propose and > > implement entity retention. WDYT? > > > > Best, > > Adnan Hemani > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <[email protected]> > > wrote: > > > > > Hi All, > > > > > > I believe PR [4705] establishes a fairly isolated module structure for > > > further OL work and does not imply much about the feature itself. I think > > > it can be merged in its current state, provided the community agrees on > > the > > > further feature direction. > > > > > > Regarding the specific Persistence implementation, from my POV the > > > following question also needs to be addressed: The proposal doc [1] > > assumes > > > that the job edges are aggregated away from the OL input data and that > > the > > > resulting inter-dataset graph is persisted with some retention > > > characteristics. This is being discussed in the comment thread rooted in > > > the "Abstract" section (at the phrase: "maintains a directed graph of > > > dataset and field relationships"). However, I did not see a clear > > > specification of this behaviour in the main doc body. > > > > > > I think this needs to be discussed in more detail, either in the doc or > > > (preferably) in this email thread. More specifically: > > > > > > * How / when such retention is performed. > > > > > > * How queries relate to data retention. Do they only "see" the latest > > state > > > of the inter-dataset graph or do they see several such graphs changing > > over > > > time (within the retention window)? > > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL > > files > > > for OL tables. I support this approach. I believe this will allow for a > > > smoother evolution of the OL schema with respect to MetaStore and other > > > schemas. Additionally this approach reduces the impact of OL changes on > > > users who elect not to enable this feature. > > > > > > [1] > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > Thanks, > > > Dmitri. > > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> wrote: > > > > > > > Hi, > > > > > > > > I’m not trying to block incremental OpenLineage scaffolding, but I > > think > > > we > > > > should make explicit what the first merges do and do not settle. > > > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they are > > > > explicitly > > > > non-committal scaffolding. > > > > > > > > My concern is that these merges should not be read as consensus on the > > > > local > > > > lineage backend, query model, persistence semantics, or JDBC schema. > > > > > > > > In particular, I do not think #4826 should be read as settling how the > > > SPI > > > > maps > > > > to JDBC schema evolution, NoSQL backends, or backend capability > > > > differences. > > > > > > > > I also think the local-store path needs explicit usability and > > > operational > > > > criteria before schema work merges: what queries are expected to be > > > > efficient, > > > > what graph size or retention window we are designing for, what > > write/read > > > > load > > > > Polaris should absorb, and what happens when lineage volume exceeds > > those > > > > bounds. > > > > > > > > Without that, we may merge a schema that is technically workable for > > > small > > > > examples but unclear for real lineage volume, query latency, pruning, > > and > > > > backend load. > > > > > > > > Those need a separate decision before #4827 or equivalent > > > > schema/persistence > > > > work merges. > > > > > > > > Robert > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > [4826] https://github.com/apache/polaris/pull/4826 > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > > > > [email protected]> wrote: > > > > > > > > > Hi Dmitri, > > > > > > > > > > Thanks for the comments. Replying point-by-point: > > > > > > > > > > > re: How to use this data > > > > > > > > > > I agree, we should list this in the document itself so it does not > > get > > > > > lost. Let me put something in the appendix for now and link it to the > > > > > relevant section so we don't lose the document's flow. Please take a > > > look > > > > > there. > > > > > > > > > > > re: Link to OL Spec? > > > > > > > > > > https://openlineage.io/apidocs/openapi/ > > > > > > > > > > > re: Does OL persistence have to be limited to JDBC? > > > > > > > > > > No, this is the LineagePersistent Contract (from earlier in this > > > thread, > > > > > linked again for comfort): > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > > > > . > > > > > This was made to ensure that different persistence layers can > > > implement a > > > > > single interface for Lineage information. > > > > > > > > > > > re: Does OL data have to be located in the same database as > > MetaStore > > > > > data? > > > > > > > > > > I don't think there is a hard requirement for this. Happy to discuss > > if > > > > > anyone else thinks differently! > > > > > > > > > > > I believe Polaris should allow downstream projects to opt in/out of > > > > > including > > > > > OL REST APIs and related persistence code. This is the general > > approach > > > > > we're following for the Metrics API proposal. > > > > > > > > > > Understood. I think this is an implementation detail and you've > > marked > > > it > > > > > on the PR as well. Let us take care of this discussion there. > > > > > > > > > > Best, > > > > > Adnan Hemani > > > > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov < > > [email protected] > > > > > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > Commenting in general. > > > > > > > > > > > > I see a few PRs have been opened related to OpenLineage. This is > > > fine. > > > > > > > > > > > > However, I believe we need to review some project-wide approaches > > > > related > > > > > > to adding this feature (among other features). > > > > > > > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > 2) Does OL data have to be located in the same database as > > MetaStore > > > > > data? > > > > > > > > > > > > 3) I believe Polaris should allow downstream projects to opt in/out > > > of > > > > > > including OL REST APIs and related persistence code. This is the > > > > general > > > > > > approach we're following for the Metrics API proposal [1] [4115] > > > > > > > > > > > > These points are not in opposition to supporting OL in Polaris. > > These > > > > are > > > > > > merely technical concerns, which I think would benefit Polaris as a > > > > whole > > > > > > if designed with flexibility and modularity in mind upfront. > > > > > > > > > > > > WDYT? > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > > > > [email protected]> wrote: > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I previously > > > posted > > > > > in a > > > > > > > different thread [1] was actually being found, given that it was > > > deep > > > > > > into > > > > > > > the thread. I request the community to review this proposal so we > > > can > > > > > > > potentially start implementation. > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > > > > > > > Best, > > > > > > > Adnan Hemani > > > > > > > > > > > > > > [1] > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > > > > > > > > > > > > > > > > > > >
