Hi Dmitri, Great question again! I think the plan still remains to ensure the OL event forwarding works first, while we continue making progress on the local storage in parallel. Here's how I see this in terms of the PRs currently open for review: * PR #4667: This is the baseline PR which adds the APIs required for OpenLineage compatibility. This PR is required regardless of local storage or forwarding modes. However, this PR is currently a blocker only for the forwarding mode, as local storage mode has other work that can be completed independently. * PR #4705: This introduces POJOs and scaffolding for the future local storage mode - but does not actually introduce the local storage functionality. It's good to keep this PR moving forward while contributors are excited to make progress on it!
Best, Adnan Hemani On Thu, Aug 6, 2026 at 4:02 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi Adnan, > > Thanks for the update. This removes my concern with timestamps in the > lineage queries. > > Re: local storage and forwarding to Marquez (or similar), it looks like > current outstanding lineage PRs only deal with the local storage aspect > now. > > So to clarify the proposal: Are you suggesting that we implement local > storage first, and deal with forwarding later? My reading of JB's email is > that he might have been expecting the implementation of these phases to > occur in the reverse order. > > Cheers, > Dmitri, > > On Tue, Aug 4, 2026 at 9:46 PM Adnan Hemani via dev < > [email protected]> > wrote: > > > Hi folks, > > > > Sorry for the delay. I have been traveling (headed to Community over Code > > Asia to present on Polaris tonight!), but I should be back next week to > > answer more questions. > > > > re Dmitri's email: I think I now understand that the source of your > > confusion surrounds the timestamps we record in the OpenLineage event. > > Querying events should only show the timestamp as a metric of "freshness" > > of the connection between different datasets - and not as a historical > log > > of all events that have ever occurred. Therefore, we don't need to > provide > > the timestamp as a "queryable" field; it only serves to answer questions > > like, "When was the last time this dataset was loaded/used successfully?" > > Does that clarification help? > > > > re JB's email: Everything you've asked can still be answered from the > > original proposal document: > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > . > > I do think Dmitri's questions are relevant to the scope presented in this > > document and are good details to ensure we are all on the same page. > > > > Best, > > Adnan Hemani > > > > On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <[email protected]> > > wrote: > > > > > Hi folks, > > > > > > I am a bit confused by the current direction of this thread. > > > > > > As far as I recall, the initial goal for OpenLineage support in > > > Polaris was to act as a "gateway" to an OpenLineage backend (such as > > > Marquez), with the potential to enrich OpenLineage events. It feels > > > like the scope and purpose have shifted. > > > > > > Could we take a step back and return to the original problem > > > statement? What exactly are we trying to achieve with OpenLineage > > > support in Polaris, and what are the primary use cases? > > > > > > Clarifying these points should help us reach a consensus and move > > > forward with the implementation details. > > > > > > Thanks! > > > > > > Regards, > > > JB > > > > > > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > > Hi Adnan, > > > > > > > > Just to clarify: My questions about data retention are not a blocker > > for > > > PR > > > > 4705. > > > > > > > > However, these questions are specific to the Open Lineage feature. > The > > > > proposal talks about removing edge information and has references to > > data > > > > retention in the gdoc and its comments. > > > > > > > > If edges are removed, the resulting inter-dataset graph will > represent > > > > "jammed" relationships for "all time", if old data is not removed. > > > > Therefore, understanding how OL-specific data is aggregated and > > retained > > > is > > > > critical to this feature. > > > > > > > > By comparison, metrics and events form a time-series data. Even > without > > > > retention, the processing of this data is pretty clear. The lack of > > > > retention affects only storage (database) size. It does not affect > > > > processing semantics. > > > > > > > > Whether OL data queries are supposed to contain time-based clauses > will > > > > affect the query API, which is in fact added in PR 4705 > > > > (cf. LineageQueryRequest). > > > > > > > > If queries are not supposed to contain time clauses (current state of > > PR > > > > 4705), then OL data retention becomes critical to understanding how > the > > > > system will work end-to-end. It is not a simple matter of growing > > storage > > > > size, but a matter of query semantics. I believe it is essential to > > > clarify > > > > this aspect. > > > > > > > > That said, assuming the "beta" status of this feature and openness to > > > > follow-up changes, PR 4705 can probably be merged "as is", but these > > > > questions still need to be addressed before opening the feature to > end > > > > users, I think. > > > > > > > > Cheers, > > > > Dmitri. > > > > > > > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev < > > > [email protected]> > > > > wrote: > > > > > > > > > Hi folks, > > > > > > > > > > I agree that #4705 is mainly just scaffolding and doesn't direct > much > > > about > > > > > the persistence contract. I have a few implementation-related > > > comments, but > > > > > once those are resolved, I think we can merge. > > > > > > > > > > > I also think the local-store path needs explicit usability and > > > > > operational criteria before schema work merges > > > > > > > > > > We can surely discuss these topics, but then we must be very clear > > > that we > > > > > are applying similar questions to all features proposed. Having to > > > answer > > > > > these questions for features has not been the case in the past - > and > > I > > > am > > > > > failing to understand why this bar is being attached specifically > to > > > the > > > > > OpenLineage feature. > > > > > > > > > > WRT Dmitri's retention concerns: We **should** build retention in > the > > > > > future - not just for Lineage events, but also for Event and > Metrics > > > > > Persistence. Basically it should apply to any entity that grows > > > linearly > > > > > with user queries. IMO retention is an orthogonal effort - I don't > > > think it > > > > > is fair to strongly attach it only to OpenLineage. We should surely > > > discuss > > > > > this in a different thread, ideally with someone willing to propose > > and > > > > > implement entity retention. WDYT? > > > > > > > > > > Best, > > > > > Adnan Hemani > > > > > > > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov < > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi All, > > > > > > > > > > > > I believe PR [4705] establishes a fairly isolated module > structure > > > for > > > > > > further OL work and does not imply much about the feature > itself. I > > > think > > > > > > it can be merged in its current state, provided the community > > agrees > > > on > > > > > the > > > > > > further feature direction. > > > > > > > > > > > > Regarding the specific Persistence implementation, from my POV > the > > > > > > following question also needs to be addressed: The proposal doc > [1] > > > > > assumes > > > > > > that the job edges are aggregated away from the OL input data and > > > that > > > > > the > > > > > > resulting inter-dataset graph is persisted with some retention > > > > > > characteristics. This is being discussed in the comment thread > > > rooted in > > > > > > the "Abstract" section (at the phrase: "maintains a directed > graph > > of > > > > > > dataset and field relationships"). However, I did not see a clear > > > > > > specification of this behaviour in the main doc body. > > > > > > > > > > > > I think this needs to be discussed in more detail, either in the > > doc > > > or > > > > > > (preferably) in this email thread. More specifically: > > > > > > > > > > > > * How / when such retention is performed. > > > > > > > > > > > > * How queries relate to data retention. Do they only "see" the > > latest > > > > > state > > > > > > of the inter-dataset graph or do they see several such graphs > > > changing > > > > > over > > > > > > time (within the retention window)? > > > > > > > > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate > > DDL > > > > > files > > > > > > for OL tables. I support this approach. I believe this will allow > > > for a > > > > > > smoother evolution of the OL schema with respect to MetaStore and > > > other > > > > > > schemas. Additionally this approach reduces the impact of OL > > changes > > > on > > > > > > users who elect not to enable this feature. > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o > > > > > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > Thanks, > > > > > > Dmitri. > > > > > > > > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> > > wrote: > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > I’m not trying to block incremental OpenLineage scaffolding, > but > > I > > > > > think > > > > > > we > > > > > > > should make explicit what the first merges do and do not > settle. > > > > > > > > > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they > are > > > > > > > explicitly > > > > > > > non-committal scaffolding. > > > > > > > > > > > > > > My concern is that these merges should not be read as consensus > > on > > > the > > > > > > > local > > > > > > > lineage backend, query model, persistence semantics, or JDBC > > > schema. > > > > > > > > > > > > > > In particular, I do not think #4826 should be read as settling > > how > > > the > > > > > > SPI > > > > > > > maps > > > > > > > to JDBC schema evolution, NoSQL backends, or backend capability > > > > > > > differences. > > > > > > > > > > > > > > I also think the local-store path needs explicit usability and > > > > > > operational > > > > > > > criteria before schema work merges: what queries are expected > to > > be > > > > > > > efficient, > > > > > > > what graph size or retention window we are designing for, what > > > > > write/read > > > > > > > load > > > > > > > Polaris should absorb, and what happens when lineage volume > > exceeds > > > > > those > > > > > > > bounds. > > > > > > > > > > > > > > Without that, we may merge a schema that is technically > workable > > > for > > > > > > small > > > > > > > examples but unclear for real lineage volume, query latency, > > > pruning, > > > > > and > > > > > > > backend load. > > > > > > > > > > > > > > Those need a separate decision before #4827 or equivalent > > > > > > > schema/persistence > > > > > > > work merges. > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > > [4826] https://github.com/apache/polaris/pull/4826 > > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > Thanks for the comments. Replying point-by-point: > > > > > > > > > > > > > > > > > re: How to use this data > > > > > > > > > > > > > > > > I agree, we should list this in the document itself so it > does > > > not > > > > > get > > > > > > > > lost. Let me put something in the appendix for now and link > it > > > to the > > > > > > > > relevant section so we don't lose the document's flow. Please > > > take a > > > > > > look > > > > > > > > there. > > > > > > > > > > > > > > > > > re: Link to OL Spec? > > > > > > > > > > > > > > > > https://openlineage.io/apidocs/openapi/ > > > > > > > > > > > > > > > > > re: Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > > > > > No, this is the LineagePersistent Contract (from earlier in > > this > > > > > > thread, > > > > > > > > linked again for comfort): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > > > > > > > . > > > > > > > > This was made to ensure that different persistence layers can > > > > > > implement a > > > > > > > > single interface for Lineage information. > > > > > > > > > > > > > > > > > re: Does OL data have to be located in the same database as > > > > > MetaStore > > > > > > > > data? > > > > > > > > > > > > > > > > I don't think there is a hard requirement for this. Happy to > > > discuss > > > > > if > > > > > > > > anyone else thinks differently! > > > > > > > > > > > > > > > > > I believe Polaris should allow downstream projects to opt > > > in/out of > > > > > > > > including > > > > > > > > OL REST APIs and related persistence code. This is the > general > > > > > approach > > > > > > > > we're following for the Metrics API proposal. > > > > > > > > > > > > > > > > Understood. I think this is an implementation detail and > you've > > > > > marked > > > > > > it > > > > > > > > on the PR as well. Let us take care of this discussion there. > > > > > > > > > > > > > > > > Best, > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov < > > > > > [email protected] > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > Commenting in general. > > > > > > > > > > > > > > > > > > I see a few PRs have been opened related to OpenLineage. > This > > > is > > > > > > fine. > > > > > > > > > > > > > > > > > > However, I believe we need to review some project-wide > > > approaches > > > > > > > related > > > > > > > > > to adding this feature (among other features). > > > > > > > > > > > > > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > > > > > > > 2) Does OL data have to be located in the same database as > > > > > MetaStore > > > > > > > > data? > > > > > > > > > > > > > > > > > > 3) I believe Polaris should allow downstream projects to > opt > > > in/out > > > > > > of > > > > > > > > > including OL REST APIs and related persistence code. This > is > > > the > > > > > > > general > > > > > > > > > approach we're following for the Metrics API proposal [1] > > > [4115] > > > > > > > > > > > > > > > > > > These points are not in opposition to supporting OL in > > Polaris. > > > > > These > > > > > > > are > > > > > > > > > merely technical concerns, which I think would benefit > > Polaris > > > as a > > > > > > > whole > > > > > > > > > if designed with flexibility and modularity in mind > upfront. > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > [1] > > > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > > > > > > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I > > previously > > > > > > posted > > > > > > > > in a > > > > > > > > > > different thread [1] was actually being found, given that > > it > > > was > > > > > > deep > > > > > > > > > into > > > > > > > > > > the thread. I request the community to review this > proposal > > > so we > > > > > > can > > > > > > > > > > potentially start implementation. > > > > > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > [1] > > > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
