Hi All, PR [4705] LGTM as an incremental step for the OL feature. I approved it in GH.
I hope to post a review for [4667] soon. [4705] https://github.com/apache/polaris/pull/4705 [4667] https://github.com/apache/polaris/pull/4667 Cheers, Dmitri. On Fri, Aug 7, 2026 at 11:13 PM Adnan Hemani via dev <[email protected]> wrote: > Hi Dmitri, > > Great question again! I think the plan still remains to ensure the OL event > forwarding works first, while we continue making progress on the local > storage in parallel. Here's how I see this in terms of the PRs currently > open for review: > * PR #4667: This is the baseline PR which adds the APIs required for > OpenLineage compatibility. This PR is required regardless of local storage > or forwarding modes. However, this PR is currently a blocker only for the > forwarding mode, as local storage mode has other work that can be completed > independently. > * PR #4705: This introduces POJOs and scaffolding for the future local > storage mode - but does not actually introduce the local storage > functionality. It's good to keep this PR moving forward while contributors > are excited to make progress on it! > > Best, > Adnan Hemani > > On Thu, Aug 6, 2026 at 4:02 AM Dmitri Bourlatchkov <[email protected]> > wrote: > > > Hi Adnan, > > > > Thanks for the update. This removes my concern with timestamps in the > > lineage queries. > > > > Re: local storage and forwarding to Marquez (or similar), it looks like > > current outstanding lineage PRs only deal with the local storage aspect > > now. > > > > So to clarify the proposal: Are you suggesting that we implement local > > storage first, and deal with forwarding later? My reading of JB's email > is > > that he might have been expecting the implementation of these phases to > > occur in the reverse order. > > > > Cheers, > > Dmitri, > > > > On Tue, Aug 4, 2026 at 9:46 PM Adnan Hemani via dev < > > [email protected]> > > wrote: > > > > > Hi folks, > > > > > > Sorry for the delay. I have been traveling (headed to Community over > Code > > > Asia to present on Polaris tonight!), but I should be back next week to > > > answer more questions. > > > > > > re Dmitri's email: I think I now understand that the source of your > > > confusion surrounds the timestamps we record in the OpenLineage event. > > > Querying events should only show the timestamp as a metric of > "freshness" > > > of the connection between different datasets - and not as a historical > > log > > > of all events that have ever occurred. Therefore, we don't need to > > provide > > > the timestamp as a "queryable" field; it only serves to answer > questions > > > like, "When was the last time this dataset was loaded/used > successfully?" > > > Does that clarification help? > > > > > > re JB's email: Everything you've asked can still be answered from the > > > original proposal document: > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > . > > > I do think Dmitri's questions are relevant to the scope presented in > this > > > document and are good details to ensure we are all on the same page. > > > > > > Best, > > > Adnan Hemani > > > > > > On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <[email protected]> > > > wrote: > > > > > > > Hi folks, > > > > > > > > I am a bit confused by the current direction of this thread. > > > > > > > > As far as I recall, the initial goal for OpenLineage support in > > > > Polaris was to act as a "gateway" to an OpenLineage backend (such as > > > > Marquez), with the potential to enrich OpenLineage events. It feels > > > > like the scope and purpose have shifted. > > > > > > > > Could we take a step back and return to the original problem > > > > statement? What exactly are we trying to achieve with OpenLineage > > > > support in Polaris, and what are the primary use cases? > > > > > > > > Clarifying these points should help us reach a consensus and move > > > > forward with the implementation details. > > > > > > > > Thanks! > > > > > > > > Regards, > > > > JB > > > > > > > > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov < > [email protected]> > > > > wrote: > > > > > > > > > > Hi Adnan, > > > > > > > > > > Just to clarify: My questions about data retention are not a > blocker > > > for > > > > PR > > > > > 4705. > > > > > > > > > > However, these questions are specific to the Open Lineage feature. > > The > > > > > proposal talks about removing edge information and has references > to > > > data > > > > > retention in the gdoc and its comments. > > > > > > > > > > If edges are removed, the resulting inter-dataset graph will > > represent > > > > > "jammed" relationships for "all time", if old data is not removed. > > > > > Therefore, understanding how OL-specific data is aggregated and > > > retained > > > > is > > > > > critical to this feature. > > > > > > > > > > By comparison, metrics and events form a time-series data. Even > > without > > > > > retention, the processing of this data is pretty clear. The lack of > > > > > retention affects only storage (database) size. It does not affect > > > > > processing semantics. > > > > > > > > > > Whether OL data queries are supposed to contain time-based clauses > > will > > > > > affect the query API, which is in fact added in PR 4705 > > > > > (cf. LineageQueryRequest). > > > > > > > > > > If queries are not supposed to contain time clauses (current state > of > > > PR > > > > > 4705), then OL data retention becomes critical to understanding how > > the > > > > > system will work end-to-end. It is not a simple matter of growing > > > storage > > > > > size, but a matter of query semantics. I believe it is essential to > > > > clarify > > > > > this aspect. > > > > > > > > > > That said, assuming the "beta" status of this feature and openness > to > > > > > follow-up changes, PR 4705 can probably be merged "as is", but > these > > > > > questions still need to be addressed before opening the feature to > > end > > > > > users, I think. > > > > > > > > > > Cheers, > > > > > Dmitri. > > > > > > > > > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev < > > > > [email protected]> > > > > > wrote: > > > > > > > > > > > Hi folks, > > > > > > > > > > > > I agree that #4705 is mainly just scaffolding and doesn't direct > > much > > > > about > > > > > > the persistence contract. I have a few implementation-related > > > > comments, but > > > > > > once those are resolved, I think we can merge. > > > > > > > > > > > > > I also think the local-store path needs explicit usability and > > > > > > operational criteria before schema work merges > > > > > > > > > > > > We can surely discuss these topics, but then we must be very > clear > > > > that we > > > > > > are applying similar questions to all features proposed. Having > to > > > > answer > > > > > > these questions for features has not been the case in the past - > > and > > > I > > > > am > > > > > > failing to understand why this bar is being attached specifically > > to > > > > the > > > > > > OpenLineage feature. > > > > > > > > > > > > WRT Dmitri's retention concerns: We **should** build retention in > > the > > > > > > future - not just for Lineage events, but also for Event and > > Metrics > > > > > > Persistence. Basically it should apply to any entity that grows > > > > linearly > > > > > > with user queries. IMO retention is an orthogonal effort - I > don't > > > > think it > > > > > > is fair to strongly attach it only to OpenLineage. We should > surely > > > > discuss > > > > > > this in a different thread, ideally with someone willing to > propose > > > and > > > > > > implement entity retention. WDYT? > > > > > > > > > > > > Best, > > > > > > Adnan Hemani > > > > > > > > > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov < > > > [email protected]> > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > I believe PR [4705] establishes a fairly isolated module > > structure > > > > for > > > > > > > further OL work and does not imply much about the feature > > itself. I > > > > think > > > > > > > it can be merged in its current state, provided the community > > > agrees > > > > on > > > > > > the > > > > > > > further feature direction. > > > > > > > > > > > > > > Regarding the specific Persistence implementation, from my POV > > the > > > > > > > following question also needs to be addressed: The proposal doc > > [1] > > > > > > assumes > > > > > > > that the job edges are aggregated away from the OL input data > and > > > > that > > > > > > the > > > > > > > resulting inter-dataset graph is persisted with some retention > > > > > > > characteristics. This is being discussed in the comment thread > > > > rooted in > > > > > > > the "Abstract" section (at the phrase: "maintains a directed > > graph > > > of > > > > > > > dataset and field relationships"). However, I did not see a > clear > > > > > > > specification of this behaviour in the main doc body. > > > > > > > > > > > > > > I think this needs to be discussed in more detail, either in > the > > > doc > > > > or > > > > > > > (preferably) in this email thread. More specifically: > > > > > > > > > > > > > > * How / when such retention is performed. > > > > > > > > > > > > > > * How queries relate to data retention. Do they only "see" the > > > latest > > > > > > state > > > > > > > of the inter-dataset graph or do they see several such graphs > > > > changing > > > > > > over > > > > > > > time (within the retention window)? > > > > > > > > > > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using > separate > > > DDL > > > > > > files > > > > > > > for OL tables. I support this approach. I believe this will > allow > > > > for a > > > > > > > smoother evolution of the OL schema with respect to MetaStore > and > > > > other > > > > > > > schemas. Additionally this approach reduces the impact of OL > > > changes > > > > on > > > > > > > users who elect not to enable this feature. > > > > > > > > > > > > > > [1] > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o > > > > > > > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > > > > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> > > > wrote: > > > > > > > > > > > > > > > Hi, > > > > > > > > > > > > > > > > I’m not trying to block incremental OpenLineage scaffolding, > > but > > > I > > > > > > think > > > > > > > we > > > > > > > > should make explicit what the first merges do and do not > > settle. > > > > > > > > > > > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they > > are > > > > > > > > explicitly > > > > > > > > non-committal scaffolding. > > > > > > > > > > > > > > > > My concern is that these merges should not be read as > consensus > > > on > > > > the > > > > > > > > local > > > > > > > > lineage backend, query model, persistence semantics, or JDBC > > > > schema. > > > > > > > > > > > > > > > > In particular, I do not think #4826 should be read as > settling > > > how > > > > the > > > > > > > SPI > > > > > > > > maps > > > > > > > > to JDBC schema evolution, NoSQL backends, or backend > capability > > > > > > > > differences. > > > > > > > > > > > > > > > > I also think the local-store path needs explicit usability > and > > > > > > > operational > > > > > > > > criteria before schema work merges: what queries are expected > > to > > > be > > > > > > > > efficient, > > > > > > > > what graph size or retention window we are designing for, > what > > > > > > write/read > > > > > > > > load > > > > > > > > Polaris should absorb, and what happens when lineage volume > > > exceeds > > > > > > those > > > > > > > > bounds. > > > > > > > > > > > > > > > > Without that, we may merge a schema that is technically > > workable > > > > for > > > > > > > small > > > > > > > > examples but unclear for real lineage volume, query latency, > > > > pruning, > > > > > > and > > > > > > > > backend load. > > > > > > > > > > > > > > > > Those need a separate decision before #4827 or equivalent > > > > > > > > schema/persistence > > > > > > > > work merges. > > > > > > > > > > > > > > > > Robert > > > > > > > > > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > > > [4826] https://github.com/apache/polaris/pull/4826 > > > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > > > > > > > Thanks for the comments. Replying point-by-point: > > > > > > > > > > > > > > > > > > > re: How to use this data > > > > > > > > > > > > > > > > > > I agree, we should list this in the document itself so it > > does > > > > not > > > > > > get > > > > > > > > > lost. Let me put something in the appendix for now and link > > it > > > > to the > > > > > > > > > relevant section so we don't lose the document's flow. > Please > > > > take a > > > > > > > look > > > > > > > > > there. > > > > > > > > > > > > > > > > > > > re: Link to OL Spec? > > > > > > > > > > > > > > > > > > https://openlineage.io/apidocs/openapi/ > > > > > > > > > > > > > > > > > > > re: Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > > > > > > > No, this is the LineagePersistent Contract (from earlier in > > > this > > > > > > > thread, > > > > > > > > > linked again for comfort): > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > > > > > > > > . > > > > > > > > > This was made to ensure that different persistence layers > can > > > > > > > implement a > > > > > > > > > single interface for Lineage information. > > > > > > > > > > > > > > > > > > > re: Does OL data have to be located in the same database > as > > > > > > MetaStore > > > > > > > > > data? > > > > > > > > > > > > > > > > > > I don't think there is a hard requirement for this. Happy > to > > > > discuss > > > > > > if > > > > > > > > > anyone else thinks differently! > > > > > > > > > > > > > > > > > > > I believe Polaris should allow downstream projects to opt > > > > in/out of > > > > > > > > > including > > > > > > > > > OL REST APIs and related persistence code. This is the > > general > > > > > > approach > > > > > > > > > we're following for the Metrics API proposal. > > > > > > > > > > > > > > > > > > Understood. I think this is an implementation detail and > > you've > > > > > > marked > > > > > > > it > > > > > > > > > on the PR as well. Let us take care of this discussion > there. > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov < > > > > > > [email protected] > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > > > > > > > Commenting in general. > > > > > > > > > > > > > > > > > > > > I see a few PRs have been opened related to OpenLineage. > > This > > > > is > > > > > > > fine. > > > > > > > > > > > > > > > > > > > > However, I believe we need to review some project-wide > > > > approaches > > > > > > > > related > > > > > > > > > > to adding this feature (among other features). > > > > > > > > > > > > > > > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > > > > > > > > > 2) Does OL data have to be located in the same database > as > > > > > > MetaStore > > > > > > > > > data? > > > > > > > > > > > > > > > > > > > > 3) I believe Polaris should allow downstream projects to > > opt > > > > in/out > > > > > > > of > > > > > > > > > > including OL REST APIs and related persistence code. This > > is > > > > the > > > > > > > > general > > > > > > > > > > approach we're following for the Metrics API proposal [1] > > > > [4115] > > > > > > > > > > > > > > > > > > > > These points are not in opposition to supporting OL in > > > Polaris. > > > > > > These > > > > > > > > are > > > > > > > > > > merely technical concerns, which I think would benefit > > > Polaris > > > > as a > > > > > > > > whole > > > > > > > > > > if designed with flexibility and modularity in mind > > upfront. > > > > > > > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > > > > > > > [1] > > > > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > > > > > > > > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > > Dmitri. > > > > > > > > > > > > > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I > > > previously > > > > > > > posted > > > > > > > > > in a > > > > > > > > > > > different thread [1] was actually being found, given > that > > > it > > > > was > > > > > > > deep > > > > > > > > > > into > > > > > > > > > > > the thread. I request the community to review this > > proposal > > > > so we > > > > > > > can > > > > > > > > > > > potentially start implementation. > > > > > > > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > > > > > > > [1] > > > > > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
