Hi folks, Sorry for the delay. I have been traveling (headed to Community over Code Asia to present on Polaris tonight!), but I should be back next week to answer more questions.
re Dmitri's email: I think I now understand that the source of your confusion surrounds the timestamps we record in the OpenLineage event. Querying events should only show the timestamp as a metric of "freshness" of the connection between different datasets - and not as a historical log of all events that have ever occurred. Therefore, we don't need to provide the timestamp as a "queryable" field; it only serves to answer questions like, "When was the last time this dataset was loaded/used successfully?" Does that clarification help? re JB's email: Everything you've asked can still be answered from the original proposal document: https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1. I do think Dmitri's questions are relevant to the scope presented in this document and are good details to ensure we are all on the same page. Best, Adnan Hemani On Wed, Jul 29, 2026 at 9:51 PM Jean-Baptiste Onofré <[email protected]> wrote: > Hi folks, > > I am a bit confused by the current direction of this thread. > > As far as I recall, the initial goal for OpenLineage support in > Polaris was to act as a "gateway" to an OpenLineage backend (such as > Marquez), with the potential to enrich OpenLineage events. It feels > like the scope and purpose have shifted. > > Could we take a step back and return to the original problem > statement? What exactly are we trying to achieve with OpenLineage > support in Polaris, and what are the primary use cases? > > Clarifying these points should help us reach a consensus and move > forward with the implementation details. > > Thanks! > > Regards, > JB > > On Wed, Jul 22, 2026 at 3:29 PM Dmitri Bourlatchkov <[email protected]> > wrote: > > > > Hi Adnan, > > > > Just to clarify: My questions about data retention are not a blocker for > PR > > 4705. > > > > However, these questions are specific to the Open Lineage feature. The > > proposal talks about removing edge information and has references to data > > retention in the gdoc and its comments. > > > > If edges are removed, the resulting inter-dataset graph will represent > > "jammed" relationships for "all time", if old data is not removed. > > Therefore, understanding how OL-specific data is aggregated and retained > is > > critical to this feature. > > > > By comparison, metrics and events form a time-series data. Even without > > retention, the processing of this data is pretty clear. The lack of > > retention affects only storage (database) size. It does not affect > > processing semantics. > > > > Whether OL data queries are supposed to contain time-based clauses will > > affect the query API, which is in fact added in PR 4705 > > (cf. LineageQueryRequest). > > > > If queries are not supposed to contain time clauses (current state of PR > > 4705), then OL data retention becomes critical to understanding how the > > system will work end-to-end. It is not a simple matter of growing storage > > size, but a matter of query semantics. I believe it is essential to > clarify > > this aspect. > > > > That said, assuming the "beta" status of this feature and openness to > > follow-up changes, PR 4705 can probably be merged "as is", but these > > questions still need to be addressed before opening the feature to end > > users, I think. > > > > Cheers, > > Dmitri. > > > > On Tue, Jul 21, 2026 at 3:12 AM Adnan Hemani via dev < > [email protected]> > > wrote: > > > > > Hi folks, > > > > > > I agree that #4705 is mainly just scaffolding and doesn't direct much > about > > > the persistence contract. I have a few implementation-related > comments, but > > > once those are resolved, I think we can merge. > > > > > > > I also think the local-store path needs explicit usability and > > > operational criteria before schema work merges > > > > > > We can surely discuss these topics, but then we must be very clear > that we > > > are applying similar questions to all features proposed. Having to > answer > > > these questions for features has not been the case in the past - and I > am > > > failing to understand why this bar is being attached specifically to > the > > > OpenLineage feature. > > > > > > WRT Dmitri's retention concerns: We **should** build retention in the > > > future - not just for Lineage events, but also for Event and Metrics > > > Persistence. Basically it should apply to any entity that grows > linearly > > > with user queries. IMO retention is an orthogonal effort - I don't > think it > > > is fair to strongly attach it only to OpenLineage. We should surely > discuss > > > this in a different thread, ideally with someone willing to propose and > > > implement entity retention. WDYT? > > > > > > Best, > > > Adnan Hemani > > > > > > On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <[email protected]> > > > wrote: > > > > > > > Hi All, > > > > > > > > I believe PR [4705] establishes a fairly isolated module structure > for > > > > further OL work and does not imply much about the feature itself. I > think > > > > it can be merged in its current state, provided the community agrees > on > > > the > > > > further feature direction. > > > > > > > > Regarding the specific Persistence implementation, from my POV the > > > > following question also needs to be addressed: The proposal doc [1] > > > assumes > > > > that the job edges are aggregated away from the OL input data and > that > > > the > > > > resulting inter-dataset graph is persisted with some retention > > > > characteristics. This is being discussed in the comment thread > rooted in > > > > the "Abstract" section (at the phrase: "maintains a directed graph of > > > > dataset and field relationships"). However, I did not see a clear > > > > specification of this behaviour in the main doc body. > > > > > > > > I think this needs to be discussed in more detail, either in the doc > or > > > > (preferably) in this email thread. More specifically: > > > > > > > > * How / when such retention is performed. > > > > > > > > * How queries relate to data retention. Do they only "see" the latest > > > state > > > > of the inter-dataset graph or do they see several such graphs > changing > > > over > > > > time (within the retention window)? > > > > > > > > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL > > > files > > > > for OL tables. I support this approach. I believe this will allow > for a > > > > smoother evolution of the OL schema with respect to MetaStore and > other > > > > schemas. Additionally this approach reduces the impact of OL changes > on > > > > users who elect not to enable this feature. > > > > > > > > [1] > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> wrote: > > > > > > > > > Hi, > > > > > > > > > > I’m not trying to block incremental OpenLineage scaffolding, but I > > > think > > > > we > > > > > should make explicit what the first merges do and do not settle. > > > > > > > > > > I’m OK with #4705, and possibly #4826, moving forward if they are > > > > > explicitly > > > > > non-committal scaffolding. > > > > > > > > > > My concern is that these merges should not be read as consensus on > the > > > > > local > > > > > lineage backend, query model, persistence semantics, or JDBC > schema. > > > > > > > > > > In particular, I do not think #4826 should be read as settling how > the > > > > SPI > > > > > maps > > > > > to JDBC schema evolution, NoSQL backends, or backend capability > > > > > differences. > > > > > > > > > > I also think the local-store path needs explicit usability and > > > > operational > > > > > criteria before schema work merges: what queries are expected to be > > > > > efficient, > > > > > what graph size or retention window we are designing for, what > > > write/read > > > > > load > > > > > Polaris should absorb, and what happens when lineage volume exceeds > > > those > > > > > bounds. > > > > > > > > > > Without that, we may merge a schema that is technically workable > for > > > > small > > > > > examples but unclear for real lineage volume, query latency, > pruning, > > > and > > > > > backend load. > > > > > > > > > > Those need a separate decision before #4827 or equivalent > > > > > schema/persistence > > > > > work merges. > > > > > > > > > > Robert > > > > > > > > > > [4705] https://github.com/apache/polaris/pull/4705 > > > > > [4826] https://github.com/apache/polaris/pull/4826 > > > > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > > > > > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > > > > > [email protected]> wrote: > > > > > > > > > > > Hi Dmitri, > > > > > > > > > > > > Thanks for the comments. Replying point-by-point: > > > > > > > > > > > > > re: How to use this data > > > > > > > > > > > > I agree, we should list this in the document itself so it does > not > > > get > > > > > > lost. Let me put something in the appendix for now and link it > to the > > > > > > relevant section so we don't lose the document's flow. Please > take a > > > > look > > > > > > there. > > > > > > > > > > > > > re: Link to OL Spec? > > > > > > > > > > > > https://openlineage.io/apidocs/openapi/ > > > > > > > > > > > > > re: Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > No, this is the LineagePersistent Contract (from earlier in this > > > > thread, > > > > > > linked again for comfort): > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > > > > > . > > > > > > This was made to ensure that different persistence layers can > > > > implement a > > > > > > single interface for Lineage information. > > > > > > > > > > > > > re: Does OL data have to be located in the same database as > > > MetaStore > > > > > > data? > > > > > > > > > > > > I don't think there is a hard requirement for this. Happy to > discuss > > > if > > > > > > anyone else thinks differently! > > > > > > > > > > > > > I believe Polaris should allow downstream projects to opt > in/out of > > > > > > including > > > > > > OL REST APIs and related persistence code. This is the general > > > approach > > > > > > we're following for the Metrics API proposal. > > > > > > > > > > > > Understood. I think this is an implementation detail and you've > > > marked > > > > it > > > > > > on the PR as well. Let us take care of this discussion there. > > > > > > > > > > > > Best, > > > > > > Adnan Hemani > > > > > > > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov < > > > [email protected] > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Commenting in general. > > > > > > > > > > > > > > I see a few PRs have been opened related to OpenLineage. This > is > > > > fine. > > > > > > > > > > > > > > However, I believe we need to review some project-wide > approaches > > > > > related > > > > > > > to adding this feature (among other features). > > > > > > > > > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > > > > > > > > > 2) Does OL data have to be located in the same database as > > > MetaStore > > > > > > data? > > > > > > > > > > > > > > 3) I believe Polaris should allow downstream projects to opt > in/out > > > > of > > > > > > > including OL REST APIs and related persistence code. This is > the > > > > > general > > > > > > > approach we're following for the Metrics API proposal [1] > [4115] > > > > > > > > > > > > > > These points are not in opposition to supporting OL in Polaris. > > > These > > > > > are > > > > > > > merely technical concerns, which I think would benefit Polaris > as a > > > > > whole > > > > > > > if designed with flexibility and modularity in mind upfront. > > > > > > > > > > > > > > WDYT? > > > > > > > > > > > > > > [1] > > > https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > > > > > > > > > Thanks, > > > > > > > Dmitri. > > > > > > > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > > > > > [email protected]> wrote: > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I previously > > > > posted > > > > > > in a > > > > > > > > different thread [1] was actually being found, given that it > was > > > > deep > > > > > > > into > > > > > > > > the thread. I request the community to review this proposal > so we > > > > can > > > > > > > > potentially start implementation. > > > > > > > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > > > > > > > > > Best, > > > > > > > > Adnan Hemani > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
