Hi folks, I agree that #4705 is mainly just scaffolding and doesn't direct much about the persistence contract. I have a few implementation-related comments, but once those are resolved, I think we can merge.
> I also think the local-store path needs explicit usability and operational criteria before schema work merges We can surely discuss these topics, but then we must be very clear that we are applying similar questions to all features proposed. Having to answer these questions for features has not been the case in the past - and I am failing to understand why this bar is being attached specifically to the OpenLineage feature. WRT Dmitri's retention concerns: We **should** build retention in the future - not just for Lineage events, but also for Event and Metrics Persistence. Basically it should apply to any entity that grows linearly with user queries. IMO retention is an orthogonal effort - I don't think it is fair to strongly attach it only to OpenLineage. We should surely discuss this in a different thread, ideally with someone willing to propose and implement entity retention. WDYT? Best, Adnan Hemani On Mon, Jul 13, 2026 at 9:00 AM Dmitri Bourlatchkov <[email protected]> wrote: > Hi All, > > I believe PR [4705] establishes a fairly isolated module structure for > further OL work and does not imply much about the feature itself. I think > it can be merged in its current state, provided the community agrees on the > further feature direction. > > Regarding the specific Persistence implementation, from my POV the > following question also needs to be addressed: The proposal doc [1] assumes > that the job edges are aggregated away from the OL input data and that the > resulting inter-dataset graph is persisted with some retention > characteristics. This is being discussed in the comment thread rooted in > the "Abstract" section (at the phrase: "maintains a directed graph of > dataset and field relationships"). However, I did not see a clear > specification of this behaviour in the main doc body. > > I think this needs to be discussed in more detail, either in the doc or > (preferably) in this email thread. More specifically: > > * How / when such retention is performed. > > * How queries relate to data retention. Do they only "see" the latest state > of the inter-dataset graph or do they see several such graphs changing over > time (within the retention window)? > > Re: the OL-related JDBC schema, PR [4827] proposes using separate DDL files > for OL tables. I support this approach. I believe this will allow for a > smoother evolution of the OL schema with respect to MetaStore and other > schemas. Additionally this approach reduces the impact of OL changes on > users who elect not to enable this feature. > > [1] > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.gnz5xpjyw73o > > [4705] https://github.com/apache/polaris/pull/4705 > > [4827] https://github.com/apache/polaris/pull/4827 > > Thanks, > Dmitri. > > On Mon, Jul 13, 2026 at 7:55 AM Robert Stupp <[email protected]> wrote: > > > Hi, > > > > I’m not trying to block incremental OpenLineage scaffolding, but I think > we > > should make explicit what the first merges do and do not settle. > > > > I’m OK with #4705, and possibly #4826, moving forward if they are > > explicitly > > non-committal scaffolding. > > > > My concern is that these merges should not be read as consensus on the > > local > > lineage backend, query model, persistence semantics, or JDBC schema. > > > > In particular, I do not think #4826 should be read as settling how the > SPI > > maps > > to JDBC schema evolution, NoSQL backends, or backend capability > > differences. > > > > I also think the local-store path needs explicit usability and > operational > > criteria before schema work merges: what queries are expected to be > > efficient, > > what graph size or retention window we are designing for, what write/read > > load > > Polaris should absorb, and what happens when lineage volume exceeds those > > bounds. > > > > Without that, we may merge a schema that is technically workable for > small > > examples but unclear for real lineage volume, query latency, pruning, and > > backend load. > > > > Those need a separate decision before #4827 or equivalent > > schema/persistence > > work merges. > > > > Robert > > > > [4705] https://github.com/apache/polaris/pull/4705 > > [4826] https://github.com/apache/polaris/pull/4826 > > [4827] https://github.com/apache/polaris/pull/4827 > > > > > > On Tue, Jun 23, 2026 at 12:12 AM Adnan Hemani via dev < > > [email protected]> wrote: > > > > > Hi Dmitri, > > > > > > Thanks for the comments. Replying point-by-point: > > > > > > > re: How to use this data > > > > > > I agree, we should list this in the document itself so it does not get > > > lost. Let me put something in the appendix for now and link it to the > > > relevant section so we don't lose the document's flow. Please take a > look > > > there. > > > > > > > re: Link to OL Spec? > > > > > > https://openlineage.io/apidocs/openapi/ > > > > > > > re: Does OL persistence have to be limited to JDBC? > > > > > > No, this is the LineagePersistent Contract (from earlier in this > thread, > > > linked again for comfort): > > > > > > > > > https://docs.google.com/document/d/1sHFR_rVG9y-35iuy_92GpAPms3RMHS_sVPQqsHxnXHs/edit?tab=t.0 > > > . > > > This was made to ensure that different persistence layers can > implement a > > > single interface for Lineage information. > > > > > > > re: Does OL data have to be located in the same database as MetaStore > > > data? > > > > > > I don't think there is a hard requirement for this. Happy to discuss if > > > anyone else thinks differently! > > > > > > > I believe Polaris should allow downstream projects to opt in/out of > > > including > > > OL REST APIs and related persistence code. This is the general approach > > > we're following for the Metrics API proposal. > > > > > > Understood. I think this is an implementation detail and you've marked > it > > > on the PR as well. Let us take care of this discussion there. > > > > > > Best, > > > Adnan Hemani > > > > > > On Fri, Jun 19, 2026 at 12:34 PM Dmitri Bourlatchkov <[email protected] > > > > > wrote: > > > > > > > Hi All, > > > > > > > > Commenting in general. > > > > > > > > I see a few PRs have been opened related to OpenLineage. This is > fine. > > > > > > > > However, I believe we need to review some project-wide approaches > > related > > > > to adding this feature (among other features). > > > > > > > > 1) Does OL persistence have to be limited to JDBC? > > > > > > > > 2) Does OL data have to be located in the same database as MetaStore > > > data? > > > > > > > > 3) I believe Polaris should allow downstream projects to opt in/out > of > > > > including OL REST APIs and related persistence code. This is the > > general > > > > approach we're following for the Metrics API proposal [1] [4115] > > > > > > > > These points are not in opposition to supporting OL in Polaris. These > > are > > > > merely technical concerns, which I think would benefit Polaris as a > > whole > > > > if designed with flexibility and modularity in mind upfront. > > > > > > > > WDYT? > > > > > > > > [1] https://lists.apache.org/thread/3t79vpmm8bw38q411o36o85fqj6584t0 > > > > > > > > [4115] https://github.com/apache/polaris/pull/4115 > > > > > > > > Thanks, > > > > Dmitri. > > > > > > > > On Thu, May 14, 2026 at 12:59 AM Adnan Hemani via dev < > > > > [email protected]> wrote: > > > > > > > > > Hi all, > > > > > > > > > > I wanted to ensure that the OpenLineage proposal I previously > posted > > > in a > > > > > different thread [1] was actually being found, given that it was > deep > > > > into > > > > > the thread. I request the community to review this proposal so we > can > > > > > potentially start implementation. > > > > > > > > > > Proposal: > > > > > > > > > > > > > > > > > > > > https://docs.google.com/document/d/1iOzIuFW66SFL2wZOADD9knMTG21OwY7VmaWVSvMUqQk/edit?tab=t.0#heading=h.59bmbnsf0gp1 > > > > > > > > > > Best, > > > > > Adnan Hemani > > > > > > > > > > [1] > https://lists.apache.org/thread/1fd6hrvx0v0s5wm6gh74cdo3yn4w1zhx > > > > > > > > > > > > > > >
