flyingImer commented on code in PR #4667: URL: https://github.com/apache/polaris/pull/4667#discussion_r3406686606
########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApiService.java: ########## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; + +/** + * Service interface implemented by the runtime to handle OpenLineage ingest. Mirrors the pattern + * used by other Polaris API modules where the JAX-RS resource sits in the API module and + * delegates to a CDI-scoped service implementation in {@code polaris-runtime-service}. + */ +public interface PolarisOpenLineageApiService { + + /** + * Handle an OpenLineage event accepted at the ingest endpoint. + * + * @param event the parsed OpenLineage event, dispatched to the correct {@code RunEvent}, {@code + * JobEvent}, or {@code DatasetEvent} variant by Jackson based on the {@code schemaURL} + * field. + * @return the JAX-RS response. OpenLineage clients expect {@code 201 Created} with no body on + * success. + */ + Response sendLineageEvent( Review Comment: Could the replacement point behind this service use OpenLineage-specific provider request/result types rather than JAX-RS/runtime types? For example: ```text OpenLineageIngestProvider.ingest(OpenLineageIngestRequest) -> OpenLineageIngestResult ``` This would let this API service stay responsible for mapping HTTP/runtime context to a provider call and mapping the provider result back to Response, while giving downstream deployments a provider contract they can implement without patching the JAX-RS resource or depending on Response/SecurityContext as the extension contract. ########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApiService.java: ########## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; + +/** + * Service interface implemented by the runtime to handle OpenLineage ingest. Mirrors the pattern + * used by other Polaris API modules where the JAX-RS resource sits in the API module and + * delegates to a CDI-scoped service implementation in {@code polaris-runtime-service}. Review Comment: This Javadoc usefully frames the type as the runtime service interface behind the JAX-RS resource. I think that distinction is important to preserve. My concern is not that this PR claims this is the lineage SPI; it does not. My concern is that this is currently the only seam behind the OpenLineage route. Since the method returns JAX-RS `Response` and accepts request/security context, I'd prefer to keep this as API/runtime delegation and introduce a separate OpenLineage-specific provider seam behind it for follow-up implementations. ########## runtime/service/src/main/java/org/apache/polaris/service/lineage/OpenLineageAdapter.java: ########## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage; + +import jakarta.enterprise.context.RequestScoped; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.service.lineage.api.PolarisLineageEvent; +import org.apache.polaris.service.lineage.api.PolarisOpenLineageApiService; + +/** + * No-op implementation of the OpenLineage ingest endpoint. + * + * <p>Accepts and discards events. Persistence, dataset resolution, and downstream forwarding will + * land in follow-up PRs as described in the Polaris OpenLineage proposal. + */ +@RequestScoped +public class OpenLineageAdapter implements PolarisOpenLineageApiService { Review Comment: The no-op scope is clear here, which I like. Can we make the no-op implementation use the same replacement point that future real implementations will use? Right now the no-op behavior lives directly in this runtime service implementation. That works for route bring-up, but it does not show where a future store/forward/custom OpenLineage implementation should plug in without modifying this runtime service. Early no-op implementations often become the extension point by inertia. If follow-up PRs add persistence or forwarding directly to this runtime service, then replacing lineage behavior means replacing runtime wiring rather than implementing a clean provider contract. That is exactly the kind of boundary ambiguity the SPI work is trying to avoid. For the initial pr, I think it would be enough to add a small `OpenLineageIngestProvider` contract plus a default `NoOpOpenLineageIngestProvider`, and have this runtime adapter call it. The default provider can still be selected/wired by CDI if that is the runtime mechanism; the key is that CDI selection remains runtime wiring, not the provider contract itself. ########## runtime/service/src/main/java/org/apache/polaris/service/lineage/OpenLineageAdapter.java: ########## @@ -0,0 +1,42 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage; + +import jakarta.enterprise.context.RequestScoped; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.service.lineage.api.PolarisLineageEvent; +import org.apache.polaris.service.lineage.api.PolarisOpenLineageApiService; + +/** + * No-op implementation of the OpenLineage ingest endpoint. + * + * <p>Accepts and discards events. Persistence, dataset resolution, and downstream forwarding will + * land in follow-up PRs as described in the Polaris OpenLineage proposal. + */ +@RequestScoped +public class OpenLineageAdapter implements PolarisOpenLineageApiService { + + @Override + public Response sendLineageEvent( + PolarisLineageEvent event, RealmContext realmContext, SecurityContext securityContext) { + return Response.status(Response.Status.CREATED).build(); Review Comment: Could this `201` come from a provider result instead of being the whole implementation? For example, even if the only provider in PR1 is no-op: ```java OpenLineageIngestResult result = provider.ingest(request); return toResponse(result); ``` and the no-op provider always returns accepted. The HTTP status is runtime behavior, while accept/reject/unavailable is provider behavior. Keeping those separate lets future implementations change lineage handling without owning JAX-RS response construction. It also gives us a place to define provider semantics before persistence/query/forwarding land. That small indirection would make the intended OpenLineage replacement point concrete without adding persistence or forwarding yet. Future implementations would change provider behavior, while this runtime class would stay responsible for HTTP response mapping. ########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/LineageEventTypeResolver.java: ########## @@ -0,0 +1,90 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import com.fasterxml.jackson.annotation.JsonTypeInfo; +import com.fasterxml.jackson.databind.DatabindContext; +import com.fasterxml.jackson.databind.JavaType; +import com.fasterxml.jackson.databind.jsontype.impl.TypeIdResolverBase; + +/** + * Resolves an OpenLineage event JSON body to one of {@link PolarisLineageEvent.OfRunEvent}, {@link + * PolarisLineageEvent.OfJobEvent}, or {@link PolarisLineageEvent.OfDatasetEvent} by inspecting the + * trailing path segment of the {@code schemaURL} field. + * + * <p>The OpenLineage spec requires every event to include a {@code schemaURL} pointing at the + * variant's JSON schema fragment, e.g. {@code + * https://openlineage.io/spec/2-0-2/OpenLineage.json#/$defs/RunEvent}. The fragment name is the + * stable discriminator across spec versions. + * + * <p>If {@code schemaURL} is missing or unrecognized, the body is parsed as a {@code RunEvent} — Review Comment: The Marquez-compatible fallback is reasonable as an OpenLineage ingress behavior, iiuc. I'd just like to keep this scoped to the OL adapter layer. The Polaris provider seam behind the adapter should receive a Polaris-owned request/result shape after this compatibility decision has been made, rather than requiring provider implementations to understand OL `schemaURL` fallback behavior. ########## spec/openlineage-service.yaml: ########## @@ -0,0 +1,132 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, +# software distributed under the License is distributed on an +# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY +# KIND, either express or implied. See the License for the +# specific language governing permissions and limitations +# under the License. +# + +# This file documents the OpenLineage ingest endpoint exposed by Apache Polaris. +# +# IMPORTANT: This spec is documentation, not a code-generation source. The +# OpenLineage event schema is a `oneOf` over RunEvent / JobEvent / DatasetEvent +# discriminated by the `schemaURL` field, which the OpenAPI Generator's Java +# templates cannot translate faithfully (they collapse the variants into a +# single class with every variant's required fields marked `@NotNull`, +# rejecting every valid event with a 400). Polaris instead hand-writes the +# JAX-RS resource on top of the official `io.openlineage:openlineage-java` +# library, exactly as Marquez (the OpenLineage reference server) does. See +# `api/openlineage-service/` for the implementation. +# +# The authoritative event schema lives upstream: +# https://openlineage.io/spec/2-0-2/OpenLineage.json +# +# This file is kept so the OpenLineage endpoint shows up alongside Polaris's +# other API specs and can be browsed by API tools. + +--- +openapi: 3.0.3 +info: + title: Apache Polaris OpenLineage Service + license: + name: Apache 2.0 + url: https://www.apache.org/licenses/LICENSE-2.0.html + version: 0.0.1 + description: + Defines the OpenLineage-compatible ingest API exposed by Apache Polaris. + Polaris accepts OpenLineage events at the standard OpenLineage path so + engines (Spark, Flink, Airflow, Trino, dbt) can be reconfigured by URL + alone. Event bodies follow the OpenLineage specification at + https://openlineage.io/spec/2-0-2/OpenLineage.json. + +servers: + - url: "{scheme}://{host}/api/v1" + description: Server URL when the port can be inferred from the scheme + variables: + scheme: + description: The scheme of the URI, either http or https. + default: https + host: + description: The host address for the specified server + default: localhost + +security: + - OAuth2: [] + - BearerAuth: [] + +paths: + /lineage: + post: + tags: + - OpenLineage API + summary: Submit an OpenLineage event + description: + Accepts an OpenLineage RunEvent, JobEvent, or DatasetEvent as defined + by the OpenLineage specification + (https://openlineage.io/spec/2-0-2/OpenLineage.json). + + + The body is dispatched to the correct event variant by inspecting the + `schemaURL` field — the fragment after the final `/` selects RunEvent, + JobEvent, or DatasetEvent. Unrecognized values fall back to RunEvent. + + + This endpoint is the standard OpenLineage ingest path, so existing + OpenLineage transports can target Polaris by URL change alone. + operationId: sendLineageEvent + requestBody: + description: + An OpenLineage event. The exact body schema is the upstream + OpenLineage 2-0-2 JSON Schema referenced above. + required: true + content: + application/json: + schema: + type: object + additionalProperties: true + externalDocs: + description: OpenLineage 2-0-2 JSON Schema (authoritative) + url: https://openlineage.io/spec/2-0-2/OpenLineage.json + responses: + 201: + description: + Created. The event was accepted for processing. Per the + OpenLineage server specification the response body is empty. + 400: + description: Bad Request - The request body is not a valid OpenLineage event + 401: + description: Unauthorized - The caller is not authenticated + 403: + description: Forbidden - The caller is not authorized to submit lineage events + 503: + description: Service Unavailable - The lineage subsystem is temporarily unable to accept events + 5XX: + description: Server error + +components: + securitySchemes: + OAuth2: + type: oauth2 + description: + OAuth2 client-credentials flow against the Polaris token endpoint. + The same client-id/secret used for catalog access is used here; only + an additional LINEAGE_INGEST privilege grant is required. Review Comment: This seems ahead of the implementation in this PR. The spec says the same client credentials need an additional `LINEAGE_INGEST` privilege, but the resource currently uses `@RolesAllowed("**")`. If privilege enforcement is not part of this pr, can we either remove this as current-contract language or mark it clearly as future work? ########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApi.java: ########## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import io.micrometer.core.annotation.Timed; +import io.micrometer.core.aop.MeterTag; +import jakarta.annotation.security.RolesAllowed; +import jakarta.inject.Inject; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.core.Context; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.eclipse.microprofile.faulttolerance.Timeout; + +/** + * JAX-RS resource for the OpenLineage ingest endpoint. + * + * <p>Mounted at the standard OpenLineage path ({@code POST /api/v1/lineage}) so that any engine + * already using the OpenLineage HTTP transport (Spark, Flink, Airflow, Trino, dbt) can target + * Polaris by URL change alone — no client-side rewriting. + * + * <p>The body is parsed into a {@link PolarisLineageEvent} which Jackson resolves to one of {@code + * OfRunEvent} / {@code OfJobEvent} / {@code OfDatasetEvent} based on the {@code schemaURL} field. + * Unrecognized {@code schemaURL} values fall back to {@code RunEvent}, matching Marquez behavior. + * + * <p>This resource is hand-written rather than generated from the OpenLineage JSON Schema because + * the OpenAPI generator's Java template does not faithfully translate the spec's {@code oneOf} + * over event variants. + */ +@Path("/api/v1/lineage") Review Comment: I'm worried this makes a one-way-door API decision. `/api/v1/lineage` reads like the generic Polaris lineage API, but this PR is specifically adding an OpenLineage-compatible ingest endpoint. Once this public route exists, it becomes hard to later introduce a Polaris-native lineage API or another lineage format without either overloading this path or adding a second, awkward namespace. From recent community discussions, Polaris is trying to be a platform with replaceable capabilities, not just an opinionated OpenLineage receiver. If the first public lineage route is generic but semantically OpenLineage-specific, OpenLineage becomes the default interpretation of "Polaris lineage" at the API boundary. That is a hard thing to unwind later because REST paths are external contracts. Could we either make the OL specificity visible in the route/API shape, or at least make it very explicit in naming/docs that this is the OpenLineage ingress endpoint and not the generic Polaris lineage API? That keeps room for a future lineage surface, if one is ever needed, without making OpenLineage the default meaning of "Polaris lineage" at the API boundary. ########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApi.java: ########## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import io.micrometer.core.annotation.Timed; +import io.micrometer.core.aop.MeterTag; +import jakarta.annotation.security.RolesAllowed; +import jakarta.inject.Inject; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.core.Context; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.eclipse.microprofile.faulttolerance.Timeout; + +/** + * JAX-RS resource for the OpenLineage ingest endpoint. + * + * <p>Mounted at the standard OpenLineage path ({@code POST /api/v1/lineage}) so that any engine + * already using the OpenLineage HTTP transport (Spark, Flink, Airflow, Trino, dbt) can target + * Polaris by URL change alone — no client-side rewriting. + * + * <p>The body is parsed into a {@link PolarisLineageEvent} which Jackson resolves to one of {@code + * OfRunEvent} / {@code OfJobEvent} / {@code OfDatasetEvent} based on the {@code schemaURL} field. + * Unrecognized {@code schemaURL} values fall back to {@code RunEvent}, matching Marquez behavior. + * + * <p>This resource is hand-written rather than generated from the OpenLineage JSON Schema because + * the OpenAPI generator's Java template does not faithfully translate the spec's {@code oneOf} + * over event variants. + */ +@Path("/api/v1/lineage") +public class PolarisOpenLineageApi { + + private final PolarisOpenLineageApiService service; + + @Inject + public PolarisOpenLineageApi(PolarisOpenLineageApiService service) { + this.service = service; + } + + @POST + @Consumes("application/json") + @RolesAllowed("**") + @Timed("polaris.OpenLineageApi.sendLineageEvent") + @Timeout + public Response sendLineageEvent( + @NotNull @Valid PolarisLineageEvent event, Review Comment: Could we avoid making the JAX-RS method's parsed body type the object that follow-up implementations build around? Right now the first object passed past the REST resource is `PolarisLineageEvent`, which wraps the current OpenLineage Java event types. That is reasonable inside an OpenLineage parser/adapter, but I'd prefer the replacement seam to be an explicitly OpenLineage ingest request/result contract rather than this runtime method signature. The first object passed across the runtime boundary tends to become the object that follow-up persistence, forwarding, and deployment-specific implementations accept in their APIs and tests. If that object is the JAX-RS method parameter, then the extension point is shaped by the resource binding and the current OpenLineage Java model, rather than by a small provider contract that Polaris controls. That makes it harder to evolve the HTTP adapter independently from provider implementations. Concretely, could the runtime path look more like: ```text raw OpenLineage request -> parse/validate as OpenLineage -> build OpenLineageIngestRequest -> call OpenLineageIngestProvider ``` rather than having future provider implementations build directly around the JAX-RS method parameter? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
