adnanhemani commented on code in PR #4667: URL: https://github.com/apache/polaris/pull/4667#discussion_r3416672278
########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApi.java: ########## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import io.micrometer.core.annotation.Timed; +import io.micrometer.core.aop.MeterTag; +import jakarta.annotation.security.RolesAllowed; +import jakarta.inject.Inject; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.core.Context; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.eclipse.microprofile.faulttolerance.Timeout; + +/** + * JAX-RS resource for the OpenLineage ingest endpoint. + * + * <p>Mounted at the standard OpenLineage path ({@code POST /api/v1/lineage}) so that any engine + * already using the OpenLineage HTTP transport (Spark, Flink, Airflow, Trino, dbt) can target + * Polaris by URL change alone — no client-side rewriting. + * + * <p>The body is parsed into a {@link PolarisLineageEvent} which Jackson resolves to one of {@code + * OfRunEvent} / {@code OfJobEvent} / {@code OfDatasetEvent} based on the {@code schemaURL} field. + * Unrecognized {@code schemaURL} values fall back to {@code RunEvent}, matching Marquez behavior. + * + * <p>This resource is hand-written rather than generated from the OpenLineage JSON Schema because + * the OpenAPI generator's Java template does not faithfully translate the spec's {@code oneOf} + * over event variants. + */ +@Path("/api/v1/lineage") Review Comment: The path `/api/v1/lineage` is the standard OpenLineage HTTP transport endpoint — it's what Marquez serves and what Spark/Flink/Airflow/dbt/Trino target when you set `transport.type=http`. The PR's core value proposition is that engines can point at Polaris with a URL change alone; a non-standard path defeats that. That said, the concern about this path becoming the default interpretation of "Polaris lineage" is real. With the `OpenLineageIngestProvider` layer the internals are unambiguously OL-specific regardless of the path, but the external path is still a one-way door. I'll make the OL-specificity explicit in the Javadoc and YAML spec. If there's appetite for a broader namespace discussion I'm happy to raise it on the dev list — though I'd want to be careful not to break wire compatibility in the process. ########## api/openlineage-service/src/main/java/org/apache/polaris/service/lineage/api/PolarisOpenLineageApi.java: ########## @@ -0,0 +1,74 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage.api; + +import io.micrometer.core.annotation.Timed; +import io.micrometer.core.aop.MeterTag; +import jakarta.annotation.security.RolesAllowed; +import jakarta.inject.Inject; +import jakarta.validation.Valid; +import jakarta.validation.constraints.NotNull; +import jakarta.ws.rs.Consumes; +import jakarta.ws.rs.POST; +import jakarta.ws.rs.Path; +import jakarta.ws.rs.core.Context; +import jakarta.ws.rs.core.Response; +import jakarta.ws.rs.core.SecurityContext; +import org.apache.polaris.core.context.RealmContext; +import org.eclipse.microprofile.faulttolerance.Timeout; + +/** + * JAX-RS resource for the OpenLineage ingest endpoint. + * + * <p>Mounted at the standard OpenLineage path ({@code POST /api/v1/lineage}) so that any engine + * already using the OpenLineage HTTP transport (Spark, Flink, Airflow, Trino, dbt) can target + * Polaris by URL change alone — no client-side rewriting. + * + * <p>The body is parsed into a {@link PolarisLineageEvent} which Jackson resolves to one of {@code + * OfRunEvent} / {@code OfJobEvent} / {@code OfDatasetEvent} based on the {@code schemaURL} field. + * Unrecognized {@code schemaURL} values fall back to {@code RunEvent}, matching Marquez behavior. + * + * <p>This resource is hand-written rather than generated from the OpenLineage JSON Schema because + * the OpenAPI generator's Java template does not faithfully translate the spec's {@code oneOf} + * over event variants. + */ +@Path("/api/v1/lineage") +public class PolarisOpenLineageApi { + + private final PolarisOpenLineageApiService service; + + @Inject + public PolarisOpenLineageApi(PolarisOpenLineageApiService service) { + this.service = service; + } + + @POST + @Consumes("application/json") + @RolesAllowed("**") + @Timed("polaris.OpenLineageApi.sendLineageEvent") + @Timeout + public Response sendLineageEvent( + @NotNull @Valid PolarisLineageEvent event, Review Comment: With the `OpenLineageIngestProvider` seam, `PolarisLineageEvent` stays inside the resource + adapter and never crosses the provider boundary. Follow-up implementations only see `OpenLineageIngestRequest`, which Polaris controls and can evolve independently from the JAX-RS binding or the OpenLineage-Java model. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
