flyingImer commented on code in PR #4826: URL: https://github.com/apache/polaris/pull/4826#discussion_r3470482312
########## runtime/service/src/main/java/org/apache/polaris/service/lineage/DefaultLineageService.java: ########## @@ -0,0 +1,82 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.service.lineage; + +import jakarta.enterprise.context.RequestScoped; +import jakarta.inject.Inject; +import java.time.Instant; +import org.apache.polaris.core.config.FeatureConfiguration; +import org.apache.polaris.core.context.CallContext; +import org.apache.polaris.core.context.RealmContext; +import org.apache.polaris.core.lineage.LineageGraph; +import org.apache.polaris.core.lineage.LineageIngestRequest; +import org.apache.polaris.core.lineage.LineagePersistence; +import org.apache.polaris.core.lineage.LineageQueryRequest; +import org.apache.polaris.core.lineage.LineageService; + +@RequestScoped +public class DefaultLineageService implements LineageService { Review Comment: why there is a default service? IMHO, it should be an default SPI impl under extensions/ ########## polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageService.java: ########## @@ -0,0 +1,26 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.lineage; + +/** Service boundary for lineage operations used by transport-layer adapters. */ +public interface LineageService { Review Comment: I see, the "Service" naming is confusing. At a glance, this is the right SPI placement. 2 cents on naming: PolarisLineage or PolarisLineageHandler or the convention. My rationale is that xxxService usually implies runtime, which should be under runtime/ ########## polaris-core/src/main/java/org/apache/polaris/core/lineage/LineageColumnEdge.java: ########## @@ -0,0 +1,29 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.lineage; + +import java.util.Objects; + +/** A field-level lineage relationship between two dataset columns. */ +public record LineageColumnEdge(LineageFieldReference source, LineageFieldReference target) { Review Comment: I feel these data models should belong to extensions/ for SPI placement, tho I am fine to leave under core/, but currently slightly inclined to put under extensions/ with the rationale of lineage is somehow "optional" capability to Polaris being a Iceberg irc spec impl. To keep core/ lean and tight towards that vision, I suggest to place under extensions/ ########## polaris-core/src/main/java/org/apache/polaris/core/lineage/LineagePersistence.java: ########## @@ -0,0 +1,64 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.lineage; + +import java.time.Instant; +import java.util.List; +import org.apache.polaris.core.context.RealmContext; + +/** + * Persistence SPI for lineage storage backends. + * + * <p>This contract is expressed in terms of Polaris's local lineage graph rather than raw + * OpenLineage run events. The service layer owns event parsing, forwarding, authorization, and + * dataset resolution. Persistence backends persist dataset nodes, dataset edges, and column edges, + * and load normalized lineage graphs. + */ +public interface LineagePersistence { Review Comment: This is seems to be durable logic related, I would suggest to name to something similar to MetastoreManager, e.g., LineageStoreManager given lineage default behavior should be no-op per last discussion, so it definitely does not belong to core/, but extensions/ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
