dimas-b commented on code in PR #1555: URL: https://github.com/apache/polaris/pull/1555#discussion_r2105125734
########## polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java: ########## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.persistence.pagination; + +import java.util.List; +import java.util.Optional; +import org.apache.polaris.core.entity.EntityNameLookupRecord; +import org.apache.polaris.core.entity.PolarisBaseEntity; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class EntityIdPageToken extends PageToken implements HasPageSize { + private static final Logger LOGGER = LoggerFactory.getLogger(EntityIdPageToken.class); + + public static final String PREFIX = "entity-id"; + + /** The minimum ID that could be attached to an entity */ + private static final long MINIMUM_ID = 0; + + /** The entity ID to use to start with. */ + public static final long BASE_ID = MINIMUM_ID - 1; + + private final long entityId; + private final int pageSize; + + public EntityIdPageToken(int pageSize) { + this.entityId = BASE_ID; + this.pageSize = pageSize; + } + + public EntityIdPageToken(long entityId, int pageSize) { + this.entityId = entityId; + this.pageSize = pageSize; + } + + /** + * Build an {@link EntityIdPageToken} from a {@link PageRequest}, or else a {@link + * ReadEverythingPageToken} if the request doesn't require pagination. + */ + public static PageToken fromPageRequest(PageRequest pageRequest) { + if (pageRequest.getPageTokenString().isEmpty()) { + if (pageRequest.getPageSize().isEmpty()) { + return PageToken.readEverything(); Review Comment: It is a bit awkward for a specific conversion method (in class `EntityIdPageToken`) to return a less specific type (`PageToken`). WDYT about using `boolean PageRequest.readEverything()` instead? ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/BasePersistence.java: ########## @@ -413,4 +415,7 @@ boolean hasChildren( default BasePersistence detach() { return this; } + + /** Construct a {@link PageToken} from a {@link PageRequest} */ + PageToken buildPageToken(PageRequest pageRequest); Review Comment: I guess this does not have to be in the `BasePersistence` interface now :thinking: ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java: ########## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.persistence.pagination; + +import java.util.Optional; + +/** + * A wrapper for pagination information passed in as part of a request. This can potentially be + * translated into a `PageToken` + */ +public class PageRequest { + private final Optional<String> pageTokenStringOpt; + private final Optional<Integer> pageSizeOpt; + + public PageRequest(String pageTokenString, Integer pageSize) { + this.pageTokenStringOpt = Optional.ofNullable(pageTokenString); + this.pageSizeOpt = Optional.ofNullable(pageSize); + } + + public static PageRequest readEverything() { Review Comment: I guess `ReadEverythingPageToken` is no longer necessary now? ########## extension/persistence/relational-jdbc/src/main/java/org/apache/polaris/extension/persistence/relational/jdbc/JdbcBasePersistenceImpl.java: ########## @@ -425,11 +436,8 @@ public <T> Page<T> listEntities( } data.forEach(results::add); }); - List<T> resultsOrEmpty = - results == null - ? Collections.emptyList() - : results.stream().filter(entityFilter).map(transformer).collect(Collectors.toList()); - return Page.fromItems(resultsOrEmpty); + List<T> resultsOrEmpty = results.stream().map(transformer).collect(Collectors.toList()); + return pageToken.buildNextPage(resultsOrEmpty); Review Comment: Side note: if we want to avoid empty last pages (when the list ends exactly on the last element from the query) we may need to request one more entry from the database, but not return it to the caller. ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/AtomicOperationMetaStoreManager.java: ########## @@ -707,15 +707,15 @@ private void revokeGrantRecord( ? 0l : catalogPath.get(catalogPath.size() - 1).getId(); Page<EntityNameLookupRecord> resultPage = - ms.listEntities(callCtx, catalogId, parentId, entityType, pageToken); + ms.listEntities(callCtx, catalogId, parentId, entityType, pageRequest); // prune the returned list with only entities matching the entity subtype if (entitySubType != PolarisEntitySubType.ANY_SUBTYPE) { resultPage = - pageToken.buildNextPage( - resultPage.items.stream() - .filter(rec -> rec.getSubTypeCode() == entitySubType.getCode()) - .collect(Collectors.toList())); + resultPage.filter( Review Comment: I believe it is preferable to keep invoking `ms.listEntities(...)` until we either exhaust the result set or build a full page... but this can be fixed later if you prefer. ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/Page.java: ########## @@ -39,4 +41,9 @@ public Page(PageToken pageToken, List<T> items) { public static <T> Page<T> fromItems(List<T> items) { return new Page<>(new DonePageToken(), items); } + + public Page<T> filter(Predicate<T> predicate) { + return new Page<>( + this.pageToken, this.items.stream().filter(predicate).collect(Collectors.toList())); Review Comment: This can result in empty pages... Is that intended? ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/PageRequest.java: ########## @@ -0,0 +1,55 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.persistence.pagination; + +import java.util.Optional; + +/** + * A wrapper for pagination information passed in as part of a request. This can potentially be + * translated into a `PageToken` + */ +public class PageRequest { + private final Optional<String> pageTokenStringOpt; + private final Optional<Integer> pageSizeOpt; Review Comment: `OptionalInt`? ########## extension/persistence/relational-jdbc/src/main/java/org/apache/polaris/extension/persistence/relational/jdbc/JdbcBasePersistenceImpl.java: ########## @@ -425,11 +436,8 @@ public <T> Page<T> listEntities( } data.forEach(results::add); }); - List<T> resultsOrEmpty = - results == null - ? Collections.emptyList() - : results.stream().filter(entityFilter).map(transformer).collect(Collectors.toList()); - return Page.fromItems(resultsOrEmpty); + List<T> resultsOrEmpty = results.stream().map(transformer).collect(Collectors.toList()); + return pageToken.buildNextPage(resultsOrEmpty); Review Comment: It looks like `buildNextPage` does not have to be a function of the previous page token. It is a function of page request + data + persistence impl. WDYT about: `PageRequest.buildPage(List<T> data, Function<T, PageToken> nextToken)`? Here, the call would look like: `return pageRequest.buildPage(resultsOrEmpty, EntityIdPageToken::fromLastItem)` Note: the `nextToken` function will be invoked only when the next page is expected (i.e. not "done" and not "everything"). ########## polaris-core/src/main/java/org/apache/polaris/core/persistence/pagination/EntityIdPageToken.java: ########## @@ -0,0 +1,120 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, + * software distributed under the License is distributed on an + * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + * KIND, either express or implied. See the License for the + * specific language governing permissions and limitations + * under the License. + */ +package org.apache.polaris.core.persistence.pagination; + +import java.util.List; +import java.util.Optional; +import org.apache.polaris.core.entity.EntityNameLookupRecord; +import org.apache.polaris.core.entity.PolarisBaseEntity; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +public class EntityIdPageToken extends PageToken implements HasPageSize { + private static final Logger LOGGER = LoggerFactory.getLogger(EntityIdPageToken.class); + + public static final String PREFIX = "entity-id"; + + /** The minimum ID that could be attached to an entity */ + private static final long MINIMUM_ID = 0; + + /** The entity ID to use to start with. */ + public static final long BASE_ID = MINIMUM_ID - 1; + + private final long entityId; + private final int pageSize; Review Comment: With the new code it looks like `pageSize` is redundant here. This information is defined by `PageRequest`. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@polaris.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
