Hello,
I would like to start this ML for discussing if we should implement the
pagination support for generic table API (in this case,
/polaris/v1/{prefix}/namespaces/{namespace}/generic-tables). Here is a
reference report issue for this matter:
https://github.com/apache/polaris/issues/5311.
Looking at the what we currently have as of today, the parameters reach the
adapter then got dropped. We are calling PageToken.readEverything() regardless
(https://github.com/apache/polaris/blob/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/generic/PolarisGenericTableCatalog.java#L196).
Now to honor this, I think we have two approaches that we can consider:
option 1: No change to SPI and use method overload instead
This would be similar to what we are doing in LocalIcebergCatalog where we will
be adding Page listTables(namespace, pageToken) via method overload. Here is
the reference:
https://github.com/apache/polaris/blame/main/runtime/service/src/main/java/org/apache/polaris/service/catalog/iceberg/LocalIcebergCatalog.java#L632
option 2: Change to SPI with listGenericTables(namespace, pageToken)
This would be similar to listSemanticModels which got implemented recently.
Here is the reference:
https://github.com/apache/polaris/blob/48126190377cfb2d0a54b4b9905a7375fde457ec/extensions/semantic-models/src/main/java/org/apache/polaris/service/catalog/semanticmodel/SemanticModelCatalogAdapter.java#L92
Thanks,
Yong
