gaborkaszab commented on code in PR #14465:
URL: https://github.com/apache/iceberg/pull/14465#discussion_r2559016818
##########
core/src/main/java/org/apache/iceberg/rest/RESTSessionCatalog.java:
##########
@@ -1010,6 +1011,82 @@ private FileIO tableFileIO(
return newFileIO(context, fullConf, storageCredentials);
}
+ /**
+ * Create a new {@link RESTTableOperations} instance for simple table
operations.
+ *
+ * <p>This method can be overridden in subclasses to provide custom table
operations
+ * implementations.
+ *
+ * @param restClient the REST client to use for communicating with the
catalog server
+ * @param path the REST path for the table
+ * @param headers a supplier for additional HTTP headers to include in
requests
+ * @param fileIO the FileIO implementation for reading and writing table
metadata and data files
+ * @param current the current table metadata
+ * @param supportedEndpoints the set of supported REST endpoints
+ * @return a new RESTTableOperations instance
+ */
+ protected RESTTableOperations newTableOps(
Review Comment:
I'm just curious here: These are protected, so not part of any API or so.
This means that there are no guarantees the signature of these functions don't
change and any change here is not considered a breaking change. I don't think
revAPI checks would break if I add a couple of extra parameters and change the
types for some of the existing ones.
However, if you make your application so that it overrides these functions
it could break with every Iceberg version bump. Isn't this an issue at your end?
##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -3066,6 +3074,113 @@ public void testCommitStateUnknownNotReconciled() {
.satisfies(ex -> assertThat(((CommitStateUnknownException)
ex).getSuppressed()).isEmpty());
}
+ @Test
+ public void testCustomTableOperationsInjection() throws IOException {
+ AtomicBoolean customTableOps = new AtomicBoolean();
+ AtomicBoolean customTxnOps = new AtomicBoolean();
+ AtomicBoolean customViewOps = new AtomicBoolean();
+
+ // Custom RESTSessionCatalog that overrides table/view operations creation
+ class CustomRESTSessionCatalog extends RESTSessionCatalog {
+ CustomRESTSessionCatalog(
+ Function<Map<String, String>, RESTClient> clientBuilder,
+ BiFunction<SessionCatalog.SessionContext, Map<String, String>,
FileIO> ioBuilder) {
+ super(clientBuilder, ioBuilder);
+ }
+
+ @Override
+ protected RESTTableOperations newTableOps(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ FileIO fileIO,
+ TableMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customTableOps.set(true);
+ return super.newTableOps(restClient, path, headers, fileIO, current,
supportedEndpoints);
+ }
+
+ @Override
+ protected RESTTableOperations newTableOpsForTransaction(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ FileIO fileIO,
+ RESTTableOperations.UpdateType updateType,
+ List<MetadataUpdate> createChanges,
+ TableMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customTxnOps.set(true);
+ return super.newTableOpsForTransaction(
+ restClient,
+ path,
+ headers,
+ fileIO,
+ updateType,
+ createChanges,
+ current,
+ supportedEndpoints);
+ }
+
+ @Override
+ protected RESTViewOperations newViewOps(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ ViewMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customViewOps.set(true);
+ return super.newViewOps(restClient, path, headers, current,
supportedEndpoints);
+ }
+ }
+
+ // Custom RESTCatalog that provides the custom session catalog
+ class CustomRESTCatalog extends RESTCatalog {
+ CustomRESTCatalog(
+ SessionCatalog.SessionContext context,
+ Function<Map<String, String>, RESTClient> clientBuilder) {
+ super(context, clientBuilder);
+ }
+
+ @Override
+ protected RESTSessionCatalog newSessionCatalog(
+ Function<Map<String, String>, RESTClient> clientBuilder) {
+ return new CustomRESTSessionCatalog(clientBuilder, null);
+ }
+ }
+
+ try (CustomRESTCatalog catalog =
+ new CustomRESTCatalog(
+ SessionCatalog.SessionContext.createEmpty(),
+ (config) -> new RESTCatalogAdapter(backendCatalog))) {
+ catalog.setConf(new Configuration());
+ catalog.initialize(
+ "test",
+ ImmutableMap.of(
+ CatalogProperties.FILE_IO_IMPL,
"org.apache.iceberg.inmemory.InMemoryFileIO"));
+
+ Namespace ns = Namespace.of("test_custom_ops");
+ catalog.createNamespace(ns);
+
+ catalog.createTable(TableIdentifier.of(ns, "table1"), SCHEMA);
+ assertThat(customTableOps).isTrue();
Review Comment:
nit: shouldn't we assert before createTable that all 3 booleans are false?
##########
core/src/main/java/org/apache/iceberg/rest/RESTTableOperations.java:
##########
@@ -44,10 +44,10 @@
import org.apache.iceberg.rest.responses.LoadTableResponse;
import org.apache.iceberg.util.LocationUtil;
-class RESTTableOperations implements TableOperations {
+public class RESTTableOperations implements TableOperations {
Review Comment:
I checked the test for this PR and apparently we test how to put a wrapper
around the ops creation by overriding RESTSessionCatalog and RESTCatalog
functionality, but we don't test the end-to-end use case. I think injecting a
custom table/view ops would be nice to see how this would work. Like injecting
an ops that adds extra header to requests, since that was the original
motivation of this PR.
##########
core/src/test/java/org/apache/iceberg/rest/TestRESTCatalog.java:
##########
@@ -3066,6 +3074,113 @@ public void testCommitStateUnknownNotReconciled() {
.satisfies(ex -> assertThat(((CommitStateUnknownException)
ex).getSuppressed()).isEmpty());
}
+ @Test
+ public void testCustomTableOperationsInjection() throws IOException {
+ AtomicBoolean customTableOps = new AtomicBoolean();
+ AtomicBoolean customTxnOps = new AtomicBoolean();
+ AtomicBoolean customViewOps = new AtomicBoolean();
+
+ // Custom RESTSessionCatalog that overrides table/view operations creation
+ class CustomRESTSessionCatalog extends RESTSessionCatalog {
+ CustomRESTSessionCatalog(
+ Function<Map<String, String>, RESTClient> clientBuilder,
+ BiFunction<SessionCatalog.SessionContext, Map<String, String>,
FileIO> ioBuilder) {
+ super(clientBuilder, ioBuilder);
+ }
+
+ @Override
+ protected RESTTableOperations newTableOps(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ FileIO fileIO,
+ TableMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customTableOps.set(true);
+ return super.newTableOps(restClient, path, headers, fileIO, current,
supportedEndpoints);
+ }
+
+ @Override
+ protected RESTTableOperations newTableOpsForTransaction(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ FileIO fileIO,
+ RESTTableOperations.UpdateType updateType,
+ List<MetadataUpdate> createChanges,
+ TableMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customTxnOps.set(true);
+ return super.newTableOpsForTransaction(
+ restClient,
+ path,
+ headers,
+ fileIO,
+ updateType,
+ createChanges,
+ current,
+ supportedEndpoints);
+ }
+
+ @Override
+ protected RESTViewOperations newViewOps(
+ RESTClient restClient,
+ String path,
+ Supplier<Map<String, String>> headers,
+ ViewMetadata current,
+ Set<Endpoint> supportedEndpoints) {
+ customViewOps.set(true);
+ return super.newViewOps(restClient, path, headers, current,
supportedEndpoints);
+ }
+ }
+
+ // Custom RESTCatalog that provides the custom session catalog
+ class CustomRESTCatalog extends RESTCatalog {
+ CustomRESTCatalog(
+ SessionCatalog.SessionContext context,
+ Function<Map<String, String>, RESTClient> clientBuilder) {
+ super(context, clientBuilder);
+ }
+
+ @Override
+ protected RESTSessionCatalog newSessionCatalog(
+ Function<Map<String, String>, RESTClient> clientBuilder) {
+ return new CustomRESTSessionCatalog(clientBuilder, null);
+ }
+ }
+
+ try (CustomRESTCatalog catalog =
+ new CustomRESTCatalog(
+ SessionCatalog.SessionContext.createEmpty(),
+ (config) -> new RESTCatalogAdapter(backendCatalog))) {
+ catalog.setConf(new Configuration());
+ catalog.initialize(
+ "test",
+ ImmutableMap.of(
+ CatalogProperties.FILE_IO_IMPL,
"org.apache.iceberg.inmemory.InMemoryFileIO"));
+
+ Namespace ns = Namespace.of("test_custom_ops");
+ catalog.createNamespace(ns);
+
+ catalog.createTable(TableIdentifier.of(ns, "table1"), SCHEMA);
Review Comment:
nit: other tests use TABLE for this. Would it make sense to keep consistency
and not create a new identifier here? Also creating new NS wouldn't be needed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
