serramatutu commented on code in PR #4694:
URL: https://github.com/apache/arrow-adbc/pull/4694#discussion_r3967449549
##########
c/include/arrow-adbc/adbc.h:
##########
@@ -1087,6 +1093,590 @@ const struct AdbcError* AdbcErrorFromArrayStream(struct
ArrowArrayStream* stream
/// @}
+/// \defgroup adbc-statement-metadata-collection Metadata Collections
+///
+/// Fetch (catalog) metadata from the database. On a statement, set
+/// ADBC_METADATA_COLLECTION to one of the collection names below, and set any
+/// filters via the options defined below. (There are driver/vendor-specific
+/// collections and filters as well.) Then call AdbcStatementExecuteQuery or
+/// AdbcStatementExecuteSchema. The result is an Arrow dataset with a schema
+/// defined by the collection. For example, a client may request a list of
+/// tables in the database, or a list of supported data types.
+///
+/// All drivers must implement a collection called "meta" (which is aliased to
+/// NULL and blank string) that defines the available collections. See
+/// ADBC_METADATA_COLLECTION_META.
+///
+/// Drivers may also implement AdbcStatementRequestSchema to (1) request
+/// different data types and (2) drop fields from the result. Drivers are not
+/// required to support this, and are not required to support other changes
+/// like reordering collection fields. Drivers are encouraged to use this to
+/// give applications flexibility over output type and shape (e.g. reducing
+/// memory pressure by run-end-encoding or dictionary-encoding primary key
+/// columns; optimizing queries by using simpler queries and eliminating joins
+/// if the application drops certain columns). Unless requested, however,
+/// drivers should return the type specified below and not substitute a
+/// logically equivalent type (e.g. utf8 below should be a string array by
+/// default, and not a utf8view or large string or other array).
+///
+/// Drivers may add more fields at the end of standard schemas to reflect
+/// vendor-specific metadata. Applications must access these by name or using
+/// an offset from the end of the schema and cannot assume that the index of
+/// the field will remain stable. Hence, it is discouraged to access these
+/// fields by ordinal as this is brittle. Drivers must add the fields at the
+/// end and must prefix field names with the vendor/driver name to
+/// differentiate them (e.g. 'POSTGRESQL:owner', not just 'owner').
+///
+/// Similarly, future standard revisions may add more fields to existing
+/// standard schemas. Applications must not assume the number of fields is
+/// fixed.
+///
+/// Drivers may implement collections beyond those defined by ADBC, but must
+/// use a vendor-specific prefix (e.g. `postgresql.`) to avoid conflicts with
+/// future standardized collections. Drivers must not use the `adbc.` prefix.
+///
+/// Drivers may not necessarily accept filter options or other options before
+/// the collection name option is set.
+///
+/// This is intended to replace AdbcConnectionGetObjects, but both APIs
+/// will be supported for the time being. A driver that implements either
+/// API should implement both APIs. AdbcConnectionGetObjects may be
+/// deprecated in a future revision.
+///
+/// \since ADBC API revision 1.2.0
+///
+/// @{
+
+/// \brief Prepare to fetch a metadata collection.
+///
+/// The type is char*.
+#define ADBC_METADATA_COLLECTION "adbc.metadata.collection"
+
+/// \brief Filter the collection on the literal catalog name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_CATALOG "adbc.metadata.filter.catalog"
+
+/// \brief Filter the collection on the catalog name, matching a search
+/// pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_CATALOG are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_CATALOG
"adbc.metadata.filter_pattern.catalog"
+
+/// \brief Filter the collection on the literal schema name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_DB_SCHEMA "adbc.metadata.filter.schema"
+
+/// \brief Filter the collection on the schema name, matching a search
+/// pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_DB_SCHEMA are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_DB_SCHEMA
"adbc.metadata.filter_pattern.schema"
+
+/// \brief Filter the collection on the literal table name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_TABLE "adbc.metadata.filter.table"
+
+/// \brief Filter the collection on the table name, matching a search pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_TABLE are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_TABLE "adbc.metadata.filter_pattern.table"
+
+/// \brief Filter the collection on the literal column name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_COLUMN "adbc.metadata.filter.column"
+
+/// \brief Filter the collection on the column name, matching a search
+/// pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_COLUMN are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_COLUMN
"adbc.metadata.filter_pattern.column"
+
+/// \brief Filter the collection on the table types.
+///
+/// Separator: comma (',').
+/// The type is char*.
+#define ADBC_METADATA_FILTER_TABLE_TYPES "adbc.metadata.filter.table_types"
+
+/// \brief Filter the collection on the literal catalog name of the foreign
+/// key.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_FOREIGN_CATALOG
"adbc.metadata.filter.foreign_catalog"
+
+/// \brief Filter the collection on the literal schema name of the foreign
+/// key.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_FOREIGN_DB_SCHEMA
"adbc.metadata.filter.foreign_schema"
+
+/// \brief Filter the collection on the literal catalog name of the foreign
+/// key.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_FOREIGN_TABLE "adbc.metadata.filter.foreign_table"
+
+/// \brief Filter the collection on the literal constraint name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_CONSTRAINT "adbc.metadata.filter.constraint"
+
+/// \brief Filter the collection on the constraint name, matching a search
+/// pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_CONSTRAINT are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_CONSTRAINT
"adbc.metadata.filter_pattern.constraint"
+
+/// \brief Filter the collection on the constraint types.
+///
+/// Separator: comma (',').
+/// The type is char*.
+#define ADBC_METADATA_FILTER_CONSTRAINT_TYPES
"adbc.metadata.filter.constraint_types"
+
+/// \brief Filter the collection on the literal routine name.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_ROUTINE "adbc.metadata.filter.routine"
+
+/// \brief Filter the collection on the routine name, matching a search
+/// pattern.
+///
+/// If both this and ADBC_METADATA_FILTER_ROUTINE are set, then the last set
+/// option wins.
+///
+/// The type is char*.
+#define ADBC_METADATA_FILTER_PATTERN_ROUTINE
"adbc.metadata.filter_pattern.routine"
+
+/// \brief Filter the collection on the routine types.
+///
+/// Separator: comma (',').
+/// The type is char*.
+#define ADBC_METADATA_FILTER_ROUTINE_TYPES "adbc.metadata.filter.routine_types"
+
+/// \brief Get or set a pagination token.
+///
+/// Some drivers may support this to allow fetching a large metadata
+/// collection in multiple calls. If the pagination token could not be used,
+/// the driver should return an error.
+#define ADBC_METADATA_OPTION_PAGINATION_TOKEN "adbc.metadata.pagination_token"
Review Comment:
Two things:
1. I assume this is an "internal" key that will be used by the driver
manager implementations to [return `RecordBatchReader`s that paginate under the
hood](https://github.com/apache/arrow-adbc/blob/150528f0fb9f1117fecce1da6a27fb548ddc6d0f/rust/driver_manager/src/lib.rs#L1016)
when you pull a new batch, right? I think it would be nice if applications
didn't need to worry about setting this option and keeping track of cursors
across multiple calls. As an application I'd like to just consume record
batches and the pagination cursors and state are handled by the driver manager.
2. Is this intended to be used as both an input option as well as an output
metadata key? If my understanding is correct, the lifecycle is:
- The app (driver manager) would call `AdbcConnectionGetInfo` without a
`adbc.metadata.pagination_token` (the first call), then the driver returns a
record batch with a `adbc.metadata.pagination_token` in the schema metadata.
- The next call, the app passes in the same
`adbc.metadata.pagination_token`. I don't understand where, since
`AdbcConnectionGetInfo` has no explicit support for options. Does this get set
as a connection-level parameter that is set before the call and unset after it
returns? Does this get added together with the filters?
- This repeats until the last page, where the driver returns no
`adbc.metadata.pagination_token`, signifying it's done listing.
I think it would be nice to document this lifecycle somewhere. Not sure the
right place for it is here, though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
