alamb commented on code in PR #7287: URL: https://github.com/apache/arrow-datafusion/pull/7287#discussion_r1294990640
########## docs/source/library-user-guide/catalogs.md: ########## @@ -0,0 +1,175 @@ +<!--- + Licensed to the Apache Software Foundation (ASF) under one + or more contributor license agreements. See the NOTICE file + distributed with this work for additional information + regarding copyright ownership. The ASF licenses this file + to you under the Apache License, Version 2.0 (the + "License"); you may not use this file except in compliance + with the License. You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + + Unless required by applicable law or agreed to in writing, + software distributed under the License is distributed on an + "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY + KIND, either express or implied. See the License for the + specific language governing permissions and limitations + under the License. +--> + +# Catalogs, Schemas, and Tables + +This section describes how to create and manage catalogs, schemas, and tables in DataFusion. For those wanting to dive into the code quickly please see the [example](../../../datafusion-examples/examples/catalog.rs). + +## General Concepts + +Catalogs, schemas, and tables are organized in a hierarchy. A catalog contains schemas, and a schema contains tables. + +Similarly to other concepts in DataFusion, you'll implement various traits to create your own catalogs, schemas, and tables. The following sections describe the traits you'll need to implement. + +The `CatalogProvider` trait has methods to set a schema to a name, get a schema by name, and list all schemas. The `SchemaProvider`, which can be registered with a `CatalogProvider`, has methods to set a table to a name, get a table by name, list all tables, deregister a table, and check for a table's existence. The `TableProvider` trait has methods to scan underlying data and use it in DataFusion. The `TableProvider` trait is covered in more detail [here](./custom-table-providers.md). + +We'll start with the `SchemaProvider` trait as we need one to register with the `CatalogProvider`. + +## Implementing `MemorySchemaProvider` + +The `MemorySchemaProvider` is a simple implementation of the `SchemaProvider` trait. It stores state (i.e. tables) in a `DashMap`, which then underlies the `SchemaProvider` trait. + +```rust +pub struct MemorySchemaProvider { + tables: DashMap<String, Arc<dyn TableProvider>>, +} +``` + +`tables` is the key-value pair described above. The underlying state could also be another data structure or other storage mechanism. Review Comment: ```suggestion `tables` is the key-value pair described above. The underlying state could also be another data structure or other storage mechanism such as a file or transactional database. ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
