markhoerth commented on code in PR #12399: URL: https://github.com/apache/gravitino/pull/12399#discussion_r3750994851
########## docs/fileset-catalog.md: ########## @@ -1,214 +1,200 @@ --- title: "Fileset Catalog" slug: "/fileset-catalog" -date: 2024-4-2 -keyword: "fileset catalog" +keywords: + - fileset + - catalog + - storage + - s3 + - gcs + - adls + - oss + - cos license: "This software is licensed under the Apache License version 2." --- -## Introduction +## Overview -Fileset catalog is a fileset catalog that using Hadoop Compatible File System (HCFS) to manage -the storage location of the fileset. It supports the local filesystem and HDFS. -Gravitino supports [S3](fileset-catalog-with-s3.md), [GCS](fileset-catalog-with-gcs.md), -[OSS](fileset-catalog-with-oss.md) and [Azure Blob Storage](fileset-catalog-with-adls.md) through Fileset catalog. -Gravitino also supports [Tencent Cloud COS](fileset-catalog-with-cos.md). +A fileset catalog manages filesets over a Hadoop Compatible File System. Gravitino owns the catalog rather than federating an external one, so no provider is needed when creating it, and the same catalog, schema, and fileset model works over HDFS, a local filesystem, or object storage. -The rest of this document will use HDFS or local file as an example to illustrate how to use the Fileset catalog. -For S3, GCS, OSS, Azure Blob Storage and COS, the configuration is similar to HDFS, -refer to the corresponding document for more details. +What changes per storage system is small: a bundle jar on the classpath, the URI scheme in the location, and a few credential properties. Creating and managing the objects is covered in [Manage Fileset Metadata](./manage-fileset-metadata-using-gravitino.md), and reading and writing the files in [How to Use GVFS](./how-to-use-gvfs.md). Neither changes because the data sits in S3 rather than HDFS, which is the point of the indirection described in [Filesets](./filesets.md). -Note that Gravitino uses Hadoop 3 dependencies to build Fileset catalog. Theoretically, it should be -compatible with both Hadoop 2.x and 3.x, since Gravitino doesn't leverage any new features in -Hadoop 3. If there's any compatibility issue, create an [issue](https://github.com/apache/gravitino/issues). +The catalog is built against Hadoop 3 but uses no Hadoop 3 features, so Hadoop 2.x should also work. Report any incompatibility as an [issue](https://github.com/apache/gravitino/issues). -## Catalog +## Catalog Properties -### Catalog Properties +These apply in addition to the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration). -Besides the [common catalog properties](./gravitino-server-config.md#catalog-properties-configuration), -the Fileset catalog has the following properties: +| Property Name | Description | Default Value | +|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------|---------------| +| `location` | Base storage location, named `unknown`. Always a directory or path prefix, never a single file | (none) | +| `location-` | Prefix for named locations, as `location-{name}={path}` | (none) | +| `credential-providers` | Credential provider types, separated by commas | (none) | +| `config.resources` | Configuration files to load, separated by commas, such as `hdfs-site.xml,core-site.xml` | (none) | +| `filesystem-conn-timeout-secs` | Timeout when obtaining a filesystem client, in seconds | `6` | +| `disable-filesystem-ops` | Stops the server creating and removing directories when schemas and filesets are created and dropped | `false` | +| `fileset-cache-eviction-interval-ms` | Fileset cache eviction interval, where `-1` never evicts | `3600000` | +| `fileset-cache-max-size` | Maximum filesets held in the cache, where `-1` is unlimited | `200000` | +| `fs.path.config.<n>` | A logical location entry set to a base URI such as `hdfs://cluster1/`. Keys sharing the prefix are forwarded to that filesystem client | (none) | Review Comment: <n> matches the code and how-to-use-gvfs.md, which documents the same key as fs.path.config.<n> and fs_path_config_<n>. Renaming it here alone would put the two pages out of step. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
