mchades commented on code in PR #7009:
URL: https://github.com/apache/gravitino/pull/7009#discussion_r2051630967
##########
docs/manage-fileset-metadata-using-gravitino.md:
##########
@@ -429,34 +465,198 @@
catalog.as_fileset_catalog().create_fileset(ident=NameIdentifier.of("test_schema
</TabItem>
</Tabs>
-The value of `storageLocation` depends on the configuration settings of the
catalog:
-- If this is a local fileset catalog, the `storageLocation` should be in the
format of `file:///path/to/fileset`.
-- If this is a HDFS fileset catalog, the `storageLocation` should be in the
format of `hdfs://namenode:port/path/to/fileset`.
+#### storageLocations
+You can also create a fileset with multiple storage locations. The
`storageLocations` is a map of location name to storage location.
+The generation rules of each location follow the generation rules of a single
location.
+The following is an example of creating a fileset with multiple storage
locations:
-For a `MANAGED` fileset, the storage location is:
+<Tabs groupId="language" queryString>
+<TabItem value="shell" label="Shell">
-1. The one specified by the user during the fileset creation, and the
placeholder will be replaced by the
- corresponding fileset property value.
-2. When the catalog property `location` is specified but the schema property
`location` isn't specified, the storage location is:
- 1. `catalog location/schema name/fileset name` if `catalog location` does
not contain any placeholder.
- 2. `catalog location` - placeholders in the catalog location will be
replaced by the corresponding fileset property value.
+```shell
+# create a catalog first
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "test_catalog",
+ "type": "FILESET",
+ "comment": "comment",
+ "provider": "hadoop",
+ "properties": {
+ "filesystem-providers": "builtin-local,builtin-hdfs,s3,gcs",
+ "location-l1":
"file:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}",
+ "location-l2":
"hdfs:///{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs
-3. When the catalog property `location` isn't specified but the schema
property `location` is specified,
- the storage location is:
- 1. `schema location/fileset name` if `schema location` does not contain any
placeholder.
- 2. `schema location` - placeholders in the schema location will be replaced
by the corresponding fileset property value.
-
-4. When both the catalog property `location` and the schema property
`location` are specified, the storage
- location is:
- 1. `schema location/fileset name` if `schema location` does not contain any
placeholder.
- 2. `schema location` - placeholders in the schema location will be replaced
by the corresponding fileset property value.
+# create a schema under the catalog
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "test_schema",
+ "comment": "comment",
+ "properties": {
+ "location-l3":
"s3a://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
+ }
+}' http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas
-5. When both the catalog property `location` and schema property `location`
isn't specified, the user
- should specify the `storageLocation` in the fileset creation.
+# create a fileset by placeholders
+curl -X POST -H "Accept: application/vnd.gravitino.v1+json" \
+-H "Content-Type: application/json" -d '{
+ "name": "example_fileset",
+ "comment": "This is an example fileset",
+ "type": "MANAGED",
+ "storageLocations": {
+ "l4": "gs://myBucket/{{catalog}}/{{schema}}/workspace_{{project}}/{{user}}"
+ },
+ "properties": {
+ "placeholder-project": "test_project",
+ "placeholder-user": "test_user",
+ "default-location-name": "l1"
+ }
+}'
http://localhost:8090/api/metalakes/metalake/catalogs/test_catalog/schemas/test_schema/filesets
-For `EXTERNAL` fileset, users should specify `storageLocation` during the
fileset creation,
-otherwise, Gravitino will throw an exception. If the `storageLocation`
contains placeholders, the
-placeholder will be replaced by the corresponding fileset property value.
+# the fileset will be created with 4 storage locations:
+{
+ "name": "example_fileset",
+ "comment": "This is an example fileset",
+ "type": "MANAGED",
+ "storageLocation": null,
+ "storageLocations": {
Review Comment:
> Is `storageLocation` a required property?
No.
> Why do we need to explicitly set a property that might be defaulted to
null?
This is not a creation request, but an example of a returned result.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
