yuqi1129 commented on code in PR #9569:
URL: https://github.com/apache/gravitino/pull/9569#discussion_r2696738507
##########
catalogs/catalog-fileset/src/main/java/org/apache/gravitino/catalog/fileset/FilesetCatalogOperations.java:
##########
@@ -164,21 +175,19 @@ public class FilesetCatalogOperations extends
ManagedSchemaOperations
static class FileSystemCacheKey {
// When the path is a path without scheme such as 'file','hdfs', etc.,
then the scheme and
// authority are both null
+
+ // NOTE: The filesystem cache key contains the schema, authority and
current user.
+ // Changed the configuration of the same fileset. will not be effected by
the cached
+ // filesystem.
Review Comment:
There exists a syntax problem between L180 and L181.
##########
docs/fileset-catalog.md:
##########
@@ -28,18 +28,19 @@ Hadoop 3. If there's any compatibility issue, please create
an [issue](https://g
Besides the [common catalog
properties](./gravitino-server-config.md#apache-gravitino-catalog-properties-configuration),
the Fileset catalog has the following properties:
-| Property Name | Description
| Default Value | Required | Since Version |
-|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------|------------------|
-| `location` | The storage location managed by
Fileset catalog. Its location name is `unknown`. The value should always a
directory(HDFS) or path prefix(cloud storage like S3, GCS.) and does not
support a single file.
| (none) | No |
0.5.0 |
-| `location-` | The property prefix. User can use
`location-{name}={path}` to set multiple locations with different names for the
catalog.
| (none) | No |
0.9.0-incubating |
-| `default-filesystem-provider` | The default filesystem provider of
this Fileset catalog if users do not specify the scheme in the URI. Candidate
values are 'builtin-local', 'builtin-hdfs', 's3', 'gcs', 'abs' and 'oss'.
Default value is `builtin-local`. For S3, if we set this value to 's3', we can
omit the prefix 's3a://' in the location. | `builtin-local` | No |
0.7.0-incubating |
-| `filesystem-providers` | The file system providers to add.
Users need to set this configuration to support cloud storage or custom HCFS.
For instance, set it to `s3` or a comma separated string that contains `s3`
like `gs,s3` to support multiple kinds of fileset including `s3`.
| (none) | Yes |
0.7.0-incubating |
-| `credential-providers` | The credential provider types,
separated by comma.
| (none) | No |
0.8.0-incubating |
-| `filesystem-conn-timeout-secs` | The timeout of getting the file
system using Hadoop FileSystem client instance. Time unit: seconds.
| 6 | No |
0.8.0-incubating |
-| `disable-filesystem-ops` | The configuration to disable file
system operations in the server side. If set to true, the Fileset catalog in
the server side will not create, drop files or folder when the schema, fileset
is created, dropped.
| false | No |
0.9.0-incubating |
-| `fileset-cache-eviction-interval-ms` | The interval in milliseconds to evict
the fileset cache, -1 means never evict.
| 3600000 | No | 0.9.0-incubating |
-| `fileset-cache-max-size` | The maximum number of the filesets
the cache may contain, -1 means no limit.
| 200000 | No |
0.9.0-incubating |
-| `config.resources` | The configuration resources,
separated by comma. For example, `hdfs-site.xml,core-site.xml`.
| (none) | No | 1.1.0
|
+| Property Name | Description
| Default Value | Required | Since
Version |
+|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------|------------------|
+| `location` | The storage location managed by
Fileset catalog. Its location name is `unknown`. The value should always a
directory(HDFS) or path prefix(cloud storage like S3, GCS.) and does not
support a single file.
| (none) |
No | 0.5.0 |
+| `location-` | The property prefix. User can use
`location-{name}={path}` to set multiple locations with different names for the
catalog.
| (none) | No |
0.9.0-incubating |
+| `default-filesystem-provider` | (deprected) The default filesystem
provider of this Fileset catalog if users do not specify the scheme in the URI.
Candidate values are 'builtin-local', 'builtin-hdfs', 's3', 'gcs', 'abs' and
'oss'. Default value is `builtin-local`. For S3, if we set this value to 's3',
we can omit the prefix 's3a://' in the location. | `builtin-local` | No |
0.7.0-incubating |
Review Comment:
I believed that you may need to explain more about how to use it. for
example `(deprecated, all you need is to place the related bundle jar to the
class path and no need to care about this configuration)
##########
docs/fileset-catalog-with-gcs.md:
##########
@@ -29,12 +29,11 @@ Apart from configurations mentioned in
[Fileset-catalog-catalog-configuration](.
| Configuration item | Description
| Default value |
Required | Since version |
|-------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------|------------------|
-| `filesystem-providers` | The file system providers to add. Set it to
`gcs` if it's a GCS fileset, a comma separated string that contains `gcs` like
`gcs,s3` to support multiple kinds of fileset including `gcs`.
| (none)
| Yes | 0.7.0-incubating |
-| `default-filesystem-provider` | The name default filesystem providers of
this Fileset catalog if users do not specify the scheme in the URI. Default
value is `builtin-local`, for GCS, if we set this value, we can omit the prefix
'gs://' in the location.
|
`builtin-local` | No | 0.7.0-incubating |
+| `filesystem-providers` | (deprected) The file system providers to
add. Set it to `gcs` if it's a GCS fileset, a comma separated string that
contains `gcs` like `gcs,s3` to support multiple kinds of fileset including
`gcs`.
|
(none) | Yes | 0.7.0-incubating |
Review Comment:
typo: deprected
##########
docs/fileset-catalog-with-s3.md:
##########
@@ -28,14 +28,14 @@ Once the server is up and running, you can proceed to
configure the Fileset cata
In addition to the basic configurations mentioned in
[Fileset-catalog-catalog-configuration](./fileset-catalog.md#catalog-properties),
the following properties are necessary to configure a Fileset catalog with S3:
-| Configuration item | Description
| Default value | Required
| Since version |
-|--------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------|------------------|
-| `filesystem-providers` | The file system providers to add. Set it to
`s3` if it's a S3 fileset, or a comma separated string that contains `s3` like
`gs,s3` to support multiple kinds of fileset including `s3`.
| (none) | Yes
| 0.7.0-incubating |
-| `default-filesystem-provider` | The name default filesystem providers of
this Fileset catalog if users do not specify the scheme in the URI. Default
value is `builtin-local`, for S3, if we set this value, we can omit the prefix
's3a://' in the location.
| `builtin-local` |
No | 0.7.0-incubating |
-| `s3-endpoint` | The endpoint of the AWS S3.
| (none) | Yes
| 0.7.0-incubating |
-| `s3-access-key-id` | The access key of the AWS S3.
| (none) | Yes
| 0.7.0-incubating |
-| `s3-secret-access-key` | The secret key of the AWS S3.
| (none) | Yes
| 0.7.0-incubating |
-| `credential-providers` | The credential provider types, separated by
comma, possible value can be `s3-token`, `s3-secret-key`. As the default
authentication type is using AKSK as the above, this configuration can enable
credential vending provided by Gravitino server and client will no longer need
to provide authentication information like AKSK to access S3 by GVFS. Once it's
set, more configuration items are needed to make it works, please see
[s3-credential-vending](security/credential-vending.md#s3-credentials) | (none)
| No | 0.8.0-incubating |
+| Configuration item | Description
| Default value | Required
| Since version |
+|-------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------------|----------|------------------|
+| `filesystem-providers` | (deprected) The file system providers to
add. Set it to `s3` if it's a S3 fileset, or a comma separated string that
contains `s3` like `gs,s3` to support multiple kinds of fileset including `s3`.
| (none) |
Yes | 0.7.0-incubating |
Review Comment:
Another point is that you may need to add information like `deprecated since
1.2.0`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
