qidaye opened a new issue #8807: URL: https://github.com/apache/incubator-doris/issues/8807
### Search before asking - [X] I had searched in the [issues](https://github.com/apache/incubator-doris/issues?q=is%3Aissue) and found no similar issues. ### Description ## Background To support the separation of hot and cold data, you need to specify the connection information of remote storage S3, such as ak, sk, url, etc., when creating an OLAP table. This information can be reused, and a Doris cluster may also be connected to multiple S3s, so these connection information needs to be managed as a remote storage resource. It is convenient for users to use, and does not need to be provided every time a table is created, only the resource name is required. At the same time, when the remote storage connection information changes, there is no need to rebuild the table, just update the corresponding resource. ## Design Currently, Doris supports the creation of Spark and MySQL resources, and the corresponding syntax can be reused. Based on the existing Resource abstraction, Resource manager resource meta-information management class, 1. Add s3 type to ResourceType 2. Create a new S3Resource class, inherit the Resource class, and implement the corresponding interface 3. Support the verification, reading and persistence of s3 resource information ### Specific syntax ```sql -- create remote storage resource create resource "remote_s3" properties ( "type" = "s3", "s3_endpoint" = "bj", "s3_region" = "bj", "s3_root_path" = "/path/to/root", "s3_access_key" = "bbb", "s3_secret_key" = "aaaa", "s3_max_connections" = "50", "s3_request_timeout_ms" = "3000", "s3_connection_timeout_ms" = "1000" ); -- create table using remote storage resource CREATE TABLE example_db.table_remote ( k1 DATE, k2 INT, k3 SMALLINT, k4 VARCHAR(2048), k5 DATETIME ) ENGINE=olap PROPERTIES ( "storage_medium" = "HDD", "storage_cold_medium" = "S3", "remote_storage_resource" = "remote_s3", "storage_cooldown_time" = "2022-06-04 00:00:00" ); ### Use case _No response_ ### Related issues _No response_ ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! ### Code of Conduct - [X] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
